This Week
Team Goal: Get the combined classification system functioning on the H7
Personal Goal: Enable localization of multiple cards by using SD card to store images, and implement Bradley's classification code on the OpenMV board using OpenMV image library functions
At the end of last week, I could successfully localize one card at a time with the OpenMV board:
The thing keeping me back from doing multiple cards at once? Memory. My process thus far is as follows:
Take a 320x240 color picture
Create a 160x120 grayscale copy which will be used to perform localization
Apply gaussian blur and adaptive threshold to the smaller copy
Use find_rects() to identify cards and their corner coordinates
For each rectangle found, apply perspective correction and crop the corresponding region in the original, higher-resolution picture
The problem with the final step is that the perspective correction and cropping would modify the original image, so it could only be done once. An analogy is helpful here: if a variable x equals 1, we'd expect the statement y = x + 2 not to modify the value of x. This effectively requires a temporary "copy" of x that is equal in value to be created, to which 2 can be added. Finally, y can be set to this result. However, in order to run on the memory-constrained H7, creating a whole copy of the original color image on which to operate would exhaust its memory capacity. The rotation_corr and crop functions by default operate destructively and modify the image on which they are called.
Images on the H7 are stored in a 512KB portion of RAM called the Frame Buffer. The original 320x240 RGB565 image is 150KB, the smaller copy converted to grayscale is 18.75KB, and the find_rects function is memory intensive. Long story short, there is not enough space to create a second 150KB copy of the original image:
My original idea for a workaround was to store copies on the included SD card and retrieve them as needed, bypassing the RAM constraints. However, after getting some errors using OpenMV's BMP image loading functions, I discovered in a forum post from 2019 that the OpenMV creators admitted the BMP reading code was not functional with BMPs created by the board itself, rendering it useless for me. In hindsight, this probably was a good thing, as saving and reading several images to and from the SD card each frame would drastically reduce speed.
The solution? Create a copy of only the portion of the full-size image containing the card (which, admittedly, sounds rather obvious after saying it). Applying the perspective transform, which requires the card's corner coordinates, required a little more effort, as the card's corner coordinates found with find_rects() are defined with respect to the global coordinate system, but in the partial copy, the card's top-left corner is now (0,0). After a bit of tweaking, I was able to create the partial copy (left) and then apply the perspective transform to it (center) and finally crop in and resize to 60x90 (right):
Just imagine my excitement when it worked after adding two more cards:
However, it was hardly detecting any rectangles after I went to 12 cards. Eventually, I concluded this was because the cards didn't appear "rectangular" enough to be identified as rectangles by find_rects when the camera was far enough away to see all 12 cards. At the time, I was using a scale factor of 4 instead of 2 for the image copy, so the input to find_rects() was an 80x60 image:
Reducing the scale factor to 2 solved the rectangle detection problem, but created another:
OpenMV's crop() function is able to resize images (in my case to 60x90), given the input image is larger than the resized version. For example, it can resize a 76x101 image to 60x90, but not a 27x49 image. This is because only enough memory to store the original size is allocated for the operation. To get around this, instead of creating a partial copy of just the card being considered, I double the needed width and height, which is still small enough to subvert any memory allocation problems.
I still occasionally get this error, though it is inconsistent and most of the times localization works fine. I suspect the issue may arise with cards on the bottom and right sides being too close to the edges, and the copy therefore being a smaller size than I intend. In future debugging I may try printing out which card is being operated on before attempting to crop, in order to verify if indeed only edge cards are causing the issue. For now, though, it happens infrequently enough to be an occasional nuisance rather than a detrimental problem.
With these adjustments, I was able to successfully localize 10 cards simultaneously:
There are some duplicates, which is caused by inconsistent ordering of the rectangles identified by find_rects() between frames, so a card found on a previous frame could be overwritten by a different card in a subsequent frame. I resolve this issue later, but first I set out to solve a different problem: it never seemed to be able to find the card with 3 purple ovals.
Red and green cards mostly appear as solid white rectangles after blurring and thresholding, but the purple shapes appeared as black cutouts in the cards:
To resolve this, I came up with an additional set of preprocessing steps:
Find all black blobs smaller in size than a card (I used a threshold of fewer than 1000 pixels)
Use flood_fill() function starting from each blob's center and fill with white
This worked on most of the blobs, but noticeably in some frames the black blobs would smudge together as above, and the center would be over a white region, causing flood_fill() to not change anything. To fix this issue, I added the following steps:
Dilate image using a kernel size of 1
Find all black blobs smaller in size than a card (I used a threshold of fewer than 1000 pixels)
Use flood_fill() function starting from each blob's center and fill with white
Erode image using a kernel size of 1
The dilate step shrinks all the black regions in the image (the same as enlarging all white areas), causing the connected shapes to separate. This allows blob detection and flood fill to work as intended, and the final erode step returns the white areas to their original size. As pictured below, this process worked almost flawlessly, and any small black regions leftover don't seem to interfere with rectangle detection.
Finally, it was time to not only localize cards, but identify where each card is positioned. As mentioned previously, find_rects() returns the cards in the image in no particular order, so there's no notion of the top-left card or the second column, third row card. Furthermore, find_rects() would inconsistently identify rectangles between frames. However, I discovered last week that find_blobs() was actually very consistent between frames:
I devised the following algorithm utilizing find_blob's strengths:
Identify all the white blobs in the preprocessed image and add their center coordinates to a list
Sort the list by ascending y-coordinates
For every four cards, sort by ascending x-coordinates
When localizing a card, its "number" is the index of the blob center coordinates within the sorted list which are contained within the card's bounding box. Assuming no overlapping cards, this will return a unique number for every card in the frame
This numbering system is a bit rigid, as it assumes the cards are arranged with 4 columns and 3 rows, as opposed to 3 columns and 4 rows, but this is a reasonable assumption for now. If time allows, later the algorithm can be generalized to detect different rows as a large jump in y-coordinates.
After implementing this algorithm, the OpenMV board successfully localized all 12 cards simultaneously and correctly recorded the position of each one!
There are still a couple of things that could be improved in the future:
On some cards, the crop is too aggressive on top, as seen here on the card with 3 purple ovals. This is most likely due to find_rects() identifying the top-left corner as slightly more on the card area rather than just outside it. On other cards, the crop is just the right amount, as with the card with 3 green ovals.
The bottom cards seem to have too little crop on bottom, as a sliver of the black background is visible.
Next Week
Team Goal: Implement algorithms for whole design
Personal Goal: Implement Bradley's classification code on the OpenMV board using OpenMV image library functions
Unfortunately, I did not achieve my personal goal this week to both localize 12 cards at once and implement Bradley's classification functions in OpenMV. Expanding localization to 12 cards ended up being a much more involved process than I anticipated. The two of us met and reworked our plan: next week, I will implement his classification algorithms while he implements the SET game logic. After that, we should be ready to combine everything together into a functional prototype!
Great work Tyler! You have so many good solutions to the issues that came up during development.