This Week
Team Goal: Finish implementing classification functions in OpenMV
Personal Goal: Continue experimenting with preprocessing steps to isolate shape bounding boxes
This is my second week working on shape counting, and spoilers: this week I successfully taught the H7 how to count to 3!!! I left off last week with some sub-optimal preprocessing and thresholding code:
I was having a difficult time differentiating between the white and colored regions of the images, so that was the first thing I set out to fix this week. The threshold function in OpenMV takes in minimum and maximum values for each color channel in the LAB color space. The LAB color system describes colors using an L (lightness) value between 0 (black) and 100 (white), an A (red to green) value between -128 and 127, and a B (blue to yellow) value between -128 and 127.
Through trial and error, I found that A from -20 to 10 and B from -39 to 10 provided a good threshold for all the cards. L from 69 to 100 worked for most shapes, but for some shapes, a lower bound of 72 or at most 80 was needed:
To determine which minimum value to use for L, I still use the automatic threshold function, but if L is less than 69 I set it to 69, if it is less than 80 I set it to 72, and for all values greater or equal to 80 I set it equal to 80. At first, I tried just using the L value returned from get_threshold(), and also simply constraining it to the range [69, 80], but sometimes the value returned would cause the background to merge with the shapes (pictured right).
One final optimization to thresholding was constraining the region of interest considered in the get_threshold() function to be 5 pixels cropped in on all sides since card localization would sometimes leave a sliver of the background in the isolated card image, which would throw off the color distributions.
Now, it was time to count some shapes! The first approach I tried was as follows (credit to Bradley for the idea):
Find the bounding box surrounding all white pixels in the thresholded image
Use the position and height of the box to determine how many shapes are on the card
A major roadblock for me last week was trying to count the number of distinct contours on the card, which failed whenever there were gaps in the perimeter of the shape causing a single shape to appear as multiple disconnected pieces. The advantage of this new approach, besides being much simpler, is that it doesn't rely on the quality of the shape contours, but rather only on where they are roughly located. Using this approach, I was able to correctly determine the number of shapes on all 12 of the shown playing cards:
The counting decision was purely based on the y-coordinate of the top-left corner of the red bounding box: if it was closest to y=30 there was 1 shape, y=18 for 2 shapes, and y=4 for 3 shapes (y=0 is the top of the image). This worked quite well but failed in cases where visual artifacts would incorrectly extend the bounding box:
I tried adding a modified erosion step before calculating the bounding box that would erode pixels with fewer than 2 white neighbors, which would help maintain the thin shape outlines on the empty fill shapes. This helped with the smaller visual artifacts but was unable to fix the larger ones.
I devised an alternative approach that to be more robust against visual artifacts:
Create masks for each number of shapes that isolate the regions containing the shapes
Going in ascending order starting from 1, apply each mask and count the number of pixels outside the masked region
If the pixel count is above a certain threshold (I found that 100 worked well) then try the next shape
Otherwise, the mask is a good fit, meaning the number of cards has been determined
It is important to start at 1 instead of 3 since the mask for 3 would cover the shape corresponding to 1, resulting in a low pixel count even though it's the wrong mask size.
This algorithm ended up being very consistently accurate and robust against random visual artifacts. It took trying out several different approaches, but I am excited to finally have a really solid shape counting system in place!
Next Week
Team Goal: Determine which algorithms/approaches we want to use in the end preprocessing and classification system
Personal Goal: Implement color and fill identification
With shape counting finished, next week I aim to move on to color and fill identification. Bradley has been exploring using a CNN to perform shape classification, so we will need to experiment with it in OpenMV to decide which classification approach we want to move forward with. Altogether, we're aiming to have all classification steps completely finished after next week to meet our week 7 milestone!
Comments