# NXT-Spy – Fixes on the flag coordinates processing and of the output of the coloured ball processing

In order to get the coordinates of the flag from the flag processing library (zxing), we are using a function called getResultPoints() on the result object outputted. The problem is that this function only outputs three points, from now on I supposed it was the top left, top right and bottom left coordinates of the flag, but actually, I just figured that it is more complicated than that.

In fact, the flag processing only provides us the coordinates of the “target style” points we can find on any QR code in a random order. The problem is that to be able to get precisely the flag position, we need first to find the coordinate of the fourth corner of the flag and then to get the smallest rectangle in which all the points can fit corresponding to the flag’s position.
That’s why today I added to the flag processing process a function that calculates and draw the two rectangle involved in the process as shown on the image below: Assuming we have a triangle ABC made from the three QR points placed randomly (their order can change if we rotate the flag), the first step is to find the biggest side of the triangle (AC in this example), we then find its centre (D in this example). We then calculate the symmetric point of B centred on D (E in this example), it is the fourth point we were looking for and we can now calculate the coordinates of the red rectangle quite easily thanks to the java.awt.Rectangle add function (a function that adds points to a Rectangle object and outputs the smallest rectangle that includes all the points passed to it).

Now, the flag processing is more accurate and works for any inclination of the flag, an other good thing is that now it outputs a java.awt.Rectangle instead of an array of points.
After done all that on the flag processing, I thought that it could be interesting if the coloured ball processing could also output a Rectangle object instead of ukdave’s BoundingBox object. BoundingBox and java.awt.Rectangle being quite similar, I quickly did the modifications in the coloured ball processing process code and so now everything is more standard: the two image processing coordinates output are a java.awt.Rectangle object.

# NXT-Spy – Let’s analyse the images!

Now that we have a fully functioning remote controllable robot with a live image from its embedded camera, we can start thinking about the next step of the project: allowing the robot to automatically recognize colored balls, go to grab them, and finally bringing them to a zone corresponding to their color.

Hopefully, I have found a color recognition algorithm written in Java on the website uk-dave.com [source]. After analyzing this code, I noticed that one of the classes, called “ImageProcessor” was taking a BufferedImage as an input and outputting an image with the isolated colored ball and its coordinates.

It was very easy to implement in my system because I only had to convert it to a process, adding it two outputs, removing the useless functions and attributes and that was it.

Here is how uk-dave’s system is working :

1. First, the user has to set intervals of hue, saturation and intensity corresponding to the color he wants to match
2. Then, the function converts the input image to the HSI (hue, saturation, intensity)  color space.
3. It then applies the hue, saturation and intensity intervals set by the user to the image, keeping only the pixels from the original image that match the interval set by the user.
4. Finally, thanks to an algorithm called “Blob coloring”, it located the locations where there is the biggest bunch of pixels on the filtered image (pixels that are linked together).

After a lot of tweaking on the three different channels (hue, saturation and intensity), I finally managed to isolate the red ball :

So it is quite efficient but I have found one problem : every time the luminosity of the room we are in is changing, we have to modify a little bit the HSI intervals. One thing I have noticed is that the hue interval is almost independent of the luminosity of the room, after some research, it is because the hue represents the color in degrees (0° is red, 120° is green, 240° is blue and 360° is red again). With the images from the phone’s camera, I have found that the hue interval for the red ball is : 7° to 20° and for the blue ball, 183° to 236°.

Because the S and I channels are based on the luminosity, it could perhaps be interesting to use the luminosity captors from Lego (RGB or light captor) to set the values automatically based on the room luminosity.

The next step is to write the algorithm that will make the robot locate an seek for the balls.

#### Supervisor’s comment:

Well done; just what I expected to see given last week’s discussions.
The fact that you have also determined its limitations is also very good