NXT-Spy – Thoughts about how to implement the Bluetooth connection

So here we go, let’s think about the different ways to create a Bluetooth connection between my Android phone and the robot.

First of all, because my supervisor (Jon Kerridge) gave me Severin Fichtl’s work on the Bluetooth connection between a PC running JCSP and a robot running JCSPre, I started to take a look at it.
After reading his code, I realized that I could not base my work on his because to create his connection, he used the NXT PC library that is not compatible with Android (a Bluetooth device driver problem).

So there is only one other way to do the things : create my own socket based net channel adapted to Android phone and add it to my JCSPae implementation. It should be something achievable because as my proof-of-concept robot control from the phone showed it, it is possible to communicate with the NXT robot with Sockets. So it should be something similar to Kevin Chalmer’s work on JCSP wi-fi implementation (that is also socket-based).

Also, I have noticed that no Bluetooth net channels existed on JCSPre, so I am probably going to write an implementation of those to have my project even more parallel!

Supervisor’s comment:

First, many thanks for the vastly superior interface to the blog; it will make life much easier, when I get used to it!
I received a first draft of some of his dissertation.  We read this together and I made lots of comments, much green ink!  Fundamentally the material was well structured but in place left the reader to do too much work on their own.  However this was a very good first piece of writing to get you in the mood.  Lets see a revised version for next week.
He also has done a lot of coding to implement the channel based connection between the android and the robot, yet to be finished, but given he was away over the weekend this represents a lot of work.
His underlying interface is a socket over which he sends data in a specific format for the control of the robot.
There is a video to demonstrate that this is working because he controls the phone from his PC using Wi-Fi, which then transfers commands to the robot using his Bluetooth capability.  The fact that he built the video at 3am in the morning after he got it working is perhaps a little worrying.  I doubt he would do the same for his dissertation at the moment, that will come later, unless I stop him from coding too much!!

NXT-Spy – A solution to my Android version problem?

After some research about how to get Android 2.2 on my HTC Hero (because HTC haven’t planed to port Android 2.2 to it yet and because I need it now), I finally fell on a team called “VillainROM” [source] who created a project called “FroydVillain” that consists in porting Android 2.2 on HTC Hero.

Their current version is 1.5 and it seems to be quite stable. The only problem is that I would have to flash my device risking to brick it and because it is not an official release of Android, my warranty probably won’t accept to fix it.

I fell on this website [source] that says that it is very unlikely to brick the phone, so it’s decided, I will try to install it.

Supervisor’s comment:

DO NOT do this

NXT-Spy – Some studies about the weight of the transfered images

The tests done previously on the camera application revealed that the images sent by the phone weighted 12 bits/pixel.

So I borrowed a book at the library called “Compressed Image File Formats” (John Miano 1999) to see how I could have a better image encoding than the one I am currently using. Very surprisingly, I learn that JPEG images size is based on the redundancy of the colors and so for two images of the same format, the size can change completely. This last point lead me to understand that what I thought was a JPEG image was actually a raw image file encoded in 12 bits.

So now that this is understood, let’s go back to the code and try to figure out why Android won’t encode my image in JPEG even if I have already clearly specified in my code the compression (parameters.setPreviewFormat(PixelFormat.JPEG)).

So after having done some tests, I realized that the setPreviewFormat function is not working at all. After some research, I realized that a lot of websites said to use the ImageFormat class, but the problem is that this class appeared only in the Android 2.2 version and my Hero runs 2.1. So I went on Android’s website to see the additions in Android 2.2 [source]. And in the “Graphics” section, it says that not only, they added the ImageFormat class but also that there is a “new YUV image format API to enable compression from YUV to JPEG and manipulation of YUV data.”. So I think that the answer is quite clear : I won’t be able to compress my images unless I wait for an Android upgrade for my phone.

Supervisor’s comment:

I think we can do something better by only sending values for the pixels that have changed but it may be that by reducing the image size the frame rate will be acceptable.

NXT-Spy – The camera images transfer now works in parallel!

So after my successful work of yesterday, I just had to code the camera image transfer today.

Si I first drew a diagram to know which classes I was going to develop and ow they were going to interact with each other:

So, the program stays idle while the Wi-Fi connection is not initialized. As soon as everything is ready, the PC’s connection class asks an image to the phone’s connection class that asks a new image to its camera and send it to the PC’s connection class. Then, the PC decodes and displays it and it loops like that again and again until one of the programs stops.

Because I already had all the elements in hand, everything came together quite easily. The only problem is that sadly but as I expected, the frame-rate is still the same : not quite satisfying yet. I did some tests and figured out that if I send a 400×240 image from the phone, it weights 144kb and I get a frame rate of 0.6 frame / sec. But if I send a 176×144 image, it weights 38.016kb and I get a frame rate of about 2 frames / sec. So exactly 1.5 bytes (12 bits) per pixel.

The only way I could perhaps enhance my frame rate would by by changing the format of the image sended by the phone.

So there seems to be two solutions to my problem :

  • The first one would be to change the format of the image being sent from the phone (for the moment it is a YUV encoded JPEG image). Event if it seems that I cannot change the YUV encoding to RGB, I can probably change the lossy compression, event if I think that JPEG is already an efficient compression.
  • The second is to simply send the images in the smallest format (176×144) and then just expand them on the computer.

So that’s it for the camera images transmission. Now the next step will be to make the Bluetooth work, it will probably be way more complicated because it is probably going to be more specific to the Android architecture than the Wi-Fi so perhaps some classes to be rewritten… I cannot work on it for the moment because my robot doesn’t have any more batteries.

Supervisor’s comment:

Well done so now all that is required is to connect the phone to the robot using Bluetooth with JCSPae.  This is non-trivial!!

NXT-Spy – First successful parallel Wi-Fi data transfer between the phone and the PC

Yes, after several hours trying to get it work, I managed to do a “full duplex” data transfer between my HTC Hero and my PC through a router.

First, I read some examples here on how the data transfer over a network were working in JCSP and tried to implement it. So I wrote a simple Android app for the phone and a simple Java app for the computer, I ran it and as usual, it did not work.

I finally figured out that I first had to initiate a server connection on each devices before I could do any client connection thanks to Kevin Chalmer’s examples [source], so I did a little UI for the phone with two buttons : “Start server” and “Connect”. Everything looked nice and should have perfectly worked well but I fell on a very strange situation : the messages from the phone arrived perfectly on the PC but the PC just could not send any messages to the phone.

I first thought that I did a mistake about the ports I was using so I changed them several times with no result. I scanned the requests with Wireshark that just confirmed the fact that the requests from the PC were not reaching the phone.

I then thought that I needed to add some permissions to tell the phone to keep its server port opened but there were no such things.

I finally, desperatly went into the org.jcsp.net sources to reverse engineer the server socket creation hoping to see where the problem was by using log report messages. As predicted, in the socket’s run method, the thread was stuck waiting for a client to come.

So the phone was waiting for a client but the computer was trying to connect, a really unusual situation. So I kept on searching for a solution until I did a modification that solved my problem : I just removed the InetAddress parameter from the ServerSocket because it was not necessary and it worked.

But I didn’t want to just modify this class and leave it like that, I wanted to find a real solution to my problem that looked like a server IP address problem. So after a little bit more studdies of the I finally found where the problem was coming from in the TCPIPLinkServer class :

In fact, when we don’t supply any IP address for the server connection, the TCPIPLinkServer class calls this piece of code to get the IP of the network card the will be used by the server :
InetAddress toUse = InetAddress.getLocalHost();
Instead of that, on Android, it returns !

So the solution to my problem was simply to provide the phone’s IP address as a parameter of the server initialization. And it worked!

So I now have all the elements to code the camera images transfer over Wi-Fi and I will probably do it soon ;).

Supervisor’s comment:

For the meeting on 12th October please will you write up how you built JCSPae to include the connection to the Android widgets