Garden Robot Update 12

The long-promised CAD file.  I used LEGO Digital Designer.  This is mainly for record purposes.

Sadly, Garden Robot development might have to come to an end until next summer.  I have a lot of things planned for this machine, not the least of which is switching out the Android phone for a more capable device (Raspberry Pi) which should allow for less latency and hopefully more options when it comes to sensors.  Of course, there's a lot I would like to do for hardware as well, including some electric solenoids to control water flow, fertilizer, etc.

In any case, this project has proven itself highly valuable so far - I've been introduced to:

• C++
• Java
• Javascript
• Jquery
• HTML

And more generally:

• Basic networking (and how not to set up a server...)
• Communication protocol through byte packages
• Bluetooth MAC addresses and handshake
• SIMPLE finite state machine
• User control vs. autonomy

It's been a great project, one that I hope to return to sooner rather than later.  But for now I have a much better feel for what goes into making this sort of telepresent, connected robot, a whole folder in My Documents with code and resources, and a series of blog posts documenting my successes and failures ;)

Garden Robot Update 11

Some images.  First of all, the current state of the GUI.

Chrome:

Chrome (Android):

Safari (IOS):

And the robot itself.  Currently missing the valve and associated water tubing...

The track, nothing more than a 2X4 with a nail in each end, propped up on books:

The robot and phone plugs can be seen from behind.  Currently, I am having it "check on" each plant (designated by the blue strips of tape) at 10 minute intervals.  I was running 1 minute intervals earlier today and haven't had a problem with this version of the program...here's to hoping.

To do:
1.  Get tubing and water container together.
2.  Figure out how to get position data from the robot to the web-page (I spent hours this morning getting bluetooth messages from phone to NXT worked out, now I have to figure out how to do it in reverse)
3.  Get some soil and actually test this thing!!!

Garden Robot Update 10

It's been a full week since the last update.  Quite a few things have changed, but not a ton.  I have continued to make changes to the GUI and have implemented a rudimentary security system.  "Rudimentary" is right...it is, for now, just a prompt box that redirects the window to another page of my choosing (mwuhaha) if you don't put in the right password.

However, the most progress has been on the robot itself.  I have finally constructed revision 1.  For now, it is comprised of 3 motors (move left/right, raise/lower attachment, open/close valve).  It also has 3 sensors (ambient light, moisture, touch sensor to reset position, and light sensor to detect marks on the track and determine position).  I am currently using a 2X4 as a track, with 3 strips of electrical tape to mark out the positions of the plants.

I have also finally gotten around to coding the position control of the robot.  This one was a little bit more tricky...I wanted a system where something (either a user request or a preprogrammed routine) could ask the robot to move to any position, regardless of where it starts.  For now, I have two parts to handle these two different requests.  (1) If the bluetooth mailbox returns a value of 0, the program switches towards a preprogrammed routine.  (2) If the bluetooth mailbox returns a value that is not 0, the user is requesting a change in position.

1. After initial reset, the main task starts another task that runs simultaneously for the rest of the program.  This sub-task simply waits one second, then increments three variables - timer1, timer2 and timer3.  The main task checks to see if any of these variables has exceeded a certain threshold - if so, it calculates the amount needed to move to position 1, 2 or 3 from where it currently is, and passes the MoveRobot subroutine this calculated value as an argument.
2. The main task sets the desired position to the value read out of the bluetooth mailbox.  This value is the user's requested position.  As before, it calculates the amount needed to move to this position and calls the MoveRobot subroutine.  

Significantly, since these two branches are on either side of an if/else statement, they cannot execute at the same time and thus cannot interfere with each other.

The MoveRobot subroutine uses the downwards-facing light sensor to check for the electrical tape markings on the track.  Depending on if the argument is negative or positive, it moves backwards or forwards, incrementing a counter variable as it detects a piece of tape. Once the argument matches the counter, it breaks out of the loop.

Couple things:

1. The direct control I currently have programmed will mess up the position of the robot, which might cause it to become confused (it thinks it is in position 1 when it is actually in position 2, etc.)  To fix this I should probably get rid of direct commands and add a third branch in addition to the above two.  This way, the robot could reset itself after the user finishes controlling it.
2. I still haven't found a way to get the robot to "push" vital information to the server without a user-initiated GET request.  For example, whenever the robot changes position (ex. from 1 to 3) it should cause this data to update on the web page.  Same for valve state and attachment position.
3. Figure out port forwarding - for now the server is only accessible to devices connected to the same network.  However I don't really yet plan on using this outside of the same network...for now.

Wow, that got long.  Pictures/screenshots/code to follow.

Garden Robot Update 9

Some aesthetic updates.  Putting things in a table makes it look a bit better...I guess.  Currently I have implemented control for left and right (rotating motor A in each direction when clicked and stop when released).  The ambient light label refreshes when clicked.

I got the app running on an older Android phone today (running Android 2.2).  I initially had some difficulty with NanoHTTPD throwing a null pointer exception, but fixed that by updating to the latest version of NanoHTTPD.  

On the hardware side, the 350 milliamp converter I hacked into the NXT seems to be working fine for the single-motor rotation, but I picked up a 2.2 amp converter that needs to be placed on.  I want to put in some kind of protection, like the fuse and diode in this example.

To do:

1.  Fix the moisture level command so that it reports the proper values.

2.  Find a way to scale the ambient light command from 0 to 100%.

3.  Start on the actual design of the robot...

4.  Get the 2.2A converter together.  

5.  Figure out how (if?) the NXT should push data to the server.  For example, I think it should display its current position and valve setting.

6.  Put in new controls for open valve and close valve.

Garden Robot Update 8

Quick update.  After WAY too long struggling how to get text info (like sensor readings) from the NXT to the webpage without refreshing, I looked into using Ajax in jquery and it turned out to be an EXTREMELY simple "get" command.  No need for Canvas text, no need to mess with the HTML (all of which, unfortunately, I tried.)  Basically, this in javascript:

$("#refresh").mousedown(function() {
    $.get('lightrefresh', function(data) {
    document.getElementById("lightlabel").innerHTML = ("Ambient Light:
" + data);
    });
    $.get('moisturerefresh', function(data) {
    document.getElementById("moisturelabel").innerHTML = ("Moisture Level:
" + data);
    });
});

(the key is the second argument to the $.get request, as "function(data)" will process whatever data you are getting back)

And in the .java file:

else if(uri.equals("lightrefresh")) {
    Log.w("HelloServer", "lightrefresh");    res = new Response(Response.Status.OK, MIME_PLAINTEXT, "999");}

else if(uri.equals("moisturerefresh")) {
    Log.w("HelloServer", "moisturerefresh");    res = new Response(Response.Status.OK, MIME_PLAINTEXT, "111");}

Of course, later, the "999" and "111" will be replaced with return values from the NXT as I discussed earlier.

To do:
1.  Continue to lay out the GUI.  (It looks so ugly now, I wonder how to make it look any better...?)
2.  Actually make it work with the NXT.  Put in the direct commands to control motors and get input values.
3.  Start to figure out some form of security...I really doubt I know enough to actually encrypt any data but I want at least a form of password-protection.  Like, really annoying password-protection.
4.  Start to design the actual thing...I certainly want some sort of CAD file, but I don't (currently) have SolidWorks, so I should probably look into MLCAD or something.  Or LDD, which I have used extensively in the past.
5.  Figure out where I can actually run this thing.  I need to talk to some people.

Garden Robot Update 7

Decent breakthrough today.

Basically, I finally got the Nxtbotguard streaming solution to work with my current implementation without modifying the Nanohttpd java file.  That's the middle canvas element in the image.  Still not entirely sure how it works, as I have borrowed a fair amount of code.

Also using the Nxtbotguard solution I have implemented the bottom two arrow buttons (as images) which respond as expected when clicked (using Jquery, which seems awesome and I really need to learn more about) Nothing works with the NXT...yet.

The biggest issue is updating text however.  I want to update the ambient light value from the NXT through dynamic text, and I am a bit of a standstill how to do that...currently trying to use Canvas, like the streaming image solution, but it doesn't work.  Yet.

Still a lot to do, but might actually see a bit of progress at last...

Garden Robot Update 6

Quick update.  I know these aren't as frequent as they should be, but hopefully I can cover most of the important things I have done since the last one.

First of all, after a few days, I finally figured out a nagging issue with the existing NXTBotGuard streaming solution.  It turned out that my browser was caching the image, causing it to only update on refresh.  I forced this out by adding this line to the NanoHTTP header:

res.addHeader( "Cache-Control", "no-store");

With this out of the way, the streaming solution worked fine, but I didn't (and still don't) understand it.  Apparently the developer " modified the NanoHTTPD  server to serve files directly out of the compressed assets folder that comes with your app package (apk)."  I don't get how he is doing that, so I took the simpler solution of just reading the index.html file via AssetManager:

try {
    // Open file from Asset Manager
    InputStream inputStream = getApplicationContext().getAssets().open("index.html");    BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));    String line = "";    while ((line = reader.readLine()) != null) {
        answer += line;    }
    reader.close();
} catch(IOException ioe) {
    Log.w("Httpd", ioe.toString());}


return new NanoHTTPD.Response(answer);

Which works perfectly for html but doesn't seem to work for Javascript, as I discovered tonight, as my index.html file will call the javascript functions as expected when I access it locally, but not when I have it read using the above method.  Grr.  (For future personal reference, this index.html is in the "Code Snippets" folder)

So in general, I think that a quick and dirty method to get all this done would be to just host an ever-changing image from the camera using a simple html tag, and have that consistently update to simulate a video.  This should work using the above method.  Although that (should?) work, the dynamic canvas property of html5 has intrigued me so much that I am dying to use it, as well as learn more about html and javascript (which I would use to update the canvas element, and maybe some other things).

So...now, I am going to look online for any more solutions that I can understand about reading directly from the Asset Manager, or just work on my dirty method in pure html and see if it works.

Garden Robot Update 5

Success!!
This took way longer than it should (and is a much more major victory to me than it should be) but I finally got a web page to host the current numerical sensor value of a light sensor running on my NXT.  This took several days due to some stupid problems:

1.  The NXT Direct Commands documentation explains that the getInputValues command sends a return byte package of 16 bytes, but I forgot that since I am using Bluetooth, another 2 bytes are tacked on the beginning.  That caused my interpretation to be offset by 2 bytes for the longest time.  Oops.
2.  This one was even worse, but I blame it on not knowing anything about Java.  So I didn't understand how the InputStream works, and I didn't at first realize that the return values from previous direct commands were already stored in the buffer, causing me to read the wrong value.  Secondly, I didn't realize that the InputStream is reset each time I called the command (still not sure why...) So as of now, I am having it skip over the 5 bytes returned from the setInputMode command before reading the 18 byte package from the getInputValues command.

Here's what I've got...

long skipped = stream.skip(streamskip);
int bytesRead = stream.read(data,0,18);
Log.w("NxtBrick", "Bytes Read: " + Integer.toString(bytesRead));
String x = Integer.toString(((data[12] & 0xff) << 8) | (data[13] & 0xff));

Log.w("NxtBrick", "Sensor Value: " + x);
return x;

And to write to NanoHTTPD:

NxtBrick nxt = ServerActivity.getNxt();
String answer="";    try {
        //nxt.playTone('a','c');        nxt.setInputMode((byte)0x00,(byte)0x06,(byte)0x80);        answer=nxt.getInputValues((byte)0x00);    } catch (IOException e) {
        e.printStackTrace();    }
return new NanoHTTPD.Response(answer);

Garden Robot Update 4

This happened today:

Still not sure how it works, but it's being hosted by my phone, which is cool.  I somehow ran across it randomly when wondering why none of the NanoHTTPD examples I found worked (doubtlessly because of my utter lack of Java knowledge...ugh).  But it works.

Next step is reading sensors from the NXT, where I am currently stuck.  I have delved deep into the NXT SDK where they have documentation on some really cool direct command protocols, which do not require a user-developed program running on the brick (excellent for direct remote control if the program is doing something wrong).  Unfortunately, while I am having no issue using the direct commands developed by the SmartLab NXTBotGuard project, I seem to be getting some bogus data when I call the getInputValues.  It changes whenever I reboot the program but not in the middle of it...it is weird.  I am sure it is a dumb Java issue.  For reference purposes, here's my buggy code:

public String getInputValues () throws IOException
{
    byte [] msg = new byte [5];            msg[0]=(byte)0x04;            msg[1]=(byte)0x02;            msg[2]=(byte)0x00;            msg[3]=(byte)0x07;            msg[4]=(byte)0x00;    sendMessage(msg);    Log.w("NxtBrick", String.valueOf(msg));
    if(in !=null)
    {

        String x= String.valueOf(in);        Log.w("NxtBrick", x);
       /*        String x = null;        int bytesRead;        byte[] contents = new byte[100];        while ((bytesRead = in.read(contents)) != -1)        {            x = new String(contents, 0, bytesRead);            System.out.print(x);            Log.v("NxtBrick", x);        }        */
        return x;    }
    else    {
        return "Null";    }

}

P.S.  Don't use it, it doesn't work.

Garden Robot Update 3

It's been a little while, but unfortunately not much has progressed.
I have contacted the developers of the MonoBrick Tunnel and haven't heard anything back yet.  This has encouraged me to go with route 2 and try to develop a custom app to handle connectivity and web serving.  It has been an interesting couple of days...

First of all, I have no clue what I am doing, so I first tried to get the NXTBotGuard project I linked earlier to work.  After downloading Android Studio, I found that I could import the Eclipse project and convert it into an Android Studio project, which is awesome.  However, running it, I ran into a lot of issues.  Firstly, the NXT brick refused to connect reliably, with the app crashing at random intervals.  Even when the Bluetooth connection succeeded and I was able to access the hosted web page, entering in the password caused the app to crash.  Reading the Logcat files, I determined that something concerning the phone's camera stream was causing it to crash, but due to my limited Java knowledge, I (still) have no clue what.

With this frustration I decided to bit the bullet and just start personal development of a custom app.  Using copious amounts of copy/paste from NXTBotGuard and a plethora of external websites, I managed to form a simple app that enabled Bluetooth, listed paired devices, scanned for devices, and displayed them in a list view for user input.  So far, it is able to connect to the NXT quite reliably, and does not crash like the NXTBotGuard app.

However, the next step is where I am (still) stuck.  NXTBotGuard makes use of a lightweight HTPP server known as NanoHTTP.  Again, I have no idea how it works, so that is my next project.

First to figure out Java's multi-threading architecture...I am so lost...

Garden Robot Update 2

Okay...don't be me.
So on the software side, using the MonoBrick C# Communication Library, I managed to set up a simple Windows Form interface that allowed for manual polling of the moisture and light sensor over the MonoBrick Tunnel running on an Android device.  But then I wanted to set up video streaming...

I couldn't get it to work.  I even tried downloading MonoBrick remote (which worked on a different computer and back when I was running Android 4.4.4) and it still wouldn't stream.  I speculate that upgrading to 5.0.2 has caused a lot of issues, including the "menu crashing" error.  Everything worked except for the streaming (and the fact I still couldn't access the menu).

So I tried re-installing the app.  This was the big mistake.  This removed the NXT from the "remembered device" list previously only accessible through the menu.  And now, since I can't get to the menu, there is no way I can reconnect the NXT.  Oops.

So while I wait for a response from the MonoBrick folks, I decided to screw around with other alternatives.  At this point, the second option from my previous post is looking pretty good...but there's still the Java problem.

Garden Robot Update 1

Okay...first update.
Couple mechanical pictures:

I spent quite a long time looking for a valve we had sitting around, and Mom found one in about 5 seconds.  In its former glory, this faded yellow device connected to a hose water spigot, allowing for the fill of water balloons as regulated by the yellow lever on the close side.  I used a 5:1 gear reduction and a single NXT motor to operate the valve (Yes, this is the first time ever I have glued a personal LEGO.) You can't see it in the picture, but the black hose is from a drip irrigation system I got years ago, and is currently connected to a crudely-cut water bottle and sealed with my mediocre hot-gluing skills.  The water bottle acts as the water reservoir and feeds via gravity through the valve.  By opening the valve for different amounts of time, I can adjust how much water is fed to the plant.

 Here is my current test setup.  Three containers of dirt (thankfully "science experiments" are encouraged in my house) and two nails for use as probes.  They are used in a basic "touch sensor" setup, in series with a 2.2kOhm resistor and connected to pins 1 and 2/3 of a sensor port.  More conductivity means more water present, and a lower raw  sensor value read by the NXT.



After a long, multi-day drama (see below), I'm currently at this code:

//Macros for moisture thresholds (raw probe values)
#define HIGH 500
#define MED 300

//Macros for watering times (ms)
#define HIGHWAIT 2000
#define MEDWAIT 1000

//Macros for moisture test wait time (ms)
#define TESTWAIT 5000

int MoistureLevel;

sub MoistureTest()
    {
int raw;
//Apparently the following 2 lines are necessary with updated firmware.
     SetSensorType(S1, SENSOR_TYPE_CUSTOM); //Set the sensor type.
     ResetSensor(S1); //Necessary following SetSensorType
     int data[10]; //Create a data vector with 10 elements
     for(int i=0; i<11 10="" epeat="" i="" p="" times="">             {
             raw=SensorRaw(S1); //Set "raw" variable equal to probe reading
             data[i]=raw; //Use the loop counter as index to save raw in vector
             ClearScreen(); //Don't forget this
             TextOut(0,LCD_LINE1,"Current value = ");
             NumOut(0,LCD_LINE2,raw); //Display current value
             Wait(100); //Wait 1/10 a second
             }
     MoistureLevel=ArrayMean(data,NA,NA); //Reduce noise by averaging vector
     }

sub Water(int WaterLevel)
    {
    if (WaterLevel==0) //Do nothing for an argument 0
       {
       }
    else //Open valve, wait the time given by the argument (>0) and close valve
        {
        RotateMotorEx(OUT_A,100,-470,NA,false,true);
        Wait(WaterLevel);
        RotateMotorEx(OUT_A,100,470,NA,false,true);
        }
    }


task main()
{
     while (true)
{
Wait(TESTWAIT); //Wait for a while before testing moisture level
           MoistureTest();
           PlayTone(262,400); //Play Tone to assert completion of subroutine
           ClearScreen();
           TextOut(0,LCD_LINE1,"Mean value = ");
           NumOut(0,LCD_LINE2,MoistureLevel); //Display this average value
           //This structure displays LOW, MEDIUM or HIGH on screen
           //and passes a watering time argument to the Water subroutine
           if (MoistureLevel <=MED)
       {
       TextOut(0,LCD_LINE3,"LOW");
       Water(0);
       }
           else
          {
               if (MoistureLevel >MED && MoistureLevel<=HIGH)
                  {
                  TextOut (0,LCD_LINE3,"MEDIUM");
                  Water(MEDWAIT);
                  }
               else
                   {
                   TextOut (0,LCD_LINE3,"HIGH");
                   Water(HIGHWAIT);
                   }
               }
           }
}


The NXC Drama:

1. My first thought:" I should have upgraded to the enhanced NXC/NBC firmware ages ago.  It has support for up to 4D matrices, which is awesome." Unfortunately, it's not that easy.  
2. Setting up BricxCC was a pain.  I didn't have the MINDSTORMS software already installed on my computer, so I needed the Fantom Drivers from this LEGO page (which they actually still host, much to my shock).
3. But I got a weird .ini file error when running the autorun.  I found someone else with the same issue, and instead used the drivers from this page (as linked by the second post in the thread).
4. With the drivers installed, BricxCC was able to connect to my NXT (I had some stock firmware already on it).  For a while I was stuck because an array command I was trying to use wasn't recognized.  I later figured out it was because the compiler will recognize the firmware you are using in your NXT and throw an error if you use commands intended for the enhanced firmware.  So I was off to find the enhanced firmware...
5. I first tried to flash the firmware linked on this page onto the NXT using BricxCC's Firmware Download feature.  It worked perfectly, but when I ran my program, I kept on getting a "0" raw value back from the sensor, which couldn't be right.  I assumed it was a firmware issue, so I looked elsewhere for a different firmware.
6. I made the mistake of getting a firmware binary from this page and attempting to use BricxCC's Firmware Download feature to flash it to my NXT...oops.  I had no idea what I was doing, and it totally failed, bringing my brick to "quiet clicking" model.  If I had read more closely, I would have been able to tell that I needed to compile it first, which I didn't.
7. I then tried to revert back to my earlier firmware using the same feature...it didn't work at all, but BricxCC didn't throw an error.  I was perplexed, because my NXT was reduced to quiet clicking, and even though I downloaded the stock LEGO firmware, it wouldn't respond.
8. Getting a bit worried, I booted up my old computer and started up NXT-G 1.1  Thankfully, I was able to flash the stock firmware onto my quiet clicking NXT.  After repeating the whole "firmware binary" episode a second time, I read the page and realized my mistake.
9. This made me realize that my first firmware flash must have been correct, and the problem was in my code.  After some searching through the documentation, I eventually stumbled upon the weird reality that you must first use SetSensorType to define the sensor, and then use ResetSensor, and you can then use SensorRaw to read it as before.
10. After some more screwing around with the firmware versions (the flash contained two different versions of the enhanced firmware, and only one worked), I settled upon using the "lms_arm_nbcnxc_132.rfw" version.

TL;DR:  Use this Fantom Driver and the "lms_arm_nbcnxc_132.rfw" file from the enhanced firmware linked on this page.  Don't use BricxCC's Firmware Download feature when restoring a clicking NXT, use NXT-G's instead.

Garden Robot

Hello World, it's been a while.
Unfortunately, real life has gotten in the way of my MINDSTORMS pursuits.  That being said, in honor of the fact that it is summer, I have decided to take a far more structured approach to a much larger project.  I won't get into it in depth here, but I essentially want to make a garden robot.  That is a bit broad.  To narrow it down, I have some specifications:

1. Use LEGO MINDSTORMS NXT for actuation.
2. Completely autonomous/maintenance-free for extended periods.
3. Water plants (only, for now).
4. Relay critical information over the internet (moisture content, PH levels, ambient light, etc.).
5. Be completely controllable over the internet (all functions accessible).

Numbers 4 and 5, particularly considering specification 1, are apparently quite complicated.  The NXT, unlike the EV3, has no embedded WiFi capability.  Thankfully, with a bit of digging, I was able to chance upon two different projects:

The folks at MonoBrick have created a set of tools to allow for enhanced communication with the NXT and EV3.  Notably, they have developed a "tunnel" Android application that allows you to connect the phone via bluetooth to the NXT and via the internet to a client.  Using their handy C# communication library, you can develop programs to send direct commands to the NXT.  Another huge plus is the capability to stream live video from the phone's camera.  For the time being, this is my method of choice, but there are a couple drawbacks:

1. Whatever you have running the program on (which for now is my laptop) has to be working...at all times.  Considering I'm a student, this is impractical.  Therefore, whatever code I write on this has to be in addition to the autonomous code that is running on the robot.  This isn't a bad thing, but it will require a bit of creativity to get the two different languages to work with each other seamlessly.
2. The tunnel app, after upgrading my phone to Android 5.0.2, seems to have lost its menu.  This is an issue if I want to adjust anything, such as which NXT the bluetooth connection pairs with.
3. I don't really understand this part, but I think that if I switch WiFi networks, I have to reconfigure the client listening port.  Not a big issue, but certainly a limitation due to my restricted programming knowledge.

Secondly, I found another project by SmartLab that does something similar - you place an Android phone on the robot, and it connects via Bluetooth to the robot and via WiFi to the internet.  But this one has a very different type of flexibility - the phone runs its own HTTP server, allowing you to access and control the machine via a webpage.  This is exciting because it removes the need to use a local program to access the robot over the internet.  But for now, this is not my method of choice for two simple reasons:

1. It's written in Java.  I don't know Java.  At all.
2. The program they used as an example works by sending direct commands to the NXT.  Considering I have specified that the robot to be autonomous, I will need to figure out how I can write it as a predetermined program on the phone.  I'm sure it's possible, but we are entering the realm of app development here, and I don't know anyone who develops Android apps who might be able to help me with this.

So, for the time being, while I am working out prototypes and proof-of concepts, I have decided to do as follows:

1. Write a NXC program (this I've dabbled in before) that runs on the NXT to control autonomous function.
2. Experiment with the MonoBrick library in VisualStudio to work out a communication method between the Windows Form application I'm planning to write (for the GUI) and the robot's data logs.
3. See if I can work out streaming video (it works in MonoBrick's remote program...I should be able to get it to work on mine too!  Right?)
4. For all of this, use haphazardly-designed prototypes (Okay, maybe not completely) to make sure I can actually build this thing mechanically.