Utility Meter Monitor

I have had a simple utility meter monitoring system working in my parents and my house for a few years.  It uses an NSLU2 single board computer connected to a simple USB input/output interface, as described at http://meterserv.webhop.net.
It worked fine for at least 3 years, but started to misbehave recently.  I have not been able to determine if it is a hardware or software fault - I suspect that I may have worn out the USB memory sticks that I used for the root filesystems.
We have gone for a hardware change to see if that is the problem, but I want to develop the software a bit more too.
Future developments include:

• Add support for cheap electricity meters (e.g. OWL USB energy monitors).
• Add support for temperature sensing.
• Better web interface (the current one uses perl based CGI scripts, which are awful to maintain).
• A simple (end user friendly) software upgrade process.

I want to get rid of the perl cgi scripts because I don't really speak perl, so they are hard to follow - will change them for an ajax based front end with python server side scripts.

The first stage is to get a sofware development environment working.  The original version used the nslu2-linux slugos operating system.  This has been changed quite a bit since I used it, so I decided that as I will need to do a complete re-build, I will standardise and use OpenWRT for all my little embedded system projects.

Compiling OpenWRT for the nslu2 was nice and easy - in the menuconfig just select the processor as an ixp4xxx and the sub-type as nslu2.

Edit target/linux/ixp4xxx/base_files/etc/config/network to give a default network configuration that will work on your system (the openwrt default ip address is 192.168.1.1, which clashes with my router....).

This creates a file called bin/ixp4xxx/openwrt-nslu2-squashfs.bin which can be flashed onto the nslu2 as follows:

• Put the NSLU2 into upgrade mode by switching off, then powering on while holding in the reset button for about 10 seconds.   When the status LED changes colour (at about 10 sec), release the reset button.  The status LED now flashes different colours (very subtly different colours if you are colour blind like me!).
• Use upslug2 (sudo apt-get install upslug2 on Ubuntu) to upgrade the NSLU2 using:  upslug2 -d wlan0 --image=openwrt-nslu2-squashfs.bin
• The NSLU2 will re-boot, but then take a while (a few minutes) to set up a writeable overlay file system before being accessible over the network.

This showed that I could build software for the NSLU2 using openwrt, but now I need to port meterserv to OpenWRT, which will be a separate post....

OpenWRT on Linksys NSLU2

I have had a couple of NSLU2's running a little application to monitor utility meter readings for a few years now (meterserv).   It worked fine until the autumn when my Dad's one stopped recording readings.  I have not been able to find out why - it looks ok.  It may be that the interface card has broken, or I do wonder if I have worn out the USB flash drive that I am using for the root filesystem.

In my inability to find a fault, I decided to do a clean install on a usb disk to rule out software and flash drive problems.   They had been using OpenSlug (now called SlugOsBE) from nslu2-linux, but it is now very out of date, because I have not updated them for a few years.  This means that all the links to package directories are broken etc.

Given the recent success in building OpenWRT for the bifferboard, I decided to use OpenWRT for the re-build.   These are my notes so I can do it again next time....

• In the MenuConfig system, set the target system to 'Intel IXP4xx', sub target to 'Generic' and target profile to 'Linksys NSLU2'.
• Because this is not my new router, I do not want it to use the default IP address of 192.168.1.1.  This can be changed in the 'Image Configuration' bit of the configuration menu.
• I would like to re-code some of the meterserv software in python, so I need python and pyusb installed in the flash image - these were selected from the 'Languages' section of the openwrt menu.
• Make builds a firmware image in bin/ixp4xxx called openwrt-nslu2-squashfs.bin, which is an 8MB image - there is a 16MB one too, but I am not convinced that my nslu2 has that much flash memory.
• You have to put the nslu2 into 'recovery mode'  by holding the re-set button in while you power up the machine - you have to release the reset button as soon as the status LED changes colour, after about 10 seconds.  The status LED then flashes a couple of different colours.
• Flash the new firmware image onto the NSLU2 using upslug2 (sudo apt-get install upslug2 on ubuntu) - the command line was sudo upslug2 --target 00:14:bf:64:df:be --image openwrt-nslu2-squashfs.bin.
• The machine now appears on the network with the right IP address.  Telnet into it and change the root password, then connect using SSH instead.

The only problem is that when I type 'python' I get an error "python: can't resolve symbol 'BC'".  So now I need to work out which package is missing - looks like openWRT missed a dependency, but I don't know which one!

I have had a bit more of a look at it.  There are two surprising things.  The first is that if I write a simple 'hello world' script and execute it, it works, so it is just interactive python that is broken.

The second is that doing "ldd /usr/bin/python" on the nslu2 and on the bifferboard give very similar results, except that the bifferboard one is linked against libgcc and libc, but the nslu2 one is linked against libc.

I think this is telling me something, but I am not sure what....

Mapnik Map Rendering

Someone asked a question on the osm-gb mailing list about viewing coverage of waterways - canals and navigable rivers.
We don't seem to have one, so I created one on my http://maps.webhop.net web site.
The trouble with that site is that although it is running on a nice powerful computer (my old laptop), it is behind my domestic broadband service from Virgin Media.   Although this service has a nice fast download speed, the upload speed is very poor ~900 kbps compared to 9600 kbps download.   This makes the web site very slow from the outside world.

I have a little virtual server provided by CloudNext - it is nowhere near as powerful as my own server, and has much less disk space, but at least it has a fast internet connection.

I loaded a british isles extract of OpenStreetMap data into the database - it took a very long time - maybe getting on for 12 hours (not sure when it finished!) - my server takes about 2 hours.   Performance rendering maps was not too bad though - rendering down to zoom level 14 took about 2 hours, which is respectable.

Just need to get the database updating to keep it up to date, and re-generate changed tiles now....
You can see the difference at http://maps.webhop.net/canals (my home server) compared to http://maps2.webhop.net/canals (the virtual server)

Personal Weather Station using BifferBoard

I have managed to re-compile openWRT to run on the bifferboard, and include python, libusb and pyusb in the main root directory.   I also included the wget utility, because I thought it would come in useful, plus a few other potential future extensions (lighttpd web server and wireless LAN support).
This fits on the bifferboard onboard flash memory, with about 1MB to spare.

I created a /home/weather directory and extracted a recent pywws tar archive into it.   I was very pleased that running TestWeatherStation.py produced a table of numbers as expected, so it looks as though the bifferboard is talking to the weather station ok.
Here it is:

Set up the initial weather database by doing

cd /home/weather 

python pywws/pywws/LogData.py -vvv /home/weather/data

This creates /home/weather/data/weather.ini and a directory /home/weather/data/raw which contains the raw data.

Run Hourly.py for the first time to process the data, and set up the weather.ini file ready for customisation:

pywws/Hourly.py /home/weather/data

Customise /home/weather/data/weather.ini to do what you want it to do - in my case update weatherunderground....

All is looking promising here - seems to send the update ok on an hourly basis.  There are two problems though:

• There is not much disk space left now I have added all of January's weather data (256kB ish)
• I am concerned that I will wreck the flash chip on the bifferboard with all this writing.

One option is to store the weather data in a ram disk and just accept that after a power-off it will need to re-initialise itself - not sure how pywws will cope with this - I will need to load some default weather.ini file from flash every time it boots, then let it update itself as best it can.

As a quick alternative I have found an old 256MB SD card and created an ext3 filesystem on it.  This detects as /dev/sda1 when plugged into the bifferboard.   I have modified /etc/rc.local to mount this as /home/weather.  This should solve my full filesystem problems.  [Note:  I used an ext3 filesystem because my openWRT build could not detect ext2 - this was a bit of a surprise because I thought you got ext2 support free with every Linux kernel...].
Then it is just a matter of a cron job to do the hourly updates - crontab -e, then add:

13 * * * *       python /home/weather/pywws/Hourly.py -v /home/weather/data >> /home/weather/Hourly.log 2>&1

If it works, you should continue to see the weather in our back garden at: http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=IHARTLEP2.

Mistake 1:  If you put an SD card into the dual USB bifferboard, it disables one of the USB ports.  This is bad if it was the one the weather station was connected to....

Mistake 2:  Believe people when they say the clock on a board is no good.  The Bifferboard clock is no good.  Not only does it loose its time when powered off, but it drifts - by a few minutes an hour.  This meant that pywws got confused about when it should update weatherunderground, because it thought time had suddenly gone back to 2009....   To avoid this I compiled ntpclient as a package and downloaded it onto the bifferboard.   I now have it set to be called once on boot, then every 10 minutes by a cron job.  This should keep it somewhere close - just hope the time server owner doesn't mind - If I get a complaint I will have to set up my own time server...

Building openWRT for BifferBoard

I had a go at building openWRT for my new bifferboard computer last weekend, but had trouble with it - it looked like the kernel started, but then panicked around the time it was supposed to be starting init. I suspect it was something to do with the j2ffs file system.

I decided to try to be clever and rather than fix openWRT, start from scratch with an OpenEmbedded based build. I think this was too big a step - the main problem was that OpenEmbedded failed to build because some sources from handhelds.org are not available, so I had to learn how to alter 'recipes' to build them from sources from another location. I got ipkg-utils to build this way, but then hit more trouble with openssh. I decided it was going to take me a long time, so went back to openWRT for now to try to get something working!

This time I followed the exact instructions from the bifferboard site, including downloading the specific revision of openWRT.   This all compiled nicely without any errors, so the next challenge was to get it onto the board.

Flashing over the serial line seemed to take a very long time, so I had a go with network setup.
I downloaded the bifferboard utilities using:

svn co https://bifferboard.svn.sourceforge.net/svnroot/bifferboard

This provides a simple python based tftp server.   I had to modify it to use a fixed ip address for my host computer, because the script assumes that you are using eth0 for the network interface, and I was using wireless.
You then need to make sure that there is a file called bzImage in your working directory and start bootp_server.py - I had to do it as root because I got some permission denied errors about opening sockets, and the easiest way to fix it was to be root - there is probably a more elegant solution though.

First I created a symbolic link to the bzImage file that openWRT had produced.   Then booted the bifferboard with the serial line connected and pressed ESC to stop the boot process.   Entering the tftpflash command downloaded the bzImage from the server and flashed it to the disk - much quicker than a serial line.
The kernel started, but failed with a lot of j2ffs errors - I think this is because I had my old root filesystem in the flash memory.
For the next attempt I created a symbolic link from openwrt-rdc-jffs2-64k-bifferboard.img to bzImage.
Re-flashed it again the same way and it boots!

A minor detail was that I could not log into it - the serial console was fine and both telnetd and dropbear (the SSH server) were running, but I got 'connection refused' when I tried to connect.   It turned out that there was something called 'firewall' running.   I deleted /etc/rc.d/S45firewall and re-booted and it worked properly - I could telnet in initially without a password, set a root password, then use ssh to connect.
Success - Now I just need to sort out the weather station software to run on it - back to the cross compiler....

running make package/symlinks in the openWRT directory adds all of the available packages to the openWRT menuconfig program.
Added the ones I know I need (libusb, python, pyusb), plus a few others that might come in useful later (atheros wireless card drivers).  Then make.....and wait...

A "Hacking Embedded Linux Devices" wiki site?

I have done a bit of work trying to hack cheap consumer devices to run different software - NSLU2, mediaMVP, musicPal, edimax IP Cameras etc.   I have also had a look at 'off the shelf' single board computers like bifferboard.
Whenever I start I have the difficult learning curve of trying to remember how to set up a cross compiler, cross compile libraries and link them into new software.
I also think it would be useful to have a nice list of which devices have been successfully hacked and which ones are difficult.
I am thinking of setting up a site to collate such basic information, which could then link out to the more specific project sites.
Unless anyone knows of one already?

I think it would need a wiki to store most of the information, and an email discussion group.  I wonder how best to do it - there are google sites and google groups, wikispaces, pbworks.

Any suggestions on the best way to do this?

Building openWRT

I have got a little bifferboard single board computer which I intend to use to run pywws to send data from our weather station to the weather underground site on the internet.

The bifferboard comes installed with a very small linux distribution called openWRT.   It has a very small python installed, but no python USB support.

I have struggled a bit to work out how the openWRT package system works - the official wiki is a bit confused about what the current version is called (kamikaze or backfire).  It implies you can add packages by downloading them from svn, but this didn't seem to do anything.
I found a useful forum post here, which seems to be the best set of instructions.
You can do

./scripts/feeds update -a

This downloads the list of packages, but does not do anything else.

To get the buildroot system to compile it for you you need to 'install' it using:

./scripts/feeds install python 
./scripts/feeds install pyusb
You can then do "make menuconfig" and python and pyusb are shown to be compiled as packages "".

'make' actually compiles it.

I'll update this when I work out how to add these to the firmware image...

Hacking the Parrot-DF3120 Picture Frame

Some clever people have been working on running Linux on a parrot DF-3120 Picture Frame, as described on http://sites.google.com/site/repurposelinux/df3120.  
The device sounds like a little computer with a colour display, USB connection, SD card and bluetooth wireless (I would have liked a wireless lan, but never mind!).   Most significantly, Amazon are selling them for less than £10, so I thought it was worth getting one to play with.   It arrived very quickly so I started to play with it yesterday.

The instructions at http://sites.google.com/site/repurposelinux/df3120 fall into two distinct parts:

1. Produce a cross compiler for the board, and use it to compile a Linux kernel, busybox and other things needed to construct a basic root filesystem.  This is all done by a single clever script (minifs).   It actually took a bit of doing to get it to work on my Ubuntu 10.10 system - when it fails you have to look at the log files to see why it has failed.   The most noticeable thing was that I had to install gtk-doc from a source tarball because Ubuntu does not have a package for it - everything else (bison, flex etc.) were installed from ubuntu packages.  
   The script produced an iso image of a root filesystem.
2. Install a boot loader on the device.   The instructions have you download and compile the u-boot boot loader for the device, and package it into  a false firmware upgrade file (.plf) file.   There is some black magic required to copy the file file onto the device in the right directory etc.

The boot loader installation apparently worked.  If you just switch on the device without touching anything, it boots normally.  If you hold down the centre and left buttons (when viewed from the front of the device) as you power it on, the screen goes blank, which is what the instructions said would happen if u-boot tries to boot linux.

In the 'black screen' mode, connecting the device to my computer creates a /dev/ttyACM0 device, which sounds promising for this being the u-boot serial console.

I have tried using a terminal emulator program to connect to the device via ttyACM0 (picocom /dev/ttyACM0, or cu -l /dev/ttyACM0).  In both cases the program connects without error, but I do not see anything on the display or in the terminal.

The copy of u-boot seems to have come from openMoko, so I tried their wiki (http://wiki.openmoko.org/wiki/U-boot).   It sounds like what I am doing should have worked - tried a few baud rates, but I am getting nothing, rather than gibberish - I think I'll read through the u-boot configuration that I just installed and try to work it out.....