Inside the Edimax IC-3010WG IP Camera

I have just added my photos of the inside of the Edimax IC-3010WG IP camera to my web page: http://ic-3010wg.webhop.net.
If anyone knows what 'J1' is, I would like to know!

Edimax IC-3010 Firmware Modification

I just discovered the 'stats' facility in blogger.com and realised that the most read posts in my blog are the ones about the Edimax IC3010 wireless ip camera. People are also looking at my web pages about it (http://ic-3010wg.webhop.net).
This got me thinking about it again so I have had a poke around inside the case - I'll post some pictures at the weekend.
There is not much to see in side - there is a mini wireless card and a single board computer. I was hoping to see a USB port, but the nearest I can find is a four pin header that is likely to be one (or more of):
- USB Port
- Serial (RS232 ish) console
- JTAG interface

I'll have to find the main system-on-chip pin-outs to try to decide what it is.

The main thing I struggle with is getting into the firmware files - I think the choice if you want to hack a device is to either connect a serial line to the console to get access to the boot loader, or modify the firmware.   Modifying the firmware sounds simplest, but you run the risk of turning the device into a brick....

That is about where I got to in the summer of 2009.  I am just downloading the latest firmware and source code from edimax to try to de-code the firmware building utility.   While I was doing this, I discovered that someone has already done it - http://www.suborbital.org.uk/canofworms/index.php?/archives/3-Getting-telnet-access-on-an-Edimax-IC3010-webcam.html.   Well done to them!

Personal Weather Station using NSLU2

While I wait for my new bifferboard to be delivered I thought I would get the weather station working and updating to weatherunderground.com using a Linksys NSLU2 that I have available.
The NSLU2 is already configured to run Linux.  There are quite a few varieties of NSLU2 linux, and I think this one was called OpenSlug when I installed it, but it now seems to be called SlugOSBE.
When I first set it up there was no python package available for it, so I had to write my utility meter monitoring program using Perl (yuk!).
Fortunately there is now a repository at http://ipkg.nslu2-linux.org/feeds/optware/slugosbe/cross/unstable/ that includes python packages.
To get python running (with USB support) on the NSLU2 I had to download the following packages from the repository, and install them with ipkg install .ipk:
libusb, libdb, libstdc++,ncursesw,readline,sqlite_3, zlib, python25, py25-usb

The python distribution did not put an executable in /usr/bin, so I had to do
ln -s /opt/bin/python2.5 /usr/bin/python

With all this installed I could plug the weather station into the NSLU2 (via a USB hub in my case) and pywws just worked.
I hope the bifferboard is as easy!

Personal Weather Station

I have just bought my wife, Sandie a personal weather station from Maplin.  It is a much better one than the simple one that broke recently - it has external humidity as well as temperature, plus wind speed and direction and rain detection.  Most importantly it has a USB connection so we should be able to connect it to an internet weather network so it can appear on our iGoogle home pages.

The maplin version is apparently a re-branded WH1080PC weather station - I chose this one rather than a more modern one because people have managed to get it working with Linux - I want to use a very low power single board computer to interface it, rather than a big PC running Windows.

I have chosen a bifferboard single board computer because it is cheap and has both USB and ethernet connections (I would have liked wireless lan too, but could not find something to do that, so I got the two USB port version of bifferboard, so will have a go at adding a USB wireless network interface card to it later).
My original idea was to use some software called meteoplug because you can buy a bifferboard with that already installed, so I know it would work.  However, when I looked into it, it is a subscription service that is not open source - all I really want to do is obtain data from the weather station and send it to weatherunderground, which can not be that difficult.
Fortunately there is an open source project, pywws, which is compatibly with the WH-1080PC weather station.
I thought it best to get it going on a desktop linux set-up first, then try to get it working on the single board computer.
From my Ubuntu 10.10 laptop I did:

svn checkout http://pywws.googlecode.com/svn/trunk/ pywws

sudo apt-get install python-usb

cd pywws 

python TestWeatherStation.py

Initially this gave an error "IOError: Claim interface failed", so I tried:

sudo python TestWeatherStation.py

Success - a string of numbers, and no errors!

To get rid of the requirement to run the program as root you have to tell 'udev' to give other users permission.
lsusb reports the vendor and product ID of the device (1941 and 8021 respectively), but this is actually reported as a USB missile launcher rather than a weather station.
I added a file called 05-weatherstation.rules to /etc/udev/rules.d, which contained the following line:

SUBSYSTEM=="usb", ATTRS{idVendor}=="1941", ATTRS{idProduct}=="8021", MODE:="0666", GROUP:="usb"

I also added a line to /etc/group to create the 'usb' group, and made myself a member of that group.
Re-started udev with 'sudo service udev restart', and now the test works without being root (note the colon in 'MODE:=' - it is very important!

pywws seems to work suspiciously well - creating a data directory (~/weather/data) and running

pywws/LogData.py -vvv ~/weather/data/

creates a file with some convincing looking numbers in it.   I have created a weather station on weatherunderground, so I think the next thing to do is install the weather station outside, then try to use pywws to send data to weatherunderground.

Ubuntu Linux on a Packard Bell OneTwo

This post is a record of me getting Ubuntu Linux (Release 10.10 - Maverick) running on our new Packard Bell OneTwo all-in-one touch screen PC.   I will update it as I make progress fixing the bits that have caused problems.

Installation
Installation was pretty easy - I booted of an Ubuntu 10.10 USB memory stick, and selected manual disk partitioning.
I kept the Windows 7 partition (/dev/sda1) unchanged, and did not touch the other two windows partitions (/dev/sda2 and /dev/sda3).   I deleted the blank 'Data' partition (/dev/sda4) and created a new swap partition (/dev/sda5) and ext4 root partition (/dev/sda6).
Installation went without problem and the machine now dual boots Ubuntu and Windows 7.

Note that I did have a funny happening yesterday when the machine completely refused to boot after I had been using windows.  I fixed this by booting off the USB memory stick and re-installing grub as described in a previous post - This has not happened again.

Audio
Worked out of the box.

Video
Ubuntu did not detect the correct screen size initially - it used a normal aspect ratio rather than wide screen, so only the centre of the screen was used, and the touch screen calibration was way out - this is because it thinks it has two displays (I don't know where the second is - there is no VGA or s-video connector on the back).
This was fixed by going to the System->Preferences->Monitor menu and un-checking the "Same image in all monitors" option.
You can then drag the main screen (labelled "Laptop") away from the second (labelled "Unknown") and set the laptop resolution to 1600x900 resolution.

This makes the screen the correct size, but every now and then it flickers twice within less than a second - it seems that the screen backlight dims momentarily before coming back to full brightness.  This seems to be associated with disk activity - I don't know what causes this - it does not do it in Windows - Any suggestions for fixing it would be appreciated!.

Touch Screen
I was impressed that the touch screen worked 'out of the box' once I had fixed the monitor resolution issue.
Update 01/01/2011 - I decided to do a clean install of Ubuntu 10.10, just in case I had mangled it installing different testing packages going through bugs on launchpad.   Having done the install and updating all packages using update manager, the touch screen is working perfectly - pressing the screen is the same as moving the mouse to the location and left click - Success!!!! (I hope!)  - Ok, I'll give it a couple of days to make sure it doesn't beak before I say success.

Unfortunately there are problems with it.   It seems that once you have done a 'drag' using the touch screen, x-windows thinks the mouse button is still pressed - moving the real mouse changes the shape of the drag region, and worse still, the real mouse button stops working.   Sometimes the keyboard stops working too (both mouse and keyboard are wireless, using a USB receiver)


Investigations so far:

sudo apt-get install input-utils

provides a program called 'lsinput'.  Doing

sudo lsinput

lists the touch screen as:

/dev/input/event6

   bustype : BUS_USB

   vendor  : 0x408

   product : 0x3001

   version : 272

   name    : "PixArt Imaging Inc. Optical Touc"

   phys    : "usb-0000:00:1a.2-1/input0"

   uniq    : ""

   bits ev : EV_SYN EV_KEY EV_ABS EV_MSC

Installing multi-touch utilities by doing:

sudo apt-get install utouch libutouch-grail-dev utouch-grail-tools

sudo apt-get install mtdev-tools

bzr branch lp:~utouch-team/utouch/mtview

cd mtview

make

Provides amongst other things a utility called mtview.  Running that lets you draw pretty patterns on the screen using multi touch - it certainly seems capable of detecting and tracking two fingers at once.

Given this I think the touch screen is basically working, and it is the interface to x-windows that is the problem.

I'll have to work out how x-windows drivers work to fix this though....

I will update this section as I investigate further.

Web Cam
Not tested.

Trouble with Ubuntu and Windows 7 dual boot

I just had a surprising problem with our new Packard Bell OneTwo.   While it had apparently worked fine after installing Ubuntu on a spare partition in the hard drive, I booted it under its original Windows 7 operating system this morning to test benjamin's program with windows.
It worked fine, but after it had been switched off it absolutely refused to boot - no error messages or anything.
Went into the bios settings to take off quite boot and quick boot, but still no errors.
It booted off the Ubuntu live CD ok, and I could mount the linux root partition.

My best guess was that the boot loader was mangled, so I re-installed grub by doing:

sudo mount /dev/sda6 /mnt      [sda6 is my root partition]

grub-install --root-directory=/mnt /dev/sda

 This worked ok with no errors, and re-booting the machine gave the familiar grub boot menu.   Ubuntu booted ok...we'll see if it works after I use Windows next time - I suspect it was Windows that killed the boot loader!

Framebuffer on Ubuntu

I would like to be able to run my benjap application in a simple framebuffer rather than using X windows.
Unfortunately the 'out of the box' ubuntu desktop does not have a framebuffer (/dev/fb0) installed by default.
It looks like you need to load the fbcon and vesafb modules (found here):

sudo modprobe fbcon vesafb

Having done this /dev/fb0 appears.   It appears to work as long as you run applications as root (a job for later to fix that!), but I have a problem with sound, which is that even running as root I get a "no available audio device" error when pygame.mixer.init() runs.  Not sure how to fix this.....

Benjap - A program for children with Autism or Learning Difficulties

I have decided to write a program for our autistic son (who also has severe learning difficulties) with my daughter, Laura.
We have bought a Packard Bell OneTwo all-in-one PC with a touch screen to use for it.
This is because Benjamin will not use a keyboard or mouse, but I hope that the cause and effect will be more obvious using a touch screen.

The basic program is written in Python using the pygame library.   The structure is of a main application with big icons to select the other applications.
The various applications are (or will be):

• Stars (a bit like the start of star-trek) you can change the origin of the stars that shoot towards the edge of the screen by pressing the touch screen.
• Bounce:  Lots of bright coloured balls bouncing off each other and the edge of the screen (not very interactive at the moment!).
• Comet:  Dragging your finger around the screen leaves a multi-coloured trail like a comet.
• Fireworks:  Pressing the screen launches a firework that explodes colourfully.

I have started a new project page for the application at http://code.google.com/p/benjap.

More to follow....

Packard Bell OneTwo with Linux

Our son, Benjamin is autistic and has severe learning disabilities. Persuading him to do anything other than his current 'favourite' activity is really difficult. We took him to a light room today and I was impressed that he showed signs of interacting with a white square which lit up where you touched it. Laura and I started to think about making him one. It looks quite difficult because as well as the hardware to mount all of the lights we will need a touch sensitive panel and produce some software to drive it. For this reason I started to think about going for a purely software solution instead - just use a simple computer and a touch sensitive monitor.

I have contacted a couple of suppliers to see what they can do, but the first quote for a whole panel PC (which would be neat - no wires!) was very expensive £1620 for a 32" one.

An alternative would be to go for something smaller - HP and Acer make 23" all-in-one touch screen PCs for <£800k, which is still quite a lot, but getting better.

PC World and Currys are selling a 20" touch sensitive monitor for £180 which is more like the price I was thinking of, but I will need to add a computer, which will mean wires. I do have a little netbook which I don't use anymore, which could do the job though, so I may try this and just invest in a bigger one if he seems interested.

When I went to PC World today they did not have any of the monitors, but they did have a Packard Bell OneTwo all in one PC with touch sensitive screen.  This was £599 so I decided to give it a go, because at least it meant no wires.

I downloaded Ubuntu 10.10 onto a USB memory stick and installed it in the spare 'Data' partition on the drive (I actually deleted the data partition and re-created a 10GB swap partition and the rest as a ext4 filesystem to use for the ubuntu filesystem.   The install went ok, and grub appears to give me the option of booting windows (I haven't tried yet).

The most noticeable thing was that the screen resolution was not very good, and it was not using the widescreen aspect ratio.   I fixed this by going to the monitor settings in the ubuntu system menu and un-clicking the 'mirror' option - it seemed to think it has two monitors, one of which is low resolution.  Once I had de-selected that option I could select the resolution of the main monitor (called 'laptop' in this case).   Now that is done it looks better, and the touch screen appears to be working.

For more information on my experiences using Linux on the Packard Bell OneTwo, see my other blog entries, in particular http://nerdytoad.blogspot.com/2010/12/ubuntu-linux-on-packard-bell-onetwo.html.

My daughter Laura and I are working on the applications for Benjamin to use - I am doing the code and she is on graphics at the moment.

The application is at an early stage, but the outline can be downloaded using

svn co https://ntmisc.googlecode.com/svn/benApp

There will be more about this application as we work on it and try to get Benjamin interested in it.

SVN Keywords

I just realised that my townguide files are no longer automatically updating their SVN keywords to record the revision number of the files.
It turns out that this is because the keyword properties in svn is set by the client, not the repository, and worse still you can not apply keyword properties to directories - you have to do each file individually.
I got around this by using the find command to add keywords to all of the python files by doing:

find . -name \*.py -exec svn propset svn:keywords "LastChangedDate Rev Author" '{}' \; 

Of course for files that do not have keywords set you have to edit them to add $Rev$ etc. into the comments.

Townguide - progress at last!

You can't beat some really bad weather to help make progress with my nerdy jobs.
I have got Waldemar's django front end to townguide working (http://dtownguide.webhop.net), and have managed to make a few changes, including adding fancy tab things to the main map selection form.
There are still a few things to sort out:

• The area selection on the map does not align properly with the mouse pointer - not sure why - must be something to do with OpenLayers projections...
• Mapnik re-sizes the output map to match the bounding box, so townguide needs to check the actual map size to make sure the grid squares are right.
• There is something funny about the output resolution - need to see what it is using - the output looks much higher resolution than I asked for, so I suspect there is a sum wrong somewhere!
• Add the GPX track and waypoint plugins (they are just templates at the moment).
• Add an option to suppress un-named ways - the 'None' in the street index can confuse people.
• Reinstate other output formats - only 'poster' is working.

Townguide - where next?

My townguide map rendering application is about a year old now, but hasn't changed much since last winter.
Waldemar did a lot of work on it for his Google Summer of Code project where he re-hashed the townguide renderer code to make it more modular and produced a django based web front end.
I have not done very well getting this working because I do not understand django well enough to sort out a few minor bugs in it.
Instead I have taken his re-hashed renderer code and developed it further to make it even more modular - as well as 'plugins' for different PDF output formats, there are plugins for the base map and for different map overlays (a grid, custom markers, gpx tracks etc.) [well nearly anyway].  It also uses mapnik2 so can now produce nice high resolution output to make it comparable to the much prettier maposmatic output.

I have been reading a book about django for a few days now (Python Web Development with Django by Forcier, Bissex and Chun) and feel that I should be able to understand both Waldemar's code and the maposmatic version, so I will have a go at the web front end now - there seems little point in developing my clanky php version with these two much slicker front ends available.

IBM ThinkPad X60S Middle Mouse Button

Every time a new release of Ubuntu linux comes out, something breaks.   Fortunately over time the somethings are becoming less and less significant!
This time it was the middle mouse button on my IBM ThinkPad X60S.    I had got it working following the instructions here in version 10.04 - it just needed a configuration file in /usr/lib/X11/xorg.conf.d.
But it stopped working following the upgrade to Ubuntu 10.10.
It turns out that the xorg.conf.d location has changed to /usr/share/X11/xorg.conf.d.
Putting a file named 20-thinkpad.conf in that directory with the following contents got middle button scrolling working nicely again!

Section "InputClass"

Identifier "Trackpoint Wheel Emulation"

MatchProduct "TPPS/2 IBM TrackPoint|DualPoint Stick|Synaptics Inc. Composite 

TouchPad / TrackPoint"

MatchDevicePath "/dev/input/event*"

Option "EmulateWheel" "true"

Option "EmulateWheelButton" "2"

Option "Emulate3Buttons" "false"

Option "XAxisMapping" "6 7"

Option "YAxisMapping" "4 5"

EndSection

 There is talk that it should work by doing:

sudo apt-get install gpointing-device-settings

But it did not work for me (at least not just like that - tried re-starting xwindows, but still no scrolling - maybe there is some configuration needed, but I like the old way of dropping a file into a configuration directory!

OS Vector Map District with PostGIS - Take 3

To recap:

• The entire UK's VectorMapDistrict dataset has been loaded into my postgresql/postgis database using my vmd2pgsql script, which is based around shp2pgsql.
• I created some stylesheets for mapnik rendering of the data based on the mapnik tutorial, but they came out with a transparent background and black lines and text, which is not what I wanted.

The rendering problem turned out to be quite easy - it is just that the mapnik tutorial is based on Mapnik 0.7, not mapnik2.  To make it work with mapnik2 you need to:

1. Replace the bgcolor parameter of the map element with background-color.
2. Remove all of the CSSParameter business in the style definitions with simple name="xyz" constructs.

It is a shame that mapnik did not complain about these issues, even in debug mode, but never mind!

Adding the different linear features works ok - you see something of an improvement from the black and white version to something resembling a map:

So now all I need to do is add the areas (water, woodland towns etc.) to make it look extra pretty, and sort out what to plot at different zoom levels so it is reasonable.

Here I hit a problem.   When I tried to add the settlement_area or naturalfeature_area items, mapnik crashed with an "invalid geometry" error, after taking quite a long time thinking about it!   It is something to do with polygons not being closed properly.   I suspect it is rounding errors, but am not too sure.  One possibility is that I should have used the shp2pgsql option to just use integers, which would have avoided rounding, or I could try to fix the database.  As it took 9 hours to import, and I don't know if integers will work, I'll try to fix it.

With much internet searching I discovered that you can check the validity of each geometric feature with the postgis st_isvalid() function.  Running a simple select statement on the database like:

select gid from naturalfeature_area where not st_isvalid(the_geom);

takes a long time (about an hour I think) and gives me a list of invalid geometries, after pages and pages of warnings about things crossing over themselves.

It seems that one trick that people use to force gemetries to be valid is to use the st_buffer() command with the buffer radius set to zero.   I checked it worked by doing:

select st_isvalid(st_buffer(the_geom,0.0)) from naturalfeature_area where gid=2342149;

(2342149 was the first entry in my list of invalid geometries) - Success - it returns 't' for true meaning the st_buffer trick fixes it.

Now to update the database.  To correct the error I had to do:

update naturalfeature_area set the_geom=st_multi(st_buffer(the_geom,0.0)) where gid=2342149;

Note that I do not really know what the st_multi thing does, but I got errors about geometry constraint violations without it - I think that st_buffer returns the simplest type of geometry it can, and st_multi forces it to be a multipolygon, but I could be wrong!.

Now all I need to do is remember enough about subqueries to write a bit of SQL that does the correction for every invalid geometry - back to that well known search engine....

Well, I have set it going using the following to try to repair the geometries:

update naturalfeature_area set the_geom=st_multi(st_buffer(the_geom,0.0)) where gid in (select gid from naturalfeature_area where not st_isvalid(the_geom));

I think I'll leave it for a few hours....

ThinkPad X60s Battery Trouble

I recently bought myself a second hand IBM Thinkpad X60s because I wanted a small laptop to use on the train, which has enough power to run postgresql/mapnik etc. so I can play with maps while I am living out of hotels (better on the body than too much food and beer, but not as pleasant!).
The problem I have is that at every charge cycle the gnome power manager in Ubuntu reports that the battery is less charged than before, and then it shuts down very quickly when running off battery because it thinks it is discharged.
I tried running it to completely empty by letting it suspend on low battery, then waiting for the battery to really die.  The next charge was higher than before, but the same problem has repeated.
I just used gconf-editor to set the action on low battery to 'nothing' so that it will run until the battery is properly dead.   The charge history is shown below:

Basically Ubuntu thinks the battery was dead nearly an hour ago, but it is still running!

I think this must be a problem with ACPI, but don't know how to sort it - a job for a very rainy afternoon.   For now I will just leave the auto-suspend switched off and try to guess when the battery is about to give up.

Using VectorMapDistrict with PostGIS

Well, vmd2pgsql took about 9 hours to import the entire UK into my postgresql database...No too bad.
Now I need to do something with it, because the mapnik stylesheets that I set up to use the shapefiles will not work - need to change it to use the postgresql database instead.
The things that need changing are:

1. Add the layers and datasources in XML (no need to do it in python now I don't have hundreds of separate datasources).
2. Convert the styles to use lowercase letters for the fields ('featcode' rather than 'FEATCODE') - postgresql seems to be case sensitive, and they have gone into the database lower case, but were upper case in the shapefiles.

Sounds easy,  but I decided I don't want it to look like the OSM stylesheet, which is a bit garbled - I want a nice structure to make it easy to maintain.  I could have used XML entities, but that seems a bit crude - define an entity, then 'use' it to actually 'call' it.   Includes seem much more suitable....But to get includes working I need mapnik2.  I have put a basic structure at code.google.com/p/ntmisc/vmdmap, but still have a few issues:

• I have had to define the database parameters manually because entities do not seem to be working in the include files, which is no good!
• The output is transparent background with black lines (see picture below), which is not what I asked for - not sure if this is a problem with the style file, or my build of mpanik2 - I'll have to run some tests on mapnik2 to make sure it works right.
• But it's too late to fix it now - job for tomorrow!

More Rendering of VectorMapDistrict Data

The nice people from Ordnance Survey have sent me a complete set of VectorMapDistrict data (which is on 6 DVDs!).   Surprisingly they didn't even charge me for the DVDs or postage.
Now I have all that data I thought I'd better do something with it!

First step was to get it onto a nice fast disk, so I copied it onto the hard disk of my server...which took a while...

Then I tried to use my vmdmap.py program to render it all, but mapnik bombed out with an error as it was adding the various shapefile layers.  I suspect that this is because every single shapefile is in its own layer, as it is a completely separate datasource, and I think I either ran out of memory or hit some mapnik internal limit.   This means I can't use vmdmap.py to render the whole country, which is a bit of a shame.

To get around this I think I need to merge them into a single datasource.  I don't know enough about manipulating shapefiles to do this, so instead am making use of shp2pgsql which allows you to import a shapefile into a postgresql database.   I have written another program, based on vmdmap.py called vmd2pgsql which will scan through a directory tree looking for the various shapefiles in the vectormap district dataset, and importing them into postgresql.   This gives a much lower number of tables - just one per shapefile name, but each one will have a lot of data.

It is importing now, so will see how long the import takes, then how well it renders.  I suppose it should render ok because I have the whole UK OSM dataset in a single database and that works, but we'll see over the weekend if it ever finishes!  The code is at my google code site.

Mapnik and OS_OpenData

I decided to compare the quality of OpenStreetMap mapping to that which has been released by Ordnance Survey OpenData.  
I started with VectorMapDistrict, which is a dataset provided as ESRI Shapefiles.   It is provided as a large number of shapefiles, so setting up a mapnik stylesheet manually would have been a pain.  Instead I separated the styles into an XML file, but added the layers using a python program which scans down the directory tree to add the various shapefiles to the map.  The python code is vmdmap.py and the stylesheet is styles.xml.

Then I tried Meridian2.  This is much simpler - just one shapefile for each sort of feature (A-Road, river etc.).  I kept the same structure with the styles defined in an XML file, and the layers in python, but it could have all been done in a single XML file this time.  The python code is md2map.py and the stylesheet is styles_md2.xml.

You can see the results below.

The most notable things are that Meridian2 is much simpler geometry, but it includes more road names than Vector Map District.  The OpenStreetMap rendering is fancier because I used the standard OSM style, rather than the home made ones I used for the OS Data.   I do like the rocks that appear in the VectorMap District rendering though - I will have to import it into OSM!.

VectorMapDistrict

Meridian2

OpenStreetMap

Mapping - OSM and Mapnik

I have been playing with mapnik quite a bit lately - I decided I needed to learn how to use it better to improve my townguide program.
I have been very impressed with Lonvia's Hiking Map, which is an overlay showing hiking routes on top of a standard OpenStreetMap map, and thought I could do something similar.

My first effort was a map showing the locations of supermarkets (handy when we go on holiday so we know where to go to stock up on provisions) - It can be seen here.
Someone has imported a lot of powerline data from Ordnance Survey into my local area, so I tried a Power Station Map.  These maps all have an 'about' link showing how they work.

I have produced a similar one to show the use of Ordnance Survey OpenData in the OpenStreetMap map of the UK, which can be seen here.

More recently I have wanted to produce maps that include contours, and also allow me to plot GPX traces to show where we have been.   To do this I have imported the SRTM data for the UK into my postgresql database using the technique described here.   This allows me to render an Openstreetmap map with contours, which is useful for the countryside.

Adding a GPX trace over the top was a bit difficult - I wanted to do it in python to save hard coding the path to the GPX file in the mapnik style file, but I haven't managed to do that - I ended up adding a GPX layer to my mapnik style file using the mapnik ogr plugin.   The result can be seen here.

The Irritating thing about Android

I have recently got an Android phone so that I can write software for it without the irritating security issues you get with J2ME on 'normal' phones (boxes popping up saying 'do you want to let this program do this?' etc.).

I was very surprised that some things that most phones do 'out of the box' are a real pain in Android, that makes me think this is still a platform for gadget enthusiasts rather than end users.

These include:

* Sharing Contacts: All of the phones I have seen with bluetooth let you send contacts from one phone to another over bluetooth. Not Android - I ended up adding my contacts from my old phone to Google Mail manually. This made them appear on the Android one.
* Using the phone as a modem: My Sony Ericsson W890i just works - plug the phone into a computer and it appears as an ethernet card - dead easy (I think I had to select this as an option in the phone menu, but it was just a simple menu action. The nearest thing I have got with Android is a program called 'Proxoid' to run on the phone. You then have to use the ADB program that comes with the Android Developers SDK to forward packets to the phone, and set up your browser to use a proxy. A long way from an end user experience. I'll do a separate post on this so I can remember how to do it next time I am stranded in Spain and my T-Mobile broadband dongle won't work!

Reprojecting Ordnance Survey Raster Data

I have been looking at the Ordnance Survey StreetView data that was released to the public on 1st April.
This has got me thinking about a previous project that I failed to make work - I have some old maps of my home town and wanted to be able to display them on the web and switch between them so you can see what changes from year to year - I have maps from 1870 and 1914.  I want to use OpenStreetMap as the recent data.  Now StreetView is available I can use that too for comparison.

The problem is projecting them onto the same coordinate system - I failed miserably last time, but my two scanned old maps were on the same scale, so you can see them at (http://maps.webhop.net/oldmaps/openlayers.html).

The maps are in Ordnance Survey projection, which has the code EPSG:27700.   The downloaded OS data has the origin specified in metres northings and eastings.
You can translate this into the same projection as OpenStreetMap (EPSG:900113) using:

gdalwarp -s_srs EPSG:27700 -t_srs EPSG:900913 *.TIF nz.tif

Check the projection information with

gdalinfo nz.tif | more

Ordnance Survey Data Released

Ordnance Survey have released quite a lot of data for free use (http://www.ordnancesurvey.co.uk/oswebsite/opendata/).
The most interesting sounding data (meridian) is a 2Gb shape file, so I chickened out of trying to do anything with that tonight.
Instead I downloaded the 50k Gazetteer and have put it into a postgresql database so it can be queried to look for things.
Work in progress is at http://www.maps2.webhop.net/openos.
It sort of works (just not very useful yet - you can do http://maps2.webhop.net/openos/gaz/www/doSearch.php, and it gives you a list of all of the items in the database containing the string 'hartlepool' - just need to transfer them back to the browser and plot them on a map now....

Android User Interfaces

I am working on a colour processing program for android (more later once it works). A fundamental part of this is to create a custom view to use in the user interface - it will eventually display a colour wheel, but it is nowhere near that yet!
The problem was that I am creating a subClass of View, which draws something on the screen.
I then simply added it to my UI XML description and tried to use it.
I got an InflateException, which was a bit of a surprise.
It turns out that I made a mistake - I had overridden the simple View(Context) constructor, which would be fine if I was adding the view directly from a program, but to use it from XML you need to override another constructor, View(Context, AttributeSet).

Now I have done that it works. Sorry this makes no sense unless you are used to Android things, and if you are used to Android, it is probably obvious, but this will help me next time I forget! I found this out here.

GPX Route Editor

I was trying to plan a family cycling trip in a rush this morning and struggling to find a map to
use (opencyclemap is no good because I like to go to un-mapped places so I can add them to the map).
I finally remembered that ages ago I wrote a little javascript application to display an Ordnance Survey map, and let you put route markers on it which you can then put onto a Garmin GPS receiver to use for navigation - I used that to record where I intended to go, so I knew where there was supposed to be a bridleway...
Anyway I thought I should add that program to my repository in case anyone else is interested. There are a few things to do to it - see the README file in the repository directory - The source code is in my ntmisc code repository under gpxedit.
You can find the program working at http://www.maps.webhop.net.

I will do a bit more work on this when I get chance. I always intended to convert it to OpenLayers rather than multimap's proprietary API, and at the moment you have to copy and paste the GPX file into a text editor to get around the irritating web browser security - need a way of getting around that - the simplest would be to add an 'echo' server side application that would allow you to download the GPX file from the server, but that feels like brute force and ignorance!

Android Development on Ubuntu

I just got an LG GW620 android mobile phone. I want to use it for mapping (http://www.openstreetmap.org). There is an android editor called Vespucci available, but it needs some work to make the UI easier to use. I thought I would have a look at the source code to see if I can work out how to modify it.
The android developer web site says that it is best to use the Eclipse IDE, but when I tried I got all sorts of errors about un-satisfied dependencies.
Fortunately someone had posted some instructions to get it working on Ubuntu (http://ubuntuforums.org/showthread.php?p=8842292#post8842292).
Now I have followed this I have a working IDE to develop Android applications, and have got a basic "Hello World" application working. I have also downloaded and complied the Vespucci source code, and run it on the SDK emulator.
Just need to understand how Android UIs work now....

Progress Mapping Hartlepool

Benjamin and I are making progress mapping Hartlepool for OpenStreetMap.
I have just run a query on my database to see how many kilometers we must have cycled to produce the map. I was really quite shocked that it was over 550 km! No wonder I am tired...
Progress with the map and the breakdown of the distances are shown on the Hartlepool OpenStreetMap page.

Townguide Web Service Working

Well, I think the townguide web service is working now (http://www.townguide.webhop.net).
There is a simple queue so it only renders one job at a time, and you can view the progress of your job through the queue.
The source code and early documentation is stored at http://code.google.com/p/townguide.
The main thing to fix now is the output resolution - if I increase the resolution (in dpi) of the output map too far, mapnik changes zoom level and reduces the size of the text, so it is no easier to read - I need to understand mapnik style sheets now....

Town Guide Progress

The good thing about so much bad weather over Christmas was that I could do some nerdy work rather than going out walking or cycling.....
The town guide Python project has therefore made good progress. The program can now generate a PDF 'poster' that can include a street index, and labelled features from the map database (pubs, shops, tourist attractions etc.). A typical example is shown at http://www.townguide.webhop.net/example1_hartlepool.pdf.

I have set up a very rudimentary web service using a PHP script to call townguide.py to generate a town guide on demand at http://www.townguide.webhop.net. It is running on a very small computer, and there is no queue, so there is no saying what will happen if more than one person tries to use it at once - that is another job!