Mapnik and OS_OpenData

I decided to compare the quality of OpenStreetMap mapping to that which has been released by Ordnance Survey OpenData.  
I started with VectorMapDistrict, which is a dataset provided as ESRI Shapefiles.   It is provided as a large number of shapefiles, so setting up a mapnik stylesheet manually would have been a pain.  Instead I separated the styles into an XML file, but added the layers using a python program which scans down the directory tree to add the various shapefiles to the map.  The python code is vmdmap.py and the stylesheet is styles.xml.

Then I tried Meridian2.  This is much simpler - just one shapefile for each sort of feature (A-Road, river etc.).  I kept the same structure with the styles defined in an XML file, and the layers in python, but it could have all been done in a single XML file this time.  The python code is md2map.py and the stylesheet is styles_md2.xml.

You can see the results below.

The most notable things are that Meridian2 is much simpler geometry, but it includes more road names than Vector Map District.  The OpenStreetMap rendering is fancier because I used the standard OSM style, rather than the home made ones I used for the OS Data.   I do like the rocks that appear in the VectorMap District rendering though - I will have to import it into OSM!.

VectorMapDistrict

Meridian2

OpenStreetMap

Mapping - OSM and Mapnik

I have been playing with mapnik quite a bit lately - I decided I needed to learn how to use it better to improve my townguide program.
I have been very impressed with Lonvia's Hiking Map, which is an overlay showing hiking routes on top of a standard OpenStreetMap map, and thought I could do something similar.

My first effort was a map showing the locations of supermarkets (handy when we go on holiday so we know where to go to stock up on provisions) - It can be seen here.
Someone has imported a lot of powerline data from Ordnance Survey into my local area, so I tried a Power Station Map.  These maps all have an 'about' link showing how they work.

I have produced a similar one to show the use of Ordnance Survey OpenData in the OpenStreetMap map of the UK, which can be seen here.

More recently I have wanted to produce maps that include contours, and also allow me to plot GPX traces to show where we have been.   To do this I have imported the SRTM data for the UK into my postgresql database using the technique described here.   This allows me to render an Openstreetmap map with contours, which is useful for the countryside.

Adding a GPX trace over the top was a bit difficult - I wanted to do it in python to save hard coding the path to the GPX file in the mapnik style file, but I haven't managed to do that - I ended up adding a GPX layer to my mapnik style file using the mapnik ogr plugin.   The result can be seen here.

The Irritating thing about Android

I have recently got an Android phone so that I can write software for it without the irritating security issues you get with J2ME on 'normal' phones (boxes popping up saying 'do you want to let this program do this?' etc.).

I was very surprised that some things that most phones do 'out of the box' are a real pain in Android, that makes me think this is still a platform for gadget enthusiasts rather than end users.

These include:

* Sharing Contacts: All of the phones I have seen with bluetooth let you send contacts from one phone to another over bluetooth. Not Android - I ended up adding my contacts from my old phone to Google Mail manually. This made them appear on the Android one.
* Using the phone as a modem: My Sony Ericsson W890i just works - plug the phone into a computer and it appears as an ethernet card - dead easy (I think I had to select this as an option in the phone menu, but it was just a simple menu action. The nearest thing I have got with Android is a program called 'Proxoid' to run on the phone. You then have to use the ADB program that comes with the Android Developers SDK to forward packets to the phone, and set up your browser to use a proxy. A long way from an end user experience. I'll do a separate post on this so I can remember how to do it next time I am stranded in Spain and my T-Mobile broadband dongle won't work!

Reprojecting Ordnance Survey Raster Data

I have been looking at the Ordnance Survey StreetView data that was released to the public on 1st April.
This has got me thinking about a previous project that I failed to make work - I have some old maps of my home town and wanted to be able to display them on the web and switch between them so you can see what changes from year to year - I have maps from 1870 and 1914.  I want to use OpenStreetMap as the recent data.  Now StreetView is available I can use that too for comparison.

The problem is projecting them onto the same coordinate system - I failed miserably last time, but my two scanned old maps were on the same scale, so you can see them at (http://maps.webhop.net/oldmaps/openlayers.html).

The maps are in Ordnance Survey projection, which has the code EPSG:27700.   The downloaded OS data has the origin specified in metres northings and eastings.
You can translate this into the same projection as OpenStreetMap (EPSG:900113) using:

gdalwarp -s_srs EPSG:27700 -t_srs EPSG:900913 *.TIF nz.tif

Check the projection information with

gdalinfo nz.tif | more

Ordnance Survey Data Released

Ordnance Survey have released quite a lot of data for free use (http://www.ordnancesurvey.co.uk/oswebsite/opendata/).
The most interesting sounding data (meridian) is a 2Gb shape file, so I chickened out of trying to do anything with that tonight.
Instead I downloaded the 50k Gazetteer and have put it into a postgresql database so it can be queried to look for things.
Work in progress is at http://www.maps2.webhop.net/openos.
It sort of works (just not very useful yet - you can do http://maps2.webhop.net/openos/gaz/www/doSearch.php, and it gives you a list of all of the items in the database containing the string 'hartlepool' - just need to transfer them back to the browser and plot them on a map now....

Android User Interfaces

I am working on a colour processing program for android (more later once it works). A fundamental part of this is to create a custom view to use in the user interface - it will eventually display a colour wheel, but it is nowhere near that yet!
The problem was that I am creating a subClass of View, which draws something on the screen.
I then simply added it to my UI XML description and tried to use it.
I got an InflateException, which was a bit of a surprise.
It turns out that I made a mistake - I had overridden the simple View(Context) constructor, which would be fine if I was adding the view directly from a program, but to use it from XML you need to override another constructor, View(Context, AttributeSet).

Now I have done that it works. Sorry this makes no sense unless you are used to Android things, and if you are used to Android, it is probably obvious, but this will help me next time I forget! I found this out here.

GPX Route Editor

I was trying to plan a family cycling trip in a rush this morning and struggling to find a map to
use (opencyclemap is no good because I like to go to un-mapped places so I can add them to the map).
I finally remembered that ages ago I wrote a little javascript application to display an Ordnance Survey map, and let you put route markers on it which you can then put onto a Garmin GPS receiver to use for navigation - I used that to record where I intended to go, so I knew where there was supposed to be a bridleway...
Anyway I thought I should add that program to my repository in case anyone else is interested. There are a few things to do to it - see the README file in the repository directory - The source code is in my ntmisc code repository under gpxedit.
You can find the program working at http://www.maps.webhop.net.

I will do a bit more work on this when I get chance. I always intended to convert it to OpenLayers rather than multimap's proprietary API, and at the moment you have to copy and paste the GPX file into a text editor to get around the irritating web browser security - need a way of getting around that - the simplest would be to add an 'echo' server side application that would allow you to download the GPX file from the server, but that feels like brute force and ignorance!

Android Development on Ubuntu

I just got an LG GW620 android mobile phone. I want to use it for mapping (http://www.openstreetmap.org). There is an android editor called Vespucci available, but it needs some work to make the UI easier to use. I thought I would have a look at the source code to see if I can work out how to modify it.
The android developer web site says that it is best to use the Eclipse IDE, but when I tried I got all sorts of errors about un-satisfied dependencies.
Fortunately someone had posted some instructions to get it working on Ubuntu (http://ubuntuforums.org/showthread.php?p=8842292#post8842292).
Now I have followed this I have a working IDE to develop Android applications, and have got a basic "Hello World" application working. I have also downloaded and complied the Vespucci source code, and run it on the SDK emulator.
Just need to understand how Android UIs work now....

Progress Mapping Hartlepool

Benjamin and I are making progress mapping Hartlepool for OpenStreetMap.
I have just run a query on my database to see how many kilometers we must have cycled to produce the map. I was really quite shocked that it was over 550 km! No wonder I am tired...
Progress with the map and the breakdown of the distances are shown on the Hartlepool OpenStreetMap page.

Townguide Web Service Working

Well, I think the townguide web service is working now (http://www.townguide.webhop.net).
There is a simple queue so it only renders one job at a time, and you can view the progress of your job through the queue.
The source code and early documentation is stored at http://code.google.com/p/townguide.
The main thing to fix now is the output resolution - if I increase the resolution (in dpi) of the output map too far, mapnik changes zoom level and reduces the size of the text, so it is no easier to read - I need to understand mapnik style sheets now....

Town Guide Progress

The good thing about so much bad weather over Christmas was that I could do some nerdy work rather than going out walking or cycling.....
The town guide Python project has therefore made good progress. The program can now generate a PDF 'poster' that can include a street index, and labelled features from the map database (pubs, shops, tourist attractions etc.). A typical example is shown at http://www.townguide.webhop.net/example1_hartlepool.pdf.

I have set up a very rudimentary web service using a PHP script to call townguide.py to generate a town guide on demand at http://www.townguide.webhop.net. It is running on a very small computer, and there is no queue, so there is no saying what will happen if more than one person tries to use it at once - that is another job!

Animation

Laura and I have been experimenting with stop-motion animation - set up a camera on a tripod with a remote shutter release, then position a scene in front of it, and change it by a very small amount between shooting frames.
10 frames per second seems to work ok for what we are doing (and you get more video for your money, so it is a bit more rewarding!).
I am using mencoder to turn the jpeg images from the camera into a mpeg4 video using the following:

mencoder mf://*.JPG -mf fps=10 -o leaves.avi -ovc lavc -lavcopts vcodec=msmpeg4v2:vbitrate=800 -vf scale=640:480

This produces an mpeg4 encoded .avi file, scaled to 640x480 pixels.
You can then run mencoder again to add a mp3 audio stream.
The results can be seen at http://flickr.com/jones139.

Town Guide (again)

Progress with the Town Guide program stalled a bit when I went on holiday, then was rather busy at work for a while. I am back to thinking about it now.
The current version has two output modes, HTML and PDF - both are quite crude, but work.
The PDF output uses LaTeX, which seemed like a good idea at the time, but I am struggling because LaTeX thinks it knows better than me what the output is supposed to look like - it keeps putting the tables etc. at the end, rather than where I ask for them.
This means I have two options - either do battle with LaTeX to sort it out, or abandon it and go for a more 'direct' PDF production.
I think that as the output I am producing is short and simple, I may as well go for 'direct' production. I am going to have a try with ReportLab's python library (http://www.reportlab.org/oss/rl-toolkit/guide/).
I'll go and read the instructions and see how it goes....

Free Town Guide

For ages I have been intending to write a program to produce a nice guide to a town or area based on OpenStreetMap data.
I finally decided that I knew enough about postgresql and python to have a go.
The result is a program called townguide.py. The idea is that you load the openstreetmap data into a postgresql database (like you do if you want to render it using Mapnik), then produce an xml file to tell townguide.py what you want rendering. It then goes and produces the nice output for you.
Version 0.1 is working and is on SVN. At the moment it only produces html output (because that is easy), and it is not very pretty.
The next stage is to make it produce postscript to give nice printable output on a defined paper size.
An example of the html output is at http://hartlepoolmap.webhop.net.

Having looked at the output it looks like it will be quite useful from a mapping point of view - if you know the area you can review the features that are summarised and look for missing banks, supermarkets etc.

Using a Graphics Tablet in Ubuntu 9.04

I have a cheap Trust graphics tablet that I used to use for drawing in Inkscape or Gimp.
It stopped working at some point (Can't remember which Ubuntu version), but I decided I needed to use it again, because drawing with a touchpad on a laptop is a pain in the thumb...
It was surprising that it didn't just work when you plug it in, because I'm sure a previous version of Ubuntu behaved like this.
I found the instructions at https://help.ubuntu.com/community/AiptekTablet.
You basically have to install the Aiptek xorg driver and put a configuration file in a directory under /etc/hal. I wondered where the X11 configuration was now that Xorg.conf is so small....

Anyway, it works now...

Progress with IP Camera Viewer

The Edimax IP Camera seems to work fairly well, but the video stream seems to get interrupted every now and again, which makes the player crash - vlc just hangs, and mplayer exits.
I decided that exiting was better so I have written a simple pyGTK front end for mplayer to play the screen. It also includes a wireless network link quality monitor to help understand why it is not working - it is here as 'bentv'.

Note that bentv uses a library that I have started work on (named ntpylib for want of a better name) - so far it only contains one class, 'prefs', which deals with saving and loading of simple key/value data from XML files. It includes a dialog box that GUI programs (such as bentv.py) can use to allow the user to edit the data. prefs.py and prefs.glade need copying or linking into the bentv directory to make it work - there is no clever installer!

Embedded Linux Firmware Binary Files

I have always struggled to understand the format of the binary files that store the firmware upgrades.
They tend to be a mixture of some simple text labels, a kernel image and one or more mountable filesystems.
I have written a little script that goes through a file in a brute force (one byte at a time) way and attempts to mount it as various filesystem types (script stored here).
I used the information found by this script, and the source code provided by Edimax to work out some information about the internals of the IC-3010WG IP Camera - it is written up here.

Practical Experience using the Edimax IP Camera

We were away on holiday last week so I got to try using the Edimax IP camera 'for real' as a monitor to keep an eye on our disabled son - known as 'BenTV-ng' (next generation) to distinguish it from 'BenTV' which was the old analogue system I used to use.

I gave up on using ad-hoc networking and instead took my wireless access point with me. This had the added advantage of improving the range of transmission because I could arrange for the access point to be between the camera and the computer. The computer I used was a Toshiba netbook running Ubuntu Linux.

The first thing I noticed was that when I powered everything up, nothing happened. At first I thought it was because the access point needed to see a wired connection as well as a wireless one, because when I connected the computer to it using a wire things started to work. I soon realised that the range was awful - it stopped working as soon as you were outside of his room. This turned out to be my fault because I had also switched on the analogue camera, which transmitted on the same frequency as the wireless network, so the noise level was huge. Things got better when I switched off the analogue camera.

At first everything appeared to work - vlc rtsp://admin:1234@bentv/ipcam.sdp played the video stream with sound.
I soon noticed that if the network got interrupted vlc froze, giving an apparently good picture, but no movement, which was no use to me. Instead I changed to mplayer - at least if that got into trouble it crashed and closed the video playback window so you knew something was wrong.

I set up a simple shell script that loops indefinitely starting mplayer then when mplayer exits it re-boots the ip camera, waits for 30 seconds and starts mplayer again.
This approach worked pretty well - the 30 second interruptions during the re-boot were not usually too troublesome.

We did find some problems when the video stream only worked for a few seconds before mplayer exited and the re-boot sequence started again, which was no good at all. The only way I found to cure this was to power off the camera all together - a software re-boot via the web interface did not cure it. I'll set up a little test at home now to see how long it takes to get into this state, and whether it makes a difference if you use wireless or wired networking.

The code to achieve this is stored at http://code.google.com/p/ntmisc/source/browse/bentv/.

The most up to date information I have on using this camera with linux can be found at http://ic-3010wg.webhop.net.

Ad-Hoc Wireless Network Troubles

Now that I have the Edimax IP Camera working I want it to connect to my Toshiba NB100 netbook computer as simply as possible - the idea is to use it when we are away on holiday so I don't want to have to set up a whole network with an access point etc. Instead I just want to set up an ad-hoc network between the computer and the IP Camera.
According to the instructions this should be easy - I tell the IP Camera to connect to an ad-hoc network called 'bentv', and then use NetworkManager in the Ubuntu desktop of the netbook to create the ad-hoc network.
This works fine on my main laptop, but fails on the netbook.
It turns out that NetworkManager is not capable of putting the wireless card into ad-hoc mode.
The work around was to have the script to start the VLC viewer shutdown network manager and set up the network manually as follows:

#!/bin/sh
/etc/init.d/NetworkManager stop
ifconfig ath1 down
wlanconfig ath1 destroy
wlanconfig ath1 create wlandev wifi0 wlanmode adhoc
iwconfig ath1 essid bentv
iwconfig ath1 channel 1
iwconfig ath1 192.168.1.35
iwconfig ath1 netmask 255.255.255.0
ifconfig ath1 up

vlc --fullscreen rtsp://192.168.1.17/ipcam.sdp

/etc/init.d/NetworkManager start

This is based on information I found here.
This seems to work, but the wireless networking is a bit iffy after the script exits - I might have to do the wlanconfig destroy bit again to put it into infrastructure mode.

More on Edimax IP Camera

I think we're getting somewhere. Searching around for references to IP cameras and mplayer yielded this thread which suggested the following:

mkfifo fifo.mpeg4
wget -q http://admin:1234@192.168.1.117:4322 -Ofifo.mpeg4 &
mplayer -V -demuxer 35 fifo.mpeg4

This seems to work ok - there is a bit of a delay in the video, but probably ~0.1 sec, which is acceptable.
Progress!
Just need to get sound working now....

Well, I did a bit more hunting around in the camera web set-up and realised I had missed a section about RTSP. The default was for RTSP to be enabled on port 554 with a rtsp path of ipcam.sdp.
When I tried

mplayer rtsp://192.168.1.117:554/ipcam.sdp

I got pictures AND sound!
I had tried rtsp before, but had missed the ipcam.sdp bit. Mplayer complains that my little netbook is too slow to play the stream, so there is a bit more work to do, but it looks promising. I'll try VLC now.

The most up to date information I have on using this camera with linux can be found at http://ic-3010wg.webhop.net.