A while ago, I decided that I needed an up-to-date gosmore routing description file for my own little routing server. As it takes some time to build, and needs a lot of memory and processing power, I decided to just let “the cloud” do the dirty work for me.

As I’m confronted with Amazons cloud services daily at work, I chose Amazons EC2/S3 services. They can be quite cost-efficient if you know what to do, but have a tendency to get uber-expensive if you don’t. From my experiences, it should be possible to rebuild a worldwide .pak file for well under $5.

I’m not going to explain all details on how EC2 and S3 work and will use several terms that might be “puzzling” if you never used Amazons services before – take care to dig them up in the Amazon FAQs or documentation. I’ve taken care to put the real crazy stuff in italic.

Continue below for the named crazy stuff

Choosing the right machine type
This was the most difficult thing to do. First of all, you need 64bit and high memory support, so the so-called “small” computing instances are ruled out – they are 32-bit only. Going for a micro-instance (which strangely enough is 64 bit as opposed to the bigger “small” ones!) is not a good idea, as they only run on EBS block devices (expensive and slow), and only sport 600MB memory. You can create a swapfile, but recreating the .pak file takes several days on these machines and I/O is more expensive than what you save on money (micro instances are close to being for free).

As gosmore only makes use of one single core, it does not make sense to use a very big machine to save on computing time; in a nutshell, what I discovered to be perfect is a “m1.large” instance, which sports 7.5GB of main memory and four ECU units in two cores. If you hunt down a spot instance in a cheap region, you might get it for something as low as $0.14 per hour.

AMI/operating system
I needed the machines volatile instance storage mounted properly, and for some reason the official Amazon unix AMIs did not do that correctly, and they all NEED a EBS block storage as a boot volume, which would have been a waste in my case. I looked around for something that is purely S3-backed, and found this one: ami-a33b06d7 to be quite usable; it originates from Amazon, so I considered it to be quite safe.

Storage
As I expected from the beginning that I’d have to try out different setups, images and machines, I decided to create a 50GB block storage (EBS volume, which is non-volatile and can be mounted into and unmounted from a virtual computing instance) to store the original Open Streetmap planet file, the gosmore compiled source code and so on. I’ve deleted it by now, and pushed all necessary files into S3. If you follow this guide, it should not be necessary to use any block storage at all, which saves money. Downloading a 10GB OSM planet file takes some time though, so if you want to be safer, store it on a mounted EBS volume that you can re-use until you are “done” with everything. S3 is an option, but copying from S3 takes a lot longer than mounting an EBS volume.

Gosmore
Use the latest version from the code repository, as opposed to the latest release. Nic Roets of Gosmore applied some minor fixes to the code/settings during my try+error phase that have been important to make building and running gosmore for rebuilding big .pak files work correctly.

Setting up the EC2 instance
First of all, create a EC2 instance in your preferred region, possibly a spot instance to save some money; use the AMI named above. After you gained shell access, you need to install the following packages to make gosmore happy:

# prepare the instance (missing packages)
sudo yum install screen libxml2-devel gtk2-devel gcc-c++ make subversion libcurl-devel gpsd-devel

Next, you need to grab gosmore from SVN. Install it on the ephemermal block device – it has enough space to store all temp files!

cd /media/ephemeral0/
svn checkout http://svn.openstreetmap.org/applications/rendering/gosmore/
cd gosmore
./configure

#build gosmore for ONLY routing and headless usage:
make CFLAGS=’-O2 -DRES_DIR=\”/usr/share/gosmore/\” -DHEADLESS -DONLY_ROUTING’

Download the .planet file you want to build into a .pak file, for example from any OSM mirror. I used a file covering whole Europe, but the world should also work out fine.

The following commands expect the file europe.osm.bz2 to exist in the same directory as gosmore.

#rebuild gosmore .pak file
screen
bzcat europe.osm.bz2 | ./gosmore rebuild

After 4-5 hours, the instance should be done with building the files; for Europe, 7.5GB of memory would be sufficient without creating a swapfile. I ended up only using a Micro instance, which took roughly 5 days due to the fact that they have dramatic CPU and bandwith limitations, and it needed to swap heavily.

In the end, I used this tool to push the final file (and my gosmore build directory for later reference) into S3:

http://www.beaconhill.com/solutions/opensource/s3cp.html

Using EC2 as a routing backend
Of course you can also run gosmore on EC2; considering the costs, it might be cheaper to go for any other web hosting service, but if you would need dramatic load balancing and high-load scaling, Amazon has what you need.

Straightforward setup guide for an m1.large instance:

# install additional packages
sudo yum install screen libxml2-devel gtk2-devel gcc-c++ make subversion libcurl-devel gpsd-devel

# install gosmore
cd /media/ephemeral0/
svn checkout http://svn.openstreetmap.org/applications/rendering/gosmore/
cd gosmore
./configure

#build gosmore for ONLY routing and headless usage:
make CFLAGS=’-O2 -DRES_DIR=\”/usr/share/gosmore/\” -DHEADLESS -DONLY_ROUTING’

Now place a .pak file in the gosmore directory, and it should be usable fine. Note that this setup will also work to build a new file.

Some additional hints

Easy setup for your fstab to mount the volume:
sudo bash
echo “” >> /etc/fstab
echo “/dev/sdf /mnt/data ext3 defaults 0 0″ >> /etc/fstab
mount /mnt/data
exit

In case you need a block storage, create and mount it, and prepare it like this:
sudo mkfs.ext3 /dev/sdf

In case you want to gamble around with micro instances, you need a swapfile. It can be created as follows (8GB example) (/mnt/data would be the volume to store it):
sudo dd if=/dev/zero of=/mnt/data/swapfile bs=1M count=8048
sudo chmod 600 /mnt/data/swapfile
sudo mkswap /mnt/data/swapfile
sudo swapon /mnt/data/swapfile

I hope this works out for you, questions can be asked in the comments anytime.

Und J. Wolfskin kriegt auch sein Fett weg

Und siehe da, der Ikea Küchenplaner ist ein wirklich schönes Stück Software, braucht für 3D natürlich noch ein PlugIn, aber sonst… stabil, hat keinen Mist gemacht, und ist eigentlich einfach zu bedienen. Ich hatte irgendwie mit schlimmerem gerechnet.

Ob wir die Küche da kaufen sei mal dahin gestellt, aber es hilft ja nun beim visualisieren und bis ich das in Blender gebaut habe…

Video von der NOZ….

und dazu die Stellungnahme vom Fraz.

&&

Happy Birthday Lars!

W000000t – ne grüne Zahl hab’ ich da schon ziemlich lange nicht mehr gesehen.

Hier zum selbrigem sattgucken.

Eintritt: 3€
Gästeliste: keine, es ist für ne gute Sache
Bier: extremst bezahlbar
Musik: lecker

Falls irgend jemand beispielsweise aus dem Süden Deutschlands anreisen möchte, und einen Schlafplatz braucht, lässt sich da sicher einiges regeln. Oder aus dem Norden, Osten, Westen… wer nicht kommt, ist definitiv kein Fonzie!

(ich hab’ den mal “gestickt” damits keiner aus versehen vergisst, gell )

Die complete series (ohne Peacekeeper Wars) zur Zeit bei amazon.com für ausgesprochen günstige $59,99 zu haben. So viel bezahlt man sonst für ne einzelne Staffel Star Trek…

Update: Tja, so ist das, Servicewüste Deutschland… amazon.com hat das Ding tatsächlich (vermutlich) so um den 28.1. herum geliefert, also binnen etwa einer Woche! Mir soll’s recht sein. Die erste DVD (eigentlich hätte es nur eine Folge sein sollen, aber nun, man kann halt nicht immer wie man sollte) habe ich mal überflogen.

Quintessenz: ganz hervorragende Bildqualität, ausgesprochen gut. Zum Ton kann ich leider nicht viel sagen, weil just vor ein paar Tagen mein DTS-Receiver (natürlich) seinen Geist aufgegeben hat, daher im Moment nur Stereo über den Fernseher. Aber ich glaub’ da geht was. Ausstattung ist super, es gibt so ca. 22 Folgen mit Audio-Kommentar, eine zusätzliche DVDs mit Extras (keine Ahnung was da drauf ist), und sämtliche Folgen verfügen über DD5.1 und Stereo (beides natürlich englisch). Staffel 1-3 sind 4:3 (so wurden sie auch gedreht), Staffel 4 dann ebenfalls korrekt in 16:9.

Also falls das Angebot bei Amazon noch besteht, kann ich jedem, der Spaß dran hatte, nur raten die Serie für die ca. 40€ eben einzutüten, so billig wird’s nimmer. Wenn ich überlege was das BSG complete gekostet hat