A while ago, I decided that I needed an up-to-date gosmore routing description file for my own little routing server. As it takes some time to build, and needs a lot of memory and processing power, I decided to just let “the cloud” do the dirty work for me.

As I’m confronted with Amazons cloud services daily at work, I chose Amazons EC2/S3 services. They can be quite cost-efficient if you know what to do, but have a tendency to get uber-expensive if you don’t. From my experiences, it should be possible to rebuild a worldwide .pak file for well under $5.

I’m not going to explain all details on how EC2 and S3 work and will use several terms that might be “puzzling” if you never used Amazons services before – take care to dig them up in the Amazon FAQs or documentation. I’ve taken care to put the real crazy stuff in italic.

Continue below for the named crazy stuff

Choosing the right machine type
This was the most difficult thing to do. First of all, you need 64bit and high memory support, so the so-called “small” computing instances are ruled out – they are 32-bit only. Going for a micro-instance (which strangely enough is 64 bit as opposed to the bigger “small” ones!) is not a good idea, as they only run on EBS block devices (expensive and slow), and only sport 600MB memory. You can create a swapfile, but recreating the .pak file takes several days on these machines and I/O is more expensive than what you save on money (micro instances are close to being for free).

As gosmore only makes use of one single core, it does not make sense to use a very big machine to save on computing time; in a nutshell, what I discovered to be perfect is a “m1.large” instance, which sports 7.5GB of main memory and four ECU units in two cores. If you hunt down a spot instance in a cheap region, you might get it for something as low as $0.14 per hour.

AMI/operating system
I needed the machines volatile instance storage mounted properly, and for some reason the official Amazon unix AMIs did not do that correctly, and they all NEED a EBS block storage as a boot volume, which would have been a waste in my case. I looked around for something that is purely S3-backed, and found this one: ami-a33b06d7 to be quite usable; it originates from Amazon, so I considered it to be quite safe.

Storage
As I expected from the beginning that I’d have to try out different setups, images and machines, I decided to create a 50GB block storage (EBS volume, which is non-volatile and can be mounted into and unmounted from a virtual computing instance) to store the original Open Streetmap planet file, the gosmore compiled source code and so on. I’ve deleted it by now, and pushed all necessary files into S3. If you follow this guide, it should not be necessary to use any block storage at all, which saves money. Downloading a 10GB OSM planet file takes some time though, so if you want to be safer, store it on a mounted EBS volume that you can re-use until you are “done” with everything. S3 is an option, but copying from S3 takes a lot longer than mounting an EBS volume.

Gosmore
Use the latest version from the code repository, as opposed to the latest release. Nic Roets of Gosmore applied some minor fixes to the code/settings during my try+error phase that have been important to make building and running gosmore for rebuilding big .pak files work correctly.

Setting up the EC2 instance
First of all, create a EC2 instance in your preferred region, possibly a spot instance to save some money; use the AMI named above. After you gained shell access, you need to install the following packages to make gosmore happy:

# prepare the instance (missing packages)
sudo yum install screen libxml2-devel gtk2-devel gcc-c++ make subversion libcurl-devel gpsd-devel

Next, you need to grab gosmore from SVN. Install it on the ephemermal block device – it has enough space to store all temp files!

cd /media/ephemeral0/
svn checkout http://svn.openstreetmap.org/applications/rendering/gosmore/
cd gosmore
./configure

#build gosmore for ONLY routing and headless usage:
make CFLAGS=’-O2 -DRES_DIR=\”/usr/share/gosmore/\” -DHEADLESS -DONLY_ROUTING’

Download the .planet file you want to build into a .pak file, for example from any OSM mirror. I used a file covering whole Europe, but the world should also work out fine.

The following commands expect the file europe.osm.bz2 to exist in the same directory as gosmore.

#rebuild gosmore .pak file
screen
bzcat europe.osm.bz2 | ./gosmore rebuild

After 4-5 hours, the instance should be done with building the files; for Europe, 7.5GB of memory would be sufficient without creating a swapfile. I ended up only using a Micro instance, which took roughly 5 days due to the fact that they have dramatic CPU and bandwith limitations, and it needed to swap heavily.

In the end, I used this tool to push the final file (and my gosmore build directory for later reference) into S3:

http://www.beaconhill.com/solutions/opensource/s3cp.html

Using EC2 as a routing backend
Of course you can also run gosmore on EC2; considering the costs, it might be cheaper to go for any other web hosting service, but if you would need dramatic load balancing and high-load scaling, Amazon has what you need.

Straightforward setup guide for an m1.large instance:

# install additional packages
sudo yum install screen libxml2-devel gtk2-devel gcc-c++ make subversion libcurl-devel gpsd-devel

# install gosmore
cd /media/ephemeral0/
svn checkout http://svn.openstreetmap.org/applications/rendering/gosmore/
cd gosmore
./configure

#build gosmore for ONLY routing and headless usage:
make CFLAGS=’-O2 -DRES_DIR=\”/usr/share/gosmore/\” -DHEADLESS -DONLY_ROUTING’

Now place a .pak file in the gosmore directory, and it should be usable fine. Note that this setup will also work to build a new file.

Some additional hints

Easy setup for your fstab to mount the volume:
sudo bash
echo “” >> /etc/fstab
echo “/dev/sdf /mnt/data ext3 defaults 0 0″ >> /etc/fstab
mount /mnt/data
exit

In case you need a block storage, create and mount it, and prepare it like this:
sudo mkfs.ext3 /dev/sdf

In case you want to gamble around with micro instances, you need a swapfile. It can be created as follows (8GB example) (/mnt/data would be the volume to store it):
sudo dd if=/dev/zero of=/mnt/data/swapfile bs=1M count=8048
sudo chmod 600 /mnt/data/swapfile
sudo mkswap /mnt/data/swapfile
sudo swapon /mnt/data/swapfile

I hope this works out for you, questions can be asked in the comments anytime.

ich dachte immer, die dübbeln die fest. tzz tzz.

zum glück habe ich keine flugangst und kann mir die beschwerliche reise mit der königin mary der zwoten schenken.

Decoding the ‘Lost’ Supper

Wie soll man auch wissen, dass man die Planke betreten muss, die lose ist, obgleich es nicht beim mouseover steht. Woher soll man wissen, das es auf Monkey Island(tm) ein Fort gibt, wenn man es doch nicht von oben sieht. Wie soll man wissen, dass man dem Affen mindestens 4Bananen geben muss.

Trotzdem herrlich.

Und das Fazit kommt von Guybrush Threepwood himself:

Zelt, Isomatte und Schlafsack – ich find das geht kompakt ab:

Merke: geh nie zum Film oder Fernsehen. Leider nix zu embedden:

http://www.myspass.de/de/ulmentv/knut/bonustracks/index.html?id=10062

Zeitungsberichten zufolge soll die arme Dame später sogar geweint haben. Dass sie das nur einmal hat, überrascht mich, wenn man sich das Bonusmaterial anschaut *hüstl*

Update: aber das Fußvolk hat die besten Witze am Start. “Das darf i net sogn, das is antisemistisch [sic]“.

Update 2: interessant. Hier ist Ulmen in einer ganz anderen Rolle unterwegs (nicht als Hansen, sondern als Kiffer Maurice), aber trotzdem läuft in Minute 2:57 die gute Plöhn (Redakteurin des Teams, oben auch genannt “Dame”) ins Bild. Doch alles nur Schauspieler? Zweimal von einem Ulmen veralbern wird sie sich ja nicht lassen. Aber das Team ist definitiv ‘wirklich’ genervt, und die Kameraleute sind ja auch keine Schauspieler (weil sie wirklich super Bilder zustande bringen). Mysteriös, mysteriös!

Nach Meinung von Niedersachsens Innenminister Uwe Schünemann (CDU) sollen Internet-Provider künftig ihre Kunden vertraglich dazu verpflichten, eine Filter-Software zu installieren, die den Zugang zu kinderpornografischen Inhalten blockiert. Er werde die rund 75 Provider in Niedersachsen, die Endkunden versorgen, schriftlich dazu auffordern, die Filterpflicht in neue Verträge aufzunehmen und bestehende entsprechend zu ändern. Die Provider sollen dann erkennen können, ob die Filtersoftware beim Kunden aktiv ist, und diesem andernfalls den Zugang zum Internet verwehren.

Das ist ja völlig bescheuert. Das ist wie… wie… ich hab keine Ahnung wie das ist. Ich bin sprachlos. Ich geh’ frühstücken. Ich… oh man.

Nein, ich hab doch weiter gelesen:

Mit der Filterpflicht würde aber zumindest die Mehrzahl der deutschen Nutzer an einem Zugriff auf kinderpornografische Internet-Seiten gehindert.

Verflucht noch mal, ihr ******** da oben, die Mehrzahl der deutschen Nutzer wird durch Verstand, ethische Grundeinstellung, und letztlich Geschmack daran gehindert, kinderpornografische Internetseiten “anzusurfen”. Ich geh’ kotzen.

Verbietet doch einfach das Denken. Ach so, ist ja schon.

Wenn irgendwer ein Katzenfoto hat zur Beruhigung…

PS: das dürfte dem Grundgesetz widersprechen. Aber das interessiert ja keine Sau mehr.

La Gomera ist mal wirklich eine entspannende Insel, mit fünf Klimazonen, wovon wir hauptsächlich in der südlichen Sonnen-SanMiguel-Zone waren.

Kann Nachahmen empfehlen