Jordo Media RSS / Atom Feed Directory

…aka 1/6th of the screen left! Yay!

As of today, I am formally a Software Designer at Sulake Dynamoid. My main focus will be working on IRC-Galleria, the largest online community in Finland. The site boasts a whopping 800,000 member visits a week and is thus one of the world’s most active online communities. I have been interested in virtual communities for some time now (heck, I started one too) and the opportunity to work in the field professionally is something of a dream come true. I anticipate a lot of challenging work ahead but also many opportunities to further my skills in an exceptional working environment.

To find me at IRC-Galleria, look for a certain yakbrain.

Last year’s gingerbread train has been added to the collection. Having just finished this year’s [as of yet secret] project and with Santa lurking around the corner, it was high time.

I may be a bit early, but Merry Christmas and a Happy New Year to everyone!!

We are now four! Renée Alexandra was born on 15 October to very happy parents and an anticipating big brother. Oh boy, now we’re busy!

The tsampa listing has been enriched with Ann Lachman’s Purple Mountain Tsampa. She takes mail orders, so you don’t have to live in Massachusetts to enjoy her meticulously produced white and purple tsampa.

Having put web sites online since 2001, a web portfolio was long overdue. It is not a coincidence that the portfolio is also available in Finnish, nor that the Curriculum Vitae and about page have also been updated. Time to write some job applications.

The need to work around a bug getting my new Nokia E61 phone to speak with Exim4 got me around to updating the chrooting scripts for Stunnel to work with Debian Etch. The few who will find pleasure in this, enjoy!

It happened again. The GNU C Library broke, this time on Debian Etch. Not being able to track down the problem to either something I did or a Debian bug, I decided on adding another layer of protection. All services at tsampa.org are now running as virtual private servers (VPS) under OpenVZ. If any VPS dies, it does not hose the main system allowing me to restore within minutes instead of reinstalling and configuring everything from scratch. This also allows several auxiliary benefits, including easy migration to newer server hardware when that time comes and easy server resource management. Thumbs up for OpenVZ!

On February 20th, a routine upgrade of the server software using the Debian package manager failed horribly. The most central software library of them all, the GNU C Library, failed to upgrade correctly leaving the system largely non-functional. And to top it off, the Windows laptop followed suit with a prompt blue screen of death, thus killing the last open shell to the server. The web servers and databases were still happily up and running thanks to being isolated in their own self-containing jail environments, but most everything else, including logging in, was dead in the water. A reboot gave the expected result when the system failed to boot. Caputt.

I forked up the $50 for a 24 hour rental of a remote console with various rescue tools and operating system reloaders to be able to make a last full backup before reinstalling. While at it, I figured I would go ahead and upgrade to Debian Etch, since it is just around the corner to become the new stable release. But the server configuration, while flexible and hard drive failure resistant, proved to be a nightmare in this scenario where data had to be rescued off an unbootable system; the server was configured to mirror the data on two identical hard drives (RAID level 1) with most of the data residing on LVM volumes formatted with the ReiserFS filesystem. With a hardware card taking care of the mirroring, the two drives still transparently appeared as a single drive to the rescue disks. Alas, the rescue disks proved not have the software libraries needed to mount the LVM volumes AND read the ReiserFS format. Goodbye quick backup, goodbye.

Plan B was to have the RAID array broken up by removing the hardware card to allow a fresh install of the operating system with the requisite tools on one drive to access the original data on the second drive. This took a bit of constructive hard drive switching and bootloader configuration as the boot information and drive partition tables turned out to be stored on the hardware RAID card itself making the drive information unaccessible without it. In the end, fed up with facing yet another several hour wait to have a hard drive removed, I bravely installed the new operating system while having the original drive plugged in as secondary master with the feared result that I whacked up the partition table for the last original drive in the process. This was four days after the server crashed, two of which was spent waiting for the remote console to actually be hooked up and then determined not to work in my server rack due to a local issue necessitating the temporary relocation of my server. At this time, I had arrived at the tranquil conclusion that losing the last few months of e-mail and a few scripts would not be the end of the world. And it wasn’t. Alea iacta est and whatnot, I still had a reasonable fresh full backup of all the web data complemented by Google’s cache so things were reasonably dandy.

Armed with a new sense of how to reconfigure the server to better take this kind of disaster scenario into account, I had the techs remove the hardware RAID card for good and set out to install Debian Etch with software RAID. Things were not smooth sailing now either, as the installer first refused to install Etch due to a hard to track down Debootstrap Error, the server chassis needed to be replaced due to a faulty keyboard port and the techs made several mistakes (it did net me free access to the remote console for a few days and a free 512MB RAM addition for a current total of 2GB). In the end, it took an incredible eight days to go come to the point where I could actually do a 30 minute fresh install of the system and have the server moved back to its regular slot.

With this much downtime already behind me and no paying clients on the server, I took the time to upgrade all software, to rewrite the jailing scripts where needed and, more than anything else, to hack on a comeback to be reckoned with for my virtual powerlifting project (soon to be relaunched, tighten your socks!)… basically doing things that would have been needed to be done soon anyway. Apologies for the downtime, stay tuned for the pay off.

The final bits and pieces have now been restored here at tsampa.org, shoot me an e-mail if something is not working like it used to.

The gory details can be found here.