The Gozer is In

MoMo CoLo

2008-11-13T22:58:36Z

MoMo CoLo
Originally uploaded by active_gozer

Finally gotten around to visiting the cage where the MoMo hardware is living. Couldn't resist snapping a few pictures, so here it is.

For the curious, here is what they are, from the top:
- Apple X-Serve
- Sun Fire 4150 x 4
- Sun Thumper (aka Sun Fire X5400) 18 Terrabytes of disk space

What's missing from the picture is the networking hardware.

We have a planet!

2008-11-12T18:19:25Z

It's actually been a couple of days, but I am just getting around to announcing it.

Mozilla Messaging now has it's own planet, planet.mozillamessaging.com, aggregating blog feeds from various Thunderbird sources. It's a complete ~~ripoff~~ copy of Planet Mozilla, just with different feed configuration.

It should get redesigned at the same time as the next incarnation of our main web site, so pay little attention to what it actually looks like, and just subscribe to the feed already.

Cheers.

P.S. If you feel like your feed (or a feed you like) should be on there, file a bug asking for it to be added

toirneach éan

2008-09-04T20:54:03Z

I've been recently working on bug 449202: Get Thunderbird L10N builds working on comm-central. It's been mostly about using KaiRo's existing work for SeaMonkey and s/SeaMonkey/Thunderbird/g in the right places.

Ran into a few more problems, mostly my fault, and some having to do with the way the MoCo build network is setup.

Turns out it's also fairly complex to test this stuff, as the current setup relies on notifiers kicking on changes to the l10n repositories, so triggering a l10n build on purpose is a bit tricky. Instead, it was simpler to just wait for somebody to change something in one of the many l10n repositories and see what happens.

Well, I am happy to report that the first localized build of Thunderbird since the move to Mercurial has been produced and can be downloaded. It's only a single build, and for Windows, but more will follow as the normal churn in l10n repositories will trigger some more.

Then, the fun begins tonight, as the nightly builders should trigger a build of all of Thunderbird supported locales in one go. So, by tomorrow morning, we should have tons (43 locales per platform, to be precise) of new localized builds waiting for us, sweet!

Oh, and in case you had been wondering, the first successfull build is here. I think it's pretty cool that the first locale that successfully build turned out to be ga-IE, so that will explain the topic of this post. Hopefully, somebody from that locale will understand the title (and apologies if I butchered your language, feel free to correct me please)

Repose en paix, Grand-Papa

2008-08-18T03:21:21Z

Sunday, August 10th, 2008, Jean-Patrice Chiasson, my father's dad, "grand-papa" to me, passed away, at the venerable age of 95.

I was able to attend his funeral this Friday. It was a very nice event, tons of people showed up, and I even saw some family members I hadn't seen in years.

To me, he was this mythical man, this doctor that cured the sick, and kept working for free until he was close to 80. Not a man of many words, but often always knew the right ones. Even if I don't think he never really understood what I do for a living, he was always able to provide me with amazing insight into my life. I remember many occasions where I was giving him an update on what's hapenning with me, and he would listen carefully. Then, almost without fail, he would ask a question, or make an observation that was always spot on. He was a very generous man, and ended up, almost secretly, financing all of his grandchildren studies, mine included.

I have so many memories of the itme I spent in their Sutton home as a kid (and later, as a grown up). The gigantic hill behind the house we loved to slide down in winter. The cold creek we build barrages in, with the woden overlooking bridge. And the maple forest where they made maple syrup every year. Every year, my grandma would drink a whole glass of the first batch they made, still warm.

La Tribune, the Sherbrooke local paper, even did a nice article on him this week, quoting my uncle Louis, among others.

It's strange to be back home like this. I was driving around town, running some errands, when I heard his name on the radio (Radio-Canada). They were talking about his passing and had interviewed a few folks about the work he did during his practice. It made me realize that, had I not moved back to Quebec, I probably wouldn't have heard it on the radio. Made me proud of my grandpa, that's for sure.

Oddly enough, he doesn't have a Wikipedia entry. I think he should.

eLOM is dead, long live iLOM!

2008-08-18T03:10:44Z

From the great c0t0d0s0 blog comes the news that Sun is working on an amazing upgrade to eLOM based systems, like the Sun X4150 we are using. Soon I'll be able to upgrade them to a real Lights-out-manager solution: iLOM.

iLOM is what's running on their Opteron-based systems, like the Sun 4500 (aka Thumper), and it's miles ahead of eLOM. Can't wait to have iLOM everywhere, instead of the current mixture of eLOM and iLOM.

Unfortunately, to perform the upgrade, you need to take the servers down... Can't have it all, I guess. Fortunately, since we are already fully redundant, it shouldn't cause visible downtime at all. All our build machines will just pause for a few seconds, and reappear on a different VM Host. Stay tuned, I'll blog about how the upgrade actually goes when it's finally available.

Thunderbird code size shrinks by 1.66Gb!

2008-08-14T14:13:56Z

First, that's only on OS X, sorry for folks running it on a different platform. Second, it's a bit of a lie, as I am talking about bug 448003.

Since we switched over to building from Mercurial, the code size tests we run were somewhat off on OS X. These tests basically measure the size of the code included in the product. For instance, right now, on Linux, it's reported at 15.7MB.

However, on OS X, they had been reporting 1.56GB! Obviously, this was a bug somewhere in the computation of this number, not the actual code size, so that's why the subject of this post is appropriately misleading.

Various hypothesis were put forward, including the fact it looked like it was off by precisely 100, so possibly a unit conversion problem, or something similar.

I've finally gotten around to it, and after some investigation, turns out it's more interesting than that. On OS X, we don't directly build universal binaries, but instead build once for i386 and once for ppc, then lipo the 2 builds together to form the final product.

To determine code size in this case, it would be misleading to count both architectures, so instead we pick one and size up that one. In our case, for historical reasons, that's the ppc version, even though our build boxes are Intel by now. Determining code size is done by parsing the output of /usr/bin/nm, an object dumping tool that nicely dumps the symbols list out of libraries and executables. Unfortunately, it dumps positions in the code, not size. So, to determine code size for a given function, just make sure addresses are sorted, then subtract the address of the previous function to the address of the current function, and you'll get the number of bytes that it takes up in the object file. Simple, right ?

Well, for some reason, not yet explained, on ppc, there is a strange symbol called trampoline_size that exists first in the list of symbols and sits at address '0'. However, right after it is the next symbol, strangely sitting at the address 0x20000000. That's 536870912 bytes further, if we believe what this is telling us. Of course, this is bogus. I am not entirely certain what this about, but if I could venture a guess, I'd have to say it has to do with the ppc executable format, OS X, and their ability to transparently run ppc binaries on Intel. Anybody that knows OS X/PPC better than me, feel free to correct me on this.


libnspr4.dylib: 00000028 a trampoline_size

libnspr4.dylib: 20000000 t __mh_dylib_header

libnspr4.dylib: 20001a7c t dyld_stub_binding_helper

Now that the problem is somewhat understood, it was a simple matter of teaching the code size stuff to ignore symbols like that on OSX/ppc and start at the next one. As you can see from the Tinderbox boxes, it was still pretty impressive to see such a large code size decrease! Oh, and SeaMonkey got the same drop in size, for the same reason.

Pretty graphs, data and what to do with it?

2008-08-13T19:01:31Z

Recently, we started receiving nightly logs of the Thunderbird start page hits. It now comes to us nightly, and I also got some historical data back to June 2008.

For the time being, I've just been feeding them all to webalizer, for lack of a better idea. There is not yet a complete plan as to what to do with all that data. Right now, it goes straight to infinite storage every evening. There is a lot of it, averaging at around 8 million hits a day, The included graph is the average traffic per hour of the day. That's in the 300k-500k hits per hour! I am glad that this traffic is being served by the mozilla.org hardware.

I know folks have already started to think of ways we could mine that data to learn useful things about Thunderbird and its user base, but it's only beginning. So, for the time being, I'd like to hear from anybody who can think of a questions that could be answered by all that data. Of course, respecting our users privacy is important, so I don't want anybody to know anything about specific individuals, just about trends. Are Russians users checking e-mails more often than us Canadians, for instance? (Not so sure that's a very useful thing to know)

Got a cool idea ? Have something you'd like to know ? Please, let me know.

Trying out slideshare

2008-06-25T05:18:15Z

More and more now, I've noticed folks posting their talk slides on SlideShare. I've decided to give it a try and post one of my talks over there and see how it goes. If it sees decent traffic, I'll probably post all my slides over there too.

| View | Upload your own

Want your buildbot to talk IRC?

2008-06-20T19:55:19Z

In setting up a buildbot master, and, reading some documentation, noticed that it has tons of status plugins, one of which is an IRC bot.

Before anything else, you need TwistedWords, an instant-messaging library. So I had to go ahead, download and install it.

cd tmp lftpget -c http://tmrc.mit.edu/mirror/twisted/Words/8.1/TwistedWords-8.1.0.tar.bz2 tar jxvf TwistedWords-8.1.0.tar.bz2 cd TwistedWords-8.1.0 python setup.py install

Once installed, all I had to do is to make a 2-line change to the buildbot master's master.cfg config file

from buildbot.status import words c['status'].append(words.IRC('irc.mozilla.org', 'thunderbot', ['maildev']))

Next thing you know, you've got an IRC bot announcing builds, and you can even ask for new builds straight from an irc /msg, cool.

Bonne fête des pères, Marc!

2008-06-16T06:23:22Z

C'est une fète fabriquée de toute pièces, mais c'est une fète à célébrer quand même!

Et puis en plus, c'est en double pour les grand-pères!

Il y a pas personne de mieux au monde pour lire des histoire des Schtroumpfs! Il l'aime son grand-père a lunettes!

Lights On!

2008-06-13T15:17:50Z

It's been a long little while, but things are finally reaching the point where I feel sufficiently satisfied to talk about it. (Plus, I've got time available now to do just that).

Life started with a rack full of brand-new Sun hardware, all wired in, mostly powered off. Thanks to mrz and Justin for all the heavy lifting!

When I was handed the 'keys', (actually, I handed my own ssh key) one of the Sun 4150s was pre-installed with CentOS. I was in!

I'll skip over an interesting learning experience with fancy Cisco switches and firewalls, maybe I'll talk about it some other time.

The first step was getting things to the point where they could boot and do something usefull. It's always possible to use remote management tools to trick the BIOS into booting from some disk image over the network, but it's very non-standard and usually requires unscriptable Java based tools, yuck.

This is where Cobbler comes in and saves the day. In short, it's an amazing tool, out of Redhat"s labs, that allows for the easy building and managment of a complete remote boot and installation solution. With it, you can easily configure systems by MAC addresses, pick what distribution you want to install on it, and you are done. Once setup, you just need to turn on the server you need, and within minutes, you have a freshly installed server, with all the specific installation instructions you wanted. The best with this is that it's 100% reproducible. You need 10 machines installed all the same, easy, just cut-n-paste their MAC addresses, turn them all on, go get a cofee, and you've got 10 identical machines. You want to re-install a system for a reason or another? Same thing. Configure it for a reinstall in cobbler and reboot the system.

It's very handy, and once you put all it's configuration in source-control, your entire server farm is easily reproductible.

Of course, having a bunch of servers all running a pristine Linux distribution isn't that usefull. You need to get them configured in various ways to actually get them to do the work you need them to do. In my particular case, first I needed basic configuration (networking/access control/logging/etc) and second, more importantly, I needed some virtual machines running to host some needed test services.

It's always possible to just login to servers and start installing packages, editing configuration files, starting services, etc. But it's such a bad idea. Even if you are a documentation fanatic, you'll always miss a step or two, and rebuilding a system exactly the way it was becomes impossible.

That's what tools like Puppet are for. They are an automated way to describe what you want on a system and perform the necessary tasks to get the system there. So, instead of installing the ntpd package, hand-editing /etc/ntp.conf and starting the ntp service yourself, you let Puppet do it for you.

Of course, it's also all in source-control, so every single change also gets tracked. The tricky part about systems like Puppet is that they are declarative. So, in the example above, you describe things much differently. You say, /etc/ntpd.conf needs to exist and needs this content. The package ntpd needs to be installed. The service ntpd : needs the ntpd package, the /etc/ntpd.conf file, and needs to be running.

Then Puppet figures what to do, in what order, and also will know that the running state of the ntpd service depends on the content of the /etc/ntpd.conf file, so if that changes, the service needs to be restarted.

There are many such systems out there, like cfengine and bcfg2. I've picked Puppet because it's very actively developped right now, plus supports Linux, Solaris & Mac OSX. The only downside to me is that it's all written in Ruby, and my Ruby-foo is pretty weak. Not a very big deal, because for most normal stuff, you deal with a configuration language, not the actual Ruby language.

At the end of last week, I had all the pieces in place, and I was finally able to directly help out Thunderbird development.

Somebody on #maildev needed access to some live IMAP servers that understood the CONDSTORE extension to test some stuff. With all these great tools already in place and working, it took very little time to get a new virtual machine configured, booted, installed and running some brand-new IMAP servers with support for CONDSTORE.

It's a small step, but it's really nice to finally get to directly contribute (even if in a small way) to the continuing success of Tunderbird.

There is still a lot of work to get done, things are not as automated as I would want them, and I've got some important missing pieces still dangling. For instance, there is no monitoring in place yet, so let's hope nothing goes down when I am not looking.

This coming week, the objective is to get a new mini-build-farm up and running, building Thunderbid continuously on Linux, Windows & OS X. While that's going on, I'll continue laying the missing bricks to the infrastructure itself.

That's enough for now.

ThreadVis: Arc Threads for Thunderbird

2008-06-09T18:21:47Z

Today, I ran into a very cool Thunderbird plugin, via David Ascher : ThreadVis

I remember having read the original IBM paper on this idea by Bernard J. Kerr
a while ago, and it was a very interesting take on thread visualization. I remember thinking at the time, bummer, I wish there was a Thunderbird plugin for this. Well, it took a little while, but it's finally here.

The idea is to use arcs to represent the relationship between the components of the thread. Why are most email threads displayed as trees, I am not sure. Looking at actual threads in my inbox with this addon confirmed to me that the idea has some merit. I especially love the fact that in this implementation, the arc length is relative to the time between 2 replies.

It's part of it's author's Master Thesis, so the extension also includes anonymized usage data extraction, to help him analyze how this is working out. It's optional, disabled by default, but do trun it on to help him out (I did).

If interested, install it directly from the official Mozilla Add-Ons site.

Power Maintenance @ Metropolitan Towers

2008-05-05T03:53:18Z

That's just too bad. The building we live in will be shutting down power for some electrical maintenance tonight. Why do I care ? Well, because I have this trusty OpenBSD Firewall that's been humming along in a closet forever. Well, not forever, but almost.

[root@dorsk root]# uname -a ; uptime

OpenBSD dorsk.bc.ectoplasm.org 3.7 GENERIC#50 i386 11:59PM

up 784 days, 10:51, 1 user, load averages: 0.86, 0.64, 0.48

[root@dorsk root]# shutdown -p -h now

Shutdown NOW!

*** FINAL System shutdown message from root@dorsk.bc.ectoplasm.org

*** System going down IMMEDIATELY

System shutdown time has arrived

Oh well, can't be helped. Go OpenBSD, GO!

openbsd

History

2008-04-14T05:14:10Z

Apparently, I've got to do like everybody else it would seem. However, I wrote the shell-foo the way I would have written it.

$> history | awk {'print $2'} | sort | uniq -c | sort -k1 -rn | head

  770   vi 
  692   ls 
  491   svn 
  381   perl 
  256   make 
   83   cat 
   75   grep 
   72   ssh 
   70   rm 
   56   rpmbuild

Vi, Subversion, Perl & make, yup, that sounds like me on productive days. ssh feels lower than it should be though.

iPhoned

2008-02-25T20:23:45Z

# uname -a

Darwin Gozer's iPhone 9.0.0d1 Darwin Kernel Version 9.0.0d1: Wed Dec 12 00:16:00 PST 2007; root:xnu-933.0.0.211.obj~2/RELEASE_ARM_S5L8900XRB iPhone1,1 unknown

Enough said, obviously