switching stacks

Usually a program only has multiple stacks if it has multiple threads, but sometimes it can be useful to have multiple stacks per thread. This is rare, but is handy for implementing certain types of language constructs like continuations or cothreads. It’s also used inside Wine and my dissertation project (which I’ll talk about more when it’s finished).

To allocate a new stack and switch to it, something like the following code will work on Linux/x86:

static __attribute__((used)) void *new_stack(int stacksize)
{
    // first argument: don't care where the stack is
    // third argument: VM permissions - stacks are read/write but not executable
    // fourth argument: VM flags - see man page
    // last arguments: unused for anonymous mappings

    void *s = mmap(NULL,  stacksize, PROT_READ|PROT_WRITE,
                   MAP_PRIVATE | MAP_GROWSDOWN | MAP_ANONYMOUS, 0, 0);
    if (s == MAP_FAILED) fatal_perror("stack mmap");

    return s;
}

..........

asm("	push $0x200000\n"  // 2 megabyte stack
    "	call new_stack\n"  // allocate it
    "	mov %eax, %esp\n"  // set %esp to its base
    "	xor %ebp, %ebp\n"  // clear %ebp
    "	call initial_function\n");  // call first function

The attribute((used)) marker is to prevent GCC optimizing the function out. Now initial_function() will be running on the newly allocated stack. If you don’t need the old one anymore remember to deallocate it using munmap. Pretty straightforward, no? Things get more fun when you need to return from initial_function() and switch back.

audiojelly

AudioJelly is a legal, paid for MP3 store which sells d&b, beatbeat, trace and other electronic music genres. I just bought my first MP3 from there, it was straightforward and no hassle. And of course works on Linux. They have an excellent Flash based preview player that gives you long previews, and even has a playlist. I bought the Burufunk remix of Ashland – Clear (click the little headphone icon to hear).

more bling

It turns out that Mirco “MacSlow” Mueller beat me to it, with his awesome Cairo Clock demos!

It’s nice to see that somebody else is producing demos of how to do this stuff – I found the code I needed by asking the GTK+ people, who pointed me towards the “testgtk.c” testcase file in CVS. A bit hard to find on your own! I wish I’d know about Mircos work before. Get your work on the Cairo examples page dude!

He has also done it far better than I did, with even a spiffy video to go with the code. That said, the code to set input area along with the drawing is quite complicated, so hopefully GTK+ will soon be supporting all this natively so we don’t have to write the boilerplate screen/cm handling code over and over (as pointed out by desrt in the comments, I got this slightly wrong).

Ray Strode says that Metacity already marks its presence in a detectable way, and I know the XFCE compositor does as well. The protocol these programs are using needs to be documented somewhere (EWMH spec?), or maybe the Composite extension home page on freedesktop.org

A few questions remain:

• Is it possible to get a pixmap containing whatever is underneath your window? Right now we can draw an alpha channel and get translucency, which is great. But how would you make a window that blurs whatever is underneath it (to give a thick-sheet-of-glass effect?).
• The window tears when resizing on my computer and in MacSlows demo. Is this fixed by the new compiz/metacity compositors?

gtk windows with alpha channels

I decided to engage in some blingineering today, and put together a self contained demo of how to make a semi-transparent window (ie with an alpha channel) using GTK+. For this, you will need GTK+ 2.8 or higher, Cairo, a modern X server and a running compositing manager. Any very recent Linux distro should meet the bill, I’m using SUSE 10.1.

And here is the heavily commented source code (python version).

This stuff should be documented in the GTK+ manual proper I guess, but hopefully Google will see this and it’ll go into the global pool of knowledge along with everything else. There are still a couple of problems:

• It’s not clear to me how to detect if a compositor is running or not. I guess there should be (maybe is) a simple X property on the root window you can check to see if one is running. Until we fix that it’s hard to use this technique in production software because people not running a compositing manager will see a black background instead of a transparent one.
• There should be a more convenient API in GTK+ for this, gtk_window_set_translucent() or something. Maybe we need a SexyWindow class, Christian?

bits and pieces – part ii

• Gimmie is one of the coolest apps I’ve seen for a while, and I now use it full time. It’s still very pre-alpha, lots of things don’t quite work yet, but it’s really awesome to see people revisiting the basic assumptions that underlie todays desktop technology.

• Chuck Norris is a pretty awesome guy. I heard he can divide by zero.

• A neat way to visualise Google news

• Recent WineHQ commits, with authors and proper descriptions. At last!

• Slides from Tim Sweeny discussing the future of programming languages. Tim is the lead programmer on the Unreal Engine and this presentation is a fascinating look at where he wants programming languges to go. Did you know that 90% of the integer variables in Unreal exist to index into arrays?

• That link came via Lambda The Ultimate, a programming languages discussion site. Quite heavy at times, but a generally genius site all round. In that link you can find comments from Tim on the performance of garbage collection in realtime 3D games, amongst other things.

• David Zeuthen blogs about using safe-for-scripting interfaces using DBUS as the privilege barrier. This is a little bit like how J2ME moderates permissions between applications, except that here RPC is being used instead of type security. It’s really cool to see these sort of next-level security systems being integrated into Linux. Go David! Elsewhere Kristian Rietveld blogs about GTK+ 2.10, which is shaping up to be a great release. <bill-and-ted>Dudes! Most Excellent!

castle food

I’m very lucky to have spent 2 years at university living in a beautiful castle:

It’s an extremely cool place that I’ll never forget. One of the nice things is that it’s catered, so in the third year we can concentrate on study rather than cooking and shopping (though we live out in the 2nd year and do all our own food then).

That said, every silver lining has a cloud, and here it’s the food. As you can imagine the rent to live in a castle is ridiculous (though not seriously worse than any other college) even though the university owns it. Unfortunately most of that rent goes into repairs and making sure the castle doesn’t fall into the moat, and as a result some budgets get cut to compensate. One of those budgets is the food budget.

The other day I arrived a bit late and all the rice was gone. We practically live off rice here, so that was a bit of a disaster. Here’s what I got for dinner:

mmm, stew
24 hours later I came down with food poisoning and was violently sick. Co-incidence? I think not

the “linux problems page”

Isak already commented on this, but I’ll do so as well as it was me who wrote that page and so I should take the blame for it. Reading Thomas’ blog entry hurt, because I don’t want people to think the project is ‘arrogant’, and it especially wasn’t intended as a slam on Python. I’ve been using Python myself a fair bit lately for various university assignments, and it’s been great.

Firstly a bit of context – as it says at the top, that page was written for a very specific meeting and should probably have been deleted or archived after that. It is insulting, partly because at the time I was feeling pretty bitter and cynical about open source and Linux especially, but mostly because there had already been (polite) conversations with the relevant projects and in each case they had been entirely unwilling to even consider change. This wasn’t due to lack of technical arguments on our part, but was usually a result of either not seeing the point (who needs binary compatibility when we have distros?) or in a few cases because it was felt that better API/ABI compatiblity would only help “evil” proprietary software.

The effect the UNIX wars had are well documented – the fragmentation and pointless incompatibilities UNIX vendors used to try and lock people into their particular flavor allowed Microsoft to step in and take over with Windows NT, a comparably well integrated and consistent solution. United we stand, divided we fall, and yet here I was writing a page about all the myriad little differences that divide the Linux software ecosystem. Seemed like history was repeating itself. I’m a bit more positive these days (a bit ) and as time went by I just forgot about the page.

More context. Virtually every single one of the points on that page has a back story which hasn’t been fully documented anywhere. Ironically I think the only one there which hasn’t been raised with the relevant upstream projects is libPython, partly because it was only encountered very shortly before the page had to be written, and partly because the primary incompatibilities are caused by distributions modifying the default configure parameters. A library which changes its exported ABI depending on how it’s compiled is never a good thing IMHO, but nobody forced distributions to configure it differently. I did file a bug with Red Hat about them using this configure switch, and IIRC it was quickly closed as not being a bug because it was felt that the (presumed?) performance improvement was worth the loss of compatibility. Not an answer I’m unfamiliar with.

The back stories behind these points are twisted and have taken place over a period of years in many different forums. In virtually every case, especially with respect to things like ELF, GCC, binutils and glibc, the issues were originally raised politely, bugs were filed, and patches were written. Sometimes discovering these issues took a great deal of effort – for instance, the GCC bugzilla bug linked to from the page took literally weeks of investigation to track down (as if compiler bugs weren’t hard enough to find, compiler bugs that cause heap/stack corruption are even worse). So imagine how I felt when it turned out that a well documented and publicised feature of GCC that we had been relying upon on in autopackage had never worked reliably, and apparently been implemented and documented without ever once testing it on a real program? Weeks of my time wasted, my end users experiencing crashes (which were blamed on us of course), no clear solution in sight and an upstream who basically didn’t care

That’s the sort of story which has been repeated over and over in different situations and with different projects. I don’t care to name them all, I did too much of that lately. In each case the story was the same – either nobody cared, or they did care but not really enough to make the necessary changes, or in the worst case they felt it was a good thing because it hurt evil proprietary software (apparently if you develop open source software you’re quite happy with requiring your users to compile things!).

4 years of this sort of conversation does make one bitter, cynical, frustrated and yeah … maybe arrogant too. I can see why it would look that way, though IMHO what people are actually seeing is frustration. Why bother holding back when nothing else that has been said ever had an impact? Might as well let off some steam.

Concern about this kind of thinking is one reason I floated the idea of somebody else becoming the autopackage maintainer and me taking a back seat. As of yet nobody has, but obviously once somebody with the necessary drive and enthusiasm comes along I’d be happy to ease them in.

other stuff

I can’t seem to post to LJ today, it just says “invalid form submission”. So here’s a quick reply to some of the comments there. Daniel Stone pointed out that Python programs which don’t use the C API aren’t affected by that. That’s true, but even in its previous burned-out-and-rude form the wiki page made it clear that those programs weren’t a problem. In practice most of the times I’ve looked at packaging a desktop program that uses Python it’s used C extensions in some way, often for performance or integration (eg using objects from libegg). So in practice this issue makes it hard to distribute Python programs in binary form, in much the same way that the shifting C++ ABI makes it hard to distribute C++ apps. It can be done, in certain situations and with some apps, but there are gotchas.

Somebody else suggested that the comparison between autopackage users and Debian packagers was invalid, because one group is users and one group is packagers, and Debian is bigger anyway. But I think it’s a valid comparison, because the conversation was something like “Me: Here’s a patch that’ll make things easier for people using your autopackage ….. DPs: Don’t do that, it’ll make things less convenient for us”. So in this case comparing the relative convenience of the two groups seems legitimate (unless you believe that a packagers time is more valuable than users time, which I think would be a tough argument to make).

on forks

One topic that has been the cause of confusion lately is my assertion that it’s wrong for Debian to do what they’re doing to XULRunner against the authors wishes. How can I both believe in the principles of free software, yet believe it’s wrong for Debian to change XULRunner in this way? Aren’t these beliefs contradictory?

No, in fact they’re not. Just because you can do something doesn’t mean you should. There are right ways to fork software like this, and there are wrong ways. Let’s quickly review the story, but changing some of the names around.

the wrong way

The Debian Project produces a series of ISO disk images that comprise the Debian Distribution. Because these ISOs are very large, mirrors are used to distribute them and spread the load out across many servers. Debian depends on this mirror network to distribute its software – not having any single predictable revenue stream means it depends entirely upon the generosity of others.

Let us imagine that Morris the mirror operator provides Debian with 30% of its bandwidth. A generous contribution indeed! One day he rsyncs his mirror and sees a new Debian release has been downloaded. Curious, he mounts the disk images and starts poking around. While he’s there he seems some obvious bugs and decides to make some simple improvements – after all, he’s allowed to by the terms of the GPL and he means well so why not? So he goes in and changes a bunch of packages, repacks the ISOs and goes on his way. Morris doesn’t bother telling the Debian project what he’s done, he’s a busy guy.

So people start downloading things that they believe to be Debian, which looks and feels and smells like Debian, but actually isn’t Debian and when they install it things crash and burn, and generally don’t work as they expect. So they log onto the net and start posting to web forums and mailing lists how crap the new Debian release is and how it didn’t work at all, and WTF are those Debian guys smoking to produce such a release? A few people will say they can’t reproduce those problems but we’re talking about computers here so nobody is surprised anyway.

So Dave the Debian developer gets a bug report he can’t reproduce, and starts investigating and eventually finds out what Morris has done. And Dave is pretty angry because he is seeing his work being trashed in public by all these angry users, through no fault of his own. So he goes to Morris and says, hey, Morris, I know you mean well but please don’t change Debian like this against our wishes. And Morris blows Dave off and points out that the GPL lets him do exactly this. So Dave posts an angry blog entry saying Morris sucks, and Morris mocks Dave saying he doesn’t understand free software, and meanwhile people are still downloading broken software and ending up thinking Debian is crap.

Now Dave does understand the GPL, but he also understands that this situation is why trademark law is invented. He wouldn’t mind if Morris had done things differently so there was no confusion over his work, and end users were able to make an informed decision about what version they wanted to use. But they can’t, the software has been silently modified during the act of distribution to the masses, and users don’t suspect anything is amiss.

the right way

Oh goodness me. What a mess. In reality, this situation couldn’t happen because we have trademark law to stop it and Debian(tm) is a trademark of Software in the Public Interest, Inc. And actually Mozilla Firefox(tm) is trademarked for exactly the same reason, and if MozCorp trademark the name XULRunner then Debian would be forced to rename the modified package to something else, DebianMozLauncher or something sufficiently different that it couldn’t cause confusion, and Debian packages that depend on XULRunner would have to depend on the actual XULRunner package and not DebianMozLauncher so no playing tricky games with dependencies thankyou very much.

Obviously though, most open source projects aren’t as large or high profile as Mozilla or Debian is and no trademarks are held on the name. In this case there’s no legal way to stop distributors modifying the software as they distribute it yet passing it off under the same name. Worse, twice now I’ve seen Debian developers use this fact as a threat, which is entirely unacceptable. But just because filing for a trademark is more effort than most open source devs are willing to go to doesn’t invalidate the underlying principle – people should know what they’re getting, and when upstream own the name they should own the software.

Don’t like it? Fork it … the right way. Create a new name, a new website, describe the differences so people know what they’re getting and have people review the products separately. Ubuntu forked Debian in the right way. Debian forked XULRunner in the wrong way.

another example

This post is a bit personal for me, because I’ve had to deal with exactly the sort of problem the XULRunner people are now experiencing, but with Wine. Debian has packaged Wine in a way different to upstream, and this can cause extremely subtle bugs. One incident that sticks in my mind is where I wasted an entire Sunday afternoon and evening working with a user to track down why a program was crashing when they selected a menu item. It turned out that when the program started it was querying a registry key that didn’t exist, and squirreling away a NULL pointer in some internal data structure. And when it tried to access that key it crashed. Why was the key missing? Because the installer invoked the regedit.exe program to merge a pre-written .reg file into the registry, which is more convenient than using the registry apis. No error checking of course because on Windows this cannot fail. And why was regedit.exe missing? Because Debian decided it looked like a “utility” and as such should be in an optional package, which the user didn’t know about and hadn’t installed.

on the future of autopackage

When I asked for ideas on what I should write about more, all the comments were from autopackage people asking me to write more about that. OK then. So be it.

## where are we now?

	The good. We have a quality product. It usually works as it was intended to work, we have a well defined target audience and these people are always happy with our work. We have a huge well of evangelical support from some of these users, and it’s good to see. Autopackage always gets great reviews from the media. When it came to controversial issues we were bold, and we justified our decisions with rational debate and argument. We mastered then documented binary compatibility. High profile projects like AbiWord, Gaim and Inkscape use what we made already, others like Mozilla want to. These things are a personal achievement for every one of us, we should rightly be proud of them.
	The bad. Autopackage has not gained widespread acceptance as we had hoped it would. Few packages exist, fewer still are of high quality and frequently updated. Whilst a few big, well known projects do use autopackage, in some cases this is not what it seems – Gaim developers except marv seem to dislike the project, and have been entirely obstructive. The Gimp package is not maintained by Gimp developers, who seem to believe that binary distribution is not their problem at all. Meanwhile, distribution acceptance sits at zero. Any mention of autopackage at all on mainstream distribution developer lists triggers instant flamewar, which further alienates developers who may not have had a strong opinion on the topic. CodeWeavers, my own employer, do not use it.
	The ugly. Some distribution people, notably Debian developers, outright hate us or insult us. This is part of a wider theme of upstream/downstream tension. Debian is by far the worst here and deserves to be singled out – their packagers routinely anger high profile developers, and the practice of subtly breaking software continues unabated. When autopackage 1.0 was released and we got a lot of media attention, one Debian developer implied that they would deliberately add autopackage runtimes modified against our wishes to their repositories. This behaviour is unacceptable, but is the status quo and as such is not seriously challenged.

## how did we get here?

The good. A strong vision of easy to use software installation, commitment to that vision, and a team with diverse skills.

The bad. Inability to change the prevailing Linux community culture in which binary packages are Not Our Problem. Refusal to make it easier for non-programmers to produce autopackages (my refusal, basically).

The ugly. Various things have been suggested, everything from the /usr vs /usr/local issue being called wrong to skipping native package manager integration, but my personal belief is that this problem was inevitable and nothing we could have done to the product itself would have changed it. At a fundamental level, distribution packagers today have huge power and nobody likes to lose power. Period.

## where do we go from here?

At heart, this is a personal question. Its answer will be different for each one of us. Faced with the following facts, what can we do?

• Autopackage works well when used correctly

• Yet it has low acceptance

• Ideas for tackling this problem are thin on the ground

For now, I think the most important thing the project needs to do is just get 1.2 out of the door. It’s dragged on for a long time …. within a month or so, it’ll have been nearly a year (!!). This is mostly my fault, I’ve been absent and uninvolved for a long time now, mostly due to coursework loads which have taken away my hacking time, and partly due to waning interest. And of course Hongli and Curtis have been busy with university and preparing for a child (congratulations Curtis!).

After 1.2 is out there are various issues to consider.

1. What should developer focus be on? Adding features like before, or on trying to tackle the causes of low acceptance?

2. Is hostility towards non-upstream packagers hurting more than it helps? I still believe that the highest quality packages will come from the people making the software themselves, but a few times now (like trying to install mplayer a week ago) I found myself wishing for an autopackage, any autopackage would have done as long as it basically worked. But our literature and workflow is entirely oriented around the idea that upstream makes the packages. Concrete idea: redesign binary relocability to be independent of code changes by using a low level trick.

3. Should I still be the maintainer? As you can see, I don’t really have the time or energy to push the project forward in the way it needs anymore. The obvious conclusion is that somebody else should be the leader. One thing I am certain of though is that at all times there must be one leader who has the final say. Letting things drift with no clear successor is not an option.

   …. and perhaps the most important one …

4. Does it make sense to make a new distro?

## a new distro?

Yes. Perhaps even a new OS entirely.

At some point, you have to take a step back and look at the bigger picture. You must ask yourself “why am I doing this?” and “is this is the best way to achieve my goals?”. When I started autopackage, I had many goals but the primary one was simply to make Linux easier to use by making software installation easier to use. That was the goal. It was a bit narrow, because it accepted that “making Linux easier to use” was the be-all and end-all, but at the time I was enamoured of the free software movement and this goal seemed to make total sense.

There were actually many ways to achieve this ideal, for instance I could have become a Debian packager and joined the never-ending push for a complete repository. Or I could have developed some mad hackz to try and make Linux binaries run whatever the source, a kind of super alien. But I decided that the best way forward was to produce a new kind of package, one that would be like a binary version of the source tarball, and which had as its primary focus usability (then a new word in my vocabulary).

Looking back, it is now clear that as of Feb 2006 it has failed to achieve the original goal. As a toolkit it doesn’t make anything easier by itself, it only provides the tools for others to make things easier. And generally others aren’t interested in that.

But times have changed and making Linux easier to use is no longer the be all and end all that it once was. What does making Linux easier achieve? What is the purpose? Beating Windows was once enough, but Apple managed to do this pretty conclusively and the bar since moved – beating Windows is no longer enough, now you must beat Apple too.

Stopping spyware was once enough, and this is why my blog has often focussed on that. But developing an entirely new OS and entirely new set of apps to go along side them seems like an inefficient solution. Linux and MacOS are currently totally undefended against spyware, and platform designs that seem to hold up like J2ME are so radically different to existing desktop models that it’s hard to imagine retro-fitting them onto Linux or Windows.

And of course, there is the original motivation behind GNU, that of existing in a world of only free software. A laudable aim, but one not in practice very compatible with reality. Whilst I could use only free software, that would be terribly inconvenient. I use the proprietary nVidia driver, the proprietary Flash player, I play proprietary games, my computer is powered by a proprietary BIOS and the circuitry inside my CPU is likewise proprietary. Putting up with inconvenience to use more free software (given that being 100% free is impossible) doesn’t always seem like a good trade.

There are clearly some goals that would be best served by creating a new distro. For instance, it would solve the problem of not having any distros support autopackage. It could solve other problems too, like no distro offering an easy (Windows-based) installer/uninstaller for the OS itself. But it’s tough to see how it could solve the cultural issues with the Linux community or the problem of binary kernel drivers.

Other ideas – shaking up the way software is managed on Linux is one possibility. For instance, union mounts change fundamental assumptions about namespace design, which haven’t really been explored yet. You could have software installed to individual prefixes in /software yet still appear to be mounted in /usr, partly solving the binary relocatability issue. You could go for the J2ME approach of totally sandboxed applications.

There are lots of potential routes for the project to explore. But I would shy away from simply adding new features as the roadmap originally implied.

Most importantly the goals and solutions must be carefully considered. So what is the goal? the solution? Good question. One I’ll leave for another time, I think.