inactivity log for davidz

The past year or so I’ve been working a lot on gnome-disk-utility and DeviceKit-disks^Wudisks and I haven’t really blogged anything about it. Time to fix that!

First of all, given that the main DeviceKit daemon was killed by libudev (if you are using GLib you want GUdev instead!), we (that is, myself and other contributors) decided to rename the project - most people were cool with that; see this message for more reasons. One improvement in this area is that we’re committing to limited ABI stability - hopefully this will make things easier on distributors.

Most of the work in the udisks project been triggered by features we want in the Palimpsest Disk Utility - which is the most prominent part of gnome-disk-utility - the other part includes various desktop shell integration bits that I don’t have room to cover in this post.

On the Disk Utility side, we’re now using an user interface that scales better with many disks - in the old user interface, we used to show all partitions as children of each disk in the tree view. In the new user interface, we’re using a grid to convey the partitioning:

Also see bgo #579211

While this interface isn’t optimal for the laptop or the machine-with-only-a-single-disk cases, it works a lot better for machines with many disks

Bunch of disks

The Disk Utility also have somewhat complete support for MD RAID (aka Software RAID on Linux) - we’ll show you running arrays and also arrays not yet running. There’s a way to add/remove components and currently I’m working on creating the UI for Online Capacity Expansion (e.g. dynamically expand a RAID-5 array from 4 to 8 drives).

The SMART stuff has been revamped and there is now a way to turn off the desktop shell notifications. As it turns out a lot of our current users are like “I’m fine with using a disk that is about to fail” - kinda shocking but there you go. Another change is that we don’t consider a disk to be failing if there’s only a couple of bad sectors - the disk will have to have many bad sectors to be considered failing. This helps with all the false positives the SMART warnings generated in e.g. Fedora 11. Anyway, it seems like the desktop notifications are useful - more than a handful of people have pinged me on IRC thanking me and Lennart for this feature. Woot!

We also now have rudimentary support for showing the SAS topology - this is really useful since machines with many disks typically use dedicated disk enclosure shelves. Currently we only show SAS Expanders but the plan is to peek (and poke!) SES-2 Enclosures (and in the future also SGPIO ones) to get information about bay numbering and to toggle the FAILURE and LOCATE leds. Ideally we want to turn on the FAILURE led if a member of a RAID array is kicked or if SMART status indicates that the disk is about to fail. Ideally, we also want to do (sensible!) things like starting a RAID rebuild if a kicked RAID member is replaced by a uninitialized and same-size disk. As always, we have to be really careful about automatically doing policy things like this - especially on Linux where it is hard to make any assumptions on how the system is used.

The goal of the udisks effort is to be useful to users. For example, if you do video editing or a lot of photography, you care about IO performance. So I added a way to easily benchmark drives and RAID arrays:

RAID-6 FTW!

This feature was inspired by various similar Windows programs (a pastime on big storage forums such as this one involves trading benchmark screenshots) and the zcav program. It’s pretty cute, actually; I learned that my Intel 80GB G1 SSD only gives me 140MB/s on my Lenovo Thinkpad X61 while I get a full 270MB/s on my workstation (connected via SAS). Looks like the X61 is only using SATA 150 MB/s - a search on the tubes confirmed this. Gee! This feature is also useful when experimenting with RAID setups - for example, a 10K 36.7GB SAS disk benchmarks like this but when you put three of them together in Software RAID-5 it turns that writing is really slow. I know it’s supposed to be slow when writing to software RAID… but, really, this slow? Maybe I need to tweak some kernel tunables (which would be bad - we should be fast out of the box etc etc).

Finally, I did a really cute hack a few weeks ago - I made Palimpsest use D-Bus over TCP/IP over SSH to speak to udisks:

Bunch of servers

With the way this works we leverage SSH for authentication and Avahi for Service Discovery. The feature is still a bit rough - it needs some UI additions (such as a disconnect button) and we probably want to replace the stock OpenSSH dialog with something, uhm, less scary. There’s also the question of authorization - right now you have to connect as root in order to modify things (udisks uses polkit for authorization) - we need to figure out how we want this to work. Right now I’m thinking udisks will ship some polkit configuration bits (a .pkla file) and a udisks-admin group that administrators can add users to. We’ll see.

I’m kinda excited about the remote connectivity bits - it means that the Disk Utility is now very useful for headless server setups and said servers won’t have to run any graphical UI in order to be managed through the Disk Utility. We probably want to make the Disk Utility (e.g. Palimpsest) run on Win32 and OS X too (for users not running Linux on the desktop) for remote connectivity to e.g. Linux boxes running udisks. It shouldn’t be too hard to do this as we mainly depend on GTK+ and D-Bus.

There’s a bunch of plans for both udisks and Palimpsest. We want to add support (meaning: udisks API and Palimpsest User Interface bits) for the following things: LVM, iSCSI (both target and initiator), Enclosure Management (as mentioned above), Event logging (e.g. “raid5:md2: read error corrected”), Multi-path, Btrfs and many other useful storage technologies. There’s also a few rough edges and missing features (e.g. file system resizing) but we’ll get to that in due time.

FWIW, it’s important to realize that udisks and the Disk Utility complements, but does not replace, existing command-line tools. This means that you can go ahead and keep using your command line tools and scripts and the udisks/GNOME/Palimpsest/etc. stack will react to changes. For example, you can start/stop RAID arrays using mdadm(8), partition your disks using parted(8), create filesystems using mkfs(8), mount/umount filesystems and so on - the UI is kept up to date. While it’s approximately ten times harder (with the 10x figure being 65% accurate) to write software this way, it’s really the only way to do things on something like Linux where our set of users is as diverse as it is.

Had a great time at Plumbers Conference last week in lovely Portland, OR. On the Friday on the conference, Kay and I gave a talk, aptly named “Replugging the Modern Desktop”. The talk covered most of the hal-ectomy that has been happening and included a couple of demos. The slides are here. UPDATE: People on IRC are telling me that the fd.o servers are not reachable, I’ve put up another copy of the slides here.

I just committed gudev-1.0 to udev-extras. GUdev isn’t a particulary big library, it’s just a thin wrapper around libudev with integration bits for the GLib main loop and support for GObject Introspection.

While GObject introspection is very neat, see this GUdev example using Seed, I must admit the whole thing feels… half-finished?

For example, we seem to be doing the wrong thing for default annotations in some cases. Another example is the fact that gjs lacks a print() function, Seed can’t handle GStrv properties (works in gjs), lack of support for string arrays in gjs (works in Seed), neither Seed nor gjs supports Shebang support (gjs bug, Seed bug).

So all in all, trying to make my new small library introspectable.. was kinda.. much more work than I expected.. and most of the time was spent with the Seed and gjs runtimes trying to figure out what was wrong.

Anyway, I don’t want to sound all negative.. with all the work going on in this area, it looks like it’s going to rock once corner cases like mentioned above are ironed out. Rock on!

The GNOME 2.26 release in Fedora 11 will ship with a completely different stack for handling storage devices. The plan is to land all this work in the upstream GNOME 2.28 release and most of that work is done already.

Basically, we’ve switched away from one daemon to another and in the process we gained a few new features.

OMG, the disk is dying - using libatasmart

including letting the user inspect what his disk is trying to tell him. Kudos goes to Lennart for adding USB support, fixing issues with false positives in libatasmart and just for providing a really good library. And it looks like it’s working.

To really make this 100% useful, we probably want some kind of directory where system vendors shipping GNOME can drop an XML file and then the user can click a button “Order new disk” and get on with their life. Of course, then we need to handle the much harder problem of backup and data migration, but, hey, we want to solve that anyway.

So I also wrote this Palimpsest Disk Utility application

Kitchen sink of disk information

which also handles things like RAID devices and support for managing things like LVM and btrfs is planned as well. We’re also looking at changing the UI to be a bit more sane - something like this perhaps

Clean, leaner, simpler? Tell us in bug #579211!

Tomas, Matthias and I also worked on a simple formatting tool suitable for replacing the venerable gfloppy tool

Formatting tool

you can see the whole series of screenshots to see how the user interaction is supposed to be like.

Finally, I just merged a series of patches to GVfs for using this new machinery

OMG, pretty RAID icon!

Building this on top of a modern storage daemon, as opposed to HAL, means that your file manager (and any other app using GIO), is instantly updated as you repartition or reformat your disks. It even works if you are using command-line tools, etc. fdisk(8) or mkfs(8) - geek comfort FTW!

We might want some extensions to GIO to make working with RAID and encrypted devices (and in the future LVM) easier (such as unlock/lock/start/stop buttons in the file manager sidepane) or maybe we’re just going to rely on the system auto-assembling such things and leave it in Palimpsest. Decisions, decisions.

Another geek-comfort I added, is the ability to show user mountable fstab entries, that’s bug #536292. I was a bit reluctant to do this, I think instead we want native support in GVfs for nfs:// yet still use the kernel filesystem driver though some kind of setuid helper. Then we’d also fix NFS servers to publish the fact they exist using Avahi and then people can discover it in Nautilus’ computer:// window. Just cleaner and easier, I think. Combined with support for bookmarks on the GIO level, this should be a dream come true for people deplying systems in the enterprise.

Most of this work is part of the bigger HAL-ectomy - my good friend Kay has a good post outlining the plans for this.

One of the cool things about the new IO library (GIO) in the GNOME stack is that it’s extensible. One set of extensions for GIO is provided by the GVfs package. The main (but not only) purpose of GVfs is to provide access to Virtual File Systems by having the file system backend (e.g. file system driver) running in a separate daemon process in user space - sort of similar to what the Filesystems in Userspace (FUSE) framework provides on Linux and, nowadays, most UNIX operating systems. GVfs currently includes backends for SFTP, FTP, OBEX for Bluetooth, Digital cameras and MTP devices (through libgphoto2), Compact Disc Audio, Webdav, HTTP and archive files (through libarchive) to name the most important ones.

One key difference between FUSE and GVfs, is that GVfs file system drivers are not constrained by the rather old-fashioned, limited and design-by-committee POSIX API. Instead, GIO provides (and exposes for implementation to file system drivers) a modern and powerful API that provides asynchronous IO, file monitoring (like inotify) and high-level operations as part of the standard API. The GIO API is at the right level of the stack, it’s part of the GLib tarball. This means that any application that today uses GTK+ has access to GIO. This is a pretty big deal, if you depend on GTK+ 2.14 on newer, you’ll also depend on GIO.

However, since it’s rather naive (not to mention arrogant) to think that people are going to be using only GTK+ applications, GVfs also ships with a FUSE daemon. This daemon provides a POSIX interface to the virtual file systems provided by GVfs. All the fancy high-level operations available when using GIO won’t be available (there’s no POSIX equivalent function to map to) of course, but the basic POSIX stuff (e.g. open(2), read(2) etc.) will work just fine. The way it works is that the GVfs FUSE daemon provides a mount in $HOME/.gvfs where each top-level directory represents a GVfs mount:


$ ls -1 $HOME/.gvfs
gphoto2 mount on usb%3A004,002
sftp on people.freedesktop.org
sftp on quad.local


For example, this means all the good old command line tools still work


$ stat /home/davidz/.gvfs/sftp\ on\ quad.local/home/davidz/.bashrc
File: `/home/davidz/.gvfs/sftp on quad.local/home/davidz/.bashrc'
Size: 313 Blocks: 0 IO Block: 4096 regular file
Device: 14h/20d Inode: 44 Links: 1
Access: (0600/-rw-------) Uid: ( 500/ davidz) Gid: ( 500/ davidz)
Access: 2008-10-01 14:35:28.000000000 -0400
Modify: 2008-04-25 11:55:02.000000000 -0400
Change: 2008-04-25 11:55:02.000000000 -0400


$ convert /home/davidz/.gvfs/gphoto2\ mount\ on\ usb%3A004\,002/DCIM/100NIKON/DSCN0001.JPG -dither out.jpg


and so forth. Lots of command-line people like this, at least they always like to complain when we didn’t have this back in the GnomeVFS days.

But it gets better. Today I committed a patch to GIO to ensure that GIO applications launching applications (such as the Nautilus file manager or the Evolution mailer) will always pass a FUSE path instead of the GIO URI. One implication of this is that if you launch a non-GIO application (such as mplayer or a KDE application) from Nautilus, that application will Just Work(tm) even if the file lives on a GVfs share.

I think this is a pretty big deal; now it just doesn’t matter what VFS system of the week is used in the application, it’s all POSIX as far as applications are concerned. Sure, there’s a small performance hit by having to go through the GVfs FUSE daemon (a couple of extra context switches) but for run-of-the-mill desktop applications this is insignificant. Also, as explained here it’s currently not a very smart idea to pass an URI to an application; there’s the problem we don’t have proper standards defining the URI’s we use; there’s also the problem of having to authenticate over and over again.

Finally, if the application itself is using GIO, this patch ensures we map back to a GIO URI and, thus, we bypass the FUSE daemon. This is best illustrated by the following screenshot:

Bastien in VFS heaven

As you can see from the terminal window, Eye of GNOME is being passed a FUSE path from Nautilus. But thanks to the aforementioned patch, this is getting mapped back to a GIO URI as shown in the properties dialog.

This feature won’t be available in GNOME until 2.26 (and GLib 2.20) - we’re concerned some broken applications might be examining the URI before it’s mapped back so we’re reluctant to provide it in GLib 2.18 / GNOME 2.24 for now. FWIW, we’ve been shipping patches for this feature since Fedora 9 and the feature will be in Fedora 10 as well. Other vendors are encouraged to ship these patches as well.

Earlier today Richard and I did a DeviceKit (the set of projects replacing HAL) presentation for some other group here at Red Hat. You can see the slides here and here.

Perhaps of interest to the GNOME community, there’s also screenshots of Palimpsest, an upcoming Disk Utility library and application for GNOME. Most of this is already available in Fedora’s development branch, dubbed Rawhide, but won’t be installed by default in Fedora 10. Right now I’m busy with PolicyKit stuff but the plan is definitely to get a gnome-disk-utility mailing list going soon and get this stuff integrated throughout GNOME (I’ve already made sure it’s easy to plug into gvfs for example).

For example, I learned the other day that my disk is failing (actually, as mjg59 pointed out, it’s wrong to use the word FAILING since it’s Old-Age. Easily fixed.). Now, it would definitely be useful to have a notification bubble indicating this. This is pretty trivial to write using the DeviceKit-disks API; just monitor the org.freedesktop.DeviceKit.Disks.Device:drive-smart-is-failing property. Of course the gnome-disk-utility libraries (there’s one at the GLib level and one at the GTK+ level) wraps this nicely. In fact the gnome-disk-utility library at the GTK+ level should probably provide the code for doing this status icon. Something to discuss. Hence why a mailing list and more community involvement in the gnome-disk-utility project is needed.

Anyway, the goal is to port the most of the Fedora desktop to use DeviceKit instead of HAL for the Fedora 11 time frame.

Got one of them big laptops where you feel tempted to use a looking glass because the pixels are so tiny? Ever feel cheated when you adjust the font size but the rest of the UI looks like crap in comparison? Or maybe ever felt dirty when hard coding pixel values in your application?

there’s also a tiny and a normal version…

Today I sent a patch to gtk-devel-list adding Resolution Independence to the GTK+ toolkit. Let’s see how that goes.

Sure, there’s already d-feet and a similar Qt tool but for people who hate using the mouse or are allergic to X, I’ve put together this cute little hack that provides bash completion to dbus-send(1). Unfortunately it depends on the XML introspection parser from dbus-glib so it’s not straightforward to add to mainline D-Bus. Could easily go in dbus-glib though. Then again, it would be nice if libdbus included an XML introspection parser. Decisions, decisions.

I rarely blog these days, doing the Twitter thing instead and all. Anyway. The Call for Papers for the Linux Plumbers Conference in September in lovely Portland, Oregon has been extended until July 31st 2008. It’s a conference about the core infrastructure of Linux systems: the part of the system where userspace and the kernel interface. It’s the first conference where the focus is specifically on getting together the kernel people who work on the userspace interfaces and the userspace people who have to deal with kernel interfaces. It’s supposed to be a place where all the people doing infrastructure work sit down and talk, so that each other understands better what the requirements and needs of the other are, and where we can work towards fixing the major problems we currently have with our lower-level APIs.

I’m running the “Desktop Integration” microconf at Plumbers. I have two goals for the microconf. One is to be informative about what’s going on in the dusty hallways between the Kernel and the Free Desktop. Which I think is important at a place like Plumbers where we’re going to a lot of people working on similar problems present. The other goal is to actually try and make some headway on actual problems that require people from multiple communities working together. Such as some of the work Jon is focusing on, e.g. how to make fast-user-switching, multi-seat and terminal services Just Work(tm).

The conference is expected to sell out pretty quickly so it’s a good idea to go ahead and register for Plumbers Conference instead of procrastinating about doing it!

See you in Portland!

Am in Melbourne, VIC, Australia for linux.conf.au 2008; just gave a talk on PolicyKit, the slides are here. Only had 20 minutes so the whole thing felt a bit rushed and I didn’t get to go into much detail. Hopefully it made some sense though, otherwise checkout the design and API docs.