Btrfs - Next generation file system for linux

Source: btrfs homepage

The most recent development version of the Linux kernel (2.6.29-rc1) included a pleasant surprise: the development version of btrfs has been added to the kernel source tree.

There has been quite a lot of filesystem work done in the past few kernel versions. The ext4 file system was added as a development version in late in 2006 (Initial git commit) and was declared stable exactly two years later. (commit to rename ext4dev to ext4) ext4 is a nice evolution of the ext series of file systems and provides a number of new features such as larger storage limits, and extent based storage. See the wikipedia article on ext4 for more detail. But ext4 is still just a file system and doesn't really compete with the newest kid on the block, ZFS.

ZFS is the feature rich, "rampant layering violating" file system that Sun has developed for it's version of UNIX, Solaris. ZFS has a few key features that make linux users green with envy: quick and easy snapshots of data, data checksumming, and integration of several tools to manage disks and file systems.

Snapshots: ZFS is based upon a Copy-on-write architecture that basically writes a new copy of the data every time it changes. Once the new version of the data is written the old version is marked as deleted and the space can be reclaimed. To implement a snapshot system on top of this is trivial, all you do is instruct the operating system to not mark the old data as deleted and and changes are preserved. This also means that snapshots do not occupy any more space then just the difference between the two versions.

Checksumming: All data that is written to a ZFS file system is checksummed to ensure it's validity. Lately it has become more critical that hardrives can silently corrupt data. This has always been an issue but due exponential growth in storage requirements corrupted data is much more common. To help mitigate the risk of silent data corruption ZFS stores a checksum of all the data it stores and validates the data again before relaying it onto the operating system. If one copy of the data has been corrupted it is identified on read and seamlessly copied from another source.

Integration of several tools: If you have a server with several hard drives in it you potentially use 3-5 different utilities to abstract away the fact that your data is stored on multiple disks. You would have a RAID setup to make several hard drives appear as one larger, or redundantly stored disk. You would have an Logical Volume created on that disk so you have the flexibility to grow/shrink/move the volume(s), and finally you would have your choice of filesystem setup in to actually allow the os to access the data. On top of this all you may have a backup system implemented so that you can roll back to previous versions of the data (snapshots), and possibly a method to identify and resolve data corruption (checksumming). ZFS unites all this functionality under one system that is more flexible, and easier to manage.

For just the three features listed above you can see why linux users would want to make have ZFS. But there is an issue. ZFS is free software, but it is not compatible with the GPL licence of the linux kernel. ZFS code (and a binary implementation in OpenSolaris) is freely available but it is released under Sun's GPL incompatible CDDL licence. Sun has released a few of their major software projects under GPL and GPL compatible licences (OpenOffice, Java), but as ZFS seems to be a major technological advantage for Solaris it doesn't seem like it will be re-licenced anytime soon.

So with the licencing incompatibility of ZFS and the "that just makes sense" factor of the feature set, Chris Mason started to work on a GPL licenced competitor to ZFS. It's called Btrfs (B-Tree FS, or Butter FS) and it implements most of the major features of ZFS. It has snapshots, checksumming, multiple hard drive support, and a few other features. Btrfs was announced on June 12, 2007 (lkml post) and already implemented a number of the key features. After over a year of development it has been merged into the linux kernel as brtfs-unstable (git commit) and will be part of the general 2.6.29 kernel release. It is still an experimental file system and not ready for production use yet, but having the code in the main kernel tree is major step on the way to becoming a workable filesystem.

Btrfs feature set is squarely aimed at ZFS, but it does have big shoes to fill. ZFS has been in continuous development for over 4 years now and was included as a fully supported production ready filesystem in the 6/06 (june 2006) update to Solaris 10.

Linux development continues on at a breakneck pace...

Related:
Oracle Presentation with Benchmarks
Article on Snapshots & Subvolumes
Wikipedia Article on btrfs
Diagrams showing RAID Levels
btrfs gitweb
btrfs utilities gitweb
The last word on ZFS

PackageKit Vs. DPKG

Source: Using PackageKit in GNOME programs



Yet another flamewar has erupted on the internet. (Surprise!)

Backstory:
A while ago Richard Hughes was looking at the various graphical programs for maintaining a desktop linux system (pirut, pup, Yum Extender, KYum, up2date, synaptic, YaST, Zypper, rpmdrake, Smart Package Manager, porthole) and thought there may be a bit of duplication of work going on here, so he started yet another package management framework.

Now he didn't do this just because he didn't like the other applications, but to combine their effort and to start fresh. By developing a common back end and front end for package management a common interface for the users is realized. By rewriting from scratch the interface can be designed to use newer technologies like PolicyKit to provide additional security. With this PackageKit was born.

"PackageKit is a system designed to make installing and updating software on your computer easier. The primary design goal is to unify all the software graphical tools used in different distributions, and use some of the latest technology like PolicyKit to make the process suck less." - PackageKit intro from the PackageKit Site

Well, as you may know, PackageKit has really taken off. It still is rather bare bones, but so far there are 12 different back ends with at least 6 of them in a usable state. For each one of these backends PackageKit presents a common command line, graphical, library, and DBUS API and therefore also be deeply integrated with other applications such as Gnome and KDE.

The offer
So, to help gain interest in this great new library/daemon/gui/API Richard Hughes posted on his blog that he would be willing to code (or help code) package kit based functionality into existing applications.

Imagine, that when you want to install an extension for epiphany you could launch a PackageKit based extension manager that would show you all the extensions available in your distro's repositories and provide all the tools to install, & update these extensions? Now you don't have to leave epiphany to launch a different package management app to install the extension saving you time, and also no matter what backend package system your distro runs (as long as the backend has been coded) the user interface is exactly the same for everyone. This would be a huge leap in usability, and a great idea.

Troll spotted!:
But guess what Jason D. Clinton looked over PackageKit's implementation and declared it to be incompatible with dpkg/apt. But he didn't stop there, he blamed Red Hat for segregating the market by not being able to use anything invented out of house.

Chaos ensued, (as you could imagine.)

A few key points though:

• Richard Hughes started PackageKit before he was hired by Red Hat, so PackageKit is not a Red Hat invention. Red Hat is only guilty of hiring a developer with vision.
  
• Foresight Linux uses PackageKit as their default package manager. If you've ever tried to understand their Conary software packaging system you will soon find out it is unlike anything out there. This proves that PackageKit can be used in systems completely unlike RPM.
  
• Bashing Red Hat like that is unfair. I highly doubt any other company in the world comes close to the contributions Red Hat makes back to Linux and its associated applications/technologies.

Outcome:
I think that tools like PackageKit are the way of the future and it's kind of sad to see such negative press about such a great idea.
The fact that dpkg/apt is designed to allow random terminal input during the package management process is really dumb from a desktop perspective and alternatives should be enforced (such as debconf).
Also, the "problems" with the dpkg/apt backend should be resolved as soon as possible because of the huge political impact. dpkg/apt = Debian = Ubuntu, without those two distributions behind you, your project is really not for every distribution, it is only for distributions that don't use dpgk/apt.


Related:
Rob Taylor's response
PackageKit's Site
Jason D. Clinton's initial post
Jason D. Clinton's follow up post
J.M. Maurer response
Richard Hughes' (the main developer for PackageKit) mocking of the whole issue
Further mocking of the issue by Richard Huges
Havoc Pennington's (past Red Hat and Debian Dev) response

Visions for OpenOffice.org - Future Direction?

Source: My Visions



OpenOffice.org (OO.o) developer Kay Ramme has just posted his visions for the future OO.o. He outlines several major points that will really help bring OO.o into the future. Following is a

Make all of OOo MVC (Model View Controller)
MVC is a long standing application architecture that really helps separate the user interface from the code and data. Major benefits of this approach are a more flexible GUI as it is not tied to the underlying code and the ability to restructure the data without affecting the presentation. The MVC architecture has been around for quite a while and is generally accepted to be a good way to structure your application. This will make OO.o more robust and will greatly help with maintaining OO.o and reducing potential for bugs. Ruby on Rails is a popular example of how a strict MVC structure for you code will benefit your application.

See: Wikipedia: MVC

Unified application with just different work approaches
Currently OO.o is split into a few different application, Draw, Writer, Calc, Base, Impress, etc. These are all the same binary but they show significantly different interfaces and functionality. This vision talks about truly unifying the applications so that no functionality is duplicated and there is less ways to do things. For example, currently in Writer there is a table feature that is completely different from the spreadsheet functionality of Calc. What would be ideal is to unify this functionality so that tables in writer & impress are the same as the spreadsheet in Calc. There will have to additional features added to the spreadsheet functionality to make it a suitable replacement to the table function but this will help unify the application and will reduce code and then potential for future problems. Also this will make the application simpler to use as there will be only one way to interact/create a spreadsheet/table.
OO.o's main competitor Microsoft Office is slowly working on this and has made some improvements in the latest Office 2007, but still does not have this. Koffice the KDE office project is working on this exact idea as the basis for Koffice 2 and could become a major alternative to OO.o in the free software arena.

Event driven architecture
This vision also requires a major architectural shift but will make for a much more robust code base. Event driven architecture is basically changing the code to make it respond only to events. These are usually user events (clicking a button, typing, scrolling the mouse wheel). The alternative to this is batch or flow driven programming where the programmer controls the flow of the application. What this means for the user is a much more reactive application that responds to user input. All interactive application have some of this functionality but by pushing it to the next level OO.o will be much more responsive and user friendly.

See: Wikipedia: Event Driven Architecture

Common threading model
This vision looks to have a common threading model throughout the OO.o code. There is an ongoing push to make all code threaded to help with utilizing new multi-core processors and OO.o is working towards this. By using a common threading architecture you reduce code and bugs by reusing the same functionality throughout the code.

See: OO.o: Uno Threading Model

Declarative Language GUI
This vision relates to the MVC as it requires the adoption of a language to describe the user interface. This allows the presentation to be separated from the code and allows for more flexible UI creation and easier changes/themeing/customizing of the UI. Another benefit of this is that this language is easier to understand then the C++ that OO.o is mainly programmed in so this could allow for people who have trouble with the complicated code base to be able to contribute better to the GUI. In other applications this separation has allowed graphic artists to easier customize the interface and make a much slicker product.
One of the best examples of this is the Firefox/Thunderbird/Sunbird/Sea Monkey programs from Mozilla. All of these applications use a language called XUL to describe the UI. This allows all theses applications to run on a very similar code base (soon to be more the same through the use of xulrunner) with most of the differences in the XUL interface code. All of these applications allow easy customization through the use of extensions and add-ons.

See: Wikipedia: User interface markup language

Improve the Build System
OO.o has the reputation of being hard to compile, and from many reports this is very true. It's immense code base and complicated structure make it one of the largest projects within the open source community. But this reputation also hurts adoption and contribution by external developers as they can have problems modifying the code and then compiling their changes. This vision is looking to make the build process much saner and only require GNU make without any external dependencies (for the building process). This vision also calls for saner output form the compiling process to help with debugging the build process.

Conclusion
So that is one developers view for the future of OO.o. Please check out the original post for more information and the included links. OO.o is a great office suite and bringing ideas and visions such as these to the public eye can only benefit the project as a whole.

Related:
OpenOffice.org
OpenOffice.org Roadmap

Proposed Fedora 8 Features

Source: Fedora 8 Proposed Features



Fedora 8 is currently under development and is scheduled for release in November of this year (2007). This is a quick overview of the proposed features. As these are proposed features and as it is still about 4 months from release some of these will change.

Better Startup Graphical, smooth and polished startup



This first feature is to bling up the boot sequence. Currently the boot sequence is a mix of text and graphical modes and doesn't look as professional as some of the other competing OS's logins. (Mac OSX, Windows, Ubuntu) This features calls for the hiding of some current boot up graphics such as the GRUB menu and ensuring that there is only one mode switch between text and graphical boot. This will help make the fedora (and the next RHEL) boot up slicker and adds a more professional image to the OS.

Bigboard Replacement for GNOME panel that uses online services via mugshot



This next feature is about integrating the Mugshot Online Desktop Project's bigboard application. Bigboard is basically a sidebar for linux that partially replaces some of the top and bottom panel's functionality but with "greater emphasis to personalization, search, presence and contacts, and documents."

Codec Buddy Helper app promotes Free alternatives and guides users trying to play content under restricted codecs



This is a similar feature to Ubuntu's automatic codec install. When a user tries to play an audio or video file with an uninstalled codec this feature will ask the user if they would like to install the relevant codec. It will also warn the user about illegal use of codecs and will try to educate the user. This is different from Ubuntu's auto codec install as it will recommend the purchase of Fluendo's gstreamer codecs, where as Ubuntu will install the "free" codecs after the user acknowledges that the codecs are illegal in some countries and should not be installed there. This feature will help with usability as often users are confused when they cannot play their audio or video files. This also goes along with Fedora's commitment to a completely free OS as they are pushing the completely legal alternative first.

KDE4 Integrating KDE 4.0, a new major version of KDE



Not much has to be said about this feature as it explains itself. KDE 4 is going to be released in October and Fedora would like to incorporate it as the default KDE version into their 8th release. They are working hard on this feature as KDE 4 is highly desired and with the expected release of Fedora in November it will be one of the first distros to completely incorporate KDE 4. The Fedora developers are also working hard on ensuring compatibility with existing KDE 3 applications by shipping a full set of compatibility libraries.

NetworkManager System wide default network manager integration



This is another feature that Ubuntu currently has. Fedora would like to integrate NetworkManager as the default network configuration method. NetworkManager was started by Red Hat and currently the easiest way to manage wireless networks. It is ideally suited for laptops with wireless cards. It also has plugins for different VPN implementations making it easy to use a VPN. The current weaknesses of the implementation is in the server market or with managing multiple network devices. Also there isn't any easy method to manage the networks without a GUI. Fedora is working on these issues and is hoping to implement NetworkManager as the default and only network manager for Fedora 8.

PolicyKit Easy and painless administration



"PolicyKit provides a flexible framework for granting users access to privileged operations." This will help integrate the various ways distros allow desktop users to preform privileged operations. PolicyKit is completely integrated with HAL and DBUS and will provide a cutting edge desktop-environment agnostic method of securing the desktop. PolicyKit will also fix some implementation flaws of current methods of providing privileged operations to desktop users. (such as running X applications as root)

PulseAudio Feature rich sound server



This is one of the most exciting features slated for inclusion in Fedora 8. PulseAudio is a sound server, which provides for network abstraction for sound. It also provides for full mixing of multiple audio streams and can be used to preform various effects on sound. PulseAudio allows for application independent volume control and also will allow any sound source or output to be shared or mixed over a network. For example with PulseAudio you can have Rhythmbox running of one computer but play the audio over the soundcard of a different computer on the network. It also will work for input allowing for a microphone input on one computer to be used on a different computer. This feature will bring the audio side of the linux desktop into the 21st century and will compliment the advanced visual effects available with Compiz and Compiz Fusion.

Presto Using delta RPM updates by default. Saves heavy amount of bandwidth and time for updates.

This is also an exciting new feature. Presto is a plugin for Yum to allow for the use of deltaRPM's to be downloaded instead of the full RPM's when updating. The current method of updating requires the full packages to be downloaded, this requires a lot of bandwith and often is not required as only a few files in the RPM package are changed. DeltaRPM's are much smaller as they only contain the differences between the two different packages. Currently there isn't an easy way to make use of these deltaRPM's and Presto hopes to change this. By using this plugin the bandwith required by a user to update is greatly decreased this will benefit the users with slow connections and will also decrease the infrastructure required to host a Fedora update mirror.

VirtSecurity Secure remote mangement for Xen, KVM & QEMU virtualization



Currently in Fedora 7 and RHEL 5 their is no secure remote management of installed VM's. This feature will incorporate full TLS for all communication between the VM and the management applications. It also incorporates secure migration of guests across hosts.

Bookmarks Fedora 8 Bookmarks

This is a very basic feature that just states which bookmarks should be default in Firefox with Fedora 8.

TexLive TeXLive 2007 inclusion



The current teTeX TeX distribution is no longer actively maintained and TeXLive is the new recommended TeX distribution. Debian and Ubuntu already are shipping TeXLive and this is a natural choice for any distribution that ships TeX.

Rsyslog New default syslog daemon

This feature calls for the inclusion of Rsyslog as a replacement for sysklogd. The sysklogd upstream is dead and also is missing a lot of requested features. Rsyslog has full security for logging (an often requested feature for sysklogd) and also can log to a MySQL database. Sysklogd is an ancient implmentation of the syslog functionality and is missing many requested features so upgrading is a no-brainer. There are several syslog replacements but Rsyslog is fully backwards compliant and provides most of the requested features and has an active upstream.

No More XFS Don't start XFS font server daemon by default and fix font packages. Savings on performance and bootup time.



This feature would like to disable the x font server that is currently enabled by default in all installations. The two main reasons are it will accelerate the bootup process and that the use of an x font server is not required for most installations. This is almost a no brainier but needs some testing to ensure that this will achieve the desired results. Also this will require checking all programs that make use of the xfs to ensure they use the local fonts.

XULrunner Use and integrate XULRunner which is the Gecko browser engine separated from the user interface.



Currently many applications depend upon Firefox just to make use of it's gecko rendering engine. XULrunner is a Mozilla project to separate the common features of Firefox, Thunderbird, Seamonkey, and any other gecko powered application into a separate program to make maintenance easier and to reduce duplication of code and functionality throughout their projects. This will require changes to many applications so that they use XULrunner instead of their own embedded gecko implementation. This will also increase security as if there is a vulnerability in the gecko engine it will be fixed in all programs that make use of it. This is another no-brainier as all distributions will eventually have to implement and the benefits are many.

Laptop Improvements Improved support for using Fedora on Laptops.



This last proposed feature is a blanket feature to improve the Fedora experience on a laptop. This is basically just a improvement to the suspend/resume support and the special keyboard keys that are often on laptops. It also would like to see an improvement in battery life to the previous Fedora release.

So, that is a rundown of the features in the works for the next fedora release. Please check out the wiki pages for more details and future changes.

Nouveau Driver 2D performance exceeds Nvidia driver

Source: Nouveau 2D progress



Richard Hughes has just published results from the gtkperf performance test.

Results:

Nouveau: 24.86 seconds
NV: 40.78 seconds

Which is a 64% improvement over the existing Nvidia sponsored nv driver.

This is a very limited 2D only test and does not represent 3D performance but it does show that the developers are making good progress.

The Nouveau driver is quickly gaining support within the various distros. Fedora 7 shipped with an alpha version of the driver and Ubuntu has committed to assisting and supporting the project.


Related:
Nouveau Driver Homepage

Updated:
To better show the improvements here is a benchmark comparing the free NV driver to the closed source Nvidia driver: NVIDIA Xserver driver Benchmark
In this benchmark (using a GeForce2 MX) the closed source Nvidia driver has a performance gain of 36% over the free NV driver. As Nouveau driver had a 64% improvement it even exceeds the Nvidia driver.

These figures should not really be treated as equal as they are using completely different Nvidia drivers. (6xxx series to hopefully the new 9x or 10x drivers.

Full OpenGL 3D Acceleration for Xen/Vmware/Kvm

Source: Thoughts on the Virtualization area.



With all the news about Vmware Fusion and Parallels upcoming 3D acceleration capabilities there hasn't been much about the free software projects work in this area.

Introducing VMGL, which provides OpenGL Hardware 3D Acceleration for Virtual Machines. It is a program that you install in both the host and guest and allows for virtualizing the OpenGL functions. When tested in xen with Quake 3 & Unreal Tournament 2004 it provides for up to 87% of the performance of the native implementation. It has been tested under vmware and xen but it should work under any of the other virtualization solutions out there. (kvm, qemu, virtual box)

This is major news for the virtualization field as 3D acceleration has been the one area where there has not been much progress. As this is a generic solution released under a BSD licence it can be incorporated into any of the current virtualization solutions. Also it is gpu independent and will work with any OpenGL accelerator.

Promo movie:


Related:
VMGL Homepage
Xen summit Slides

New SSL secured Apt in works for Debian/Ubuntu

As part of the new Gusty 7.10 release of Ubuntu, Apt has be upgraded to the latest 0.7.x series slated for Lenny. Some of the benefits of this new apt have already been seen in Ubuntu. The automatic dependency removal, the unattended upgrades, and the dpkg breaks field.

One upgrade in this latest series is the addition of a https transport method. This transport method should help to further increase apt's security.

Already with Etch (and many past releases of Ubuntu) secure apt has been included. Secure apt includes cryptological signed packages. These signed packages ensure that they are the ones the developers compiled, ensuring security. This was a major step forward in apt security.

The addition of the apt-transport-https package allows for ssl secured communication between the webserver and the apt client. Now with secure apt ensuring that the packages are what they say they are, and https secured communication, the entire debian/ubuntu package stack is more secure.



Related:
Secure Apt
Apt 0.7.0 Changes
Debian: apt-transport-https
Ubuntu: apt-transport-https

OpenMoko.com goes live

Source: OpenMoko.com



Today marks the launch of openmoko.com the corporate site for the openmoko project. If you didn't know already the openmoko is a new cell phone designed from the ground up in an open manner. All the hardware is designed in an open manner and the software on the phone itself is gnu/linux.

Previously only the community oriented openmoko.org site was live but this did not allow actual purchase of the Neo1973 phone. (The first phone openmoko will run on) Today anyone can go to the openmoko.com store and purchase a Neo1973 phone running openmoko. The phone is initially aimed at developers and is not quite ready for mass consumption.

The Neo Base model retails for $300 US, and the Neo Advanced is $450 US. They are not tied to any network and will function on any GSM provider.

This hopefully will provide some competition to Apple's latest creation, the iPhone.

Related:
OpenMoko.org - Development Site
- OpenMoko.org - Wiki
- OpenMoko.org - Planet
- OpenMoko.org - Mailing List
- OpenMoko.org - Getting Started with your Neo1973

New exciting Gnome developments

Source: Developing GNOME is going to rock even more very soon!



Gnome development is starting to see some interesting changes. The first is the previously mentioned gtkbuilder support in GTK+ 2.12. This adds new features over the previous libglade implementation and also is now part of GTK+ so it can be used everywhere GTK+ is used.

Another ongoing development is GVFS. Currently gnome uses something called gnome-vfs which is great but is showing its age when it comes to newer functionality such as FUSE.

These new features alongside the amazing existing language bindings for GTK+/Gnome make for a very exciting time for Gnome.

Related:
Initial plans to replace gnome-vfs
GVFS announcement
Upcoming talk on GVFS at GUADEC
GVFS Benchmarking Ideas

Kernel Change Logs for linux in English

Source: Linux Kernel Newbies



Ever want to know what was changed in the current release of the linux kernel but can't understand the cryptic changelogs? This site rewrites the changelogs in English and adds references. A great resource for anyone interested in the linux kernel.


Related:
Kernel Terms Glossary
Current Stable Kernel Changelog