Shared posts

03 Oct 21:06

Tmux and Vim: the perfect combination

Last night at Vim London I spoke about some of my favourite plugins and settings for Tmux and Vim that allow me to work with them as I do. It’s very rare these days that I won’t be editing code from Vim, within a Tmux session, but out of the box the immediate advantages of this pairing are not always obvious. It takes some time to get things working.

The talk was recorded and I will update the post with it once it’s public, but for now I just wanted to link to some of the plugins and settings I mentioned in the talk that I use in my dotfiles.

tnew alias

I create all my tmux sessions through an alias I have which I’ve called tnew, which simply calls a function called new-tmux-from-dir-name (this is taken from the thoughbot dotfiles. The function looks like so:

sh function new-tmux-from-dir-name { tmux new-session -As `basename $PWD` }

This does two things: - when you create a new session, it names it based on the directory name - if you are in a directory that already has a tmux session associated, tnew will attach you back into the already existing session

Tmux + OS X, pbcopy

By default tmux on OS X breaks the pbcopy command. I’m not going to go into details, but thankfully Chris Johnsen’s fix has detailed instructions on how to get that functionality back, and it’s not too tricky either.

Renumber windows

Out of the box, if you have 3 tmux windows numbered 0, 1, 2 and then you delete the one numbered ‘1’, you’re left with two windows: 0, and 2. This is rather odd when you navigate between them. However, you can tell tmux to renumber existing windows automatically with this line in your tmux config:

set-option -g renumber-windows on


Without a doubt, this is the one thing that makes working with Vim and Tmux so much better, Chris Toomey’s vim-tmux-navigator plugin. By installing this and adding a few lines to your tmux config (instructions are in the README), this lets you seamlessly jump between Vim and tmux panes (or splits, as Vim calls them) with the same keystrokes. This means if you have a Vim split next to a tmux pane, you can hop between them even though one of them is Vim, and the other is tmux. It’s fantastic, and really ups the intergration of these two tools by another level.


Although there are many choices, I personally have taken to using Vimux, a plugin which lets you send any command to a tmux pane (which it will create if it doesn’t already exist). This means I can set up mappings for running specific commands, such as for testing:

viml map <leader>nt :call VimuxRunCommand("clear; npm test")<CR>

It also has a function which you can run that will prompt you for a command, which I have mapped to <leader>x:

viml map <leader>x :VimuxPromptCommand<CR>


This plugin adds any word visible in a tmux pane to a completion function that you can trigger with <C-X><C-U>. It’s rare that I want to type something in Vim that I can see in a tmux pane, but when I do this plugin is always useful and saves coming out of Vim to grab some text.

So that’s just some of the plugins I use daily when working with tmux and Vim. If you have any to recommend, please do leave a comment!

18 Apr 19:43

managers are awesome / managers are cool when they’re part of your team

Apropos the Julie Ann Horvath Github shitshow, I’ve been thinking this weekend about management, generally.

I don’t know details about the particular Github situation so I won’t say much about it, but I was present for Tom Preston-Werner’s 2013 OSCON talk about Github. After a strong core message about open source licenses, liability, and freedom (tl;dr: avoid the WTFPL), Tom talked a bit about Github’s management model.

Management is about subjugation; it’s about control.

At Github, Tom described a setup where the power structure of the company is defined by the social structures of the employees. He showed a network hairball to illustrate his point, said that Github employees can work on what they feel like, subject to the strategic direction set for the company. There are no managers.

This bothered me a bit when I heard it last summer, and it’s gotten increasingly more uncomfortable since. I’ve been paraphrasing this part of the talk as “management is a form of workplace violence,” and the still-evolving story of Julie Ann Horvath suggests that the removal of one form of workplace violence has resulted in the reintroduction of another, much worse form. In my first post-college job, I was blessed with an awesome manager who described his work as “firefighter up and cheerleader down,” an idea I’ve tried to live by as I’ve moved into positions of authority myself. The idea of having no managers, echoed in other companies like Valve Software, suggests the presence of major cultural problems at a company like Github. As Shanley Kane wrote in What Your Culture Really Says, “we don’t have an explicit power structure, which makes it easier for the unspoken power dynamics in the company to play out without investigation or criticism.” Managers might be difficult, hostile, or useless, but because they are parts of an explicit power structure they can be evaluted explicitly. For people on the wrong side of a power dynamic, engaging with explicit structure is often the only means possible to fix a problem.

Implicit power can be a liability as well as a strength. In the popular imagination, implicit power elites close sweetheart deals in smoke-filled rooms. In reality, the need for implicit power to stay in the shadows can cripple it in the face of an outside context problem. Aaron Bady wrote of Julian Assange and Wikileaks that “while an organization structured by direct and open lines of communication will be much more vulnerable to outside penetration, the more opaque it becomes to itself (as a defense against the outside gaze), the less able it will be to “think” as a system, to communicate with itself. The more conspiratorial it becomes, in a certain sense, the less effective it will be as a conspiracy.”

Going back to the social diagram, this lack of ability to communicate internally seems to be an eventual property of purely bottoms-up social structures. Github has been enormously successful on the strength of a single core strategy: the creation of a delightful, easy-to-use web UI on top of a work-sharing system designed for distributed use. I’ve been a user since 2009, and my belief is that the product has consistently improved, but not meaningfully changed. Github’s central, most powerful innovation is the Pull Request. Github has annexed adjoining territory, but has not yet had to respond to a threat that may force it to abandon territory or change approach entirely.

Without a structured means of communication, the company is left with the vague notion that employees can do what they feel like, as long as it’s compliant with the company’s strategic direction. Who sets that direction, and how might it be possible to change it? There’s your implicit power and first point of weakness.

This is incidentally what’s so fascinating about the government technology position I’m in at Code for America. I believe that we’re in the midst of a shift in power from abusive tech vendor relationships to something driven by a city’s own digital capabilities. The amazing thing about GOV.UK is that a government has decided it has the know-how to hire its own team of designers and developers, and exercised its authority. That it’s a cost-saving measure is beside the point. It’s the change I want to see in the world: for governments large and small to stop copy-pasting RFP line items and cargo-culting tech trends (including the OMFG Ur On Github trend) and start thinking for themselves about their relationship with digital communication.

Comments (9)
27 Feb 04:20

The heart of juju

by niemeyer

The very first time the concepts behind the juju project were presented, by then still under the prototype name of Ubuntu Pipes, was about four years ago, in July of 2009. It was a short meeting with Mark Shuttleworth, Simon Wardley, and myself, when Canonical still had an office on a tall building by the Thames. That was just the seed of a long road of meetings and presentations that eventually led to the codification of these ideas into what today is a major component of the Ubuntu strategy on servers.

Despite having covered the core concepts many times in those meetings and presentations, it recently occurred to me that they were never properly written down in any reasonable form. This is an omission that I’ll attempt to fix with this post while still holding the proper context in mind and while things haven’t changed too much.

It’s worth noting that I’ve stepped aside as the project technical lead in January, which makes more likely for some of these ideas to take a turn, but they are still of historical value, and true for the time being.


This post is long enough to deserve an index, but these sections do build up concepts incrementally, so for a full understanding sequential reading is best:

Classical deployments

In a simplistic sense, deploying an application means configuring and running a set of processes in one or more machines to compose an integrated system. This procedure includes not only configuring the processes for particular needs, but also appropriately interconnecting the processes that compose the system.

The following figure depicts a simple example of such a scenario, with two frontend machines that had the Wordpress software configured on them to serve the same content out of a single backend machine running the MySQL database.

Deploying even that simple environment already requires the administrator to deal with a variety of tasks, such as setting up physical or virtual machines, provisioning the operating system, installing the applications and the necessary dependencies, configuring web servers, configuring the database, configuring the communication across the processes including addresses and credentials, firewall rules, and so on. Then, once the system is up, the deployed system must be managed throughout its whole lifecycle, with upgrades, configuration changes, new services integrated, and more.

The lack of a good mechanism to turn all of these tasks into high-level operations that are convenient, repeatable, and extensible, is what motivated the development of juju. The next sections provide an overview of how these problems are solved.

Preparing a blank slate

Before diving into the way in which juju environments are organized, a few words must be said about what a juju environment is in the first place.

All resources managed by juju are said to be within a juju environment, and such an environment may be prepared by juju itself as long as the administrator has access to one of the supported infrastructure providers (AWS, OpenStack, MAAS, etc).

In practice, creating an environment is done by running juju’s bootstrap command:

$ juju bootstrap

This will start a machine in the configured infrastructure provider and prepare the machine for running the juju state server to control the whole environment. Once the machine and the state server are up, they’ll wait for future instructions that are provided via follow up commands or alternative user interfaces.

Service topologies

The high-level perspective that juju takes about an environment and its lifecycle is similar to the perspective that a person has about them. For instance, although the classical deployment example provided above is simple, the mental model that describes it is even simpler, and consists of just a couple of communicating services:

That’s pretty much the model that an administrator using juju has to input into the system for that deployment to be realized. This may be achieved with the following commands:

$ juju deploy cs:precise/wordpress
$ juju deploy cs:precise/mysql
$ juju add-relation wordpress mysql

These commands will communicate with the previously bootstrapped environment, and will input into the system the desired model. The commands themselves don’t actually change the current state of the deployed software, but rather inform the juju infrastructure of the state that the environment should be in. After the commands take place, the juju state server will act to transform the current state of the deployment into the desired one.

In the example described, for instance, juju starts by deploying two new machines that are able to run the service units responsible for Wordpress and MySQL, and configures the machines to run agents that manipulate the system as needed to realize the requested model. An intermediate stage of that process might conceptually be represented as:


The service units are then provided with the information necessary to configure and start the real software that is responsible for the requested workload (Wordpress and MySQL themselves, in this example), and are also provided with a mechanism that enables service units that were related together to easily exchange data such as addresses, credentials, and so on.

At this point, the service units are able to realize the requested model:


This is close to the original scenario described, except that there’s a single frontend machine running Wordpress. The next section details how to add that second frontend machine.

Scaling services horizontally

The next step to match the original scenario described is to add a second service unit that can run Wordpress, and that can be achieved by the single command:

$ juju add-unit wordpress

No further commands or information are necessary, because the juju state server understands what the model of the deployment is. That model includes both the configuration of the involved services and the fact that units of the wordpress service should talk to units of the mysql service.

This final step makes the deployed system look equivalent to the original scenario depicted:


Although that is equivalent to the classic deployment first described, as hinted by these examples an environment managed by juju isn’t static. Services may be added, removed, reconfigured, upgraded, expanded, contracted, and related together, and these actions may take place at any time during the lifetime of an environment.

The way that the service reacts to such changes isn’t enforced by the juju infrastructure. Instead, juju delegates service-specific decisions to the charm that implements the service behavior, as described in the following section.


A juju-managed environment wouldn't be nearly as interesting if all it could do was constrained by preconceived ideas that the juju developers had about what services should be supported and how they should interact among themselves and with the world.

Instead, the activities within a service deployed by juju are all orchestrated by a juju charm, which is generally named after the main software it exposes. A charm is defined by its metadata, one or more executable hooks that are called after certain events take place, and optionally some custom content.

The charm metadata contains basic declarative information, such as the name and description of the charm, relationships the charm may participate in, and configuration options that the charm is able to handle.

The charm hooks are executable files with well-defined names that may be written in any language. These hooks are run non-concurrently to inform the charm that something happened, and they give a chance for the charm to react to such events in arbitrary ways. There are hooks to inform that the service is supposed to be first installed, or started, or configured, or for when a relation was joined, departed, and so on.

This means that in the previous example the service units depicted are in fact reporting relevant events to the hooks that live within the wordpress charm, and those hooks are the ones responsible for bringing the Wordpress software and any other dependencies up.


The interface offered by juju to the charm implementation is the same, independently from which infrastructure provider is being used. As long as the charm author takes some care, one can create entire service stacks that can be moved around among different infrastructure providers.


In the examples above, the concept of service relationships was introduced naturally, because it’s indeed a common and critical aspect of any system that depends on more than a single process. Interestingly, despite it being such a foundational idea, most management systems in fact pay little attention to how the interconnections are modeled.

With juju, it’s fair to say that service relations were part of the system since inception, and have driven the whole mindset around it.

Relations in juju have three main properties: an interface, a kind, and a name.

The relation interface is simply a unique name that represents the protocol that is conventionally followed by the service units to exchange information via their respective hooks. As long as the name is the same, the charms are assumed to have been written in a compatible way, and thus the relation is allowed to be established via the user interface. Relations with different interfaces cannot be established.

The relation kind informs whether a service unit that deploys the given charm will act as a provider, a requirer, or a peer in the relation. Providers and requirers are complementary, in the sense that a service that provides an interface can only have that specific relation established with a service that requires the same interface, and vice-versa. Peer relations are automatically established internally across the units of the service that declares the relation, and enable easily clustering together these units to setup masters and slaves, rings, or any other structural organization that the underlying software supports.

The relation name uniquely identifies the given relation within the charm, and allows a single charm (and service and service units that use it) to have multiple relations with the same interface but different purposes. That identifier is then used in hook names relative to the given relation, user interfaces, and so on.

For example, the two communicating services described in examples might hold relations defined as:


When that service model is realized, juju will eventually inform all service units of the wordpress service that a relation was established with the respective service units of the mysql service. That event is communicated via hooks being called on both units, in a way resembling the following representation:


As depicted above, such an exchange might take the following form:

  1. The administrator establishes a relation between the wordpress service and the mysql service, which causes the service units of these services (wordpress/1 and mysql/0 in the example) to relate.
  2. Both service units concurrently call the relation-joined hook for the respective relation. Note that the hook is named after the local relation name for each unit. Given the conventions established for the mysql interface, the requirer side of the relation does nothing, and the provider informs the credentials and database name that should be used.
  3. The requirer side of the relation is informed that relation settings have changed via the relation-changed hook. This hook implementation may pick up the provided settings and configure the software to talk to the remote side.
  4. The Wordpress software itself is run, and establishes the required TCP connection to the configured database.

In that workflow, neither side knows for sure what service is being related to. It would be feasible (and probably welcome) to have the mysql service replaced by a mariadb service that provided a compatible mysql interface, and the wordpress charm wouldn’t have to be changed to communicate with it.

Also, although this example and many real world scenarios will have relations reflecting TCP connections, this may not always be the case. It’s reasonable to have relations conveying any kind of metadata across the related services.


Service configuration follows the same model of metadata plus executable hooks that was described above for relations. A charm can declare what configuration settings it expects in its metadata, and how to react to setting changes in an executable hook named config-changed. Then, once a valid setting is changed for a service, all of the respective service units will have that hook called to reflect the new configuration.

Changing a service setting via the command line may be as simple as:

$ juju set wordpress title="My Blog"

This will communicate with the juju state server, record the new configuration, and consequently incite the service units to realize the new configuration as described. For clarity, this process may be represented as:


Taking from here

This conceptual overview hopefully provides some insight into the original thinking that went into designing the juju project. For more in-depth information on any of the topics covered here, the following resources are good starting points:

22 Feb 05:17

Quickly run Ubuntu cloud images locally using uvtool

by s3hh

We have long been able to test Ubuntu isos very easily by
using ‘testdrive’. It syncs releases/architectures you are interested
in and starts them in kvm. Very nice. But nowadays, in addition to
the isos, we also distribute cloud images. They are the basis for
cloud instances and ubuntu-cloud containers, but starting a local vm
based on them took some manual steps. Now you can use ‘uvtool’ to
easily sync and launch vms with cloud images.

uvtool is in the archive in trusty and saucy, but if you’re on precise
you’ll need the ppa:

sudo add-apt-repository ppa:uvtool-dev/trunk
sudo apt-get update
sudo apt-get install uvtool

Now you can sync the release you like, using a command like:

uvt-simplestreams-libvirt sync release=saucy
uvt-simplestreams-libvirt sync --source release=trusty arch=amd64

See what you’ve got syncd:

uvt-simplestreams-libvirt query

then launch a vm

uvt-kvm create xxx release=saucy arch=amd64
uvt-kvm list

and connect to it

uvt-kvm ssh --insecure -l ubuntu xxx

While it exists you can manage it using libvirt,

virsh list
virsh destroy xxx
virsh start xxx
virsh dumpxml xxx

Doing so, you can find out that the backing store is a qcow snapshot
of the ‘canonical’ image. If you decided you wanted to publish a
resulting vm, you could of course convert the backing store to a
raw file or whatever:

sudo qemu-img convert -f qcow2 -O raw /var/lib/uvtool/libvirt/images/xxx.qcow xxx.img

When you’re done, destroy it using

uvt-kvm destroy xxx

Very nice!

22 Feb 05:16

Emulating tagged views in unity

by s3hh

I like tagged tiling window managers. I like tiling because it lets me avoid tedious window move+resize. I like tagged wm because I can add multiple tags to windows so that different tag views can show different subsets of my windows – irc and mail, irc and task1, task1 and browsers, task2 and email…

Unity doesn’t tile, but has the grid plugin which is quite nice. But what about a tagged view? There used to be a compiz plugin called group. In the past when I’ve tried it it didn’t seem to quite fit my needs, but beyond that I couldn’t find it in recent releases.

I briefly considered building it straight into unity, but I really just wanted something to work with < 1 hr work. So I implemented it as a script, winmark. Winmark takes a single-character mark (think of marking in vi, ma, 'a) and stores or restores the layout of the currently un-minimized windows under that mark (in ~/.local/share/winmark/a). Another tiny c program grabs the keyboard to read a mark character, then calls winmark with that character.

So now I can hit shift-F9 a to store the current layout, set up a different layout, hit shift-f9 b to store that, then restore them with F9 a and F9 b.

I’m not packaging this right now as I *suspect* this is the sort of thing noone but me would want. However I’m mentioning it here in case I’m wrong. The source is at lp:~serge-hallyn/+junk/markwindows.

There’s definite room for improvement, but I’ve hit my hour time limit, and it is useful as is :) Potential improvements would include showing overlay previews as with application switching, and restoring the stacking order.

14 Feb 23:59

A No Excuses Guide to Blogging (PDF, EPUB, MOBI – free!); also, notes on publishing

by Sacha Chua

Advice for ... "I have all these ideas, but I never finish posts..."

"Turn your ideas into small questions and then answer those."

This entry is part 1 of 19 in the series A No-Excuses Guide to Blogging

First, a quick announcement: A No-Excuses Guide to Blogging is now available as a free/pay-what-you-want e-book so that you can work your way through your excuses without having to click through lots of blog posts. =)

Mock-up by Ramon Williamson

Mock-up by Ramon Williamson

The cover I made for Amazon

The cover I made for Amazon

The PDF looks prettiest if you’re reading it on your computer or tablet, and the EPUB/MOBI version is handy for other e-readers. You get all three (and any future updates) if you grab it from, or you can get the MOBI version from the Kindle store (currently $0.99, but eventually they’ll price-match it to $0). The book is under Creative Commons Attribution, so feel free to share it with other people. =)

UPDATE 2014-02-13: Here’s a one-page summary!

2014-02-13 A No-Excuses Guide to Blogging - Summary of 10 blogging excuses and how to work around them

2014-02-13 A No-Excuses Guide to Blogging – Summary of 10 blogging excuses and how to work around them


- Behind the scenes stuff! - 

So this was about 8 hours of packaging after I’d identified the topics and asked an assistant to compile all the blog posts in a Word document. I edited the text to make it fit better in a collection, fiddled with the graphics, added more sketches, tweaked the layout some more, fought with section headers, and eventually created a PDF that I was reasonably happy with. I contacted a bunch of people on Fiverr about converting the DOCX into EPUB and MOBI. While waiting for a response, I decided to try doing it myself. It took me some time to clean up the messy HTML, but I’m reasonably happy with how the EPUB worked out. I had to tweak the EPUB table of contents in order to get it to pass the validator used by Lulu, but eventually I got it worked out with Calibre and Sigil. The MOBI was a straight conversion from the EPUB, although I wonder if I can get the background colour to be white…

2014-02-05 Notes on publishing

2014-02-05 Notes on publishing

So that was interesting and useful, and it would be good to do more. Here are some ideas for other “No Excuses”-type guides I might put together. Maybe self-directed learning and delegation, actually, since other people are covering the sketchnoting bit quite well.

2014-02-06 What other excuses can I collect and work around

2014-02-06 What other excuses can I collect and work around

There’s also this list of other book ideas, Thinking with Emacs, Tracking Your Time, Accelerating Your Learning with Sketchnotes, and this big outline. Lots to do. Looking forward to figuring out how I can get more of these out the door. =)

In the meantime… tada!

Read the original or check out the comments on: A No Excuses Guide to Blogging (PDF, EPUB, MOBI – free!); also, notes on publishing (Sacha Chua's blog)

The post A No Excuses Guide to Blogging (PDF, EPUB, MOBI – free!); also, notes on publishing appeared first on sacha chua :: living an awesome life.

14 Feb 23:57

Using github for homeworks

by C. Titus Brown

Compare this to my own experiences using github in my class.

This term, I'm once again teaching my upper-division CSE undergrad course in Web Dev here at MSU. For the second time, I'm requiring students to use github for their homework; unlike last year, I now understand pull requests and have integrated them into the process.

How does it work, basically?

The instructions for handing in homework are as follows:

Merge hw3 into your master. Please don't delete the 'hw3' branch :)

Hand in this homework on branch 'hw4' on your github account, and set up a pull request between hw4 and your master branch. Don't merge it, just set up the PR.

This sets up a standing pull request for HW 3 that lets me (and other students -- more on that in a bit) comment on the commit.

When I go to grade homework, I go into a checkout that I keep of each student's directory; importantly, it's named for their github repo. Here I can easily do:

git fetch
git checkout hw4

and get their HW 4.

Next, I run stuff, fix minor bugs, comment on code, commit changes, and otherwise interact with their homework. I can then push my changes and comments to a branch on my github repo:

#! /bin/bash
dir=$(basename $PWD)
git push hw4:hw4-$dir-comments

and create a pull request automatically by constructing the right URL:

#! /bin/bash
dir=$(basename $PWD)
open -a "/Applications/Google"$dir:hw4...ctb:hw4-$dir-comments?expand=1

I typically send a private e-mail with their grade in it at this point, along with any requests or general comments I have (fix your tests, think about refactoring, etc.)

Note that leaving the hw3 branch lying around lets me backtrack and check old code more easily. Sure, I could do this with 'git log' but it's nice to have the branch there.

Code sharing policy

The github repos are public, which presents a challenge: how do you prevent cheating?

Well, my general policy in this class has always been that "code should work; I don't care where you got it." My experience has been that some number of students simply fail to even copy code effectively, and that level of evaluation is sufficient for me.

That having been said, this year I now require that students have their own names on the commits -- they can't just completely swap in someone else's commits. This is antithetical to the whole version control and credit-where-credit-is-due thing but I found that git made it too easy to copy functioning code. I'll let you know how this goes.

Grading only a subset of HWs

Grading this stuff is quite a chore; I can do 5-10 HWs in a single sitting, but ~40-45 is too much. I don't hand this off to TAs -- I prefer to look at the code myself, and I also generally find that my TAs don't know the material that well (they're usually my grad students but that doesn't mean they're web devs!)

So, this year, inspired by a post on Tomorrow's Professor, I am only grading 5-10 homeworks a week. I am choosing them more or less randomly, and I expect to grade each person's work a few times during the term. As the term goes on I will target people who need a bit more help. I do take requests, too, so if people want my feedback I'm sure to look at their code.

So how do I assign grades to the ones I don't grade? The rule is that if I don't grade your HW, you automatically get 100%. However, a key component of this is that you must have handed in something approximating the homework, and it must have been handed in on time; if I find a bunch of empty HW branches (and I _will_ notice), the student will be in trouble.

Thus far I like the way this is working out. I've been able to do a much better (less hurried, better comments) job of grading than in previous years, and of course the students don't mind. Because the homework is cumulative, I expect this approach to still effectively sort the students.

Grading efficiently

I use tests (nose) and code coverage (Ned Batchelder's excellent coverage module ) to target my grading. If there are uncovered bits of code, that's the first place to look for bugs. I also do some exploratory testing (aka "clicking on things and waiting for stuff to break) and skim the code, but targeting missing code coverage is an amazingly effective way to find bugs.


I am planning to set up continuous integration on each repo and pull request, largely as a tool to help the students. I'll let you know how that goes.

To find people who may not have handed in HW, I can run a little shell script,

#! /bin/bash
for i in $(cat github-list.txt)
     cd $i
     echo $i $(git branch -r | grep hw4) >> ../out
     cd ../

that lets me see who does not have a hw4 branch.

I'm thinking about writing a little RSS feed examiner to see who has turned in stuff, too. Probably redundant at this point.

Enabling code review, and a crazy idea

I'd really like to get the students into code review, but it's remarkably hard to take people who are not that experienced at software development and get them to critique each other's code. I don't scale particularly well, and so I can't really help the students learn to do this 1-1. So... how?

One thing I'm trying out is to have students do code review in class, on each others' pull requests. This works ... OK, but there's not really enough time in class to do it.

What else could we do?

Since I have a range of student expertise in the class, why not get the more expert students to help out? It will give the more expert students some code review experience, and it will help the less expert students work through problems.

Here's my idea:

  1. if a student is having trouble, they can request an extension.
  2. as part of the extension, I will assign them a mentor from a pool of people who are generally doing well.
  3. the mentor will go through their pull request and make comments.
  4. the mentee will then be able to work through the problems, potentially with back-and-forth with the mentor.
  5. the mentee gets an extension and probably a better grade; the mentor gets project credit towards their final grade.

This can be gamed, and undoubtedly will; in particular, I expect people who are perfectly capable of getting it to work but who just want an extension to ask for one. But do the benefits significantly outweigh the loss from gaming? I'll let you know how I feel about it :)

Suggestions on modifications welcome, of course. I'm thinking that there will be a non-zero chance that I will refuse the extension request, which should keep people on their toes. Other thoughts?

What isn't working?

The students are still fairly new to git, and I haven't been forcing them to do tricky stuff with it -- as we know from Software Carpentry, git is very difficult to learn, so I'm taking it slow. (This is the students' first introduction to distributed version control, and I don't think many of them are really experienced at version control, period; we don't introduce it at MSU until 3rd year!)

This leads to one big problem -- the pull requests often contain significant extraneous code, like entire virtualenvs; and sometimes the PRs aren't useful for other reasons, like someone swapping in a whole bunch of new code. But this is still a fairly rare problem.

Concluding thoughts

This is all an experiment, but so far I've been impressed with how well git and github are working. I'd love to hear about other experiences; drop me a line or comment below.

In previous years, I've gotten feedback that students really find the testing and continuous integration stuff useful for their next jobs. This year, I hope to get the same comments about git and github.


14 Feb 23:56

From networking with people to networking around ideas: How I stopped worrying about keeping in touch

by Sacha Chua

More thoughtful ideas and sketches-- as always.

I used to think a lot about how to keep track of the people I know, writing down notes about their interests in Emacs’ aptly named Big Brother Database (BBDB). I wanted to be the kind of person who could remember people’s goals and dreams, who paid attention to trivia from long-ago conversations. A little configuration allowed me to see my notes when reading or replying to e-mail from people, and I even wrote some code to track when I last contacted someone and filter the list of people so that I could focus on people I hadn’t talked to in a while. It was amusing to see a record and realize I’d actually known someone for years (these things sneak up on you before you know it!), and it was occasionally gratifying to be able to look up someone based on an interest I vaguely remembered them having and to make the connection with a book I’d just read or someone else they should meet.

I kept tweaking this system. Maybe there was a contact relationship management tool out there that could help me remember what was important to people’s lives or discover small-talk topics from people’s social media updates. Maybe there was an application that could help me act thoughtfully, beyond the usual flood of Facebook wall messages on someone’s birthday. Maybe there was a way for me to learn how to keep in touch. I asked people who networked for a living (mostly salespeople and politicians) how they managed to remember all these little details and reach out. They told me about spreadsheets and index cards, phone calls and e-mails: “Hi, how are you doing? Was just thinking of you. What’s up?” I was inspired by examples of super-connectors who somehow managed to remember everyone’s faces and names, asking after kids and pets and hobbies, effortlessly breaking through the limits of Dunbar’s number. (Or at least it seemed effortless; many worked very hard at this.) I tried out systems like Nimble and Contactually as a way to see people’s activity across platforms and remember to keep in touch. I tried ConnectedHQ (no longer available?) and Sunrise for people-related notes in my calendar. I followed Facebook and LinkedIn’s suggestions to greet people on their birthdays or job changes.

Still, those practices never quite felt natural. Maybe I’m just not a people person, at least not that kind. I never became comfortable with calling someone out of the blue, and I preferred posting stories on my blog instead of telling them one-on-one through e-mail or conversation.

I decided to try a different approach: I stopped worrying about keeping in touch. If people came into my life, great. If people drifted out, that was fine too. Instead of working on keeping those relationships going, I focused instead on being open to the conversations as they came up. Instead of reaching out so that I can hear about what people are working on and stay on their radar, I decided to focus on what I was curious about and be open to the people I’d bump into along the way.

Sure, I might have missed a few serendipitous connections with people who were quietly working on interesting things–but the world is full of other people who make it easy to keep up with them, like the way my blog makes it easy to keep up with what I’m interested in. I subscribe to a number of people’s blogs and I follow people on social networks. But for the most part, I care about ideas and conversations first, and then getting to know and keep in touch with people happens along the way.

It’s a very different form of networking, I think, than what I’ve read about in countless books. I still recommend books like How to Talk to Anyone (Lowndes) and Never Eat Alone (Ferrazzi) to people who are curious about building relationships with specific people or making the most of networking events. Me, I’m learning about how to talk to everyone. Instead of building a relationship with a specific person, I look for opportunities to help people around an idea or contribute to a group. Instead of asking a specific person for help, I ask in general, and people step forward if they’re interested. I don’t have to fear rejection, and no one has to look for tactful ways to say no; and if a request doesn’t get any takers, well, I’m where I would’ve been anyway.

In order to make that work, I need to make sure I have good ideas that people want to be part of, the confidence to share them, a platform for sharing them with, specific requests and ways people can get involved, and a way to celebrate how people are helping. I like that. It turns networking into something more than “I think you’re cool and I want to stay in touch” or even “You’re working on cool stuff; How can I help?” – it becomes “We’re all part of something bigger. Let’s make things happen.”

2014-01-05 How can I learn to bring people together
2014-01-05 How can I learn to bring people together

I’d been trying this approach over the past few years. Watching my dad do it was a real eye-opener. My dad works Facebook like a pro, posting ideas and stories and the occasional request. He has probably never taken notes on someone’s interests. He probably doesn’t spend time cataloging people’s goals and passions. As it turns out, he does take a few notes, and he asks my mom if he needs help remembering. (At least my sister says so!) Still, I don’t even know if it’s possible to remember all these little details about the thousands of Facebook friends that he has. If he got the hang of working with fan pages, he’d probably have even more fans. He might ask one or two people who come to mind, but in general, he shares and asks generally. And stuff happens!

2014-01-02 Making things happen

2014-01-02 Making things happen

I’m working on sharing, learning from people, asking for help, and making it easy for people to build on what we have in common. I’ll get there someday. It’s not so much that I want to keep in touch with a specific person; it’s that we’re both actively thinking about a topic, so maybe we’ll set up a conversation so we can exchange notes and learn together. It’s not so much that I want to have a wide range of skills in my roster of contacts; it’s more that I want to provide value to a wide range of people, and maybe some of them will want to help me when the opportunity comes up.

I was thinking about the way people are related to topics so that I can nudge myself to reach out more for questions and conversations. I like the idea of confederates: people who are actively learning about similar things, and whom I’m comfortable asking if I want a different perspective or I’m curious about what they’re learning. I don’t have any set schedule for reaching out to these people. If I learn something they might be interested in, I’ll mention it to them, but I’ll also open up the conversation to everyone who might be interested. When I started drawing out the map, I realized how lucky I was to have so many people I can learn from and who might find stuff I’m learning useful too. This is how I keep in touch now – through the exchange of questions and ideas.

2014-01-30 Confederates

2014-01-30 Confederates

So I might not build as close a relationship as you might with someone who takes the time to send you birthday cards or calls you up every quarter, but then do you really feel all that close to insurance or real estate agents who send you cards like clockwork? I don’t send “Hi, what’s up?” e-mails and I tend to not be very good at responding to them, but I’ll dive deeply into topics of mutual curiosity. My network is shifting to people who share what they know and what they’re learning. While that means I’m probably missing out on a lot of stuff that never gets shared beyond the small circles most people are comfortable in, there are so many good people out there focused on bringing everyone up–not just the people they’ve bumped into. It might also mean that I’m not as close to the people for whom these little gestures mean a lot, but it means that I draw closer to people who don’t have that need. It’s like the difference between someone who puts you on the spot when you forget your name and someone who helpfully supplies it at a hint of hesitation; someone who makes you feel bad for forgetting their birthday or anniversary or whatever, and someone who’s more focused on what’s good instead of what’s missing.

You don’t have to wish me a happy birthday, and you never have to apologize for replying slowly or not being in touch. Better to share your questions and ideas, and better to share that not just with me but with the world.

Read the original or check out the comments on: From networking with people to networking around ideas: How I stopped worrying about keeping in touch (Sacha Chua's blog)

The post From networking with people to networking around ideas: How I stopped worrying about keeping in touch appeared first on sacha chua :: living an awesome life.

14 Feb 23:53

Living in the matrix?

by mf344

There's no doubt that maths is very good at describing the world around us. Could this be because the Universe we live in is itself a mathematical structure? We talk to Max Tegmark.

You can listen to our interview with Tegmark in our podcast.

It's something that most of us dread: to find out one day that we aren't real, that we are living in a computer simulation. Yet some people who work at the forefront of science believe that the ultimate nature of reality might not be so far off this idea.

read more

08 Jul 04:35

Use a Hot Spoon to Instantly Relieve Itchy Bug Bites

by Shep McAllister

Use a Hot Spoon to Instantly Relieve Itchy Bug Bites

Tis the season for annoying bug bites, but a surprisingly simple remedy exists that can eliminate all of the itch within minutes.



08 Jul 04:30

Reprint: Stop Disabling SELinux!

by Major Hayden

This article appeared in SC Magazine and I’ve posted it here as well. For those of you who were left wanting more from my previous SELinux post, this should help. If it doesn’t help, leave a comment. ;)

The push to cloud transforms the way we apply information security principles to systems and applications. Perimeters of the past, secured heavily with traditional network devices in the outermost ring, lose effectiveness day by day. Shifting the focus to “defense in depth” brings the perimeter down to the individual cloud instances running your application. Security-Enhanced Linux, or SELinux, forms an effective part of that perimeter.

SELinux operates in the realm of mandatory access control, or MAC. The design of MAC involves placing constraints on what a user (a subject) can do to a particular object (a target) on the system. In contrast, discretionary access control, or DAC, allows a user with certain access to use discretion to limit or allow access to certain files, directories, or devices. You can set any file system permissions that you want but SELinux can override them with ease at the operating system level.

Consider a typical server running a web application. An attacker compromises the web application and executes malicious code via the web server daemon itself. SELinux has default policies that prevent the daemon from initiating communication on the network. That limits the attacker’s options to attack other services or servers.

In addition, SELinux sets policies on which files and directories the web server can access, regardless of any file system permissions. This protection limits the attacker’s access to other sensitive parts of the file system even if the administrator set the files to be readable to the world.

This is where SELinux shines. Oddly enough, this is the point where many system administrators actually disable SELinux on their systems.

Troubleshooting these events, called AVC denials, without some helpful tools is challenging and frustrating. Each denial flows into to your audit log as a cryptic message. Most administrators will check the usual suspects, like firewall rules and file system permissions. As frustration builds, they disable SELinux and notice that their application begins working as expected. SELinux remains disabled and hundreds of helpful policies lie dormant solely because one policy caused a problem.

Disabling SELinux without investigation frustrated me to the point where I started a site at The site is a snarky response to Linux administrators who reach for the disable switch as soon as SELinux gets in their way.

All jokes aside, here are some helpful tips to use SELinux effectively:

Use the setroubleshoot helpers to understand denials
Working through denials is easy with the setroubleshoot-server package. When a denial occurs, you still receive a cryptic log message in your audit logs. However, you also receive a message via syslog that is very easy to read. Your server can email you these messages as well. The message contains guidance about adjusting SELinux booleans, setting contexts, or generating new SELinux policies to work around a really unusual problem. When I say guidance, I mean that the tools give you commands to copy and paste to adjust your policies, booleans and contexts.

Review SELinux booleans for quick adjustments
Although the myriad of SELinux user-space tools isn’t within the scope of this article, getsebool and togglesebool deserve a mention. Frequently adjusted policies are controlled by booleans that are toggled on and off with togglesebool. Start with getsebool –a for a full list of booleans and then use togglesebool to enable or disable the policy.

Quickly restore file or directory contexts
Shuffling files or directories around a server can cause SELinux denials due to contexts not matching their original values. This happens to me frequently if I move a configuration file from one system to another. Correcting the context problem involves one of two simple commands. The restorecon command applies the default contexts specific to the file or directory. If you have a file in the directory with the correct context, use chcon to fix the context on the wrong file by giving it the path to the file with the correct context.

Here are some additional links with helpful SELinux documentation:

Reprint: Stop Disabling SELinux! is a post from: Major Hayden's blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

07 Jul 08:47

Scaling variant detection pipelines for whole genome sequencing analysis

by Brad Chapman

Scaling for whole genome sequencing

Moving from exome to whole genome sequencing introduces a myriad of scaling and informatics challenges. In addition to the biological component of correctly identifying biological variation, it’s equally important to be able to handle the informatics complexities that come with scaling up to whole genomes.

At Harvard School of Public Health, we are processing an increasing number of whole genome samples and the goal of this post is to share experiences scaling the bcbio-nextgen pipeline to handle the associated increase in file sizes and computational requirements. We’ll provide an overview of the pipeline architecture in bcbio-nextgen and detail the four areas we found most useful to overcome processing bottlenecks:

  • Support heterogeneous cluster creation to maximize resource usage.
  • Increase parallelism by developing flexible methods to split and process by genomic regions.
  • Avoid file IO and prefer streaming piped processing pipelines.
  • Explore distributed file systems to better handle file IO.

This overview isn’t meant as a prescription, but rather as a description of experiences so far. The work is a collaboration between the HSPH Bioinformatics Core, the research computing team at Harvard FAS and Dell Research. We welcome suggestions and thoughts from others working on these problems.

Pipeline architecture

The bcbio-nextgen pipeline runs in parallel on single multicore machines or distributed on job scheduler managed clusters like LSF, SGE, and TORQUE. The IPython parallel framework manages the set up of parallel engines and handling communication between them. These abstractions allow the same pipeline to scale from a single processor to hundreds of node on a cluster.

The high level diagram of the analysis pipeline shows the major steps in the process. For whole genome samples we start with large 100Gb+ files of reads in FASTQ or BAM format and perform alignment, post-alignment processing, variant calling and variant post processing. These steps involve numerous externally developed software tools with different processing and memory requirements.

Variant calling overview

Heterogeneous clusters

A major change in the pipeline was supporting creation of heterogeneous processing environments targeted for specific programs. This moves away from our previous architecture, which attempted to flatten processing and utilize single cores throughout. Due to algorithm restrictions, some software requires the entire set of reads for analysis. For instance, GATK’s base quality recalibrator uses the entire set of aligned reads to accurately calculate inputs for read recalibration. Other software operates more efficiently on entire files: the alignment step scales better by running using multiple cores on a single machine, since the IO penalty for splitting the input file is so severe.

To support this, bcbio-nextgen creates an appropriate type of cluster environment for each step:

  • Multicore: Allocates groups of same machine processors, allowing analysis of individual samples with multiple cores. For example, this enables running bwa alignment with 16 cores on multiprocessor machines.
  • Full usage of single cores: Maximize usage of single cores for processes that scale beyond the number of samples. For example, we run variant calling in parallel across subsets of the genome.
  • Per sample single core usage: Some steps do not currently parallelize beyond the number of samples, so require a single core per sample.

IPython parallel provides the distributed framework for creating these processing setups, working on top of existing schedulers like LSF, SGE and TORQUE. It creates processing engines on distributed cores within the cluster, using ZeroMQ to communicate job information between machines.

Cluster schedulers allow this type of control over core usage, but an additional future step is to include memory and disk IO requirements as part of heterogeneous environment creation. Amazon Web Services allows selection of exact memory, disk and compute resources to match the computational step. Eucalyptus and OpenStack bring this control to local hardware and virtual machines.

Variant calling overview

Parallelism by genomic regions

While the initial alignment and preparation steps require analysis of a full set of reads due to IO and algorithm restrictions, subsequent steps can run with increased parallelism by splitting across genomic regions. Variant detection algorithms do require processing continuous blocks of reads together, allowing local realignment algorithms to correctly characterize closely spaced SNPs and indels. Previously, we’d split analyses by chromosome but this has the downside of tying analysis times to chromosome 1, the largest chromosome.

The pipeline now identifies chromosome blocks without callable reads. These blocks group by either genomic features like repetitive hard to align sequence or analysis requirements like defined target regions. Using the globally shared callable regions across samples, we fraction the genome into more uniform sections for processing. As a result we can work on smaller chunks of reads during time critical parts of the process: applying base recalibration, de-duplication, realignment and variant calling.

Parallel block selection from genome

Streaming pipelines

A key bottleneck throughout the pipeline is disk usage. Steps requiring reading and writing large BAM or FASTQ files slow down dramatically once they overburden disk IO, distributed filesystem capabilities or ethernet connectivity between storage nodes. A practical solution to this problem is to avoid intermediate files and use unix pipes to stream results between processes.

We reworked our alignment step specifically to eliminate these issues. The previous attempt took a disk centric approach that allowed scaling out to multiple single cores in a cluster. We split an input FASTQ or BAM file into individual chunks of reads, and then aligned each of these chunks independently. Finally, we merged all the individual BAMs together to produce a final BAM file to pass on to the next step in the process. While nicely generalized, it did not scale when running multiple concurrent whole genomes.

The updated pipeline uses multicore support in samtools and aligners like bwa-mem and novoalign to pipe all steps as a stream: preparation of input reads, alignment, conversion to BAM and coordinate sorting of aligned reads. This results in improved scaling at the cost of only being able to increase single sample throughput to the maximum processors on a machine.

More generally, the entire process creates numerous temporary file intermediates that are a cause of scaling issues. Commonly used best-practice toolkits like Picard and GATK primarily require intermediate files. In contrast, tools in the Marth lab’s gkno pipeline handle streaming input and output making it possible to create alignment post-processing pipelines which minimize temporary file creation. As a general rule, supporting streaming algorithms amenable to piping can ameliorate file load issues associated with scaling up variant calling pipelines. This echos the focus on streaming algorithms Titus Brown advocates for dealing with large metagenomic datasets.

Distributed file systems

While all three of CPU, memory and disk speed limit individual steps during processing, the hardest variable to tweak is disk throughput. CPU and memory limitations have understandable solutions: buy faster CPUs and more memory. Improving disk access is not as easily solved, even with monetary resources, as it’s not clear what combination of disk and distributed filesystem will produce the best results for this type of pipeline.

We’ve experimented with NFS, GlusterFS and Lustre for handling disk access associated with high throughput variant calling. Each requires extensive tweaking and none has been unanimously better for all parts of the process. Much credit is due to John Morrissey and the research computing team at Harvard FAS for help performing incredible GlusterFS and network improvements as we worked through scaling issues, and Glen Otero, Will Cottay and Neil Klosterman at Dell for configuring an environment for NFS and Lustre testing. We can summarize what we’ve learned so far in two points:

  • A key variable is the network connectivity between storage nodes. We’ve worked with the pipeline on networks ranging from 1 GigE to InfiniBand connectivity, and increased throughput delays scaling slowdowns.
  • Different part of the processes stress different distributed file systems in complex ways. NFS provides the best speed compared to single machine processing until you hit scaling issues, then it slows down dramatically. Lustre and GlusterFS result in a reasonable performance hit for less disk intensive processing, but delay the dramatic slowdowns seen with NFS. However, when these systems reach their limits they hit a slowdown wall as bad or worse than NFS. One especially slow process identified on Gluster is SQLite indexing, although we need to do more investigation to identify specific underlying causes of the slowdown.

Other approaches we’re considering include utilizing high speed local temporary disk, reducing writes to long term distributed storage file systems. This introduces another set of challenges: avoiding stressing or filling up local disk when running multiple processes. We’ve also had good reports about using MooseFS but haven’t yet explored setting up and configuring another distributed file system. I’d love to hear experiences and suggestions from anyone with good or bad experiences using distributed file systems for this type of disk intensive high throughput sequencing analysis.

A final challenge associated with improving disk throughput is designing a pipeline that is not overly engineered to a specific system. We’d like to be able to take advantage of systems with large SSD attached temporary disk or wonderfully configured distributed file systems, while maintaining the ability scale on other systems. This is critical for building a community framework that multiple groups can use and contribute to.

Timing results

Providing detailed timing estimates for large, heterogeneous pipelines is difficult since they will be highly depending on the architecture and input files. Here we’ll present some concrete numbers that provide more insight into the conclusions presented above. These are more useful as a side by side comparison between approaches, rather than hard numbers to predict scaling on your own systems.

In partnership with Dell Solutions Center, we’ve been performing benchmarking of the pipeline on dedicated cluster hardware. The Dell system has 32 16-core machines connected with high speed InfiniBand to distributed NFS and Lustre file systems. We’re incredibly appreciative of Dell’s generosity in configuring, benchmarking and scaling out this system.

As a benchmark, we use 10x coverage whole genome human sequencing data from the Illumina platinum genomes project. Detailed instructions on setting up and running the analysis are available as part of the bcbio-nextgen example pipeline documentation.

Below are wall-clock timing results, in total hours, for scaling from 1 to 30 samples on both Lustre and NFS fileystems:

primary 1 sample 1 sample 1 sample 30 samples 30 samples
bottle 16 cores 96 cores 96 cores 480 cores 480 cores
neck Lustre Lustre NFS Lustre NFS
alignment cpu/mem 4.3h 4.3h 3.9h 4.5h 6.1h
align post-process io 3.7h 1.0h 0.9h 7.0h 20.7h
variant calling cpu/mem 2.9h 0.5h 0.5h 3.0h 1.8h
variant post-process io 1.0h 1.0h 0.6h 4.0h 1.5h
total 11.9h 6.8h 5.9h 18.5h 30.1h

Some interesting conclusions:

  • Scaling single samples to additional cores (16 to 96) provides a 40% reduction in processing time due to increased parallelism during post-processing and variant calling.
  • Lustre provides the best scale out from 1 to 30 samples, with 30 sample concurrent processing taking only 1.5x as along as a single sample.
  • NFS provides slightly better performance than Lustre for single sample scaling.
  • In contrast, NFS runs into scaling issues at 30 samples, proceeding 5.5 times slower during the IO intensive alignment post-processing step.

This is preliminary work as we continue to optimize code parallelism and work on cluster and distributed file system setup. We welcome feedback and thoughts to improve pipeline throughput and scaling recommendations.

07 Jul 02:13

Using AUFS and LXC to Manage Dependencies at Cloud Scale

by Ken Park

Environments in a Nutshell

At PiCloud we strive to be the easiest service out-of-the-box, yet versatile enough to be used for the most complex applications. This ideal was especially challenging to realize when expanding our compute platform beyond Python-only modules, to support the whole gamut of non-Python software. How could we extend the same ease-of-use that our cloud library offers to the complex world of software dependencies?

In this blog post, I’ll explore how we’ve harnessed three different technologies to achieve our goal of a highly-scalable software dependency manager. We call it Environments, and for many PaaS users, it’s what sets us apart.

Where We Were

PiCloud originated as a Python-only platform. In that simplified setup, our Dependency Analyzer could identify and ship all pure-Python dependencies at job submission time. Why only pure-Python? Anything that requires compilation cannot be expected to work when shipped to a computer with a different architecture. Unfortunately, for performance purposes, many Python libraries are actually compiled C-extensions, and this meant if you used a Python library with C-extensions, our compute nodes needed to have that library pre-installed. Not surprisingly, it didn’t take long before this limitation became an issue for some of our users.

Where We Wanted To Be

When the time came to revamp our dependency manager, we sought a generic solution that would work for Python and non-Python software alike. We boiled our requirements for the new dependency management down to the following:

  • Simplicity. It must be no more difficult than what a user would do on their own machine. Importantly, it should not require users to learn a new tool or configuration language.
  • Powerful. There should be minimal limitations on what packages or programs a user can install. Such flexibility necessitates that escalated privileges (for dependency installation purposes) are given in controlled situations.
  • Low Overhead. It must not introduce significant overhead to job processing. Among other things, this means satisfying a user’s dependency must not require booting a new machine or rebooting an existing one.
  • Parallelizable. The value of PiCloud is giving users the freedom to use as many cores as needed to accomplish their tasks. Deploying dependencies should not hinder parallelization across many machines.

How We Got There

Filesystem Virtualization

Fundamentally, satisfying the dependencies of a program means ensuring that the appropriate files can be found in the proper places in the filesystem. We can then rephrase the role of the dependency manager as making sure a job is run in whatever filesystem environment it needs for proper function.

We took inspiration from the Python tool virtualenv, which lets you create, on one machine, multiple, isolated Python environments, each with its own packages and settings. In other words, it virtualizes those portions of the filesystem relevant to Python. Generalizing this concept, what we wanted was a virtualenv for the entire filesystem.

On *nix machines, the simplest way to virtualize the filesystem is through the chroot program, which allows you to set any point in your filesystem hierarchy as the new filesystem “root”. Unfortunately, chroot lacks mechanisms for isolation and resource management. At the other end of the virtualization spectrum is a virtual machine, which gives you full control and isolation but comes with the huge overhead of starting and running the VM. However, in between the simple chroot utility and full VMs are container technologies that share the host’s kernel but have the capacity to virtualize most everything else. Combined with other recent Linux kernel developments, like cgroups and namespaces, container technologies provide resource management and isolation, respectively. And starting a container is fast, because there is no operating system to boot up – the host’s kernel is already up and running. We ended up choosing LXC (LinuX Container) as our container technology, because it has been mainlined into the Linux kernel.

Our new dependency management system was beginning to take shape. A user would start with an LXC-bootable base filesystem, and install whatever programs and files their jobs need. We would store this modified filesystem, which we’ll call an Environment from this point on. When the user runs a job, we would start an LXC container using their Environment as the filesystem. The next question is, how do we store and send these big Linux filesystems across the network to compute nodes?

Union Mount System

What we realized was that saving the entire Environment was wasteful. In practice, Environments would be mostly the same, with only a relatively small number of files added in or changed. Theoretically, given the base filesystem the Environment was built on, we only need the changes the user made on top of this base to recreate the full Environment. What would enable us to save just this difference, rather than the whole Environment?

The answer is union mounts. A union mount allows you to stack multiple filesystems and present a unioned view at a different mount point. It is the technology behind many Live CDs. A Live CD contains a bootable operating system but is not typically writable. Hence, upon bootup from a Live CD, the system will create a temporary scratch space in RAM, then union mount this RAMFS on top of the filesystem of the CD. After chrooting into this unioned mount, the user is presented with a machine seemingly running off a read-only CD, but giving you the ability to tinker with and change the operating system files as you try it out. And due to the magic of the union mount, all changes are being written to the RAM filesystem, even when modifying files from the CD.

Several union filesystem tools exist, but after some research, we settled on AUFS (Another Union FileSystem), a nifty module created and maintained by Junjiro Akajima. We chose AUFS, because it is relatively performant and reliable, and we have been impressed with Junjiro’s responsiveness to support requests.

So, how much does AUFS help? On PiCloud, the average size of Environment modifications is around 220 MB. Given a base filesystem size of around 2.5 GB (which includes many pre-installed libraries for convenience and performance), this leads to roughly a 12-fold savings in terms of storage and network transfer.

Environment In Action

So, putting all that together, here’s how PiCloud Environments work:

  1. PiCloud user requests creation of a new Environment through our web UI or client CLI.
  2. PiCloud launches a machine the user can log into, that is actually an AUFS stack of an empty filesystem (read-writable) on top of a base Environment (read-only), much like a Live CD.
  3. User logs into the “setup server” and installs dependencies as they would on their own machine.

Environment Creation

  1. When the user is done setting up the Environment and requests to save it, user changes are sanitized for security and saved into our distributed file system.

Environment Saving

  1. User can now submit jobs specifying it should be run in the custom Environment. In Python, Environment is specified with the “_env” keyword, and in bash, the “-e” flag.
  2. PiCloud recreates the AUFS stack on our compute nodes, and runs the user’s jobs in LXC containers.

Using Environment

DFS Features That Improve Performance

As mentioned above, AUFS reduces the size of an Environment by roughly 12-fold. In practice, we end up doing much better than that thanks to our distributed file system (DFS), which has two important properties:

  1. Files from an environment are transferred at the block level to the compute node on-demand (only when a program accesses them).
  2. Transferred files are cached on the compute node automatically.

To understand the benefits of these properties, consider the Matlab Compiler Runtime (MCR), which enables the stand-alone execution of compiled Matlab applications. When a user installs the MCR in an Environment, close to a gigabyte of files are added. But, for the typical MCR-compiled application, a small minority (

Check Out the New Features

If you’re currently a PiCloud user and have wondered how Environments worked, I hope this shed some light into the black box.

If you haven’t already, you should check out our recent updates to the Environment system. We’ve redone the web UI and added client-side CLI support for managing your Environments. Also, we are excited about the new sharing feature that lets users share Environments they’ve created with colleagues and the general public.