Aculich

Last night at Vim London I spoke about some of my favourite plugins and settings for Tmux and Vim that allow me to work with them as I do. It’s very rare these days that I won’t be editing code from Vim, within a Tmux session, but out of the box the immediate advantages of this pairing are not always obvious. It takes some time to get things working.

The talk was recorded and I will update the post with it once it’s public, but for now I just wanted to link to some of the plugins and settings I mentioned in the talk that I use in my dotfiles.

tnew alias

I create all my tmux sessions through an alias I have which I’ve called tnew, which simply calls a function called new-tmux-from-dir-name (this is taken from the thoughbot dotfiles. The function looks like so:

sh function new-tmux-from-dir-name { tmux new-session -As `basename $PWD` }

This does two things: - when you create a new session, it names it based on the directory name - if you are in a directory that already has a tmux session associated, tnew will attach you back into the already existing session

Tmux + OS X, pbcopy

By default tmux on OS X breaks the pbcopy command. I’m not going to go into details, but thankfully Chris Johnsen’s fix has detailed instructions on how to get that functionality back, and it’s not too tricky either.

Renumber windows

Out of the box, if you have 3 tmux windows numbered 0, 1, 2 and then you delete the one numbered ‘1’, you’re left with two windows: 0, and 2. This is rather odd when you navigate between them. However, you can tell tmux to renumber existing windows automatically with this line in your tmux config:

set-option -g renumber-windows on

vim-tmux-navigator

Without a doubt, this is the one thing that makes working with Vim and Tmux so much better, Chris Toomey’s vim-tmux-navigator plugin. By installing this and adding a few lines to your tmux config (instructions are in the README), this lets you seamlessly jump between Vim and tmux panes (or splits, as Vim calls them) with the same keystrokes. This means if you have a Vim split next to a tmux pane, you can hop between them even though one of them is Vim, and the other is tmux. It’s fantastic, and really ups the intergration of these two tools by another level.

Vimux

Although there are many choices, I personally have taken to using Vimux, a plugin which lets you send any command to a tmux pane (which it will create if it doesn’t already exist). This means I can set up mappings for running specific commands, such as for testing:

viml map <leader>nt :call VimuxRunCommand("clear; npm test")<CR>

It also has a function which you can run that will prompt you for a command, which I have mapped to <leader>x:

viml map <leader>x :VimuxPromptCommand<CR>

tmux-complete

This plugin adds any word visible in a tmux pane to a completion function that you can trigger with <C-X><C-U>. It’s rare that I want to type something in Vim that I can see in a tmux pane, but when I do this plugin is always useful and saves coming out of Vim to grab some text.

So that’s just some of the plugins I use daily when working with tmux and Vim. If you have any to recommend, please do leave a comment!

Apropos the Julie Ann Horvath Github shitshow, I’ve been thinking this weekend about management, generally.

I don’t know details about the particular Github situation so I won’t say much about it, but I was present for Tom Preston-Werner’s 2013 OSCON talk about Github. After a strong core message about open source licenses, liability, and freedom (tl;dr: avoid the WTFPL), Tom talked a bit about Github’s management model.

Management is about subjugation; it’s about control.

At Github, Tom described a setup where the power structure of the company is defined by the social structures of the employees. He showed a network hairball to illustrate his point, said that Github employees can work on what they feel like, subject to the strategic direction set for the company. There are no managers.

This bothered me a bit when I heard it last summer, and it’s gotten increasingly more uncomfortable since. I’ve been paraphrasing this part of the talk as “management is a form of workplace violence,” and the still-evolving story of Julie Ann Horvath suggests that the removal of one form of workplace violence has resulted in the reintroduction of another, much worse form. In my first post-college job, I was blessed with an awesome manager who described his work as “firefighter up and cheerleader down,” an idea I’ve tried to live by as I’ve moved into positions of authority myself. The idea of having no managers, echoed in other companies like Valve Software, suggests the presence of major cultural problems at a company like Github. As Shanley Kane wrote in What Your Culture Really Says, “we don’t have an explicit power structure, which makes it easier for the unspoken power dynamics in the company to play out without investigation or criticism.” Managers might be difficult, hostile, or useless, but because they are parts of an explicit power structure they can be evaluted explicitly. For people on the wrong side of a power dynamic, engaging with explicit structure is often the only means possible to fix a problem.

Implicit power can be a liability as well as a strength. In the popular imagination, implicit power elites close sweetheart deals in smoke-filled rooms. In reality, the need for implicit power to stay in the shadows can cripple it in the face of an outside context problem. Aaron Bady wrote of Julian Assange and Wikileaks that “while an organization structured by direct and open lines of communication will be much more vulnerable to outside penetration, the more opaque it becomes to itself (as a defense against the outside gaze), the less able it will be to “think” as a system, to communicate with itself. The more conspiratorial it becomes, in a certain sense, the less effective it will be as a conspiracy.”

Going back to the social diagram, this lack of ability to communicate internally seems to be an eventual property of purely bottoms-up social structures. Github has been enormously successful on the strength of a single core strategy: the creation of a delightful, easy-to-use web UI on top of a work-sharing system designed for distributed use. I’ve been a user since 2009, and my belief is that the product has consistently improved, but not meaningfully changed. Github’s central, most powerful innovation is the Pull Request. Github has annexed adjoining territory, but has not yet had to respond to a threat that may force it to abandon territory or change approach entirely.

Without a structured means of communication, the company is left with the vague notion that employees can do what they feel like, as long as it’s compliant with the company’s strategic direction. Who sets that direction, and how might it be possible to change it? There’s your implicit power and first point of weakness.

This is incidentally what’s so fascinating about the government technology position I’m in at Code for America. I believe that we’re in the midst of a shift in power from abusive tech vendor relationships to something driven by a city’s own digital capabilities. The amazing thing about GOV.UK is that a government has decided it has the know-how to hire its own team of designers and developers, and exercised its authority. That it’s a cost-saving measure is beside the point. It’s the change I want to see in the world: for governments large and small to stop copy-pasting RFP line items and cargo-culting tech trends (including the OMFG Ur On Github trend) and start thinking for themselves about their relationship with digital communication.

The very first time the concepts behind the juju project were presented, by then still under the prototype name of Ubuntu Pipes, was about four years ago, in July of 2009. It was a short meeting with Mark Shuttleworth, Simon Wardley, and myself, when Canonical still had an office on a tall building by the Thames. That was just the seed of a long road of meetings and presentations that eventually led to the codification of these ideas into what today is a major component of the Ubuntu strategy on servers.

Despite having covered the core concepts many times in those meetings and presentations, it recently occurred to me that they were never properly written down in any reasonable form. This is an omission that I’ll attempt to fix with this post while still holding the proper context in mind and while things haven’t changed too much.

It’s worth noting that I’ve stepped aside as the project technical lead in January, which makes more likely for some of these ideas to take a turn, but they are still of historical value, and true for the time being.

Contents

This post is long enough to deserve an index, but these sections do build up concepts incrementally, so for a full understanding sequential reading is best:

• Classical deployments
• Preparing a blank slate
• Service topologies
• Scaling services horizontally
• Charms
• Relations
• Configuration
• Taking from here

Classical deployments

In a simplistic sense, deploying an application means configuring and running a set of processes in one or more machines to compose an integrated system. This procedure includes not only configuring the processes for particular needs, but also appropriately interconnecting the processes that compose the system.

The following figure depicts a simple example of such a scenario, with two frontend machines that had the Wordpress software configured on them to serve the same content out of a single backend machine running the MySQL database.

Deploying even that simple environment already requires the administrator to deal with a variety of tasks, such as setting up physical or virtual machines, provisioning the operating system, installing the applications and the necessary dependencies, configuring web servers, configuring the database, configuring the communication across the processes including addresses and credentials, firewall rules, and so on. Then, once the system is up, the deployed system must be managed throughout its whole lifecycle, with upgrades, configuration changes, new services integrated, and more.

The lack of a good mechanism to turn all of these tasks into high-level operations that are convenient, repeatable, and extensible, is what motivated the development of juju. The next sections provide an overview of how these problems are solved.

Preparing a blank slate

Before diving into the way in which juju environments are organized, a few words must be said about what a juju environment is in the first place.

All resources managed by juju are said to be within a juju environment, and such an environment may be prepared by juju itself as long as the administrator has access to one of the supported infrastructure providers (AWS, OpenStack, MAAS, etc).

In practice, creating an environment is done by running juju’s bootstrap command:

$ juju bootstrap

This will start a machine in the configured infrastructure provider and prepare the machine for running the juju state server to control the whole environment. Once the machine and the state server are up, they’ll wait for future instructions that are provided via follow up commands or alternative user interfaces.

Service topologies

The high-level perspective that juju takes about an environment and its lifecycle is similar to the perspective that a person has about them. For instance, although the classical deployment example provided above is simple, the mental model that describes it is even simpler, and consists of just a couple of communicating services:

That’s pretty much the model that an administrator using juju has to input into the system for that deployment to be realized. This may be achieved with the following commands:

$ juju deploy cs:precise/wordpress
$ juju deploy cs:precise/mysql
$ juju add-relation wordpress mysql

These commands will communicate with the previously bootstrapped environment, and will input into the system the desired model. The commands themselves don’t actually change the current state of the deployed software, but rather inform the juju infrastructure of the state that the environment should be in. After the commands take place, the juju state server will act to transform the current state of the deployment into the desired one.

In the example described, for instance, juju starts by deploying two new machines that are able to run the service units responsible for Wordpress and MySQL, and configures the machines to run agents that manipulate the system as needed to realize the requested model. An intermediate stage of that process might conceptually be represented as:

The service units are then provided with the information necessary to configure and start the real software that is responsible for the requested workload (Wordpress and MySQL themselves, in this example), and are also provided with a mechanism that enables service units that were related together to easily exchange data such as addresses, credentials, and so on.

At this point, the service units are able to realize the requested model:

This is close to the original scenario described, except that there’s a single frontend machine running Wordpress. The next section details how to add that second frontend machine.

Scaling services horizontally

The next step to match the original scenario described is to add a second service unit that can run Wordpress, and that can be achieved by the single command:

$ juju add-unit wordpress

No further commands or information are necessary, because the juju state server understands what the model of the deployment is. That model includes both the configuration of the involved services and the fact that units of the wordpress service should talk to units of the mysql service.

This final step makes the deployed system look equivalent to the original scenario depicted:

Although that is equivalent to the classic deployment first described, as hinted by these examples an environment managed by juju isn’t static. Services may be added, removed, reconfigured, upgraded, expanded, contracted, and related together, and these actions may take place at any time during the lifetime of an environment.

The way that the service reacts to such changes isn’t enforced by the juju infrastructure. Instead, juju delegates service-specific decisions to the charm that implements the service behavior, as described in the following section.

Charms

A juju-managed environment wouldn't be nearly as interesting if all it could do was constrained by preconceived ideas that the juju developers had about what services should be supported and how they should interact among themselves and with the world.

Instead, the activities within a service deployed by juju are all orchestrated by a juju charm, which is generally named after the main software it exposes. A charm is defined by its metadata, one or more executable hooks that are called after certain events take place, and optionally some custom content.

The charm metadata contains basic declarative information, such as the name and description of the charm, relationships the charm may participate in, and configuration options that the charm is able to handle.

The charm hooks are executable files with well-defined names that may be written in any language. These hooks are run non-concurrently to inform the charm that something happened, and they give a chance for the charm to react to such events in arbitrary ways. There are hooks to inform that the service is supposed to be first installed, or started, or configured, or for when a relation was joined, departed, and so on.

This means that in the previous example the service units depicted are in fact reporting relevant events to the hooks that live within the wordpress charm, and those hooks are the ones responsible for bringing the Wordpress software and any other dependencies up.

The interface offered by juju to the charm implementation is the same, independently from which infrastructure provider is being used. As long as the charm author takes some care, one can create entire service stacks that can be moved around among different infrastructure providers.

Relations

In the examples above, the concept of service relationships was introduced naturally, because it’s indeed a common and critical aspect of any system that depends on more than a single process. Interestingly, despite it being such a foundational idea, most management systems in fact pay little attention to how the interconnections are modeled.

With juju, it’s fair to say that service relations were part of the system since inception, and have driven the whole mindset around it.

Relations in juju have three main properties: an interface, a kind, and a name.

The relation interface is simply a unique name that represents the protocol that is conventionally followed by the service units to exchange information via their respective hooks. As long as the name is the same, the charms are assumed to have been written in a compatible way, and thus the relation is allowed to be established via the user interface. Relations with different interfaces cannot be established.

The relation kind informs whether a service unit that deploys the given charm will act as a provider, a requirer, or a peer in the relation. Providers and requirers are complementary, in the sense that a service that provides an interface can only have that specific relation established with a service that requires the same interface, and vice-versa. Peer relations are automatically established internally across the units of the service that declares the relation, and enable easily clustering together these units to setup masters and slaves, rings, or any other structural organization that the underlying software supports.

The relation name uniquely identifies the given relation within the charm, and allows a single charm (and service and service units that use it) to have multiple relations with the same interface but different purposes. That identifier is then used in hook names relative to the given relation, user interfaces, and so on.

For example, the two communicating services described in examples might hold relations defined as:

When that service model is realized, juju will eventually inform all service units of the wordpress service that a relation was established with the respective service units of the mysql service. That event is communicated via hooks being called on both units, in a way resembling the following representation:

As depicted above, such an exchange might take the following form:

1. The administrator establishes a relation between the wordpress service and the mysql service, which causes the service units of these services (wordpress/1 and mysql/0 in the example) to relate.
2. Both service units concurrently call the relation-joined hook for the respective relation. Note that the hook is named after the local relation name for each unit. Given the conventions established for the mysql interface, the requirer side of the relation does nothing, and the provider informs the credentials and database name that should be used.
3. The requirer side of the relation is informed that relation settings have changed via the relation-changed hook. This hook implementation may pick up the provided settings and configure the software to talk to the remote side.
4. The Wordpress software itself is run, and establishes the required TCP connection to the configured database.

In that workflow, neither side knows for sure what service is being related to. It would be feasible (and probably welcome) to have the mysql service replaced by a mariadb service that provided a compatible mysql interface, and the wordpress charm wouldn’t have to be changed to communicate with it.

Also, although this example and many real world scenarios will have relations reflecting TCP connections, this may not always be the case. It’s reasonable to have relations conveying any kind of metadata across the related services.

Configuration

Service configuration follows the same model of metadata plus executable hooks that was described above for relations. A charm can declare what configuration settings it expects in its metadata, and how to react to setting changes in an executable hook named config-changed. Then, once a valid setting is changed for a service, all of the respective service units will have that hook called to reflect the new configuration.

Changing a service setting via the command line may be as simple as:

$ juju set wordpress title="My Blog"

This will communicate with the juju state server, record the new configuration, and consequently incite the service units to realize the new configuration as described. For clarity, this process may be represented as:

Taking from here

This conceptual overview hopefully provides some insight into the original thinking that went into designing the juju project. For more in-depth information on any of the topics covered here, the following resources are good starting points:

• The project page
• Getting started guide
• In-depth technical documentation
• Overview presentation at Google I/O 2012

We have long been able to test Ubuntu isos very easily by
using ‘testdrive’. It syncs releases/architectures you are interested
in and starts them in kvm. Very nice. But nowadays, in addition to
the isos, we also distribute cloud images. They are the basis for
cloud instances and ubuntu-cloud containers, but starting a local vm
based on them took some manual steps. Now you can use ‘uvtool’ to
easily sync and launch vms with cloud images.

uvtool is in the archive in trusty and saucy, but if you’re on precise
you’ll need the ppa:

sudo add-apt-repository ppa:uvtool-dev/trunk
sudo apt-get update
sudo apt-get install uvtool

Now you can sync the release you like, using a command like:

uvt-simplestreams-libvirt sync release=saucy
or
uvt-simplestreams-libvirt sync --source http://cloud-images.ubuntu.com/daily release=trusty arch=amd64

See what you’ve got syncd:

uvt-simplestreams-libvirt query

then launch a vm

uvt-kvm create xxx release=saucy arch=amd64
uvt-kvm list

and connect to it

uvt-kvm ssh --insecure -l ubuntu xxx

While it exists you can manage it using libvirt,

virsh list
virsh destroy xxx
virsh start xxx
virsh dumpxml xxx

Doing so, you can find out that the backing store is a qcow snapshot
of the ‘canonical’ image. If you decided you wanted to publish a
resulting vm, you could of course convert the backing store to a
raw file or whatever:

sudo qemu-img convert -f qcow2 -O raw /var/lib/uvtool/libvirt/images/xxx.qcow xxx.img

When you’re done, destroy it using

uvt-kvm destroy xxx


Very nice!

I like tagged tiling window managers. I like tiling because it lets me avoid tedious window move+resize. I like tagged wm because I can add multiple tags to windows so that different tag views can show different subsets of my windows – irc and mail, irc and task1, task1 and browsers, task2 and email…

Unity doesn’t tile, but has the grid plugin which is quite nice. But what about a tagged view? There used to be a compiz plugin called group. In the past when I’ve tried it it didn’t seem to quite fit my needs, but beyond that I couldn’t find it in recent releases.

I briefly considered building it straight into unity, but I really just wanted something to work with < 1 hr work. So I implemented it as a script, winmark. Winmark takes a single-character mark (think of marking in vi, ma, 'a) and stores or restores the layout of the currently un-minimized windows under that mark (in ~/.local/share/winmark/a). Another tiny c program grabs the keyboard to read a mark character, then calls winmark with that character.

So now I can hit shift-F9 a to store the current layout, set up a different layout, hit shift-f9 b to store that, then restore them with F9 a and F9 b.

I’m not packaging this right now as I *suspect* this is the sort of thing noone but me would want. However I’m mentioning it here in case I’m wrong. The source is at lp:~serge-hallyn/+junk/markwindows.

There’s definite room for improvement, but I’ve hit my hour time limit, and it is useful as is :) Potential improvements would include showing overlay previews as with application switching, and restoring the stacking order.

First, a quick announcement: A No-Excuses Guide to Blogging is now available as a free/pay-what-you-want e-book so that you can work your way through your excuses without having to click through lots of blog posts. =)

Mock-up by Ramon Williamson

The cover I made for Amazon

The PDF looks prettiest if you’re reading it on your computer or tablet, and the EPUB/MOBI version is handy for other e-readers. You get all three (and any future updates) if you grab it from http://sachachua.com/no-excuses-blogging, or you can get the MOBI version from the Kindle store (currently $0.99, but eventually they’ll price-match it to $0). The book is under Creative Commons Attribution, so feel free to share it with other people. =)

UPDATE 2014-02-13: Here’s a one-page summary!

2014-02-13 A No-Excuses Guide to Blogging – Summary of 10 blogging excuses and how to work around them

- Behind the scenes stuff! - 

So this was about 8 hours of packaging after I’d identified the topics and asked an assistant to compile all the blog posts in a Word document. I edited the text to make it fit better in a collection, fiddled with the graphics, added more sketches, tweaked the layout some more, fought with section headers, and eventually created a PDF that I was reasonably happy with. I contacted a bunch of people on Fiverr about converting the DOCX into EPUB and MOBI. While waiting for a response, I decided to try doing it myself. It took me some time to clean up the messy HTML, but I’m reasonably happy with how the EPUB worked out. I had to tweak the EPUB table of contents in order to get it to pass the validator used by Lulu, but eventually I got it worked out with Calibre and Sigil. The MOBI was a straight conversion from the EPUB, although I wonder if I can get the background colour to be white…

2014-02-05 Notes on publishing

So that was interesting and useful, and it would be good to do more. Here are some ideas for other “No Excuses”-type guides I might put together. Maybe self-directed learning and delegation, actually, since other people are covering the sketchnoting bit quite well.

2014-02-06 What other excuses can I collect and work around

There’s also this list of other book ideas, Thinking with Emacs, Tracking Your Time, Accelerating Your Learning with Sketchnotes, and this big outline. Lots to do. Looking forward to figuring out how I can get more of these out the door. =)

In the meantime… tada! http://sach.ac/no-excuses-blogging

Read the original or check out the comments on: A No Excuses Guide to Blogging (PDF, EPUB, MOBI – free!); also, notes on publishing (Sacha Chua's blog)

The post A No Excuses Guide to Blogging (PDF, EPUB, MOBI – free!); also, notes on publishing appeared first on sacha chua :: living an awesome life.

This term, I'm once again teaching my upper-division CSE undergrad course in Web Dev here at MSU. For the second time, I'm requiring students to use github for their homework; unlike last year, I now understand pull requests and have integrated them into the process.

git fetch
git checkout hw4

#! /bin/bash
dir=$(basename $PWD)
git push git@github.com:ctb/cse491-serverz.git hw4:hw4-$dir-comments

#! /bin/bash
dir=$(basename $PWD)
open -a "/Applications/Google Chrome.app/" https://github.com/ctb/cse491-serverz/compare/$dir:hw4...ctb:hw4-$dir-comments?expand=1

#! /bin/bash
for i in $(cat github-list.txt)
do
     cd $i
     echo $i $(git branch -r | grep hw4) >> ../out
     cd ../
done

I used to think a lot about how to keep track of the people I know, writing down notes about their interests in Emacs’ aptly named Big Brother Database (BBDB). I wanted to be the kind of person who could remember people’s goals and dreams, who paid attention to trivia from long-ago conversations. A little configuration allowed me to see my notes when reading or replying to e-mail from people, and I even wrote some code to track when I last contacted someone and filter the list of people so that I could focus on people I hadn’t talked to in a while. It was amusing to see a record and realize I’d actually known someone for years (these things sneak up on you before you know it!), and it was occasionally gratifying to be able to look up someone based on an interest I vaguely remembered them having and to make the connection with a book I’d just read or someone else they should meet.

I kept tweaking this system. Maybe there was a contact relationship management tool out there that could help me remember what was important to people’s lives or discover small-talk topics from people’s social media updates. Maybe there was an application that could help me act thoughtfully, beyond the usual flood of Facebook wall messages on someone’s birthday. Maybe there was a way for me to learn how to keep in touch. I asked people who networked for a living (mostly salespeople and politicians) how they managed to remember all these little details and reach out. They told me about spreadsheets and index cards, phone calls and e-mails: “Hi, how are you doing? Was just thinking of you. What’s up?” I was inspired by examples of super-connectors who somehow managed to remember everyone’s faces and names, asking after kids and pets and hobbies, effortlessly breaking through the limits of Dunbar’s number. (Or at least it seemed effortless; many worked very hard at this.) I tried out systems like Nimble and Contactually as a way to see people’s activity across platforms and remember to keep in touch. I tried ConnectedHQ (no longer available?) and Sunrise for people-related notes in my calendar. I followed Facebook and LinkedIn’s suggestions to greet people on their birthdays or job changes.

Still, those practices never quite felt natural. Maybe I’m just not a people person, at least not that kind. I never became comfortable with calling someone out of the blue, and I preferred posting stories on my blog instead of telling them one-on-one through e-mail or conversation.

I decided to try a different approach: I stopped worrying about keeping in touch. If people came into my life, great. If people drifted out, that was fine too. Instead of working on keeping those relationships going, I focused instead on being open to the conversations as they came up. Instead of reaching out so that I can hear about what people are working on and stay on their radar, I decided to focus on what I was curious about and be open to the people I’d bump into along the way.

Sure, I might have missed a few serendipitous connections with people who were quietly working on interesting things–but the world is full of other people who make it easy to keep up with them, like the way my blog makes it easy to keep up with what I’m interested in. I subscribe to a number of people’s blogs and I follow people on social networks. But for the most part, I care about ideas and conversations first, and then getting to know and keep in touch with people happens along the way.

It’s a very different form of networking, I think, than what I’ve read about in countless books. I still recommend books like How to Talk to Anyone (Lowndes) and Never Eat Alone (Ferrazzi) to people who are curious about building relationships with specific people or making the most of networking events. Me, I’m learning about how to talk to everyone. Instead of building a relationship with a specific person, I look for opportunities to help people around an idea or contribute to a group. Instead of asking a specific person for help, I ask in general, and people step forward if they’re interested. I don’t have to fear rejection, and no one has to look for tactful ways to say no; and if a request doesn’t get any takers, well, I’m where I would’ve been anyway.

In order to make that work, I need to make sure I have good ideas that people want to be part of, the confidence to share them, a platform for sharing them with, specific requests and ways people can get involved, and a way to celebrate how people are helping. I like that. It turns networking into something more than “I think you’re cool and I want to stay in touch” or even “You’re working on cool stuff; How can I help?” – it becomes “We’re all part of something bigger. Let’s make things happen.”

2014-01-02 Making things happen

I’m working on sharing, learning from people, asking for help, and making it easy for people to build on what we have in common. I’ll get there someday. It’s not so much that I want to keep in touch with a specific person; it’s that we’re both actively thinking about a topic, so maybe we’ll set up a conversation so we can exchange notes and learn together. It’s not so much that I want to have a wide range of skills in my roster of contacts; it’s more that I want to provide value to a wide range of people, and maybe some of them will want to help me when the opportunity comes up.

I was thinking about the way people are related to topics so that I can nudge myself to reach out more for questions and conversations. I like the idea of confederates: people who are actively learning about similar things, and whom I’m comfortable asking if I want a different perspective or I’m curious about what they’re learning. I don’t have any set schedule for reaching out to these people. If I learn something they might be interested in, I’ll mention it to them, but I’ll also open up the conversation to everyone who might be interested. When I started drawing out the map, I realized how lucky I was to have so many people I can learn from and who might find stuff I’m learning useful too. This is how I keep in touch now – through the exchange of questions and ideas.

2014-01-30 Confederates

So I might not build as close a relationship as you might with someone who takes the time to send you birthday cards or calls you up every quarter, but then do you really feel all that close to insurance or real estate agents who send you cards like clockwork? I don’t send “Hi, what’s up?” e-mails and I tend to not be very good at responding to them, but I’ll dive deeply into topics of mutual curiosity. My network is shifting to people who share what they know and what they’re learning. While that means I’m probably missing out on a lot of stuff that never gets shared beyond the small circles most people are comfortable in, there are so many good people out there focused on bringing everyone up–not just the people they’ve bumped into. It might also mean that I’m not as close to the people for whom these little gestures mean a lot, but it means that I draw closer to people who don’t have that need. It’s like the difference between someone who puts you on the spot when you forget your name and someone who helpfully supplies it at a hint of hesitation; someone who makes you feel bad for forgetting their birthday or anniversary or whatever, and someone who’s more focused on what’s good instead of what’s missing.

You don’t have to wish me a happy birthday, and you never have to apologize for replying slowly or not being in touch. Better to share your questions and ideas, and better to share that not just with me but with the world.

Read the original or check out the comments on: From networking with people to networking around ideas: How I stopped worrying about keeping in touch (Sacha Chua's blog)

The post From networking with people to networking around ideas: How I stopped worrying about keeping in touch appeared first on sacha chua :: living an awesome life.

It's something that most of us dread: to find out one day that we aren't real, that we are living in a computer simulation. Yet some people who work at the forefront of science believe that the ultimate nature of reality might not be so far off this idea.

read more

Tis the season for annoying bug bites, but a surprisingly simple remedy exists that can eliminate all of the itch within minutes.

Read more...

This article appeared in SC Magazine and I’ve posted it here as well. For those of you who were left wanting more from my previous SELinux post, this should help. If it doesn’t help, leave a comment. ;)

SELinux operates in the realm of mandatory access control, or MAC. The design of MAC involves placing constraints on what a user (a subject) can do to a particular object (a target) on the system. In contrast, discretionary access control, or DAC, allows a user with certain access to use discretion to limit or allow access to certain files, directories, or devices. You can set any file system permissions that you want but SELinux can override them with ease at the operating system level.

Consider a typical server running a web application. An attacker compromises the web application and executes malicious code via the web server daemon itself. SELinux has default policies that prevent the daemon from initiating communication on the network. That limits the attacker’s options to attack other services or servers.

In addition, SELinux sets policies on which files and directories the web server can access, regardless of any file system permissions. This protection limits the attacker’s access to other sensitive parts of the file system even if the administrator set the files to be readable to the world.

This is where SELinux shines. Oddly enough, this is the point where many system administrators actually disable SELinux on their systems.

Troubleshooting these events, called AVC denials, without some helpful tools is challenging and frustrating. Each denial flows into to your audit log as a cryptic message. Most administrators will check the usual suspects, like firewall rules and file system permissions. As frustration builds, they disable SELinux and notice that their application begins working as expected. SELinux remains disabled and hundreds of helpful policies lie dormant solely because one policy caused a problem.

Disabling SELinux without investigation frustrated me to the point where I started a site at stopdisablingselinux.com. The site is a snarky response to Linux administrators who reach for the disable switch as soon as SELinux gets in their way.

All jokes aside, here are some helpful tips to use SELinux effectively:

Use the setroubleshoot helpers to understand denials
Working through denials is easy with the setroubleshoot-server package. When a denial occurs, you still receive a cryptic log message in your audit logs. However, you also receive a message via syslog that is very easy to read. Your server can email you these messages as well. The message contains guidance about adjusting SELinux booleans, setting contexts, or generating new SELinux policies to work around a really unusual problem. When I say guidance, I mean that the tools give you commands to copy and paste to adjust your policies, booleans and contexts.

Review SELinux booleans for quick adjustments
Although the myriad of SELinux user-space tools isn’t within the scope of this article, getsebool and togglesebool deserve a mention. Frequently adjusted policies are controlled by booleans that are toggled on and off with togglesebool. Start with getsebool –a for a full list of booleans and then use togglesebool to enable or disable the policy.

Quickly restore file or directory contexts
Shuffling files or directories around a server can cause SELinux denials due to contexts not matching their original values. This happens to me frequently if I move a configuration file from one system to another. Correcting the context problem involves one of two simple commands. The restorecon command applies the default contexts specific to the file or directory. If you have a file in the directory with the correct context, use chcon to fix the context on the wrong file by giving it the path to the file with the correct context.

Here are some additional links with helpful SELinux documentation:

• SELinux Project Wiki
• Red Hat Enterprise Linux 6 SELinux Guide
• Dan Walsh’s Blog

Reprint: Stop Disabling SELinux! is a post from: Major Hayden's blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

At PiCloud we strive to be the easiest service out-of-the-box, yet versatile enough to be used for the most complex applications. This ideal was especially challenging to realize when expanding our compute platform beyond Python-only modules, to support the whole gamut of non-Python software. How could we extend the same ease-of-use that our cloud library offers to the complex world of software dependencies?

In this blog post, I’ll explore how we’ve harnessed three different technologies to achieve our goal of a highly-scalable software dependency manager. We call it Environments, and for many PaaS users, it’s what sets us apart.

Where We Were

PiCloud originated as a Python-only platform. In that simplified setup, our Dependency Analyzer could identify and ship all pure-Python dependencies at job submission time. Why only pure-Python? Anything that requires compilation cannot be expected to work when shipped to a computer with a different architecture. Unfortunately, for performance purposes, many Python libraries are actually compiled C-extensions, and this meant if you used a Python library with C-extensions, our compute nodes needed to have that library pre-installed. Not surprisingly, it didn’t take long before this limitation became an issue for some of our users.

Where We Wanted To Be

When the time came to revamp our dependency manager, we sought a generic solution that would work for Python and non-Python software alike. We boiled our requirements for the new dependency management down to the following:

• Simplicity. It must be no more difficult than what a user would do on their own machine. Importantly, it should not require users to learn a new tool or configuration language.
• Powerful. There should be minimal limitations on what packages or programs a user can install. Such flexibility necessitates that escalated privileges (for dependency installation purposes) are given in controlled situations.
• Low Overhead. It must not introduce significant overhead to job processing. Among other things, this means satisfying a user’s dependency must not require booting a new machine or rebooting an existing one.
• Parallelizable. The value of PiCloud is giving users the freedom to use as many cores as needed to accomplish their tasks. Deploying dependencies should not hinder parallelization across many machines.

How We Got There

Filesystem Virtualization

Fundamentally, satisfying the dependencies of a program means ensuring that the appropriate files can be found in the proper places in the filesystem. We can then rephrase the role of the dependency manager as making sure a job is run in whatever filesystem environment it needs for proper function.

We took inspiration from the Python tool virtualenv, which lets you create, on one machine, multiple, isolated Python environments, each with its own packages and settings. In other words, it virtualizes those portions of the filesystem relevant to Python. Generalizing this concept, what we wanted was a virtualenv for the entire filesystem.

On *nix machines, the simplest way to virtualize the filesystem is through the chroot program, which allows you to set any point in your filesystem hierarchy as the new filesystem “root”. Unfortunately, chroot lacks mechanisms for isolation and resource management. At the other end of the virtualization spectrum is a virtual machine, which gives you full control and isolation but comes with the huge overhead of starting and running the VM. However, in between the simple chroot utility and full VMs are container technologies that share the host’s kernel but have the capacity to virtualize most everything else. Combined with other recent Linux kernel developments, like cgroups and namespaces, container technologies provide resource management and isolation, respectively. And starting a container is fast, because there is no operating system to boot up – the host’s kernel is already up and running. We ended up choosing LXC (LinuX Container) as our container technology, because it has been mainlined into the Linux kernel.

Our new dependency management system was beginning to take shape. A user would start with an LXC-bootable base filesystem, and install whatever programs and files their jobs need. We would store this modified filesystem, which we’ll call an Environment from this point on. When the user runs a job, we would start an LXC container using their Environment as the filesystem. The next question is, how do we store and send these big Linux filesystems across the network to compute nodes?

Union Mount System

What we realized was that saving the entire Environment was wasteful. In practice, Environments would be mostly the same, with only a relatively small number of files added in or changed. Theoretically, given the base filesystem the Environment was built on, we only need the changes the user made on top of this base to recreate the full Environment. What would enable us to save just this difference, rather than the whole Environment?

The answer is union mounts. A union mount allows you to stack multiple filesystems and present a unioned view at a different mount point. It is the technology behind many Live CDs. A Live CD contains a bootable operating system but is not typically writable. Hence, upon bootup from a Live CD, the system will create a temporary scratch space in RAM, then union mount this RAMFS on top of the filesystem of the CD. After chrooting into this unioned mount, the user is presented with a machine seemingly running off a read-only CD, but giving you the ability to tinker with and change the operating system files as you try it out. And due to the magic of the union mount, all changes are being written to the RAM filesystem, even when modifying files from the CD.

Several union filesystem tools exist, but after some research, we settled on AUFS (Another Union FileSystem), a nifty module created and maintained by Junjiro Akajima. We chose AUFS, because it is relatively performant and reliable, and we have been impressed with Junjiro’s responsiveness to support requests.

So, how much does AUFS help? On PiCloud, the average size of Environment modifications is around 220 MB. Given a base filesystem size of around 2.5 GB (which includes many pre-installed libraries for convenience and performance), this leads to roughly a 12-fold savings in terms of storage and network transfer.

Environment In Action

So, putting all that together, here’s how PiCloud Environments work:

1. PiCloud user requests creation of a new Environment through our web UI or client CLI.
2. PiCloud launches a machine the user can log into, that is actually an AUFS stack of an empty filesystem (read-writable) on top of a base Environment (read-only), much like a Live CD.
3. User logs into the “setup server” and installs dependencies as they would on their own machine.

4. When the user is done setting up the Environment and requests to save it, user changes are sanitized for security and saved into our distributed file system.

5. User can now submit jobs specifying it should be run in the custom Environment. In Python, Environment is specified with the “_env” keyword, and in bash, the “-e” flag.
6. PiCloud recreates the AUFS stack on our compute nodes, and runs the user’s jobs in LXC containers.

DFS Features That Improve Performance

As mentioned above, AUFS reduces the size of an Environment by roughly 12-fold. In practice, we end up doing much better than that thanks to our distributed file system (DFS), which has two important properties:

1. Files from an environment are transferred at the block level to the compute node on-demand (only when a program accesses them).
2. Transferred files are cached on the compute node automatically.

To understand the benefits of these properties, consider the Matlab Compiler Runtime (MCR), which enables the stand-alone execution of compiled Matlab applications. When a user installs the MCR in an Environment, close to a gigabyte of files are added. But, for the typical MCR-compiled application, a small minority (

Check Out the New Features

If you’re currently a PiCloud user and have wondered how Environments worked, I hope this shed some light into the black box.

If you haven’t already, you should check out our recent updates to the Environment system. We’ve redone the web UI and added client-side CLI support for managing your Environments. Also, we are excited about the new sharing feature that lets users share Environments they’ve created with colleagues and the general public.