Kevin van Zonneveld

If you’re anything like me, you probably log in and out of a half dozen remote servers (or these days, local virtual machines) on a daily basis. And if you’re even more like me, you have trouble remembering all of the various usernames, remote addresses and command line options for things like specifying a non-standard connection port or forwarding local ports to the remote machine.

Shell Aliases

Let’s say that you have a remote server named dev.example.com, which has not been set up with public/private keys for password-less logins. The username to the remote account is fooey, and to reduce the number of scripted login attempts, you’ve decided to change the default SSH port to 2200 from the normal default of 22. This means that a typical command would look like:

Not too bad.

We can make things simpler and more secure by using a public/private key pair; I highly recommend using ssh-copy-id for moving your public keys around. It will save you quite a few folder/file permission headaches.

Now this doesn’t seem all that bad. To cut down on the verbosity you could create a simple alias in your shell as well:

This works surprisingly well: Every new server you need to connect to, just add an alias to your .bashrc (or .zshrc if you hang with the cool kids), and voilà.

~/.ssh/config

However, there’s a much more elegant and flexible solution to this problem. Enter the SSH config file:

This means that I can simply  $ ssh dev, and the options will be read from the configuration file. Easy peasy. Let’s see what else we can do with just a few simple configuration directives.

Personally, I use quite a few public/private keypairs for the various servers and services that I use, to ensure that in the event of having one of my keys compromised the dammage is as restricted as possible. For example, I have a key that I use uniquely for my github account. Let’s set it up so that that particular private key is used for all my github-related operations:

The use of IdentityFile allows me to specify exactly which private key I wish to use for authentification with the given host. You can, of course, simply specify this as a command line option for “normal” connections:

but the use of a config file with IdentityFile is pretty much your only option if you want to specify which identity to use for any git commands. This also opens up the very interesting concept of further segmenting your github keys on something like a per-project or per-organization basis:

Which means that if I want to clone a repository using my organization credentials, I would use the following:

Going further

As any security-conscious developer would do, I set up firewalls on all of my servers and make them as restrictive as possible; in many cases, this means that the only ports that I leave open are 80/443 (for webservers), and port 22 for SSH (or whatever I might have remapped it to for obfuscation purposes). On the surface, this seems to prevent me from using things like a desktop MySQL GUI client, which expect port 3306 to be open and accessible on the remote server in question. The informed reader will note, however, that a simple local port forward can save you:

This will forward all local port 9906 traffic to port 3306 on the remote dev.example.com server, letting me point my desktop GUI to localhost (127.0.0.1:9906) and have it behave exactly as if I had exposed port 3306 on the remote server and connected directly to it.

Now I don’t know about you, but remembering that sequence of flags and options for SSH can be a complete pain. Luckily, our config file can help alleviate that:

Which means I can simply do:

And my local port forwarding will be enabled using all of the configuration directives I set up for the tunnel host. Slick.

Homework

There are quite a few configuration options that you can specify in ~/.ssh/config, and I highly suggest consulting the online documentation or the ssh_config man page. Some interesting/useful things that you can do include: change the default number of connection attempts, specify local environment variables to be passed to the remote server upon connection, and even the use of * and ? wildcards for matching hosts.

I hope that some of this is useful to a few of you. Leave a note in the comments if you have any cool tricks for the SSH config file; I’m always on the lookout for fun hacks.

If you’ve looked at my previous entry Getting WordPress Running on HHVM, you’ve already got a general idea for how to create a config.hdf file and get a web server up and running. Now you want to make sure it’s running as quickly, and efficiently as possible. But first, it’s worth understanding how HHVM runs your PHP scripts.

How HHVM serves up a page

When a request comes in, HHVM considers the hostname of the request against VirtualHost and Sandbox configuration directives to determine the right base directory to serve files from.

Once it’s identified the file, it checks a code cache stored in a SQLite database to see if it has already compiled the file. If so, and the file hasn’t changed, HHVM will execute that cached version very similarly to how APC caches PHP’s compilation process. If not, then HHVM’s analysis engine is invoked to compile the script, perform type inference, optimization, and store the result for next time.

As you might expect, there is a short warm-up period when a new server is started with a fresh, empty cache. Just like any other compilation cache, this only slows the first request of a given page. Unlike APC, however, HHVM’s cache is on disk, and therefore survives restarts.

If you’re curious about the code cache repository, you can find it in the location specified by the setting Repo.Central.Path, which defaults to ~/.hhvm.hhbc (you probably want to change that). Refer to doc/repo for more info on Repo settings.

Repo {
  Central {
    Path = /var/run/hhvm.hhbc.sq3
  }
}

Just In Time

For production servers, you’ll almost certainly want to enable JIT (Just in Time) compilation to progressively turn the bytecode contained in hhvm.hhbc.sq3 into native code. Depending on exactly what your code base is doing, this will result in performance improvements of anywhere from 50% to 300% (100% seems to be a fairly common average).

To do so, just add the following snippet to your config.hdf file. Yes, it really is a “go faster” button.

Eval {
  Jit = true
}

Bear in mind that JIT compilation is meant for long-running web servers where the extra effort of compiling to native code pays off after a few page views. For single-run uses like command line scripts, the extra work of compiling to native code will actually slow it down compared to just executing bytecode.

Note that as of Oct 18, 2012, HHVM will enable JIT by default for server, daemon, and replay modes, and disable by default for cli, debug, and translate modes.  Ref: 57f063d2 e5824bea

Pre-analyzing

As stated already, all compilation from source to bytecode is cached on disk, so restarting your web server doesn’t lose any of that information and there will be no “warm-up” period on your second and later server starts; But why have that first warm-up period at all?

When you built HipHop (with USE_HHVM=1), the build process actually made two binaries. One, in src/hhvm/hhvm, is the web server and PHP engine you know and love. The other, in src/hphp/hphp (aka hhvm-analyze), is a version of HPHPc meant for HHVM use. It’s this latter binary which will allow you to pre-generate that cache before your web server ever starts up.

If you’re using our pre-built `hiphop-php` package for Ubuntu 12.04, this HHVM version of hphp has been delivered as /usr/bin/hhvm-analyze. The name has been changed to avoid confusion with /usr/bin/hphp which will be the normal HPHPc compiler in non-HHVM mode. For the purpose of this article, I’ll refer to running hhvm-analyze. Those who compiled HipHop themselves should take this to mean src/hphp/hphp built using USE_HHVM=1

To build the cache, you’ll need to identify the files you want included and invoke the analyzer:

$ cd /var/www
$ find . -name '*.php' > /tmp/files.list
$ hhvm --hphp --target hhbc --input-list /tmp/files.list --output-file hhvm.hhbc.sq3 -k1 -l3

The newly generated bytecode cache will be in /tmp/hphp_XXXXX/hhvm.hhbc.sq3 where XXXXX is a unique label produced in the output.

In addition to avoiding the initial warm-up by doing that ahead of time, hhvm-analyze will actually work a little harder than hhvm would at runtime. It’ll find a few more optimizations, be a little more sure about type inference, and produce better bytecode.

Repo.Authoritative

Even with the cache completely primed by running the analysis step above, HHVM is still going to default to checking the subject file each time it’s requested to make sure it hasn’t changed. That means disk I/O, and that means time. For a production server where source files are confidently not-changing, that’s just a waste. APC users will be familiar with this concept from the “apc.stat=0″ option.

To ignore the state of the source files during runtime on your production servers, just tell HHVM that the code cache (or Repo in HipHop parlance) is the authoritative source of all scripts on the system. With this setting enabled, if the file is found in the cache, it’s used without checking for updates. If it’s not found in the cache, the server will offer up a 404 without even trying to grab the file.

Repo {
  Authoritative = true
}

Note that the bytecode cache is also tagged with the build ID of the hhvm which generated it. If a new hhvm binary is used, it will invalidate any previous cache and start fresh. So when running in Repo/Authoritative mode, be sure to regenerate your cache any time you replace your hhvm binary.

For a while now I’ve been a huge fan of Gittip. I think they’ve created one of the most interesting models for funding Open Source development.

One of the missing pieces, for most Open Source developers, is having consistent, reliable, income backing your development. Some developers are sponsored by their work place, others have a job but do Open Source work on the side. Even others may not have reliable incomes and still want to create software that isn’t tied to a specific revenue-generator. I personally use Gittip to fund hosting costs for some of my side projects.

Gittip is capable of freeing Open Source developers by allowing people to give them money on a week-by-week basis. This is intriguing as the money that comes in will come slowly – people will be less likely to give and they’ll likely give smaller amounts – but it’ll also be relatively stable (once someone is giving there will be not much reason to decrease that amount or stop giving).

I’ve been providing feedback to Chad Whitacre and the rest of the Gittip team wherever I can because this is something that I really want to succeed.

Taking a further step I looked to see what were the major pain points to growth. Right now they’re:

1. Getting more developers signed up and asking for donations.
2. Getting more people giving money to those developers.

I’m attempting to work with the Gittip team to help make #1 more feasible, and while I’m giving money personally, I wanted to try and get more money into the system.

I saw a great opportunity for the development team at Khan Academy to give back to the Open Source community in ways that were not just code-related.

Talking with Ben Kamens we came upon a good solution: Every developer at Khan Academy (interns included) has $5/week to give to the Open Source developers of their choice. (You’re restricted from giving that money to employees of Khan Academy though, heh.) That money would then be rolled up into a single contribution coming from the Khan Academy Gittip account.

This seemed like the ideal scenario: Developers gets full control of how their money should be allocated and Khan Academy gets the benefit of their financial contributions coming from a unified source. (Which helps to communicate to developers that we care about Open Source and that, hopefully, we’re a cool place to work, which we are.)

In the spirit of shipping beats perfection I hacked together a Google Spreadsheet which Khan Academy devs could modify to allocate their contributions.

At the moment 21 Khan Academy devs are collectively giving $57.75/week to 15 different Open Source devs. This is a great start! Unfortunately we’re trying to give $46.25 to an additional 8 devs who aren’t yet on Gittip, so our contribution is stuck in limbo. (At the moment the best we can hope for is to pester them on Twitter, etc. and get them to sign up. In some cases devs seem to be disinclined to receive donations if they already have a full-time job — in that case they should feel free to donate the money to the charity of their choice, or even just back to other devs!)

Update July 17th: We’re now up to 26 Khan Academy devs giving $89/week to 17 Open Source devs on Gittip. Another $40.25/week is being held until some additional Open Source devs join Gittip.

We’ve already started the process of contacting some of the devs that we want to support:

@jashkenas @fat @cloudhead @stuk @ariyahidayat Heya! Could you join @Gittip? We want to thank you for your OSS work: http://t.co/jr6OHRDoVm

— Khan Academy (@khanacademy) July 16, 2013

There is a script that I’ve made to automate the process of extracting the tips from the spreadsheet and uploading them to Gittip (run as a weekly cron job). You can find it on Github:

https://github.com/Khan/gittip-gdoc

I want to thank Chad for helping to add in an API that would make it easier to create this script!

I hope others are as excited about Gittip as we are at Khan Academy. I strongly think that this is the future of providing developers with a financial incentive for creating useful Open Source work. Gittip is still quite young and our understanding of how this mode of contribution should work is still open for debate so I hope that this will help inspire others to give, and strive towards a financial ideal, as well.

Often in various stages of development we perform repetitive tasks e.g minification, compilation, unit testing, linting and we would definitely want to automate them. But each of these task require different configurations or installations. What if there is a system where in we can easily configure and run these tasks from a single place. Also it would be really nice if it gels well with Node.js system. Meet Grunt a task runner for simplifying chores. Apart from being simple Grunt ecosystem is huge and it's growing every day with literally hundreds of plugins to choose from.

Note that I will be using grunt task/plugin interchangeably which means the same.

Installation

In order to get started first we need to install grunt-cli.

npm install -g grunt-cli

This will install grunt command in your system. Note that this is different from the grunt task runner. Notice that we are installing grunt-cli in global scope. After this install grunt in your package.json directory using

npm install grunt --save-dev

This will install grunt task runner and add it to devDependencies section of your package.json.

You can also install grunt by adding it to devDependency of package.json manually and run npm install

Why is grunt-cli required

grunt-cli is used to run the local grunt task runner using node.js require() system. This helps us in running grunt task runner from any sub-directory of your project. The job of the grunt-cli is simple: run the version of grunt which has been installed next to a Gruntfile i.e under the node_modules directory.

What is a Gruntfile

Gruntfile is where we define configurations for the task which we are planning to run. Think of it as what package.json is for npm. Gruntfile is always placed at the top level of project along with package.json. Below is a directory structure showing the Gruntfile for project.

Please note that though we call it Gruntfile name of the file will be Gruntfile.js.

Installing a grunt task

Best way to learn something is by example. Lets take a very common task of minifying set of css files and then combine them to a single file.

As I have mentioned earlier grunt has an excellent repository of plugins to perform these tasks, which can be found on http://gruntjs.com/plugins. Doing a quick search on this page gives cssmin as one of the best plugin for it. To see installation and configuration instructions, head over to the plugins npm page.

As given in the npm page of cssmin install it with the given below command.

npm install grunt-contrib-cssmin

Now we have successfully installed a grunt task, its time to use it in Gruntfile.

Using a task in Gruntfile

Since we have installed our task into our project its time to dive into Gruntfile and use the task in it. Below is the Gruntfile for our task. Most of the parts could be understood with the comments given.

Note that we have to wrap your Gruntfile in module.exports. Lets understand the various important parts of this Gruntfile.

Understanding parts of Gruntfile

grunt.initConfig - Block where we define configurations specific to a plugin. In the example above i have shown configuration for 1 plugin but there could be more than 1 in it. e.g

cssmin: {
  // task/plugin 1
},
uglify: {
  // task/plugin 2
}

grunt.loadNpmTasks - This is where we load the task/plugin.

grunt.registerTask - This is where we register the task. I am going to explain this part a bit more in the next section where we will learn to run this task of ours.

Running a grunt task

Before diving into how to run a grunt task lets understand grunt.registerTask in depth. Recall from our Gruntfile below block of code.

// Default task(s).
// This is the task which runs if we just execute grunt command
  grunt.registerTask('default', ['cssmin']);
  // cssmin task
  // This is a custom task which runs if we execute grunt buildcss command
  grunt.registerTask('buildcss', ['cssmin']);

A Gruntfile is executed either by running grunt command or grunt task-name. In our Gruntfile when we say grunt.registerTask('default', ['cssmin']), this is what is going to run when we only type grunt command and that's why name of the task is default . Also note that second argument to grunt.registerTask is an array of tasks to be run.

grunt.registerTask('buildcss', ['cssmin']) defines a new custom task and perform cssmin. We can also add some other task to it. This is what is going to run when we type grunt buildcss command.

To summarize

grunt

Runs default grunt task defined with grunt.registerTask('default', ['cssmin'])

grunt buildcss

Runs custom grunt task defined with grunt.registerTask('buildcss', ['cssmin'])

Running any of these commands should generate our minified file with the following output.

In both the cases, second argument to grunt.registerTask can have more than one task. For example if we have JavaScript minification task uglify in our Gruntfile with default grunt.registerTask as:

grunt.registerTask('default', ['cssmin','uglify'])

This tutorial has just touched base on Grunt but there are in-numerous things which we can do using it. I use Grunt for css/js minification, templates compilation, appending cache busting param, string replacements etc. Grunt community has built tasks for almost everything we do. So check them out and use them to automate your chores.