Now I can use the sssh command to connect to hosts using the firewall machine as a proxy. Like most good hacks, this uses netcat.
Eg:
$ sssh 10.1.2.3
Will connect me directly to a machine on the server network, via the firewall box. Seeing as it passes all parameters to ssh (the $* bit) you can do port forwards and X-forwarding as usual too:
$ sssh -L 5432:localhost:5432 my-vm
This lets me tunnel the port for a PostgreSQL running on my development vm (my-vm) in a single command. I have all my keys installed, so no passwords needed - I estimate this will save me about 60 seconds every day.
Splitsider is thrilled to offer our first digital download, The Exquisite Corpse Project, a fantastic film from the former members of legendary sketch group Olde English. I recently sat down with the guys to look through some of their classic Olde English sketches. Here, the group looked back at a uniquely Olde English rap video.
Caleb: Well I would say that if you look at like the '08 sketches, there's I Hate Nature, but then I think in the same year, there's also Totally Crazy. I could see another group trying, maybe not executing as well, but trying to do I Hate Nature, but I don't think anyone would have made Totally Crazy.
Raphael: Totally Crazy's a very Raphael sketch.
Caleb: But it works really well.
Ben: I think Totally Crazy is an awesome video, by the way.
Raphael: Totally Crazy I guess is the best hip-hop song I could possible write.
Ben: The premise is basically, it's the annoying guy at work who tells stories that are seemingly pointless and don't have a direction. But it's his hip hop song about a mundane night of his life.
Caleb: But how crazy he thinks it is.
Ben: And it has no direction, but it's done in the form of an exciting music video. And I think that was a really cool format, and it's a really fun video.
Raphael: I think what makes it really work, if I may toot my own horn a little bit, even though that's all we've been doing, but one thing I'm really proud of with that sketch is that the rap doesn't rhyme at all. And that's what really makes it this awkward long story. Even if it had the same kind of style to it but there was a beat, like a clear rhythm and a rhyme to it, I feel like it would be like not as funny. But it really is just like a terrible rap It doesn't work on any level.
Caleb: It works on one level, for a comedy sketch.
Raphael: Yeah. And again, the production value really makes it look like something approaching a real rap music video. It has all this like bombast and special effects and dancing, and we're in this club, and there's beautiful women, but it's just this awkward guy trying to tell his story about a thing that nobody really cares about.
I live in a country generally assumed to be a dictatorship. One of the Arab spring countries. I have lived through curfews and have seen the outcomes of the sort of surveillance now being revealed in the US. People here talking about curfews aren't realizing what that actually FEELS like. It isn't about having to go inside, and the practicality of that. It's about creating the feeling that everyone, everything is watching. A few points:
1) the purpose of this surveillance from the governments point of view is to control enemies of the state. Not terrorists. People who are coalescing around ideas that would destabilize the status quo. These could be religious ideas. These could be groups like anon who are too good with tech for the governments liking. It makes it very easy to know who these people are. It also makes it very simple to control these people.
Lets say you are a college student and you get in with some people who want to stop farming practices that hurt animals. So you make a plan and go to protest these practices. You get there, and wow, the protest is huge. You never expected this, you were just goofing off. Well now everyone who was there is suspect. Even though you technically had the right to protest, you're now considered a dangerous person. With this tech in place, the government doesn't have to put you in jail. They can do something more sinister. They can just email you a sexy picture you took with a girlfriend. Or they can email you a note saying that they can prove your dad is cheating on his taxes. Or they can threaten to get your dad fired. All you have to do, the email says, is help them catch your friends in the group. You have to report back every week, or you dad might lose his job. So you do. You turn in your friends and even though they try to keep meetings off grid, you're reporting on them to protect your dad.
2) Let's say number one goes on. The country is a weird place now. Really weird. Pretty soon, a movement springs up like occupy, except its bigger this time. People are really serious, and they are saying they want a government without this power. I guess people are realizing that it is a serious deal. You see on the news that tear gas was fired. Your friend calls you, frantic. They're shooting people. Oh my god. you never signed up for this. You say, fuck it. My dad might lose his job but I won't be responsible for anyone dying. That's going too far. You refuse to report anymore. You just stop going to meetings. You stay at home, and try not to watch the news. Three days later, police come to your door and arrest you. They confiscate your computer and phones, and they beat you up a bit. No one can help you so they all just sit quietly. They know if they say anything they're next. This happened in the country I live in. It is not a joke.
3) Its hard to say how long you were in there. What you saw was horrible. Most of the time, you only heard screams. People begging to be killed. Noises you've never heard before. You, you were lucky. You got kicked every day when they threw your moldy food at you, but no one shocked you. No one used sexual violence on you, at least that you remember. There were some times they gave you pills, and you can't say for sure what happened then. To be honest, sometimes the pills were the best part of your day, because at least then you didn't feel anything. You have scars on you from the way you were treated. You learn in prison that torture is now common. But everyone who uploads videos or pictures of this torture is labeled a leaker. Its considered a threat to national security. Pretty soon, a cut you got on your leg is looking really bad. You think it's infected. There were no doctors in prison, and it was so overcrowded, who knows what got in the cut. You go to the doctor, but he refuses to see you. He knows if he does the government can see the records that he treated you. Even you calling his office prompts a visit from the local police.
You decide to go home and see your parents. Maybe they can help. This leg is getting really bad. You get to their house. They aren't home. You can't reach them no matter how hard you try. A neighbor pulls you aside, and he quickly tells you they were arrested three weeks ago and haven't been seen since. You vaguely remember mentioning to them on the phone you were going to that protest. Even your little brother isn't there.
4) Is this even really happening? You look at the news. Sports scores. Celebrity news. It's like nothing is wrong. What the hell is going on? A stranger smirks at you reading the paper. You lose it. You shout at him "fuck you dude what are you laughing at can't you see I've got a fucking wound on my leg?" "Sorry," he says. "I just didn't know anyone read the news anymore." There haven't been any real journalists for months. They're all in jail.
Everyone walking around is scared. They can't talk to anyone else because they don't know who is reporting for the government. Hell, at one time YOU were reporting for the government. Maybe they just want their kid to get through school. Maybe they want to keep their job. Maybe they're sick and want to be able to visit the doctor. It's always a simple reason. Good people always do bad things for simple reasons.
You want to protest. You want your family back. You need help for your leg. This is way beyond anything you ever wanted. It started because you just wanted to see fair treatment in farms. Now you're basically considered a terrorist, and everyone around you might be reporting on you. You definitely can't use a phone or email. You can't get a job. You can't even trust people face to face anymore. On every corner, there are people with guns. They are as scared as you are. They just don't want to lose their jobs. They don't want to be labeled as traitors.
This all happened in the country where I live. You want to know why revolutions happen? Because little by little by little things get worse and worse. But this thing that is happening now is big. This is the key ingredient. This allows them to know everything they need to know to accomplish the above. The fact that they are doing it is proof that they are the sort of people who might use it in the way I described. In the country I live in, they also claimed it was for the safety of the people. Same in Soviet Russia. Same in East Germany. In fact, that is always the excuse that is used to surveil everyone. But it has never ONCE proven to be the reality.
Maybe Obama won't do it. Maybe the next guy won't, or the one after him. Maybe this story isn't about you. Maybe it happens 10 or 20 years from now, when a big war is happening, or after another big attack. Maybe it's about your daughter or your son. We just don't know yet. But what we do know is that right now, in this moment we have a choice. Are we okay with this, or not? Do we want this power to exist, or not?
You know for me, the reason I'm upset is that I grew up in school saying the pledge of allegiance. I was taught that the United States meant "liberty and justice for all." You get older, you learn that in this country we define that phrase based on the constitution. That's what tells us what liberty is and what justice is. Well, the government just violated that ideal. So if they aren't standing for liberty and justice anymore, what are they standing for? Safety?
Ask yourself a question. In the story I told above, does anyone sound safe?
I didn't make anything up. These things happened to people I know. We used to think it couldn't happen in America. But guess what? It's starting to happen. I actually get really upset when people say "I don't have anything to hide. Let them read everything." People saying that have no idea what they are bringing down on their own heads. They are naive, and we need to listen to people in other countries who are clearly telling us that this is a horrible horrible sign and it is time to stand up and say no.
A problem from the 2003 Moscow Mathematical Olympiad:
A store has three floors, which are connected only by an elevator. At night the store is empty, and during the workday:
(1) Of the customers who enter the elevator on the second floor, half go to the first floor and half to the third floor.
(2) The number of customers who get out the elevator on the third floor is less than 1/3 the total number of customers who get out of the elevator.
Which is greater, the number of customers who go from the first floor to the second on a given workday, or the number who go from the first floor to the third?
A good friend of mine at a customer said an important thing to me once:
When you're up to your ass in alligators, it's hard to remember you're meant to be draining the swamp
He was spot on right, especially given the customer and the situation that
he was a part of.
We were doing our best firefighting the problems that had built up over
years of sales guys pressing a dev team for features right now and there
being no senior/leader figure among the devs who could explain technical
debt to them in a way they could understand.
Wild Bill fixed that. He could speak fluent business guy and he could speak
fluent tech guy, so he'd listen to what the business needs and tell their
team plus the shadowcat staff helping out what they needed ... and we knew
what we needed to do. And we knew if we did it, then it'd be what their
business people needed to make the world work for them and for the customers
making the whole shebang go round.
He taught them an important lesson - that if you're always shooting
alligators the swamp will never get drained, so no matter how bad it gets
you have to assign some of your developer time to draining the swamp because
if you don't nothing actually gets better so there's no point to any of it.
I think most serious technical staff have been in this situation again and
again - but the distinction between alligator hunting and swamp draining is
a metaphor and a half, and the first one I've ever seen every team in a
company agree on as a description of what we're doing and why we need to
work on both.
He showed the business side why they wanted it, and gave us time to do it,
and the firefighting still got done because we all knew it had to but we
were faster because we all had hope.
By the time you read this, my colleague, my friend, will probably be dead.
So let's call the idea of doing it that way, of dedicating some percentage of
tech time to making things better no matter how much firefighting has to go
on anyway, "The Wild Bill Protocol".
It's a damn important thing and I don't know that it's ever had a name. So
I'm going to call it that because it's the only tribute I can think of that
matters to me, and because this is a damn good excuse to give it a name.
This post is dedicated to Wild Bill, and may whatever afterlife you find
be largely free of alligators.
There's this common notion of "10x programmers" who are 10x more productive than the average programmer. We can't quantify productivity so we don't know if it's true. But definitely, enough people appear unusually productive to sustain the "10x programmer" notion.
How do they do it?
People often assume that 10x more productivity results from 10x more aptitude or 10x more knowledge. I don't think so. Now I'm not saying aptitude and knowledge don't help. But what I've noticed over the years is that the number one factor is 10x more selectivity. The trick is to consistently avoid shit work.
And by shit work, I don't necessarily mean "intellectually unrewarding". Rather, the definition of shit work is that its output goes down the toilet.
I've done quite a lot of shit work myself, especially when I was inexperienced and gullible. (One of the big advantages of experience is that one becomes less gullible that way - which more than compensates for much of the school knowledge having faded from memory.)
Let me supply you with a textbook example of hard, stimulating, down-the-toilet-going work: my decade-old adventures with fixed point.
You know what "fixed point arithmetic" is? I'll tell you. It's when you work with integers and pretend they're fractions, by implicitly assuming that your integer x actually represents x/2^N for some value of N.
So to add two numbers, you just do x+y. To multiply, you need to do x*y>>N, because plain x*y would represent x*y/2^2N, right? You also need to be careful so that this shit doesn't overflow, deal with different Ns in the same expression, etc.
Now in the early noughties, I was porting software to an in-house chip which was under development. It wasn't supposed to have hardware floating point units - "we'll do everything in fixed point".
Here's a selection of things that I did:
There was a half-assed C++ template class called InteliFixed<N> (there still is; I kid you not). I put a lot of effort into making it, erm, full-assed (what's the opposite of half-assed?) This included things like making operator+ commutative when it gets two fixed point numbers of different types (what's the type of the result?); making sure the dreadful inline assembly implementing 64-bit intermediate multiplications inlines well; etc. etc.
My boss told me to keep two versions of the code - one using floating point, for the noble algorithm developers, and one using fixed point, for us grunt workers fiddling with production code. So I manually kept the two in sync.
My boss also told me to think of a way to run some of the code in float, some not, to help find precision bugs. So I wrote a heuristic C++ parser that automatically merged the two versions into one. It took some functions from the "float" version and others from the "fixed" version, based on a header-file-like input telling it what should come from which version.
Of course this merged shit would not run or even compile just like that, would it? So I implemented macros where you'd pass to functions, instead of vector<float>&, a REFERENCE(vector<float>), and a horrendous bulk of code making this work at runtime when you actually passed a vector<InteliFixed> (which the code inside the function then tried to treat as a vector<float>.)
And apart from all that meta-programming, there was the programming itself of course. For example, solving 5×5 equation systems to fit polynomials to noisy data points, in fixed point. I managed to get this to work using hideous normalization tricks and assembly code using something like 96 bits of integer precision. My code even worked better than single-precision floating point without normalization! Yay!
For months and months, I worked as hard as ever, cranking out as much complicated, working code as ever.
And here's what I should have done:
Convince management to put the damned hardware floating point unit into the damned chip. It didn't cost that many square millimeters of silicon - I should have insisted on finding out how many. (FPUs were added in the next chip generation.)
Failing that, lay my hands on the chip simulator, measure the cost of floating point emulation, and use it wherever it was affordable. (This is what we ended up doing in many places.)
Tell my boss that maintaining two versions in sync like he wanted isn't going to work - they're going to diverge completely, so that no tool in hell will be able to partially merge them and run the result. (Of course this is exactly what happened.)
Why did this end up in many months of shit work instead of doing the right thing? Because I didn't know what's what, because I didn't think I could argue with my management, and because the work was challenging and interesting. It then promptly went down the toilet.
The hardest part of "managing" these 10x folks - people widely known as extremely productive - is actually convincing them to work on something. (The rest of managing them tends to be easy - they know what's what; once they decide to do something, it's done.)
You'd expect the opposite, kind of, right? I mean if you're so productive, why do you care? You work quickly; the worst thing that happens is, nothing comes out of it - then you'll just do the next thing quickly, right? I mean it's the slow, less productive folks that ought to be picky - they're slower and so get less shots at new stuff to work on to begin with, right?
But that's the optical illusion at work: the more productive folks aren't that much quicker - not 10x quicker. The reason they appear 10x quicker is that almost nothing they do is thrown away - unlike a whole lot of stuff that other people do.
And you don't count that thrown-away stuff as productivity. You think of a person as "the guy who did X" where X was famously useful - and forget all the Ys which weren't that useful, despite the effort and talent going into those Ys. Even if something else was "at fault", like the manager, or the timing, or whatever.
To pick famous examples, you remember Ken Thompson for C and Unix - but not for Plan 9, not really, and not for Go, not yet - on the contrary, Go gets your attention because it's a language by those Unix guys. You remember Linus Torvalds even though Linux is a Unix clone and git is a BitKeeper clone - in fact because they're clones of successful products which therefore had great chances to succeed due to good timing.
The first thing you care about is not how original something is or how hard it was to write or how good it is along any dimension: you care about its uses.
The 10x programmer will typically fight very hard to not work on something that is likely enough to not get used.
One of these wise guys asked me the other day about checkedthreads which I've just finished, "so is anyone using that?" with that trademark irony. I said I didn't know; there was a comment on HN saying that maybe someone will give it a try.
I mean it's a great thing; it's going to find all of your threading bugs, basically. But it's not a drop-in replacement for pthreads or such, you need to write the code using its interfaces - nice, simple interfaces, but not the ones you're already using. So there's a good chance few people will bother; whereas Helgrind or the thread sanitizer, which have tons of false negatives and false positives, at least work with the interfaces that people use today.
Why did I bother then? Because the first version took an afternoon to write (that was before I decided I want to have parallel nested loops and stuff), and I figured I had a chance because I'd blog about it (as I do, for example, right now). If I wrote a few posts explaining how you could actually hunt down bugs in old-school shared-memory parallel C code even easier than with Rust/Go/Erlang, maybe people would notice.
But there's already too much chances of a flop here for most of the 10x crowd I personally know to bother trying. Even though we use something like checkedthreads internally and it's a runaway success. In fact the ironic question came from the guy who put a lot of work in that internal version - because internally, it was very likely to be used.
See? Not working on potential flops - that's productivity.
How to pick what to work on? There are a lot of things one can look at:
Is there an alternative already available? How bad is it? If it's passable, then don't do it - it's hard to improve on a good thing and even harder to convince that improvements are worth the switch.
How "optional" is this thing? Will nothing work without it, or is it a bell/whistle type of thing that can easily go unnoticed?
How much work do users need to put in to get benefits? Does it work with their existing code or data? Do they need to learn new tricks or can they keep working as usual?
How many people must know about the thing for it to get distributed, let alone used? Will users mostly run the code unknowingly because it gets bundled together with code already distributed to them, or do they need to actively install something? (Getting the feature automatically and then having to learn things in order to use it is often better than having to install something and then working as usual. Think of how many people end up using a new Excel feature vs how many people use software running backups in the background.)
How much code to deliver how much value? Optimizing the hell out of a small kernel doing mpeg decompression sounds better than going over a million lines of code to get a 1.2x overall speed-up (even though the latter may be worth it by itself; it just necessarily requires 10x the programmers, not one "10x programmer").
Does it have teeth?If users do something wrong (or "wrong"), does it silently become useless to them (like static code analysis when it no longer understands a program), or does it halt their progress until they fix the error (like a bounds-checked array)?
…
You could easily expand this list; the basic underlying question is, what are the chances of me finishing this thing and then it being actually used? This applies recursively to every feature, sub-feature and line of code: does it contribute to the larger thing being used? And is there something else I could do with the time that would contribute more?
Of course it's more complicated than that; some useful things are held in higher regard than others for various reasons. Which is where Richard Stallman enters and requires us to call Linux "GNU/Linux" because GNU provided much of the original userspace stuff. And while I'm not going to call it "Gah-noo Lee-nux", there's sadly some merit to the argument, in the sense that yeah, unfortunately some hard, important work is less noticed than other hard, important work.
But how fair things are is beside the point. After all, it's not like 10x the perceived productivity is very likely to give you 10x the compensation. So there's not a whole lot of reasons to "cheat" and appear more productive than you are. The main reason to be productive is because there's fire raging up one's arse, more than any tangible benefit.
The point I do want to make is, to get more done, you don't need to succeed more quickly (although that helps) as much as you need to failless often. And not all failures are due to lack of knowledge or skill; most of them are due to quitting before something is actually usable - or due to there being few chances for it to be used in the first place.
So I believe, having authored a lot of code that went down the toilet, that you don't get productive by working as much as by not working - not on stuff that is likely to get thrown away.
and a good the bad and ugly drawing out of my sketchbook with them quoting KRS and Crooked I: oh lee van clief, you pussy hound.
The last 3 days were freelance for money days so I'm excited to get back to my own shit. Multiple warheads issue 3 woo. This morning Marian went off to island cabin solitude and I ate burritos and watched new Tron.
Dude, that movie barely has Tron in it. It needs a "this movie contains less than .02% Tron" warning label. It'd be like if they called the first star wars, cantina bartender. or if Tribe called quest was just called Jarobi.
Seems like if any movie was to get deep into Arcade logic It shouldve been that. A life bar would have been nice. also the dudes making it just threw out all the the rad moebius designs from the old one. Like taking off gold plating so you can cover it in plastic.
I drew this while watching it. The idea for this is that as Flynn aged in the Grid his user being would get bigger and bigger. He'd be like Cloak from cloak and dagger with infinate space inside his shell. and the programs all wear their frizbe disks like halos.
The gentleman who runs the Moebius Tumblr put some of the old Tron designs this week too: http://theairtightgarage.tumblr.com/tagged/tron If there's a division of nobel peace prizes for running a Tumblr it should go to that guy. ------------------------------------------------------------------------- For Xmas My Lady gave me a tortoise toy and made a comic to go along with it. It is the best thing ever.
My pal Moritat sent me images from a sketchbook he did. He was my gateway into drawing porn comics a million years ago and now is drawing the Sprit over at DC. Rooster in the hen house. I really like his sketches. I think theres copies of the sketcbook on http://www.stuartngbooks.com
My homie Simon Roy sent me some of his newer shorts. I was thinking about how artists like Marian and Stokeo and Simon are pulling so much more influence from reading real real books and I think it shows.
Reading Simons stuff makes me feel like I'm seeing an amazing 60's science fiction author reincarnated to make comics. His work is so good and so nicely subtle.
Also Simon made this personal map about making his Jan's Atomic heart book. Bigger here: http://www.flickr.com/photos/povorot/5261821802/ ------------------------------------------------------------------------ I spent Xmas over at Marians parents place she baked stuff with her sister while I played Marians childhood copy of Castlevania: Symphony of the Night in the basementvania.
I like the idea that it's set in this giant living ever changing castle. But I had a hard time relating to the weird overdressed (I'm so mad at you dad)dracula hunters.
400 and 17 is such an age difference it'll never work.
I like the monsters. Like this shit, a naked lady on top of some giant wolf heads with eeels coming out of them-- what the shit is that!?
I made it to the upsidedown 2nd castle. -----------------------------------------------------------------------
And I read some comics this week. Moebius 8, Mississippi river and the Dominion: Phantom of the audience short. Man, look at that Clark Gable/ Rhett Butler mutherfucker on the left. I like to think that Jean-Michel Charlier and Moebius had just watched Gone with the wind and we're like "fuck that guy, lets kill him"
Dominion always feels like Shirow just fucking around. This one has some great batshit crazy in it.
I really like this scene where a pollitician tries to negotiate with fire.
Or the crazy shit with these fireman. where he's told to take a nap and in the next panel hes goot a full bed set up and is sleeping on the spot.-- and that Boar saying Groovy as she races by it in a tank.
Mississippi was cool, I liked it more than the Blueberry stuff Moebius does with Charlier. I like our yankie hero Jim Cutlass. (jim cut-a-bitch) Here's a scene where he tricks the snake handelers out for him by sleeping under his bed.
There's something severly badass about making a perfectly good soft bed as a decoy and then sleeping hidden in the same room. Like this Appleseed page.
Mr Moritat showed me this Phil Winslade Batman cover that he said was the best ever. I think I agree. Also unrelated, reminds me of the best rap line I've heard this week where Joell Ortiz says--"I've got my back to the future like I'm mad at Biff" and I like this Moebius poster for Kurosawa's Bday
in the 80's Heavy metal ran a ton of great Adamov comics. I really like his Waters of Deadmoon about post apocalyptic cannibal french aristocrats. and Dayak ghetto 9 about a white dude with an albino ape in future Africa. But yeah if I ever reach comic book Valhalla I expect Adamov to be there.
Here's some other French books that I haven't read but I really like the look of. L’Arbre des Deux Printemps on the left and Sheol on right. The shadows on that house are great.
And I'm really excited about Pat McEown's(of back ups in Cooper's Weasle fame)Hair shirt McEown is such a Tom Herpich , (one of the on the low secret best to ever touch a comic. shhhh)-- For me artists like that are proof that so much of the best comics are rare and hard to find.
And here's a cover by Enki Bilal I hadn't seen before and a nice Milo Manara drawing. I never get tired of these dudes.
And some Miyazaki drawing bearded war pigs.--Just like witches at black massessss..
Dirty pair blade runner.
Aight, I'm going back to Multiple warheads pages and then later make my bed and not even sleep in it. DO SOMETHING!
One of the things that Git makes safe and practical is bulk editing by way of “sledgehammer-and-review”:
Apply some powerful (but potentially dangerous) sledgehammer to a bunch of files.
Review the sledgehammer’s effects using git diff to make sure there was no collateral damage.
If everything looks right, commit the effects; otherwise, reset them back to step 1 and try again using a more-refined sledgehammer.
That technique is handy, but it’s not the trick I want to share with you.
The trick I want to share with you is a refinement of sledgehammer-and-review, one that allows you to adjust the sledgehammer iteratively without having to re-review changes you’ve already reviewed.
Instead, you just have to review the changes to the changes, hence the “second-order-diff trick.”
The idea is to replace git reset with git stash, should you need to back up and try again with an adjusted sledgehammer.
Using stash not only resets your working tree back to the starting point but also stashes the result of the applying the previous sledgehammer.
Then you can apply your new sledgehammer and use git diff stash to review how it changed your original working tree differently than the previous sledgehammer.
In this way, you can more easily verify that adjusting your sledgehammer worked as expected.
Once everything looks good, git commit as usual.
Example
Recently, I moved my blog from the Typo system to the Hakyll system.
The hardest part was translating a decade’s worth of posts from Typo-flavored Textile to Hakyll-flavored Markdown.
I automated what I could using Pandoc, but a lot of Textile markup evaded translation.
In particular, over 100 links spread across 30 posts remained in Textile form.
In Textile, a link looks like this:
"Link Title":http://example.com/
My challenge was to convert links like that one, across all posts, into Markdown:
[Link Title](http://example.com/)
Rather than convert them by hand, I reached for a sledgehammer.
In my head, the sledgehammer took shape:
Use git grep to locate posts that probably contained Textile-flavored links.
This I did by looking for a double quote followed by a colon, which I figured unlikely to occur in normal text.
Pipe the matching posts into a Perl one-liner that would translate probable Textile links into Markdown links.
Because the sledgehammer makes changes that are “probably” right, I had the obligation to review its effects for collateral damage.
Thus began the process of sledgehammer-and-review.
Sledgehammer-and-review
At first, I used plain sledgehammer-and-review to home in on a regular expression that matched Textile links semi-reliably.
The first iteration went like this:
git status # verify that we're starting w/ a clean slate
# Iteration 1: take an educated guess at a good sledgehammer
# 1. Try out the candidate sledgehammer
git grep -lF '":' -- posts |
xargs perl -i -pe's{"(.+?)":([^.,:\s]+)}{[$1]($2)}g'
# 2. Review results and judge
git diff # => judgment: not right
# 3. Roll back to clean slate in preparation for the next attempt
git checkout -- posts # (also could have used git reset --hard)
The next iterations took only a few seconds.
For each, I just hit the up-arrow key from the command line to recall and refine the previous iteration:
# Iteration 2: refine the sledgehammer
git grep -lF '":' -- posts |
xargs perl -i -pe's{"(.+?)":([^.,!\)\]\s]+)}{[$1]($2)}g'
git diff # => judgment: still not right
git checkout -- posts
# Iteration 3: refine the sledgehammer more
git grep -lF '":' -- posts |
xargs perl -i -pe's{"(.+?)":([^,!\)\]\s]+)\b}{[$1]($2)}g'
git diff # => judgment: still not right
git checkout -- posts
But on iteration four I hit on a regular expression that worked well:
# Iteration 4: refine the sledgehammer just a bit more
git grep -lF '":' -- posts |
xargs perl -i -pe's{"(.+?)":([^,!\)\]\s]+)}{[$1]($2)}g'
git diff # => judgment: hey! looking pretty good...
This time, I spent a good ten minutes in git diff because most of the changes looked good and I ended up reviewing them all.
But there was one tiny problem.
If a link occurred at the end of a sentence, the sentence-ending period got absorbed into the link.
Here’s one of the dozen such errors I spotted during the review:
--Do check it out: "Try Ruby":http://tryruby.hobix.com/.
++Do check it out: [Try Ruby](http://tryruby.hobix.com/.)
So I needed to adjust my sledgehammer to not break sentence-ending links.
And, of course, after adjustment I would need to verify that everything now worked as expected.
But I didn’t want to re-do the entire review.
Why review all 100+ links, when the adjustment should affect only a dozen?
Time for the second-order-diff trick!
The second-order-diff trick
This time, instead of blowing away the previous sledgehammer’s results, I stashed them:
git stash save "link fix: almost there"
Now I had my original clean slate back, but I also had the previous results stashed away for later comparison.
Time to refine the sledgehammer and try again:
This time for review, however, I didn’t compare the sledgehammer’s results to the original clean slate but to the previous sledgehammer’s results, as stashed away:
git diff 'stash@{0}'
And, sure enough, there were only a dozen changes to review, and they were the expected end-of-sentence fixes:
-Do check it out: [Try Ruby](http://tryruby.hobix.com/.)
+Do check it out: [Try Ruby](http://tryruby.hobix.com/).
Mission accomplished! All that was left was to commit the reviewed changes:
On Hacker News, I’m seeing some common questions, so I’ll try to answer them here.
Why not use git add -p to incrementally accept the good changes while you refine the sledgehammer?
The reason is that the sledgehammer is not incremental: it expects to be applied to the original clean slate.
Every time.
Using git stash will get me back to that clean slate.
Using git add -p will not, for both the approved and unapproved changes will remain in the working tree, where the sledgehammer expects neither of them to be.
Instead of using the stash, can’t you just create a temporary branch?
Using the stash is just creating a temporary branch.
The stash is just a stack of local temporary branches.
But, unlike a hand-made branch, the stash has a more-convenient interface for work that follows a review-reset-retry cycle.
Update 2013-02-21
I expanded my explanation of what was happening during each step of the sledgehammer-and-review process.
That touched off a jumbled Twitter conversation in which I did a typically poor job of explaining why, in my opinion, the cancellation of Google Reader signals the end of Google as a responsible custodian of the world’s data. I think Chris gave up on me towards the end, possibly mistaking me for an angry troll instead of the true-blue (but now completely disillusioned) fan of Google products and services that I have been, these past ten years. Just a few years ago I was heaping derision on Microsoft for daring to compete with Google in search. I loved Google. Now I’m sort of thinking maybe Bing and DuckDuckGo aren’t such a bad idea.
Rumblings
How can the cancellation of an obscure Google service that many people were never aware of signal the coming demise of Google as a thought leader? The answer is a bit hard to approach because so much of it is locked up in the world’s psychological perception of Google, a nebulous and fluid thing at best. When Chris asked me the same question over Twitter, I struggled for a response and ended up with this:
@chromosundrift Either I was having a bad day or Google has crossed some sort of psychological threshold.
Actually, as it turns out, that’s not half wrong. A psychological threshold has been crossed. It’s not so much what Google did with Reader—although that was evil and shameful—it’s that what Google did with Reader follows a long trail of obnoxious Google behavior. Google has revealed itself to be as trigger-happy in the creation and cancellation of dubious products as Microsoft ever was, a thought echoed by former Googler Scott Whittaker, who simply dismissed the entire company as another Microsoft. Yikes!
But it gets worse. After reading Ed Bott’s scathing rebuke of Google’s embrace, extend, extinguish behavior in the marketplace it’s difficult to see Google as other than a power-hungry megacorp willing to play dirty under the guise of “free”; a faceless machine that cares nothing about you or your privacy; that offers no solution or satisfaction when you slip through the cracks of their services; and for whom “free” is simply a way to devour a market space. Behold the new Google, which Aldo Cortesi famously dubbed a destroyer of ecosystems.
The truth is this: Google destroyed the RSS feed reader ecosystem with a subsidized product, stifling its competitors and killing innovation. It then neglected Google Reader itself for years, after it had effectively become the only player. Today it does further damage by buggering up the already beleaguered links between publishers and readers. It would have been better for the Internet if Reader had never been at all.
What are we to make of this? What are we to make of it when Om Malik casually brushes aside the release of Google Keep, actively recruiting people to use Evernote instead? (A recommendation I endorse, by the way. Keep away from Google Keep and all non-essential Google services.) What of security guru Bruce Schneier who this week announced a transition away from Google servicesincluding search?
Who is this new Google and how did you get control of our internet?
It might be naive, but my own little protest is avoiding Google+ whenever possible, as a user and in projects I build.
RSS isn’t dying of natural causes. Plenty of small-medium sites are getting orders of magnitude more traffic from Reader than from G+. Google and other major companies are trying to deprecate it in order to replace with their own, tightly controlled, solutions.
Another commenter replied:
G+ can’t be avoided for too long. Google is using it to tie together Author Rank, which helps for SEO. If you want your content to rank higher post-Panda update you’ll leverage G+.
To which I replied, somewhat incredulously:
If that’s true then G+ is more of a danger to Google than Facebook ever was. Using the index as a way to strong-arm publishers into using a half-baked Facebook clone may even work, short-term. Long-term this is one way Google can lose Search.
To which someone else replied (factually!):
It is true, Eric Schmidt said: “Within search results, information tied to verified online profiles will be ranked higher than content without such verification, which will result in most users naturally clicking on the top (verified) results. The true cost of remaining anonymous, then, might be irrelevance.”
And indeed he did. Because wait, let me get this straight—this obtuse, confusing, unoriginal Facebook wannabe called Google Plus, which we’ve all been dutifully ignoring because it’s confusing and broken and sort of a top-down corporate attempt to “do social”, is going to become a requirement for our very existence on the web, because we all supported and loved Google and gave them the capability to even consider pulling such a move in the first place? That I’m going to have to participate in this absurd concoction by force?
It’s enough to make me stop using Google entirely, and especially search. Because fuck you.
“Don’t be evil, except…”—the new Google
We stand at a crossroads where Google is losing ground seemingly on every front—the best Google products (Search, Maps, etc.) are just a few iterations ahead of competitors (Bing, OpenMaps, etc.) while the less popular services live in the perpetual shadow of the executioner’s axe. There are substantial and compelling technical reasons to move away from Google and toward open (or commercial-but-dedicated) solutions, and this would be true even if Google had the best of intentions.
What the cancellation of Google Reader showed us is that Google does not always have the best intentions, or even good intentions. At worst, Google can be a wolf in sheep’s clothing; at best, Google’s motivations are simply and mundanely corporate. Maybe we shouldn’t be surprised that corporations behave like corporations, but I believe Google was once different. Nowhere does the old magical spirit of “Don’t Be Evil” enter this new equation, and gone is the sense of optimism and openness that characterized the early Google. Even my beloved Google Search page, which once conquered the world by being simple and robust, has mutated into an all-encompassing confederated “web portal” from the land of privacy hell with integrated logins, “personalized” search, and in short, basically everything the world hated about AOL.
Within search results, information tied to verified online profiles will be ranked higher than content without such verification, which will result in most users naturally clicking on the top (verified) results. The true cost of remaining anonymous, then, might be irrelevance.
The arrogance of this “message from the CEO” should be a concern to every man, woman, and child now using Google services. Google is involved in nothing less than a blatant attempt to confute the notion of privacy with that of anonymity, making it seem naughty, and then to posit a Google Plus account as being equivalent to an identity. Therefore Google Reader must die an ignominious death to an upstart Facebook clone. But we never asked for anonymity or identity; we asked for some basic privacy. What we got, was Google Plus. So maybe Schmidt is right. Maybe the true cost of remaining anonymous will be irrelevance—but if I were Schmidt I’d be more concerned about the true cost of Google Plus, that social albatross. Otherwise the thing that becomes irrelevant may well be Google itself.
