Shared posts

22 Jun 14:42

Array covariance: not just ugly, but slow too

by skeet

It seems to be quite a long time since I've written a genuine "code" blog post. Time to fix that.

This material may well be covered elsewhere – it's certainly not terrifically original, and I've been meaning to post about it for a long time. In particular, I remember mentioning it at CodeMash in 2012. Anyway, the time has now come.

Refresher on array covariance

Just as a bit of background before we delve into the performance aspect, let me remind you what array covariance is, and when it applies. The basic idea is that C# allows a reference conversion from type TDerived[] to type TBase[], so long as:

  • TDerived and TBase are both reference types (potentially interfaces)
  • There's a reference conversion from TDerived to TBase (so either TDerived is the same as TBase, or a subclass, or an implementing class etc)

Just to remind you about reference conversions, those are conversions from one reference type to another, where the result (on success) is never a reference to a different object. To quote section 6.1.6 of the C# 5 spec:

Reference conversions, implicit or explicit, never change the referential identity of the object being converted. In other words, while a reference conversion may change the type of the reference, it never changes the type or value of the object being referred to.

So as a simple example, there's a reference conversion from string to object, therefore there's a reference conversion from string[] to object[]:

string[] strings = new string[10];
object[] objects = strings;

// strings and objects now refer to the same object

There is not a reference conversion between value type arrays, so you can't use the same code to conver from int[] to object[].

The nasty part is that every store operation into a reference type array now has to be checked at execution time for type safety. So to extend our sample code very slightly:

string[] strings = new string[10]; 
object[] objects = strings;

objects[0] = "string"; // This is fine
objects[0] = new Button(); // This will fail

The last line here will fail with an ArrayTypeMismatchException, to avoid storing a Button reference in a String[] object. When I said that every store operation has to be checked, that's a slight exaggeration: in theory, if the compile-time type is an array with an element type which is a sealed class, the check can be avoided as it can't fail.

Avoiding array covariance

I would rather arrays weren't covariant in the first place, but there's not a lot that can be done about that now. However, we can work around this, if we really need to. We know that value type arrays are not covariant... so how about we use a value type array instead, even if we want to store reference types?

All we need is a value type which can store the reference type we need – which is dead easy with a wrapper type:

public struct Wrapper<T> where T : class
{
    private readonly T value;
    public T Value { get { return value; } }
    
    public Wrapper(T value)
    {
        this.value = value;
    }
    
    public static implicit operator Wrapper<T>(T value)
    {
        return new Wrapper<T>(value);
    }
}

Now if we have a Wrapper<string>[], we can't assign that to a Wrapper<object>[] variable – the types are incompatible. If that feels a bit clunky, we can put the array into its own type:

public sealed class InvariantArray<T> where T : class
{
    private readonly Wrapper<T>[] array;
    
    public InvariantArray(int size)
    {
        array = new Wrapper<T>[size];
    }
    
    public T this[int index]
    {
        get { return array[index].Value; }
        set { array[index] = value; }
    }
}

Just to clarify, we now only have value type arrays, but ones where each value is a plain wrapper for a reference. We now can't accidentally violate type-safety at compile-time, and the CLR doesn't need to validate write operations.

There's no memory overhead here – aside from the type information at the start, I'd actually expect the contents of a Wrapper<T>[] to be indistinguishable from a T[] in memory.

Benchmarking

So, how does it perform? I've written a small console app to test it. You can download the full code, but the gist of it is that we use a stopwatch to measure how long it takes to either repeatedly write to an array, or repeatedly read from an array (validating that the value read is non-null, just to prove that we've really read something). I'm hoping I haven't fallen foul of any of the various mistakes in benchmarking which are so easy to make.

The test tries four scenarios:

  • object[] (but still storing strings)
  • string[]
  • Wrapper<string>[]
  • InvariantArray<string>

Running against an array size of 100, with 100 million iterations per test, I get the following results on my Thinkpad Twist :

Array type Read time (ms) Write time
object[] 11842 44827
string[] 12000 40865
Wrapper<string>[] 11843 29338
InvariantArray<string> 11825 32973

That's just one run, but the results are fairly consistent across runs. The one interesting deviation is the write time for object[] – I've observed it sometimes being the same as for string[], but not consistently. I don't understand this, but it does seem that the JIT isn't performing the optimization for string[] that it could if it spotted that string is sealed.

Both of the workarounds to avoid array covariance make a noticeable difference to the performance of writing to the array, without affecting read performance. Hooray!

Conclusion

I think it would be a very rare application which noticed a significant performance boost here, but I do like the fact that this is one of those situations where a cleaner design also leads to better performance, without many obvious technical downsides.

That said, I doubt that I'll actually be using this in real code any time soon – the fact that it's just "different" to normal C# code is a big downside in itself. Hope you found it interesting though :)

13 Jun 21:26

Bliki: ConfigurationSynchronization

Automated configuration tools (such as CFEngine, Puppet, or Chef) allow you to avoid SnowflakeServers by providing recipes to describe the configuration of elements of a server. Configuration synchronization continually applies these specifications, either on a regular schedule or when it changes, to server instances throughout their lifetime. If someone makes a change to a server outside the tool, it will be reverted to the centrally specified configuration the next time the server is synchronized. If some configuration change is needed, it's made in the configuration specification (recipes, manifests, or whatever the particular configuration tool calls it), and is then applied to all relevant servers across the infrastructure.

In theory, once a server has been created, configuration synchronization will keep it up to date, applying upgrades and patches and preventing configuration drift for a potentially long lifetime.

In practice, however, it isn't feasible to keep servers completely consistent using current automated configuration management tools, so over time configuration synchronization leads to inconsistency. [1]

Each element of server configuration managed by an automated tool requires work to write, test, and maintain the recipe or manifest. It simply isn't reasonable to attempt to manage every single element of a typical server using these tools. There are far too many of them. And for each additional element you do specify, the work involved grows non-linearly thanks to the potential interactions, integrations, and dependencies between them.

Software packages are a particular challenge, given the dependencies they may have on yet more packages. Tools such as yum, apt-get, gems, etc. automatically work out dependencies and install them from repositories. So a large number of the packages on a system are only implicitly managed, and may change without notice. Although you can micro-manage the versions and dependencies of software packages installed, it's an intractable job considering the number of packages involved.

If managing the stuff you want to have on your server is difficult, managing the things you don't want is worse, given there are an effectively inifinite number of them. Current configuration management tools require you to explicitly list each file, package, or other element you want removed if found. This means additional things can be manually added onto some servers, creating inconsistent and unexpected behaviour.

In reality, people using automated configuration tools get by well enough without specifying 100% of the configurable surface area of their servers. Teams apply the 80/20 rule to automated configuration, focusing 80% (or more like 95%) of their attention on defining that 20% (or 5%) of the system which is most relevant to their particular needs, leaving the rest to the defaults installed by the base OS. So the majority of the system is not under automated configuration, and generally, this works. Unfortuantely, when it doesn't - when some part of the system not managed by automation tools does cause an issue, the effects are unexpected and can be difficult to track down.

This problem grows worse over the lifespan of a server, and with the frequency of change. The longer a server stays up, the more it may deviate from other servers, especially newer ones.

This issue leads some people to use PhoenixServers. By ensuring that a server's lifespan is kept short, frequently rebuilding fresh ones from a base image, the potential for configuration drift is kept small, without the overhead of specifying more of the server's configuration in a management tool than is really necessary. Taking the phoenix server to its logical conclusion leads to ImmutableServers, which avoid any changes made to a server during it's lifespan.

Notes

1: Vendor Aspirations

My colleague Max Lincoln provided these references to configuration tool vendors' aspirations around total system configuration:

  • "Create a blueprint of your infrastructure - so it can be built or rebuilt consistently from scratch in minutes" - Opscode Chef
  • "Eliminate configuration drift and reduce outages" - Puppetlabs
  • "CFEngine then continuously corrects configuration drift, keeping systems in compliance with their Desired State." - CFEngine
13 Jun 21:25

Bliki: ImmutableServer

Automated configuration tools (such as CFEngine, Puppet, or Chef) allow you to specify how servers should be configured, and bring new and existing machines into compliance. This helps to avoid the problem of fragile SnowflakeServers. Such tools can create PhoenixServers that can be torn down and rebuilt at will. An Immutable Server is the logical conclusion of this approach, a server that once deployed, is never modified, merely replaced with a new updated instance.

Automated configuration tools are usually used with ConfigurationSynchronization where you leave a server running for a potentially long period of time, repeatedly applying configuration to bring it into line with the latest specification. In theory servers can be allowed to run indefinitely, and they'll be kept completely consistent and up to date. In practice it's not possible to manage a server's configuration completely, so there is considerable scope for configuration drift, and unexpected changes to running servers.

This is where a PhoenixServer is helpful.

By frequently destroying and rebuilding servers from the base image, 100% of the server's elements are reset to a known state, without spending a ridiculous amount of time specifying and maintaining detailed configuration specifications.

Once you're using phoenixes, the focus of configuration management shifts to the management of base images. Fixes, changes, and updates are applied to the base image rather than to running systems. Each time you want a new update you modify the base instance and run it through an automated test harness. You only create new servers when they pass these steps.

So a phoenix server's complete state is built from a combination of base image + automated configuration management + data [1], which reduces the pressure to have automated configuration manage 100% of the server.

But while we can continue to run configuration management updates on a server during its brief lifetime, there's less value in doing so. In fact, there is considerable value in not doing so, since any change to a running system introduces risk.

This leads naturally to immutable servers [2].

Once you've spun up a server instance from a well-tested base image, you shouldn't run configuration management tools, since they create opportunities for untested changes to the instance. Any changes that are needed can be made to the base image, tested, and then rolled out. Servers without the change are torn down and replaced.

If this sounds familiar, it's because it follows the practices of ContinuousIntegration and ContinuousDelivery. With Continuous Delivery of software, it's safer to compile a given version of an application into a deployable artifact only once, and know that you are deploying and running a consistently built application in all environments. With an immutable server, you make each change to a base image, and then you know that all instances created from that image are consistent.

The main differences between instances of a server role come from configuration settings, which should come from outside the server. For example, most virtualization and cloud platforms offer a way to set metadata values when provisioning a new instance, which can then be read by the running server. New servers may also pull configuration values from a central registry like Zookeeper.

It's advisable to minimize the number and scope of per-instance configuration items for immutable servers, and to run changes to these through automated testing where feasible.


Handling data

If servers are disposable, the data that lives on them often is not. When implementing phoenix or immutable servers you should consider what data needs to be persisted as servers are destroyed and created, and what data must be replicated in order to scale by adding additional servers.

You can ship data off of the instance when it has value but isn't needed at runtime, for example sending logfiles to a central syslog server. A shared file system like NFS can make files available to servers, perhaps living on a SAN. Cloud platforms generally offer mountable storage devices like AWS EBS volumes which can be attached to new servers instances when the old ones are destroyed, or quickly duplicated and attached to replicas when scaling a cluster. Often you can pass the buck to a service which someone else maintains, like Amazon's RDS database service.

Further Reading

Netflix has been at the forefront in using the ImmutableServer pattern, although I'm not aware that they've used the term. They have open-sourced the Aminator tool they developed to manage AMI instances for use on Amazon's AWS cloud and blogged about how their use of this pattern has evolved with experience. Interestingly, the speed of instantiating new instances has been a key driver for them, since this helps them to automatically scale and recover.

Notes

1: Data covers a variety of things, including database files and other application-managed data, runtime state, other runtime-generated data such as log files, and externally supplied configuration.

2: My colleague Ben Butler-Cole coined the term "Immutable Server" for this pattern on an internal ThoughtWorks mailing list, and has provided insights from his experiences with it.

09 Jun 17:38

Applying the benefits of Yoga to product development and vice versa

by Steven Sinofsky

Yoga scene from "Forgetting Sarah Marshall"The physiological benefits of exercise are well known, and regular exercise can also bring benefit to your work. Any exercise will do of course, starting with just walking as often as possible. The summer brings with it opportunities for broad ranges of outdoor activities. Almost all health studies point to the need for rigorous exercise to be a regular part of your routine in order to achieve the maximum benefit for health and work productivity.

Please be sure to take the survey on exercise and productivity found here.

Yoga is an especially good exercise for those that value a routine (or require a routine to better stick to a program). Rain or shine, hot or cold, day or night, yoga is always there. Centuries old, the health benefits of yoga are highly regarded by practitioners around the world and many religions, psychologists and biologists agree that taking time to rest, focus and center is an important part of human productivity. In the US, a variety of styles of yoga performed in room heated anywhere from 90 to 105+ degrees F has become increasingly popular. The heat provides additional benefits and an extra level of challenge as well. If you value routine, Bikram yoga is especially good since it is precisely the same every class.

Yoga’s history is rooted in meditation and ritual and for many today the spiritual aspect is the high order bit. For others, the spiritual side of yoga is appreciated in the context of other beliefs or just your daily personal life. Even the aum (om) sign or chant carries with it a deep ritual meaning for some as well as a more personally defined spiritual meaning for others.

I was asked about practicing yoga at the All Things D conference by Katie Boehret and she captured a few seconds on the KatieCam. I wanted to elaborate on the benefits by using 5 sayings/expressions that I’ve learned from many of the wonderful yoga instructors I’ve had over the years.

  1. Be present. At the start of most yoga classes, instructors will remind everyone to “be present”. That means to set aside all that is going on outside the class and to spend the next 90 minutes present in the room, on your mat, and in the practice–no electronics or distractions. How often at work do things outside the context of what you’re working on interfere with the work—did you bring the challenges from the previous meeting into the next meeting or is something going on outside work showing through how you are at work?  Take the time at the start of a day, as a meeting starts, or while coding to be present in what you’re doing.
  2. All that matters is on your mat. Yoga is not a competitive sport.* Yoga is not a race—you can’t finish first, you can’t be faster or lift more. Yoga is about making sure you are focused on what you can do best and that you are doing your best at that. So when practicing, making sure you’re focused on what you are supposed to be focused on is a path to success. The workplace isn’t a competitive sport either. In the workplace, this can mean doing the work you’re supposed to be doing and assuming those around you are doing the same.
  3. Drishti. Focus is a big part  of Yoga. If you lose focus during some balancing pose you probably just fall over. Or if you lose focus on your breathing you very quickly hyperventilate and get exhausted or just turn blue!  Drishti is a Sanskrit word that means focus, but a distinct form of focus. You focus but not so intensely that you lose sight of all around you. Rather it is the opposite, where you focus but with a full awareness of the rest of your mat and body. So rather than staring at a dot on the wall in front of you, you gaze at the dot but focus on breathing, your balance, and more. In software projects it is important to be focused—but if you’re too head’s down you miss important connections to what is going on around you or around the code. The full definition of drishti means vision, point of view, or even intelligence and wisdom. It also means being equal in all directions you look and maintaining self-control. A lot of collaboration in product development can be summed up in drishti.
  4. Yoga practice is not yoga perfect. Many “Type A” personalities find a way to compete in exercise—running times, weight lifted, miles biked and so on. Yoga is designed for life long exercise. There’s always more to do or a way to connect one pose to another you never thought of. Any yogi who has browsed the advanced videos online is quickly humbled by what they cannot do. During those difficult postures, yoga instructors always remind the class that yoga is a practice and it is not called yoga perfect. You do the best you can with the body you have that day, and you commit to practicing the next day. Product development is like this as well—we often say the enemy of the good is the perfect. No product is perfect, but the least perfect product is one that doesn’t ship. Shipping gives you the right to come back the time with improvements and a better product based on what you learned as a team.
  5. How you do anything is how you do everything. In class, especially when it is hot, you can easily find a way to slack off or find a way to do a pose that might look like you’re posing but in reality you are missing the benefits. Of course you’re just cheating yourself. When an instructor sees this you might hear the most gentle of reminders, “how you do anything is how you do everything”. Put simply this just means that if you are willing to take a shortcut on one pose then where else in life (or in your product) are you willing to take a shortcut. If you’re willing to cheat yourself out of your best efforts, then won’t you cheat others?  Always put forth your best efforts, even when you’re pushed to the point of thinking you can’t possibly continue.

That’s a yoga perspective on exercise and well-being that are critical parts of contributing to your work, your project, and your team. While yoga has been my personal approach, what is really important is that you find your approach to physical and mental well-being. Whatever that might be is sure to be a critical tool in your own success.

What do you do to maintain your physical as well as spiritual health in the workplace?  What lessons do you bring back to the workplace from your avocation?

Namaste,

Steven Sinofsky

*While controversial there do exist yoga competitions — check this out.


06 Jun 17:38

UltraMon

UltraMon:

UltraMon is a utility for multi-monitor [Windows] systems, designed to increase productivity and unlock the full potential of multiple monitors.

Features include per-monitor taskbars, window positioning shortcuts and display mirroring.

05 Jun 21:43

storm is a command line tool to manage your ssh connections.



storm is a command line tool to manage your ssh connections.

15 May 14:57

Push to Kindle

Push to Kindle:

Push to Kindle lets you send web articles (news stories, blog posts, Wikipedia entries) to your Kindle app or device for a better reading experience.

Add nicely-formatted articles to your Kindle via extensions for Chrome, Firefox, Opera & Safari, or an Android app.

Amazon Appstore / Google Play

08 May 20:38

Conversation #38– disrupt or die

by Steven Sinofsky

thCAM164QKAnyone worth their salt in product development knows that listening to customers through any and all means possible is the means to innovation. Wait a minute, anyone worth their salt in product development knows that listening to customers leads to a faster horse.

Deciding your own product choices within these varying perspectives is perhaps the seminal challenge in product development, tech products or otherwise. This truly is a tyranny of or, but one in which changing the rules of the game is the very objective.

In this discussion, which is such a common dialog in the halls of HBS as well tech companies everywhere it should probably be a numbered conversation (for this blog let’s call this Conversation #38 for shorthand—disrupt or die).

For a recent discussion about why it is so difficult for large companies to face changes in the marketplace, see this post Why Corporate Giants Fail to Change.

“Disrupt or die” or “disrupt and die”?

Failure to evolve a product as technologies change or as customer scenarios change is sure to lead to obsolescence or elimination from the marketplace. It is difficult to go a day in tech product development without hearing about technology disruption or “innovator’s dilemma”. The biggest fear we all have in tech is failing to keep up with the changing landscape of technologies and customers, and how those intersect.

At the same time, hopefully we all get to that lucky moment when our product is being used actively by customers who are paying. We’re in that feedback loop. We are improving the product, more is being sold, and we’re on a roll.

That’s when innovation over time looks like this:

Incremental

In this case as time progresses the product improves in a fairly linear way. Listening to customers becomes a critical skill of the product team. Product improvements are touted as “listening to customers” and things seem to go well. This predictability is comforting for the business and for customers.

That is, until one day when needs change or perhaps in addition a new product from a competitor is released. Seemingly out of nowhere the great feedback loop we had looks like it won’t help. If we’re fortunate enough to be in tune to changing dynamics outside our core (and growing) customer base we have time to react and change our own product’s trajectory.

That’s when innovation looks like this:

New Product

This is a time when the market is receptive to a different point of view, and a different product — one that redefines, or reimagines, the category. Sometimes customers don’t even realize they are making a category choice, but all of a sudden they are working differently. People just have stuff to get done and find tools that help.

We’re faced with what seems like an obvious choice—adjust the product feature set and focus to keep up with the new needs of customers. Failing to do so risks losing out on new sales, depth usage, or even marginalization. Of course features/capabilities is a long list that can include price, performance, battery life, reliability, simplicity, APIs, different integration points or service connections, and any other attributes that might be used by a new entrant to deliver a unique point of view around a similar scenario.

Many folks will be quick to point out that such is only the case if a new product is a “substitute” for the product people are newly excited about. There is truth to this. But there is also a reality shown time and time again which gets to the heart of tech bets. It is almost always the case that a new product that is “adjacent” to your product has some elements of more expensive, more complex in some dimensions, less functional, or less than ideal. Then what seems like an obvious choice, which is to adjust your own product, quickly looks like a fool’s bet. Why would you chase an inferior product?  Why go after something that can’t really replace you?

The examples of this are too numerous to count. The iPhone famously sucked at making phone calls (a case where the category of “mobile phone” was under reinvention and making calls turned out to be less important). Solid State storage is famously more expensive and lower capacity than spindle drives (a case where the low power, light weight, small size are more valued in mobile devices). Of course tablets are famously unable to provide apps to replace some common professional PC experiences (a case where the value of mobility, all day battery life, always connected seem more valued than a set of platform capabilities). Even within a large organization we can see how limited feature set cloud storage products are being used actively by employees as “substitutes” for enterprise portals and file shares (a case where cross-organization sharing, available on the internet, and mobile access are more valued than the full enterprise feature set). The list goes on and on.

As product managers we all wish it was such a simple choice when we face these situations. Simply leapfrog the limited feature set product with some features on our profitable product. Unfortunately, not every new product that might compete with us is going to disrupt us. So in addition to facing the challenges of evolving the product, we also have to decide which competitors to go after. Often it takes several different attempts by competitive products to offer just enough in the way of new / different approaches to begin to impact an established product.

Consider for example of how much effort the Linux community put into desktop Linux. And while this was going on, Android and iOS were developed and offered a completely different approach that brings new scenarios to life. A good lesson is that usually a head-on alternative will quite often struggle and might even result in missing other disruptive technologies. Having a unique point of view is pretty important.

The reality of this situation is that it is only apparent in hindsight. While it is going on the changes are so small, the product features so minimal, and the base of the customers choosing a new path so narrow that you don’t realize what is going on. In fact, the new product is also on an incremental innovation path, having attained a small amount of traction, and that incremental innovation rapidly accumulates. There is a tipping point.

That is what makes acting during such a “crisis” so urgent. Since no one is first all the time (almost by definition when you’re the leader), deciding when and how to enter a space is the critical decision point. The irony is that the urgency to act comes at a time when it appears from the inside to be the least urgent.

Choosing to innovate means accepting the challenges

We’ve looked at the landscape and we’ve decided as a team that our own product needs to change course. There is a real risk that our product (business) will be marginalized by a new entry adjacent to us.

We get together and we come up with the features and design to go after these new scenarios and capabilities.

The challenge is that some of what we need to do involves changing course—this is by definition what is going on. You’re Apple and you decide that making phone calls is not the number 1 feature of your new mobile phone or your new tablet won’t run OS X apps. Those are product challenges. You also might face all sorts of challenges in pricing, positioning, and all the things that come from having a stable business model. For example, your competitor offers a free substitute for what you are selling.

The problem is your existing customers have become conditioned to expect improvements along the path we were traveling together. Worse, they are by definition not expecting an “different” product in lieu of a new version of their favorite product. These customers have built up not just expectations, but workflows, extensions, and whole jobs around your product.

But this is not about your existing and best customers, no matter how many, it is about the foundation of your product shifting and you’re seeing new customers use a new product or existing customers use your product less and less.

Moving forward the product gets built and it is time to get it into market for some testing or maybe you just release it.

BOOM!

All that work your marketing team has done over the years to establish what it means to “win” in the space that you were winning is now used against you. All the “criteria” you established against every competitor that came along are used to show that the new product is not a winning product. Except it is not winning in the old way. What you’ve done is become your own worst enemy.

But even then, the new way appears to be the less than optimal way—more expensive, less features, more clicks, or simply not the same at doing things the product used to do.

The early adopters or influential users (that was an old term in the literature, “IEU” or sometimes “lead user”) are immediately taken aback by the change in direction. The workflows, keystroke memory, add-ins, and more are just not the same or no longer optimal–there’s no regard for the new scenarios or capabilities when the old ones are different. Worse, they project their views across all customer segments. “I can’t figure this out, so imagine how hard it will be for my parents” or “this will never be acceptable in the enterprise” are common refrains in tech.

This happens no matter who a product is geared towards or how complex the product was in the first place. It is not how it does anything but the change in how it did things people were familiar with. This could be in user experience, pricing, performance, platform requirements or more.

You’re clearly faced with a set of choices that just don’t look good. In Lean Startup, Eric Ries talks in detail about the transition from early users of a new product to a wider audience. In this context, what happens is that the early users expect (or tolerate) a very different set of features and have very different expectations about what is difficult or easy. His conclusion is that it is painful to make the transition, but at some point your learning is complete and it is time to restart the process of learning by focusing on the broader set of customers.

In evolving an existing product, the usage of a pre-release is going to look a lot like the usage of the current release. The telemetry proves this for you, just to make this an even more brutal challenge. In addition, because of the years of effort the enthusiasts put into doing things a certain way and all that work establishing criteria for how a product should work, the obvious thing to do when testing a new release is to try everything out the old release did and compare to the old product (the one you are changing course of) and then maybe some new stuff.  This looks a lot like what Eric describes for startups. For products in market, the moment is pretty much like the startup moment since your new product is sort of a startup, but for a new trajectory.

Remember what brought us here, two things:

  • The environment of usage or business around the product was changing and a bet was made that changes were material to the team. With enough activity in the market, someone will always argue that this change is different and the old and new will coexist and not cannibalize each other (tell that to PalmPilot owners who swore phones would be separate from calendar and contacts, or GPS makers who believe in stand-alone units, or…).
  • A reminder that if Henry Ford had asked customers what they wanted from a car they would have said a faster horse. The market was conditioned to ask for and/or expect improvements along a certain trajectory and no matter what you are changing that trajectory.

All the data is flowing in that shows the new product is not the old product on the old path. Not every customer is interested in doing new things, especially the influential testers who generally focus on the existing ways of doing things, have domain expertise, and are often the most connected to the existing product and all that it encompasses. There is an irony in that for tech these customers are also the most tech-savvy.

Pretty quickly, listening to customers is looking exceedingly difficult.

If you listen to customers (and vector back to the previous path in some way: undo, product modes, multiple products/SKUs, etc.) you will probably cede the market to the new entrants or at least give them more precious time. If technology product history is any guide, pundits will declare you will be roadkill in fairly short order as you lack a strategic response. There’s a good chance your influential customers will rejoice as they can go back and do what they always did.  You will then be left without an answer for what comes next for your declining usage patterns.

If you don’t listen to customers (and stick to your guns) you are going to “alienate” folks and cede the market to someone who listens. If technology product history is any guide, pundits will declare that your new product is not resonating with the core audience. Pundits will also declare that you are stubborn and not listening to customers.

All of this is monumentally difficult simply because you had a successful product. Such is the price of success. Disrupting is never easy, but it is easier if you have nothing to lose.

Many folks will be quick to say that new products are fine but they should just have the old product’s way of doing things. This can seem like asking for a Prius with a switch to turn off the battery (my 2002 Prius came with a training DVD, parking attendant reference card, and more!). There are many challenges with the “side by side” approach. The most apparent is that it only delays the change (meaning delays your entry into the new market or meeting of new scenarios). Perhaps in a world of cloud-services this is more routine where you have less of a “choice” in the change, but the operational costs are real. In client code/apps the challenge becomes very quickly doing things twice. The more complex the changes are the more costly this becomes. In software nothing is free.

Product development is a social science.

People and time

In this numbered conversation, “disrupt or die” there are a few factors that are not often discussed in detail when all the debates happen.

First, people adapt. The assumption, especially about complex tech products, is that people have difficulty or lack of desire to change. While you can always overshoot the learning people can or are willing to do, people are the most adaptable part of a system. One way to think about this is that every successful product in use today, those that we all take for granted, were introduced to a customer base that had to change behavior.  We would not be where we are today without changing and adapting.  If one reflects, the suboptimal change (whether for the people that are customers or the people running a business) is apparent with every transition we have made.  Even today’s tablets are evidence of this.  Some say they are still for “media consumption” and others say they are “productivity tools”.  But behind the scenes, people (and developers) are rapidly and actively changing and adapting to the capabilities of tablets because the value proposition is so significantly improved in some dimensions.

Second, time matters. Change is only relative to knowledge people have at a moment in time and the customers you have at the moment. New people are entering the customer base all the time and there is a renewal in skills, scenarios, and usage patterns. Five years ago almost no one used a touch screen for very much.  Today, touch is a universally accepted (and expected) input method.  The customer base has adapted and also renewed around touch.  Universities are the world’s experts at understanding this notion of renewal. They know that any change to policy at a university is met with student resistance (especially in the spring). They also know that next year, 25% of the “customer base” will be replaced. And in 3 summers all the students on campus will only know the new way. One could call that cynical. One could also call that practical.

Finally time means that major product change, disruption, is always a multi-step process. Whether you make a bet to build a new product that disrupts the market dynamics or change an existing product that disrupts your own product, it rarely happens in one step.  Phones added copy/paste and APIs and even got better at the basics.  The pivot is the tool of the new endeavor until there is some traction.  Feedback, refinement, and balancing the need to move to a new space with the need to satisfy the installed base are the tools of the established product “pivoting” in response to a changed world.  It takes time and iteration–just the same way it took time and iteration to get to the first summit.  Never lose sight of the fact that disrupting is also product development and all the challenges that come from that remain–just because you’re disrupting does not mean what you do will be perfect–but that’s a given we all work with all the time.  We always operate knowing there is more change to come, improvements and fixes, as we all to learn by shipping.

Part of these factors almost always demonstrate, at least in the medium term, that disruption is not synonymous with elimination.  Those championing disruption often over-estimate progress towards elimination in the short term.  Though history has shown the long term to be fairly predictable.  Black cars are still popular.  They just aren’t the only cars.

Product development choices are based on social science. There is never a right answer. Context is everything.  You cannot A/B test your way to big bets or decisions about technology disruption. That’s what makes all of this so fun!!

Go change the rules of the game!

–Steven Sinofsky

Note.  I believe “disrupt or die” is the name of a highly-regarded management class at General Electric’s management school.


07 May 15:46

MobaXterm

MobaXterm:

MobaXterm is an enhanced terminal for Windows with an X11 server, a SSH client and several other network tools for remote computing (VNC, RDP, telnet, rlogin). MobaXterm brings all the essential Unix commands to Windows desktop, in a single portable exe file which works out of the box.

07 May 12:02

git? tig!

by Antoine Büsch

I’m a big fan of Git, but I’m not such a big fan of most UIs for it, especially the ones integrated into IDEs. I find them convoluted and confusing. They try to map some generic “VCS” language onto the commands, or try to hide too much, making it hard to understand what’s going on. Or worse: they’re written in Tcl/Tk…

In short, I don’t trust ‘em.

So command line it is for me, which is fine because I love my command line. Except once in a while, it’s nice to be able to see a “graphical” view of your history, or to have a bit of help when preparing your commits.

Enter tig. tig is a text-mode, ncurses-based UI for git written by Jonas Fonseca. In short, it is the mutt to your Outlook, the Vim to your Emacs, the w3m to your Firefox. I use it all the time, but I haven’t seen many people using it so I thought I’d publicize it a bit. So, why is it awesome?

It’s insanely fast

Seriously. Three letters to type, and it instantly shows up. No fussing around with your mouse, or alt-tabbing to another window, or waiting for a JVM to start. It’s there at your fingertips whenever you need it. It literally loads the 50,000 commits of the JIRA codebase in a fraction of a second.

History view

The default view when launching tig is the history view. It’s basically a git log, with a bit of ASCII-art to represent the history (you can hide the graph with g). Without any argument, it displays the current branch. You can also pass it the name of a branch, or --all for all branches. You can also pass it a range of commits as git log would expect (e.g. tig master..branch). You can navigate through the commits with arrow keys or j/k. Use the / key to search (match is done on summary or author).

Pressing Enter on a commit opens that commit in a split diff view.

tig-2

You can then navigate in the diff view using the j/k keys. At the beginning of the diff is the list of files that the commit touched. If you select one of them and press Enter, you will directly jump to the beginning of that file in the diff. Press q to close the diff view (and press q a second time to quit tig).

If you press t on a commit in the history view, you will open the tree view. In this view, you browse the content of your repository in exactly the state it was at that point in time (just use the arrow keys and Enter), without having to do a git checkout.

Status view

The is the killer feature in my opinion, and the one I use the most. If you start it with tig status, or if you press shift-S in the main view, you will go to the status view. This is like git status, but interactive. You can see the files that you’ve modified and staged, the files you’ve modified but haven’t staged yet, and the files that are no yet tracked by git. If you select a file that is not staged or untracked and press u, it will stage it. If you select a staged file and press u, it will unstage it. No need to remember that pesky git reset HEAD <file> command. You can also press ! on a file to revert all uncommitted changes to that file.

If you press Enter on a file, you will see a diff view of the changes you’ve made to that file (press q to close that view), making it easy to review what you want to commit or not. Even better: you can navigate in the diff with the j/k keys (line by line), or the @ key (chunk by chunk). If you then press u it will only stage the current diff chunk (a section in the diff that starts with @@). You can also stage a single line with 1. It’s all the power of git add -i, except easy to use.

When you are ready to commit,  press shift-C. and it will open your favourite editor to enter the commit message (make sure you close any open diff view first).

tig-4

Blame view

Want to know who messed up your beautiful code or introduced that nasty bug? Just do tig blame <file>. It is similar to git blame or Git -> Annotate in IDEA. Just select a line and press Enter, and it will show you the last commit that touched that line.

tig-6

Pager mode

Got a fancy git log command that you’ve tweaked to display just the information you want? Just pipe its output into tig to have it nicely colorised, and easy to scroll back and forth. Also, if you’re on a line that contains a commit hash, you can just press enter to view that commit. This should work for most Git commands.

How do I get it?

If you use OS X, you can easily install it via homebrew, and most Linux distributions should have a package for it. If not, you can always get the source from http://jonas.nitro.dk/tig/ and compile it yourself. I think it is even available for Cygwin, so there’s no reason not to try it!

Don’t hesitate to look at the man page, or press h inside tig: There’s a lot more you can do with it.

06 May 17:24

Reaching for harmony in org changes

by Steven Sinofsky

HarmonyReorgs are a part of an organization of any size. As business changes, development teams resize, code evolves, or products pivot, the organization can and should change as well. Given the frequency and challenges of reorgs it is worth looking a bit at the complexity, rationale and some challenges of reorganization. While the first reaction to a reorg could range from a sigh of relief to  groan or worse, the most important thing is to keep calm and make sure the work continues.

Be sure to take our three question survey on reorgs after reading this post, here (https://www.surveymonkey.com/s/WS8TNMP) and to check out survey results below from the last survey about “Meeting effectively”.

A first-year  MBA student I recently met took the occasion of a reorg as time to career pivot and attend business school, which motivated this post. 

Reorgs (this post is about structural and management changes, not changes in staffing levels) are sometimes a popular topic in blogs where they take on a certain level of drama or mystique (for example, some blogs talk about org changes as solutions to perceived design challenges). Lacking context, some tend to see reorgs as either the solution to or the cause of a change in strategy or execution. That itself can be the source of reorg angst. In practice, a reorg should be the outcome of a strategic decision not the decision itself-—reorgs don’t cause change or things to happen, but are (hopefully) a better way to execute on strategic changes that have been decided upon.

Reorgs can be a natural way to make sure a team is aligned to deliver on a strategy and a tool to allocate resources effectively towards a shared product plan. When done well, reorgs go from something that happens to you to something that happens with and for you, even if things don’t always feel that way for every member of the team at the start. At the same time, reorgs are enormously challenging by their very nature–organizations are never perfect and there can always be unpredictable outcomes as members of the team implement org changes.

I’ve been part of and executed a few “big” reorgs and always find them incredibly challenging, humbling, stressful, and much more work than is often expected. That’s why I tend to view reorgs as a tool of final resort rather than a tool to routinely drive change, which was something discussed on another blog a while back (and motivated this post).  Executing a reorg involves doing everything you can to “precompute” actions, reactions, and further reactions as best you can while also compensating for them in the plan.

Reorgs are complex and can be thought of from many perspectives. As blunt as they might sometimes seem, there is a great deal of subtlety and nuance to reorgs. While we’re focused on product development organizations, the concepts and implications of reorgs are a pretty general topic.

Reaching for harmony is something to strive for in any organizational change.

Do keep in mind, like so many things in the social science of business, organization and reorganization context dependent—there’s no right or wrong outside the context being discussed.  By definition, reorgs are forward looking and so past history might not always be the best guide.

Perspective and context

Discussing a (potential) reorg can stretch many in an organization. Much like the group describing an elephant, a reorg can mean very different things to different people. A good way to think of things is to refer to a well-known description of organization dynamics that is often used in training classes: tops, middles, bottoms. We’ll return to this often in this blog as it is always a good reminder of patterns and practices that one can generally (emphasis on generally) see repeated.

Bottoms are the folks that do the work. Of course this is an awful moniker, but is the one chosen in the original work (See http://www.powerandsystems.com/resources-a-thought-starters/books/the-possibilities-of-organization.html). Bottoms also make up the bulk of an organization. In a typical, large, development organization (>100) you usually need fewer than 20% of the team middles and tops, which means more than 80% of your resources are bottoms. Whenever possible, you probably want to be better than that (meaning fewer managers, though one should caution a metric like this should not be abused as a scorecard goal as context matters).

Middles are the line managers in an organization. Middles are where the work and collaboration get defined, where friction is either created or eliminated in getting work done, and where information can flow freely or stop. Healthy middles are an essential part of any organization. It is why practices such as skip-level 1:1s, communication that goes broadly to the middles, and shared view of plans are all such critical tools in a product team-—those are the tools of middles managing up and across a team (emphasis on helping the middles, not the middles helping the tops, which is a common dysfunction). Middles can also be tops. For example, if you are the most senior developer in an organization and your manager is not a developer then when it comes to development stuff you are a top.

Tops are the big bosses in an organization. The top is where a certain organization function “ends”. You can be the boss of product design, the boss of the test schedule, the boss of marketing, or (but not necessarily) the boss of the whole organization or company. It is worth noting that nearly all tops are also middles at some point. It just depends on the context. CEOs are middles relative to the board (and also Customers). Your VP is a middle relative to the CEO even if you don’t think of him/her as a middle.

To be complete, the framework also includes Customers. Their role in will be touched on later in the post.

I would encourage folks to check out this framework and book just because it succinctly sums up many of the core challenges within an organization. While there are many insights and many specifics to teams, a key understanding is that members of a team should do far more to understand each other’s context (and problems) than they do in practice during times of change–simply walk in each other’s shoes. Of course this is blindingly obvious, yet terribly difficult for even the best folks on a team. For example:

  • Tops should sometimes spend less time worrying about their big strategic views and needs and consider how their choices (based on those needs) can ripple through an organization and impact execution. Tops would do well to listen more (see this great discussion of 1:1s from Ben Horowitz) and perhaps worry less about what is on their mind.
  • Middles might spend more time talking to other middles and sharing what they are actually doing, what are their real execution issues, and how they are really progressing. All too often middles get caught up communicating idealized situations and plans which can cause confusion, misplaced bets, or just poor choices in other parts of a team and organization. Middles might spend too much energy on describing problems rather than solutions, or even trying to account for things not going well. Middles can spend more time informing their tops about what is going on, but that also depends on tops spending time listening or asking to be informed.
  • Bottoms might also spend more time listening or asking questions and a little less time feeling like “victims”. It is easy when middles and tops are communicating poorly to assume the worst or to assume folks don’t know what is going on. It is equally challenging if the communication that does take place is not taken advantage of, so more listening here can be beneficial as well.

If you think about these typical patterns (remember, this is a generalized sociology framework not a description of your team/behavior), one can see how any discussion of reorgs can quickly degrade. In fact, few things tap into the typical patterns of this behavior framework better than a reorg. Why is that?

Reorgs, by definition, are usually kicked off by the tops. So out of the gate the assumptions that go into making an org change are from a top perspective. The biggest changes in a reorg generally affect the middles since work is reassigned, people’s responsibility changes, and so on. Middles have a tendency to view reorgs at the extreme of “whatever” or “oh my gosh this is really messed up” — as a middle so much of your role depends on context, connections, and structure changes can significantly impact execution.

For the bottoms, a reorg can appear like a bunch of people rearranging deck chairs on the Titanic since ultimately the organization doesn’t really change all the work of individuals (much of the same code still needs to get written, tested, maintained and changing the people with that expertise seems the opposite of progress). Throughout the process, communication is less than and often later than many would like or expect.

The process of a reorganization is one where perspectives of each on the team need to come together to define the problem, scope the alternatives, and implement the solution. Absent these steps a reorg goes from a potential solution to a certain problem.

Why reorg?

There are many reasons for doing an org change. In fact, the most important first step of a reorg is to be able to articulate to those who ask why you might do a reorg.

It is often in this very first step where most reorgs hit a snag. The reason is because the tops have a set of reasons in their context about what a change is for and what it will accomplish and then quickly find out others don’t share the perspective (or problems) or view it as incomplete. Yet the process often continues.

For the tops, this can be a real pain or just frustrating and worse it can bring out the worst of bottoms and middles in terms of how they dig in their heels and get defensive about the change. They begin to immediately dispense the reasons why a reorg won’t work and the bottoms pick up on these and start to feel like victims. All the while the process keeps moving forward.

Reorgs are typically instituted for a pretty common set of reasons, some of which on their own can cause people to retreat to a defensive or cynical state of mind.  Some common drivers include:

  • Resource efficiency. The role of management is to effectively allocate resources and in fact is really often the only tool management has. As a product and team evolve, resource allocations that seemed perfect at one point can seem less than optimal. An organization change has the potential to allocate resources more effectively towards the problems as they are today.
  • Duplication of efforts. In any organization of size, over time efforts will start to converge or overlap. This is especially true in technology companies. This can be at a very visible level, for example if many groups are working on basic tools for editing photos or user names. This can also be at an infrastructure level such as how many teams have people buying servers or running labs.
  • Individual bandwidth. Sometimes teams or responsibility grow and the management of the work becomes too challenging or individuals are spread across multiple projects too frequently. Managers at any level can systematically have too many direct reports, for example. Alternatively, the product line can change or evolve over time and folks on the team find themselves context switching between somewhat unrelated projects more than actually managing. This lack of bandwidth becomes a problem for the team overall as everyone evolves to having more overhead than work.
  • Structural challenges. Organizations evolve over time in a way that suits the time, problem space, and skills. Sometimes when you take a step back, the current state ends up being suboptimal going forward. The alignment of resources, decision making, even core roles and responsibilities are not yielding the results. More often than not, this type organizational pain is felt broadly by the team or by customers.
  • Synergy / Strategy. The notion of increased synergy or strategic change generally drives the most challenging of org changes. Many are familiar with these challenges-—the effort to move large blocks of work in sort of an architectural view. Motivation is this sort often is about “proximity” or “relationship” and has the feel of architecting a product except it is about the team that builds the product.  There’s a tendency to create “portfolios” of products and teams when organizing along these lines.
  • Alignment.  Alignment is slightly different than synergy/strategy in that it speaks to how the organization should be viewed moving forward.  A long time ago, for example, the Office team at Microsoft shifted from building Office “apps” to building the Office “suite”.  Alignment also could include many mechanical elements of businesses/products like customer definition, business models, ship dates, and so on.

Even though these have the potential to sound Dilbert-esque, the reality is that when problems are identified that most people on a team share, then these can form the basis of not just a useful reorganization but a reorg that people want to do. Each one of these motivations (and others not listed) can serve as the basis of a successful reorg. That might not reduce the stress, uncertainty, or even dislike of a change but it does say that reorgs do not have to be a priori negative or random for a team.

Ultimately, changes to an organization should be rooted in getting more and better work done. Few would disagree with that. The question is really whether the team believes an org change will do that. It sounds easy enough.

Challenges

Even with the best of initial intentions, reorgs can (and often) do hit rough spots. Rarely are reorgs stopped once started (just as it is rare that products are stopped once under development). It is a good idea to have a taxonomy of why reorgs can hit snags or challenges, since it is likely they will.

The question is not how do you avoid these necessarily, but how do you identify a specific hiccup the reorg is going through (much like how you identify problems in product development and address them) rather than just stopping. This preparation should take on elements of chess-play as changes and reactions are mapped out and reconsidered based on feedback. Some potential challenges include:

  • Rushing. A potential failure with any reorg is rushing. The funny thing is that the tops usually don’t think they are rushing and everyone else feels things are going too fast. During a reorg process most tops think it is dragging on forever and are just in closure mode simply because tops have likely been thinking about the reorg for quite some time already and most other people have not. In practice, most people only get a short time to hear, absorb, and reflect on the potential change. Skipping a communication and feedback step or skipping deep 1:1 conversations in a consistent and thoughtful manger can make for a very tricky reorg. When people feel the changes are rushed, the process loses structural integrity.
  • Reasoning. Failure to effectively communicate the rationale commonly plagues reorgs. Think of a reorg like any “launch” in that you want to be clear, concise, and appeal the folks with your message. If your message is not the problem your customers have then only challenges follow. The reasoning should appeal to the people who will experience the changes—the organization is what most people in a job and on a team experience day in and day out so reasoning needs to resonate with them. Reorgs announcements that leave too many questions as “exercises for the reader” might be viewed cynically and folks might believe that not enough thought has gone into the change.
  • Strategy. Sometimes a reorg is being done in place of a strategy– “when all else fails, lets reorg” is how victims of such a reorg might characterize things. Reorgs are not a substitute for a strategic choice an organization must make. In fact, a reorg is a tool to use after you have made a strategic choice. Hearing objections to reorgs based on differences in strategy is a real warning sign that the first order problem has not been addressed. If the team has a strategic choice to make (less people, fewer managers, align products, etc.) then first make that choice, then decide if a reorg is needed to accomplish the choice. More often than not, clarifying and then making a strategic choice is the more difficult, but useful, way to spend energy.
  • Timing. A complaint bottoms and middles might raise about a reorg is when it happens—“the timing isn’t right”. A complaint many tops might have with reorgs is that everyone is always telling them the timing is wrong. In practice reorgs can be like a “stand down” for a product team. For some period of time, proportional to the number of people who change managers and/or responsibility, the team will effectively stop working. Therefore no matter how urgent the rationale, the timing of a reorg needs to minimize the impact on the work. On big teams, org inefficiencies trickle on to a team throughout a product cycle (no matter how long or short) due to people coming/going or even things like acquisitions. Unless the point of the reorg is to pivot the product, the potential loss of time to market due to a reorg is a high price to pay.
  • New problems. Any reorg can and will introduce new problems. A common technique for middles is to quickly identify the things that get “more difficult” or for bottoms to ask “well who will do X now”. From a top driving a reorg these often look like self-preservation rather than constructive input. It is a safe bet that almost everything one hears at this time is going to come become issues as middles and bottoms know their jobs. Even if it is presented in a selfish manner, the reality is that tops are not in touch enough with all the details of the work to just keep moving without adjusting. There’s a real balance to understanding what new problems are introduced in any org change and the impact those problems might have on the work.
  • Too much change, too little problem. If the reasoning of a change is not sound for most people or there is a lot of feedback about strategy then there’s a chance that the reorg being executed is outsized relative to the problem. The feedback loop in this case is really pointing to an incomplete problem definition or simply a solution that doesn’t match the problem. This is a case where listening to the feedback can be especially enlightening.
  • Fatigue. Reorgs can also be too much of a (good) thing. Teams can grow tired of the churn that comes from reorgs and enter a state of reorg fatigue. Finding the right cadence for org changes and finding the ability to get the reorg done and over with are important parts of an effective process.  When more than one person starts sending mail saying how many managers or office moves they have had, then it might be time to consider this challenge.
  • Org distance.  Getting work done every day is how most people will evaluate an org change.  The “org distance” between routine collaborators and resources is one measure commonly used.  Org changes can potentially run into resistance when people perceive the changes mean they are “further away” from those they work with routinely.  Commonly people will just count the org intersection point and see how far it moves or how different it becomes.
  • Accountability for the present and future.  Ultimately any organization needs to land clearly with who is accountable for what.  This is a statement about specific people, code, and job functions.  Every accountability has a “30,000 foot” view as well as an “on the ground” view.  It is usually accountability at the detail level that matters in terms of selling through an org change.  People will naturally want to know who “decides” which is another way of asking who is accountable.  To answer who is accountable also requires one to answer where the resources are that “own” the code, designs, tests, etc.  The transition from the present and all the work in flight to the future is a key part of any reorg effort.
  • Leadership and people.  One of the most challenging aspects of reorgs, particularly those that are about restructuring, is the ripple effect on staffing.  At each level of the change, leaders need to be put in place.  Some might be the existing leaders and others might be new.  The image of musical chairs can come to mind, which is always stressful.  Alternatively it is entirely possible to create an organization where there are more jobs of a certain type than people to fill them, which is equally stressful.  As is always the case, making sure that when roles are created the people filling them are truly the right choice for the intended role is paramount.  A new organization that is poorly staffed gets off to a challenging start.

In addition to these conceptual challenges, there are always potential pitfalls with respect to the process of reorganization. The tools of communication, listening, planning, empathy, adapting, are all absolutely critical. My own efforts at blogging started as part of the learning, sharing, and feedback loop for the team as we geared up for Windows 7 development (see our book) and re-organization. Blogging was one tool of many, but an effective way to drive a two-way dialog about changes (many posts were the result of questions or follow-up).

Finally, accountability for a reorg rests with management, specifically the line manager driving the org change. Reorgs are not something HR does for or on behalf of management. HR has valuable tools and a position of objectivity to assist, but they are not accountable or there to drive the process, pick up the pieces, or otherwise appear out in front of a reorg. A way to think of this is that as a manager resource allocation is your primary tool, therefore you can’t really delegate org design and implementation because it is a primary job function—-it is like a developer outsourcing coding (wait didn’t we recently read about the dev that did that?). A common source of frustration is when someone is referred to HR when they raise issues about the goals and execution of a reorg.

Tools

While there are many human resources and management tools to support the communication, feedback, and discussion of a reorg, there are also some specific work management needs to do in order to drive an effective process. A big part of the use of these tools is the contribution from a large set of people on the team who are enrolled in driving the change.

The initial burden for getting things going well falls to the tops to communicate clearly. The reasons for implementing an org change need to be clear and resonate with the team and discussed separately from the solution. This is the problem statement and explains the why behind a reorg. The first sign of skipping steps in a reorg is that the first words, slide or paragraph show a reporting structure. Any reorg that leads with reporting structure is likely to be hit head-on with resistance. Of course most people will be anxious and want to know the structure first anyway, but as a leader of a reorg there is a real responsibility to explain the problems being solved first. This is not burying the real news because the real news is management waking up to a problem that needs to be solved.

With those two tools in place (the why and what), there are a few other tools that can help smooth over what is bound to be an emotional change for a team.

  • How. The next thing to identify is how the work will get done. This is not the job of the top at a very gross “whole product” level but at the level down to some granular level that shows the implications of an org change are understood. Even in the largest organization, understanding at a level of 10-15 developers (engineers, marketing people, etc.) is really an acid test for knowing if an org change has been thought through.
  • Who. The funny thing about reorgs is that the success of them depends on the most local of variables-—individuals want to know what they work on and how the org affects their work, their career, and their place on the team. This is the “who” of a change. In an information based team (software!) this is your asset, not the code. So failing to really understand the who of an org change is going to make it rough. For this tool you need to enlist the help of managers throughout the team to make sure everyone is clear on who does what.
  • When. The timeline of a reorg is critical for everyone. You need to take the time and yet not drag it out. How you balance this depends on the scope of the change and size of the organization.

Whenever a reorg is taking place, whether people agree or not, ultimately the members of the team will want to know about their own careers, skills, knowledge, and place in the new structure.  As much as reorgs are about the big picture, successful reorgs are about the individuals that do the bulk of the work on any product team.

Or not…

One more tool for reorgs is simply not to do them. As strange as that sounds, the reality is that no organization is perfect and even if an organization is perfect it won’t remain so for very long just because of the dynamic nature of product development and teams. People move around, features become more or less important as the technology landscape changes, some areas require more resources than planned or some require less, business models change and more.

This is why more often than not a reorg might not be the best place to spend the team’s limited energy. Reorgs have the potential to substitute activity for progress and can cause an organization to be looking inward right when it needs to be outward focused the most.

That’s not always the case, but it certainly is worth considering.

Yet that doesn’t cure any problems a team or organization might be having. What are some changes tops can initiate or help to drive that can be substitutes for addressing the root cause of challenges that might be equally challenging but perhaps focus on the root cause more than an org change? Here are some examples for tech teams:

  • Align planning and execution. Any time an organization has more than two products (or projects) that connect to each other (two unique products, front end/back end, etc.) there could be a need for alignment. The easiest way to have alignment is to align the planning and execution calendars. Teams that are joined by a calendar have the easiest time working together when it comes to hard decisions like what code to write and when. This alignment needs to be supported by tops–meaning once the bet is made to align, then you have to work within the constraints of release cadence, scope of product, external communication, and more.  The converse of this is that putting teams together that have different schedules does not bring alignment–alignment in product design and code sharing essentially requires some degree of schedule alignment (at least in my experience).
  • Process alignment. Teams that do the same things but do them differently will always have a hard time working together. From even abstract things like roles and responsibilities to extremely concrete things like how to categorize bugs or deliver daily builds, differences in processes can really make it hard to work together. A good thing to do is pick the processes that matter most to your orgs and just align (perhaps see Managing through disagreement).
  • Strategic choice. Perhaps the real problem is not one that can be solved by organization at all and an org change is a substitute for a strategic choice (exit or enter a business, combine businesses, etc.) In this case, as painful as it may be, the org change only pushes accountability and delegates responsibility for something that should just be decided.
  • Decide to share code. The hardest thing for dev teams to do is share code with other dev teams—50 years after the invention of the subroutine. Yet it is magical when teams do commit to doing so. How to share code effectively and how to manage the provider and consumer roles, especially in a complex org in many businesses, is an art form, but one that needs to perfected, locally. As we all know, sharing code is great and also constraining–so again support from the broader perspective regarding additional constraints is critical.  Sharing code is also a lot easier if teams are aligned on planning and execution timelines.

Implementing a reorg is a big step. It is always wise to think first about your problem statement and decide if you can attack the root cause in a much less disruptive way. This is especially true in a large organization where changing things “at the top” has much less of an impact on product evolution than many believe.

Customers

The Oshry framework also includes customers. Customers of course define the reason for making products in the first place.

The biggest challenge any multi-product organization faces is that customers want products and technologies (relevant to them, keeping in mind many products serve many different customer types) to appear to work together. From the outside, that is the customer perspective, when products don’t appear to work together or appear to have arbitrary differences/redundancies then the obvious culprit is the org. The org was not structured to work on that problem as an integrated whole. This can be seen as “shipping the org chart”.

In this case, the org chart for the products is not right-—some things need to be “closer” or “one person” needs to be in charge of a couple of products. This goes a step further. When the design or quality is not right according to customers then the org is not right because the designers or testers were not organizationally working closely enough with developers.

You can see this multi-dimensional problem. It all boils down to graph theory and how you can connect all the parts of all of the products with the highest bandwidth and always connected flow of information, decisions, and more.  This means it is much more difficult than it appears to use organization to address these perceived challenges.  The side-effects of moving some things closer include moving other things farther apart, and the implications of the solution might be worse than the problem.

In the idealized world of small teams you can get everyone in the same conference room and decide everything. This tops out at about 40 -50 people. For example, Excel 5 had about that many developers. After that, organization is a tool that can help you to overcome this limitation. While it would be great to work on product families that always take fewer people, that isn’t always possible just on the basis of the number of features it takes to be competitive in the market place over any period of time.

The substitute of anointing someone to oversee all aspects of a product is also a scale challenge. There are just so many hours in a day and only so many people that might fill such a role (if that is even possible to do). Once a person is managing a large number of related, but different, projects or just a large number of people then the ability for the large/complex team to act like a small team is limited. In other words, just joining two entities at the top does not necessarily mean they will appear to work better together for customers.

Yet, what everyone wants to avoid is a dynamic where your collective efforts result in “shipping the org chart” to customers.

Since you have to have an organization, which might be divided by geography, discipline, products, architectural layer, product release timing, business models, or more, the real tools to avoid shipping the org chart are planning, communication, and accountability. You can really never solve the multi-dimensional matrix of responsibility without making teams so large or structurally complex, or relying on a superhero manager that any value that might come from being on one team is lost.  The converse to this is that designing products by a committee doesn’t work either. Just taking a lot of complexity and sort of saying “work it out” usually fails to be optimal for any customer.

Because of the complexity of org changes in a large team, the best lesson I have learned is that a culture that adapts to solving problems turns out to be the best organization structure. Combine that with common views of roles/responsibilities, clear and reliable plans, and accountability and you can have the makings of an agile and flexible organization that can move work around, partner across projects, and deliver without using org structure as a high-order bit for strategic change.

–Steven Sinofsky

Be sure to check out this week’s survey on org changes https://www.surveymonkey.com/s/WS8TNMP.

Thanks for everyone that responded to our survey for “Using meetings to be more effective”.  In this survey, we hoped to learn together about the tools and characteristics that make meetings successful.

 Here are the results:

  • About half of our most recent meetings include a phone bridge, with about one third connecting via Voice over IP (i.e. Skype)
  • In about one in six meetings, at least one person joins via a cell phone
  • About half of our meetings take advantage of screen sharing and about half involve PowerPoint, though only in about one third was a projector used
  • When asked about whether our last meeting was a success, on average (mean and median) we “neither agree nor disagree” that it was a success

In looking at drivers for what made us rate a meeting a success, there were some interesting findings:

  • Regarding technologies, of the technologies queried (phone, cell, VoIP, screen sharing, PowerPoint, projector, and meeting software), only the use of a projector had a statistically significant impact on our success rating.  However, meetings with a projector ranked half a point lower on a five point scale, than those without projectors
  • Interestingly, presenters rated meetings with projectors lower than members of the audience, with a difference of about a half point, it’s worth noting this was not correlated with slideshow software like PowerPoint
  • Of the tips for success discussed,  “a fully understood context” drove the success factor up a third-point , and  a “concise” meeting (brevity) drove success up nearly a half-point.
  • Interestingly, presenters rated meetings with “a fully understood context” higher than members of the audience

Bottom Line:

Modern meetings leverage online tools like to get everyone on the same page, though care should be taken during in-person meetings to not let the audio/visuals detract from your message as a presenter.  Taking time before and during the meeting to create a shared sense of context and keeping your message concise seem to drive the best outcomes for everyone, presenter and audience alike.

Thanks, Cameron

###


02 May 20:11

Learning by slipping

by Steven Sinofsky

Countdown“Slipping” or missing the intended completion or milestone date of software projects is as old as software itself. There’s a rich history of our industry tracking intended v. actual ship dates and speculating as to the length of the slip and the cause.  Even with all this history, slipping is a complex and nuanced topic worth a bit of discussion about slipping as an engineering concept.

Slipping

I’ve certainly had my fair share of experience slipping.  Projects I’ve worked on have run the full spectrum from landing exactly on time to slipping 20-30% from the original date.  There’s never a nice or positive way to look at slipping since as an engineer you’re only as good as your word.  So you can bet the end of every project includes a healthy amount of introspection about the slip.

Big software projects are pretty unique.  The biggest challenge is that large scale projects are rarely “repeated” so the ability to get better through iteration keeping some things constant is limited.  This is different than building a bridge or a road where many of the steps and processes can be improvements from previous projects.  In large scale software you rarely do the same thing with the same approach a second or third time.

While software is everywhere, software engineering is still a very young discipline that rapidly changes.  The tools and techniques are wildly different today than they were just a few years ago.  Whether you think about the languages, the operating systems, or the user experience so much of what is new software today is architected and implemented in totally new ways.

Whenever one talks about slipping, at some basic level there is a target date and a reality and slipping just means that the two are not the same (Note:  I’ve yet to see a software project truly finish early).  There’s so much more to slipping than that.

What’s a ship date

In order to slip you need to know the ship date.  For many large scale projects the actual date is speculation and of course there are complexities such as the release date and the availability date to “confuse” people.  This means that discussions about slipping might themselves be built on a foundation of speculation.

The first order of business is that a ship date is in fact a single date.  When people talk about projects shipping “first quarter” that is about 90 different dates and so that leaves everyone (on the team and elsewhere) guessing what the ship date might be.  A date is a date.  All projects should have a date.  While software itself is not launching to hit a Mars orbit, it is important that everyone agree on a single date.  Whether that date is public or not is a different question.

In the world of continuously shipping, there’s even more of a challenge in understanding a slip.  Some argue that “shipping” itself is not really a concept as code flows to servers all the time.  In reality, the developers on the team are working to a date—they know that one day they come to work and their code is live which is a decidedly different state than the day before.  That is shipping.

Interestingly, the error rate in short-term, continuous projects can often (in my experience) be much higher.  The view of continuously shipping can lead to a “project” lasting only a month or two.  The brain doesn’t think much of missing by a week or two, but that can be a 25 – 50% error rate.  On a 12 month project that can mean it would stretch to 15-18 months, which does sound like a disaster.

There’s nothing about having a ship date that says it needs to be far off.  Everything about having a date and hitting it or slipping can apply to an 8 week sprint or a 3 year trek.  Small errors are a bigger part of a short project but small errors can be amplified over a long schedule.  Slipping is a potential reality regardless of the length of the schedule.

The key thing from the team’s perspective about a ship date is that there is one and everyone agrees.  The date is supported by the evidence of a plan, specifications, and the tools and resources to support the plan.  As with almost all of engineering, errors early in the process get magnified as time goes by.  So if the schedule is not believable or credible up front, things will only get worse.

On the other hand, a powerful tool for the team is everyone working towards this date.  This is especially true for collaboration across multiple parts of the team or across different teams in a very large organization.  When everyone has the same date in mind then everyone is doing the same sorts of work at the same time, making the same sorts of choices, using the same sorts of criteria.  Agreeing on a ship date is one of the most potent cross-group collaboration tools I know.

Reasons to slip

Even with a great plan, a team on the same page, and a well-known date, stuff can happen.  When stuff happens, the schedule pressure grows.  What are some of the reasons for slipping?

  • Too much work, aka “we picked too much stuff”.  The most common reason for slipping is that the team signed up to do more work than could be done.  The most obvious solution is to do less stuff.  In practice it is almost impossible to do less once you start (have you ever tried to cut the budget on a kitchen remodel once it starts?  You cut and cut and end up saving no money but costing a lot of time.) The challenge is the inter-connected nature of work.  You might want to cut a feature, but more often than not it connected to another feature either upstream or downstream.
  • Some stuff isn’t working, aka “we picked the wrong architecture”.  This causal factor comes from realizing that the approach that is halfway done just won’t work, but to redo things will take more time than is available.  Most architecturally oriented developers in this position point to a lack of time up front thinking about the best approach.  More agile minded developers assume this is a normal part of “throw away the first version” for implementing new areas.  In all cases, there’s not much you can do but stick with what you have or spend the time you don’t have (i.e. slipping).
  • Didn’t know what you know now, aka “we picked the wrong stuff”.  No matter how long or short a project, you’re learning along the way.  You’re learning about how good your ideas were or what your competitors are doing, for example.  Sometimes that learning tells you that what you’re doing just won’t fly.  The implications for this can run from minimal (if the area is not key) to fairly significant (if the area is a core part of the value).  The result in the latter case can be a big impact on the date.
  • Change management, aka “we changed too much stuff”.  As the project moves forward, things are changing from the initial plans.  Features are being added or removed or reworked, for example.  This is all normal and expected.  But at some point you can get into a position where there’s simply been too much change and the time to get to a known or pre-determined is more than the available time.

The specifics of any slip can also be a combination of these and it should be clear how these are all interrelated.  In practice, once the project is not on a schedule all of these reasons for slipping begin to surface.  Pretty soon it just looks like there’s too much stuff, too much is changing, and too many things aren’t “right”.

That is the nature of slipping.  It is no one single thing or one part of a project.  The interrelationships across people, code, and goals mean that a slip is almost always a systemic problem. Recognizing the nature of slipping leads to a better understanding of project realities.

Reality

In reality, slips are what they are and you just have to deal with them.  In software, as in most other forms of engineering, once you get in the position of missing your date things get pretty deterministic pretty quickly.

In the collective memories of most large projects that slipped are the heroes or heroic work that saved a project.  That could very well happen and does, but from a reliable or repeatable engineering perspective these events are circumstantial and hard to reproduce project over project.  Thus the reality of slipping is that you just have to deal with it.

The most famous description of project scheduling comes from Frederic P. Brooks who authored “The Mythical Man-Month” in 1975.  While his domain was the mainframe, the ideas and even the metrics are just as relevant almost 40 years later.  His most famous aphorism about trying to solve a late project by adding resources is:

When a task cannot be partitioned because of sequential constraints, the application of more effort has no effect on schedule.  The bearing of a child takes nine months, no matter how many women are assigned.

Software projects are generally poorly partitioned engineering – much like doing a remodel in a tiny place you just can’t have all the different contractors in a tiny place.

There are phases and parts of a project in large scale software that are very amenable to scale with more resources, particularly in testing and code coverage work, for example.  Adding resources to make code changes runs right up against the classic man-month reality. Most experienced folks refer to this as “physics” implying that these are relatively immutable laws.  Of course as with everything we do, context matters (unlike physics) and so there are ways to make things work and that’s where experience in management and most importantly experience as a team working together on the code matters.

The triad of software projects can be thought of as features, quality, and schedule.  At any given point you’re just trading off against each of those.  But if it were only that easy.

Usually it is easy to add features at the start, unaware of precisely how much the schedule or quality will be impacted.  Conversely, changing features at other times becomes increasingly difficult and obviously so.  From a product management/program management perspective, this is why feature selection, feature set understanding, and so on is so critical and why this part of the team must be so crisp at the start of a project.  In reality, the features of a product are far less adaptable than one might suspect.  Products where features planned are not delivered can sometimes feel incomplete or somehow less coherent.

It is almost impossible to ever shorten a schedule.  And once you start missing dates there is almost no way to “make up for time”.  If you have an intermediate step you miss by two weeks, there’s a good chance the impact will be more than two weeks by the end of a project.  The developers/software engineers of a project are where managing this work is so critical.  Their estimates of how long things will take and dependencies across the system can make or break the understanding of reality.

Quality is the most difficult to manage and why the test leadership is such a critical part of the management structure of any project.  Quality is not something you think about at the end of the project nor is it particularly malleable.  While a great test manager knows quality is not binary at a global level, he/she knows that much like error bars in physics a little bit of sub-par quality across many parts of the project compounds and leads to a highly problematic, or buggy, product.  Quality is not just bugs but also includes scale, performance, reliability, security, and more.

Quality is difficult to manage because it is often where people want to cut corners.  A product might work for most cases but the boundary conditions or edge cases show much different results.  As we all know, you only get one chance to make a first impression.

On a project of any size there are many moving parts.  This leads to the reality that when a project is slipping, it is never one thing—one team, one feature, one discipline.  A project that is slipping is a product of all aspects of a project.  Views of what is “critical path” will need to be reconciled with reality across the whole project, taking into account many factors.  Views from other parts of the organization, the rumor mill, or just opinions of what is holding up the project are highly suspect and often as disruptive to the project as the slip itself.  That’s why when faced with a slipping project, the role of management and managing through the slip is so critical.

What to do

When faced with a slip, assuming you don’t try to toss some features off the side, throw some more resources at the code, or just settle for lower quality there are a few things to work on.

First and foremost, it is important to make sure the team is not spending energy finger pointing.  As obvious as that sounds, there’s a natural human tendency to avoid having the spotlight at moments like this.  One way to accomplish that, improperly, is to shine the light on another part of the project.  So the first rule of slipping is “we’re all slipping”.  What to do about that might be localized, but it is a team effort.

What else can be done?

  • Don’t move the goalposts (quality, features, architecture).  The first thing to do is to avoid taking drastic actions with hard to measure consequences.  Saying you’re going to settle for “lower quality” is impossible to measure.  Ripping out code that might not work but you understand has a very different risk profile than the “rewrite”.  For the most part, in the face of slipping the best thing to do is keep the goals the same and move the date to accomplish what you set out to do.
  • Think globally, act locally. Teams will often take actions that are very local at times of slipping.  They look to cut or modify features that don’t seem critical to them but have important upstream or downstream impact, sometimes not well understood on a large project.  Or feature changes that might seem small can have a big impact on planned positioning, pricing, partnerships, etc.  The approach of making sure everyone is checking/double checking on changes is a way to avoid these “surprises”.
  • Everyone focuses on being first, not avoiding being last.  When a project has more than a couple of teams contributing and is faced with a tight schedule, there’s a tendency for a team to look around to just make sure they are not the team that is worse off.  A great leader I once worked with used to take these moments to remind every part of the project to focus on being first rather than focusing on being “not last”.  That’s always good advice, especially when followed in a constructive manner.
  • Be calm, carry on. Most of all, slipping is painful and even though it is all too common in software, the most important thing to do during crunch time is to remain calm and carry on.  No one does good work in a panic and for the most part the quality of decisions and choices degrades if folks are operating under too many constraints that can’t get met.  It is always bad for business, customers, and the team to slip.  But if you are slipping you have to work with what you’ve got since most of the choices are usually even less desirable.

Managing a software project is one of the more complex engineering endeavors because of the infinite nature of change, complexity of interactions, and even the “art” that still permeates this relatively new form of engineering.  Scheduling is not yet something we have all mastered and slipping is still a part of most large projects.  The more that Software Eats the World ($), the more the challenges of software project management will be part of all product and service development.

Given that, this post tried to outline some of the causes, realities, and actions one could take in the face of learning by slipping.

–Steven Sinofsky


27 Apr 13:40

Two New Videos About Testing at Google

by Google Testing Bloggers
by Anthony Vallone

We have two excellent, new videos to share about testing at Google. If you are curious about the work that our Test Engineers (TEs) and Software Engineers in Test (SETs) do, you’ll find both of these videos very interesting.

The Life at Google team produced a video series called Do Cool Things That Matter. This series includes a video from an SET and TE on the Maps team (Sean Jordan and Yvette Nameth) discussing their work on the Google Maps team.

Meet Yvette and Sean from the Google Maps Test Team



The Google Students team hosted a Hangouts On Air event with several Google SETs (Diego Salas, Karin Lundberg, Jonathan Velasquez, Chaitali Narla, and Dave Chen) discussing the SET role.

Software Engineers in Test at Google - Covering your (Code)Bases



Interested in joining the ranks of TEs or SETs at Google? Search for Google test jobs.