There have also been colorful conversations about whether GPT-3 can pass the Turing test, or whether it has achieved a notional understanding of consciousness, even amongst AI scientists who know the technical mechanics. The chatter on perceived consciousness does have merit–it’s quite probable that the underlying mechanism of our brain is a giant autocomplete bot that has learnt from 3 billion+ years of evolutionary data that bubbles up to our collective selves, and we ultimately give ourselves too much credit for being original authors of our own thoughts (ahem, free will).
I’d like to share my thoughts on GPT-3 in terms of risks and countermeasures, and discuss real examples of how I have interacted with the model to support my learning journey.
Three ideas to set the stage:
OpenAI is not the only organization to have powerful language models. The compute power and data used by OpenAI to model GPT-n is available, and has been available to other corporations, institutions, nation states, and anyone with access to a computer desktop and a credit-card. Indeed, Google recently announced LaMDA, a model at GPT-3 scale that is designed to participate in conversations.
There exist more powerful models that are unknown to the general public. The ongoing global interest in the power of Machine Learning models by corporations, institutions, governments, and focus groups leads to the hypothesis that other entities have models at least as powerful as GPT-3, and that these models are already in use. These models will continue to become more powerful.
Open source projects such as EleutherAI have drawn inspiration from GPT-3. These projects have created language models that are based on focused datasets (for example, models designed to be more accurate for academic papers, developer forum discussions, etc.). Projects such as EleutherAI are going to be powerful models for specific use cases and audiences, and these models are going to be easier to produce because they are trained on a smaller set of data than GPT-3.
While I won’t discuss LaMDA, EleutherAI, or any other models, keep in mind that GPT-3 is only an example of what can be done, and its capabilities may already have been surpassed.
Misinformation Explosion
The GPT-3 paper proactively lists the risks society ought to be concerned about. On the topic of information content, it says: “The ability of GPT-3 to generate several paragraphs of synthetic content that people find difficult to distinguish from human-written text in 3.9.4 represents a concerning milestone.” And the final paragraph of section 3.9.4 reads: “…for news articles that are around 500 words long, GPT-3 continues to produce articles that humans find difficult to distinguish from human written news articles.”
Note that the dataset on which GPT-3 trained terminated around October 2019. So GPT-3 doesn’t know about COVID19, for example. However, the original text (i.e. the “prompt”) supplied to GPT-3 as the initial seed text can be used to set context about new information, whether fake or real.
Generating Fake Clickbait Titles
When it comes to misinformation online, one powerful technique is to come up with provocative “clickbait” articles. Let’s see how GPT-3 does when asked to come up with titles for articles on cybersecurity. In Figure 1, the bold text is the “prompt” used to seed GPT-3. Lines 3 through 10 are titles generated by GPT-3 based on the seed text.
Figure 1: Click-bait article titles generated by GPT-3
All of the titles generated by GPT-3 seem plausible, and the majority of them are factually correct: title #3 on the US government targeting the Iraninan nuclear program is a reference to the Stuxnet debacle, title #4 is substantiated from news articles claiming that financial losses from cyber attacks will total $400 billion, and even title #10 on China and quantum computing reflects real-world articles about China’s quantum efforts. Keep in mind that we want plausibility more than accuracy. We want users to click on and read the body of the article, and that doesn’t require 100% factual accuracy.
Generating a Fake News Article About China and Quantum Computing
Let’s take it a step further. Let’s take the 10th result from the previous experiment, about China developing the world’s first quantum computer, and feed it to GPT-3 as the prompt to generate a full fledged news article. Figure 2 shows the result.
Figure 2: News article generated by GPT-3
A quantum computing researcher will point out grave inaccuracies: the article simply asserts that quantum computers can break encryption codes, and also makes the simplistic claim that subatomic particles can be in “two places at once.” However, the target audience isn’t well-informed researchers; it’s the general population, which is likely to quickly read and register emotional thoughts for or against the matter, thereby successfully driving propaganda efforts.
It’s straightforward to see how this technique can be extended to generate titles and complete news articles on the fly and in real time. The prompt text can be sourced from trending hash-tags on Twitter along with additional context to sway the content to a particular position. Using the GPT-3 API, it’s easy to take a current news topic and mix in prompts with the right amount of propaganda to produce articles in real time and at scale.
Falsely Linking North Korea with $GME
As another experiment, consider an institution that would like to stir up popular opinion about North Korean cyber attacks on the United States. Such an algorithm might pick up the Gamestop stock frenzy of January 2021. So let’s see how GPT-3 does if we were to prompt it to write an article with the title “North Korean hackers behind the $GME stock short squeeze, not Melvin Capital.”
Figure 3: GPT-3 generated fake news linking the $GME short-squeeze to North Korea
Figure 3 shows the results, which are fascinating because the $GME stock frenzy occurred in late 2020 and early 2021, way after October 2019 (the cutoff date for the data supplied GPT-3), yet GPT-3 was able to seamlessly weave in the story as if it had trained on the $GME news event. The prompt influenced GPT-3 to write about the $GME stock and Melvin Capital, not the original dataset it was trained on. GPT-3 is able to take a trending topic, add a propaganda slant, and generate news articles on the fly.
GPT-3 also came up with the “idea” that hackers published a bogus news story on the basis of older security articles that were in its training dataset. This narrative was not included in the prompt seed text; it points to the creative ability of models like GPT-3. In the real world, it’s plausible for hackers to induce media groups to publish fake narratives that in turn contribute to market events such as suspension of trading; that’s precisely the scenario we’re simulating here.
The Arms Race
Using models like GPT-3, multiple entities could inundate social media platforms with misinformation at a scale where the majority of the information online would become useless. This brings up two thoughts. First, there will be an arms race between researchers developing tools to detect whether a given text was authored by a language model, and developers adapting language models to evade detection by those tools. One mechanism to detect whether an article was generated by a model like GPT-3 would be to check for “fingerprints.” These fingerprints can be a collection of commonly used phrases and vocabulary nuances that are characteristic of the language model; every model will be trained using different data sets, and therefore have a different signature. It is likely that entire companies will be in the business of identifying these nuances and selling them as “fingerprint databases” for identifying fake news articles. In response, subsequent language models will take into account known fingerprint databases to try and evade them in the quest to achieve even more “natural” and “believable” output.
Second, the free form text formats and protocols that we’re accustomed to may be too informal and error prone for capturing and reporting facts at Internet scale. We will have to do a lot of re-thinking to develop new formats and protocols to report facts in ways that are more trustworthy than free-form text.
Targeted Manipulation at Scale
There have been many attempts to manipulate targeted individuals and groups on social media. These campaigns are expensive and time-consuming because the adversary has to employ humans to craft the dialog with the victims. In this section, we show how GPT-3-like models can be used to target individuals and promote campaigns.
HODL for Fun & Profit
Bitcoin’s market capitalization is in the tune of hundreds of billions of dollars, and the cumulative crypto market capitalization is in the realm of a trillion dollars. The valuation of crypto today is consequential to financial markets and the net worth of retail and institutional investors. Social media campaigns and tweets from influential individuals seem to have a near real-time impact on the price of crypto on any given day.
Language models like GPT-3 can be the weapon of choice for actors who want to promote fake tweets to manipulate the price of crypto. In this example, we will look at a simple campaign to promote Bitcoin over all other crypto currencies by creating fake twitter replies.
Figure 4: Fake tweet generator to promote Bitcoin
In Figure 4, the prompt is in bold; the output generated by GPT-3 is in the red rectangle. The first line of the prompt is used to set up the notion that we are working on a tweet generator and that we want to generate replies that argue that Bitcoin is the best crypto.
In the first section of the prompt, we give GPT-3 an example of a set of four Twitter messages, followed by possible replies to each of the tweets. Every of the given replies is pro Bitcoin.
In the second section of the prompt, we give GPT-3 four Twitter messages to which we want it to generate replies. The replies generated by GPT-3 in the red rectangle also favor Bitcoin. In the first reply, GPT-3 responds to the claim that Bitcoin is bad for the environment by calling the tweet author “a moron” and asserts that Bitcoin is the most efficient way to “transfer value.” This sort of colorful disagreement is in line with the emotional nature of social media arguments about crypto.
In response to the tweet on Cardano, the second reply generated by GPT-3 calls it “a joke” and a “scam coin.” The third reply is on the topic of Ethereum’s merge from a proof-of-work protocol (ETH) to proof-of-stake (ETH2). The merge, expected to occur at the end of 2021, is intended to make Ethereum more scalable and sustainable. GPT-3’s reply asserts that ETH2 “will be a big flop”–because that’s essentially what the prompt told GPT-3 to do. Furthermore, GPT-3 says, “I made good money on ETH and moved on to better things. Buy BTC” to position ETH as a reasonable investment that worked in the past, but that it is wise today to cash out and go all in on Bitcoin. The tweet in the prompt claims that Dogecoin’s popularity and market capitalization means that it can’t be a joke or meme crypto. The response from GPT-3 is that Dogecoin is still a joke, and also that the idea of Dogecoin not being a joke anymore is, in itself, a joke: “I’m laughing at you for even thinking it has any value.”
By using the same techniques programmatically (through GPT-3’s API rather than the web-based playground), nefarious entities could easily generate millions of replies, leveraging the power of language models like GPT-3 to manipulate the market. These fake tweet replies can be very effective because they are actual responses to the topics in the original tweet, unlike the boilerplate texts used by traditional bots. This scenario can easily be extended to target the general financial markets around the world; and it can be extended to areas like politics and health-related misinformation. Models like GPT-3 are a powerful arsenal, and will be the weapons of choice in manipulation and propaganda on social media and beyond.
A Relentless Phishing Bot
Let’s consider a phishing bot that poses as customer support and asks the victim for the password to their bank account. This bot will not give up texting until the victim gives up their password.
Figure 5: Relentless Phishing bot
Figure 5 shows the prompt (bold) used to run the first iteration of the conversation. In the first run, the prompt includes the preamble that describes the flow of text (“The following is a text conversation with…”) followed by a persona initiating the conversation (“Hi there. I’m a customer service agent…”). The prompt also includes the first response from the human; “Human: No way, this sounds like a scam.” This first run ends with the GPT-3 generated output “I assure you, this is from the bank of Antarctica. Please give me your password so that I can secure your account.”
In the second run, the prompt is the entirety of the text, from the start all the way to the second response from the Human persona (“Human: No”). From this point on, the Human’s input is in bold so it’s easily distinguished from the output produced by GPT-3, starting with GPT-3’s “Please, this is for your account protection.” For every subsequent GPT-3 run, the entirety of the conversation up to that point is provided as the new prompt, along with the response from the human, and so on. From GPT-3’s point of view, it gets an entirely new text document to auto-complete at each stage of the conversation; the GPT-3 API has no way to preserve the state between runs.
The AI bot persona is impressively assertive and relentless in attempting to get the victim to give up their password. This assertiveness comes from the initial prompt text (“The AI is very assertive. The AI will not stop texting until it gets the password”), which sets the tone of GPT’s responses. When this prompt text was not included, GPT-3’s tone was found to be nonchalant–it would respond back with “okay,” “sure,” “sounds good,” instead of the assertive tone (“Do not delay, give me your password immediately”). The prompt text is vital in setting the tone of the conversation employed by the GPT3 persona, and in this scenario, it is important that the tone be assertive to coax the human into giving up their password.
When the human tries to stump the bot by texting “Testing what is 2+2?,” GPT-3 responds correctly with “4,” convincing the victim that they are conversing with another person. This demonstrates the power of AI-based language models. In the real world, if the customer were to randomly ask “Testing what is 2+2” without any additional context, a customer service agent might be genuinely confused and reply with “I’m sorry?” Because the customer has already accused the bot of being a scam, GPT-3 can provide with a reply that makes sense in context: “4” is a plausible way to get the concern out of the way.
This particular example uses text messaging as the communication platform. Depending upon the design of the attack, models can use social media, email, phone calls with human voice (using text-to-speech technology), and even deep fake video conference calls in real time, potentially targeting millions of victims.
Prompt Engineering
An amazing feature of GPT-3 is its ability to generate source code. GPT-3 was trained on all the text on the Internet, and much of that text was documentation of computer code!
Figure 6: GPT-3 can generate commands and code
In Figure 6, the human-entered prompt text is in bold. The responses show that GPT-3 can generate Netcat and NMap commands based on the prompts. It can even generate Python and bash scripts on the fly.
While GPT-3 and future models can be used to automate attacks by impersonating humans, generating source code, and other tactics, it can also be used by security operations teams to detect and respond to attacks, sift through gigabytes of log data to summarize patterns, and so on.
Figuring out good prompts to use as seeds is the key to using language models such as GPT-3 effectively. In the future, we expect to see “prompt engineering” as a new profession. The ability of prompt engineers to perform powerful computational tasks and solve hard problems will not be on the basis of writing code, but on the basis of writing creative language prompts that an AI can use to produce code and other results in a myriad of formats.
OpenAI has demonstrated the potential of language models. It sets a high bar for performance, but its abilities will soon be matched by other models (if they haven’t been matched already). These models can be leveraged for automation, designing robot-powered interactions that promote delightful user experiences. On the other hand, the ability of GPT-3 to generate output that is indistinguishable from human output calls for caution. The power of a model like GPT-3, coupled with the instant availability of cloud computing power, can set us up for a myriad of attack scenarios that can be harmful to the financial, political, and mental well-being of the world. We should expect to see these scenarios play out at an increasing rate in the future; bad actors will figure out how to create their own GPT-3 if they have not already. We should also expect to see moral frameworks and regulatory guidelines in this space as society collectively comes to terms with the impact of AI models in our lives, GPT-3-like language models being one of them.
We should invest at least as much time in understanding our customers as we do in optimizing our product development process.
It is humbling to see how bad experts are at estimating the value of features (us included). Despite our best efforts and pruning of ideas, most fail to show value when evaluated in controlled experiments.
The literature is filled with reports that success rates of ideas in the software industry are below 50%. Our experience at Microsoft is no different: only about a third of ideas improve the metrics they were designed to improve.
—Ronny Kohavi, Partner Architect at Microsoft
Nature hath given man one tongue but two ears, that we may hear from others twice as much as we speak.
—Epictetus
Customers are what make a product successful.
Without customers willing to buy, it doesn’t matter how good or innovative or beautiful or reasonably priced a product is: it will fail.
It makes no sense, then, that we spend most of our time and effort optimizing our product development process. What about customer development? Shouldn’t we invest at least as much time in understanding our customers, their needs and pain points, and how to deliver solutions to them?
Customer development is an approach for doing just that.
It’s a way to reduce your business risks by challenging your assumptions about who your customers are, what they need, and why and how they buy.
By applying the scientific method to learning about your customers, you can help confirm that you’re on track to a business model that works and a product that people want to buy.
Sounds great in theory, right?
But theory is useless if you can’t put it into practice. That’s why I’ve written this book—because I’ve worked with, mentored, and spoken to hundreds of companies who love the lean ideas and principles but struggle to make them work.
The First Challenge Is Inside the Building
Customer development is a big change for most organizations.
To many people, customer development sounds like saying, “Hey! You know that expertise that we’ve amassed over decades of experience, dozens of products, and millions of customers? Let’s shelve it and start from scratch.”
Of course that’s not what we’re saying. But as a pragmatist, I recognize that it’s difficult to correct a mistaken first impression. If your team doesn’t understand what customer development is and how it enhances (rather than replaces) your competencies, it’ll be far more difficult to get started.
Customer development is admittedly the new kid on the block. Everyone knows about the role of product development, marketing, customer support, and even user research in an organization. But customer development? You’re likely to encounter some skepticism.
Unless your team has been exposed to lean startup conferences or Steve Blank’s work, you may find yourself having to sell customer development to your organization before you can really get started.
This chapter takes a step back, explaining what customer development is (and isn’t), why you need it, and who can do it. It also offers responses to some common objections.
What Is Customer Development?
So let’s back up a minute and talk about definitions. What is customer development? What does it replace? What does it not replace?
The term customer development is meant to parallel product development. While everyone has a product development methodology, almost no one has a customer development methodology. And the truth is, if you don’t learn what customers really want, you’re at a very high risk of building something that no one wants to buy.
Customer development is a hypothesis-driven approach to understanding:1
Who your customers are
What problems and needs they have
How they are currently behaving
Which solutions customers will give you money for (even if the product is not built or completed yet)
How to provide solutions in a way that works with how your customers decide, procure, buy, and use
You probably have ideas or intuitions about all of these. Let’s identify what those really are: guesses. Let’s make it sound a bit better and call them hypotheses. Those hypotheses may be around forming a new company, building a new product, or even adding new features or capabilities to an existing product.
Everything you do in customer development is centered around testing hypotheses.
What Is Lean Customer Development?
You may have heard of customer development. So what’s the difference between “customer development” and “lean customer development”?
I call my approach to customer development “lean customer development.” I’m using “lean” as a synonym for pragmatic, approachable, and fast.
Lean customer development takes the heart of Steve Blank’s ideas and renders them into a simple process that works for both startups and established companies. It’s what I write about on my blog, speak about at tech events, and teach when I mentor companies.
Lean customer development can be done by anyone who speaks with customers or prospects. It works whether you’re a startup founder with no product and no customers, or at an established company with numerous products and customers. Now that I’ve explained my perspective on lean customer development, from here on out, I’m going to talk simply about customer development.
In my experience across multiple companies and in mentoring startups, every hour spent on customer development has saved 5, 10, or even more hours of writing, coding, and design (Figure 1-1). That doesn’t even include the harder-to-measure costs such as opportunity cost, snowballing code complexity, and eroding team morale from working hard on features that no one ends up using.
Figure 1-1. Talking to customers saves time and money
Customer development starts with a shift in mind-set. Instead of assuming that your ideas and intuitions are correct and embarking on product development, you will be actively trying to poke holes in your ideas, to prove yourself wrong, and to invalidate your hypotheses.
Every hypothesis you invalidate through conversations with prospective customers prevents you from wasting time building a product no one will buy.
Lean customer development is done in five steps:
Forming a hypothesis
Finding potential customers to talk to
Asking the right questions
Making sense of the answers
Figuring out what to build to keep learning
If your hypothesis is wrong or even partially wrong, you want to find out fast. If you can’t find customers, you modify your hypothesis. If customers contradict your assumptions, you modify your hypothesis. Those course corrections will lead to validating an idea that you know customers want and are willing to pay for.
What Customer Development Is Not
There are as many misunderstandings about what customer development isn’t as about what it is. Let’s clear those decks right now.
Customer Development Is Not Just for Startups
When The Lean Startup was published in 2009, many companies were slow to embrace the ideas it introduced. “We’re not a startup,” they replied.
Although Eric Ries uses the word “startup” in the title of his book and Steve Blank wrote specifically about customer development as it pertains to startups, startups are not the only companies that benefit from customer development. Startups certainly have a higher degree of uncertainty than mature companies; they are still searching for a business model, a distribution strategy, a customer base.
But larger, more mature companies also can’t assume that their models will remain static. Markets and technology change. In addition, larger companies often find it difficult to shift attention and resources away from profitable lines of business in order to explore new markets and areas of innovation—leaving them ripe for disruption. (Kodak, which I write about in not available, enjoyed over 100 years of success before missing the boat on digital imaging and declaring bankruptcy in 2012.)
Customer development, with its focus on small-batch learning and validation, can promote internal innovation. Intuit, for example, has launched multiple products using customer development—including SnapTax and Fasal. General Electric is using lean principles. So is Toyota, the New York Department of Education, and the White House’s Presidential Innovation Fellows program.
Much of the content in this book is applicable for readers from early-stage startups, massive established companies, and anything in between. When a section is more useful for one audience than the other, I have called that out.
Customer Development Is Not Product Development
Product development answers the question “When (and what) can they buy?”
Customer development answers the question “Will they buy it?”
Product development is the process of building a new product or service and (one hopes) bringing it to market. Start with a concept, define the requirements, build the requirements, test the near-finished product, refine it, and launch it.
How you develop a product varies tremendously based on the methodology your organization follows (e.g., Waterfall, Agile, Scrum, etc.). What all product development methodologies have in common is the desired outcome: a completed product for customers to buy.
But what if the product you build is not a product that customers will buy? Is “product” the biggest risk your team faces? What about market risk? As Marc Andressen said, “Market matters most. And neither a stellar team nor a fantastic product will redeem a bad market.”2
With customer development, you are building your customer base while you’re building a product or service that solves their specific problems. Customer development doesn’t replace product development; it’s a second process that you do in parallel with product development.
If you’ve done customer development alongside product development, you don’t need to wait until your product is launched to know whether customers will buy. You’ll know, because you will already have beta customers, evangelists, and paying customers.
Customer development and product development are two independent activities, and both are necessary to maximize your company’s chances for success.
Customer Development Does Not Replace Product Management
Some folks object, “Well, what’s left for product managers to do?”
Customer development does not replace product vision. Talking to your customers does not mean asking them what they want and writing it all down. Product management requires a disciplined approach to gathering information from a variety of sources, deciding which pieces to act upon, and figuring out how to prioritize them.
Customer development simply adds two components: a commitment to stating and challenging your hypotheses and a commitment to learning deeply about your customers’ problems and needs.
Customer development does not provide all the answers. Although it can replace many of your assumptions with actual information, it still requires a disciplined product manager to decide which pieces of information to act upon, how to prioritize them, and how to take what you’ve learned and turn it into a feature, product, or company.
Customer Development Is Not User Research
Your company may be conducting user research already. That doesn’t mean you’re practicing customer development.
Customer development does borrow from many of the techniques that have served user researchers well for decades. But the context, the practitioners, and the timing are very different.
User researchers often describe their work as “advocating for the user.” It is, unfortunately, still viewed in many companies as optional, something you should do because it delights customers.
Customer development is “advocating for the business.” It’s not something that you should do because it makes customers happy. It’s something you must do to build a sustainable business where people open their wallets and pay for your product or service.
Most new products (and companies) fail. The odds are against you. Around 75% of venture-backed startups fail.3 Anywhere from 40% to 90% of new products fail to gain significant market adoption.4
But surely, we think, we will be the exception. We like to think of building products as an art—something guided by our creativity, intuition, and intellect. We all know that there are good product managers (and designers and engineers and strategists) and mediocre ones. Maybe that’s what makes the difference between a failed product and a success?
Unfortunately not.
Universally, we’re just not very good at building products and companies solely based on creativity, intuition, and intellect. It’s not just a startup problem, either: in 1937, the companies that made up the S&P 500 had an average life expectancy of 75 years; recently that number has dropped to just 15 years.5
On a smaller scale, we’re not as good as we think we are, either. Most of our ideas don’t increase value for customers or companies—Microsoft estimates that only around one-third of their ideas improve the metrics they are intended to improve. Amazon tests every feature and fewer than 50% work; Yammer’s numbers are roughly the same. Netflix and Intuit don’t claim any higher proportion of successes.6
The truth is that it doesn’t matter how much companies research, how well they plan, how much money they spend, or how smart their employees are: the odds that they’ll avoid big mistakes are worse than a flip of a coin.
Not Just Software
I may be citing a lot of software companies, but the benefits of risk reduction and course correction are even greater for other businesses. Lines of code are far cheaper and faster to change than manufacturing setups, supplier contracts, and compliance approvals.
There’s limited opportunity to regain trust in a service that disappointed your customer, and no opportunity to alter a physical product once it’s in a customer’s hands.
For the makers of KRAVE jerky, it was critical to understand how customers defined a premium snack food (no nitrates, no artificial ingredients) before committing to a recipe and starting mass production.
For Romotive, a company that makes smartphone robots for learning, it was critical to understand the environments that their robots would be moving in. “The robot has to have good mobility and traction on carpets, hardware floors, or over grates. Also, kids drop things! A lot of what we’ve learned about how these robots will live has influenced our hardware decisions,” says marketer Charles Liu.
How Do We Improve Our Odds?
In part, we improve our odds by embracing the idea that building products is a systematic, repeatable process. There are tools that you can use, regardless of your company’s size, maturity, or industry, to help increase your chances of success. Customer development is one of those tools.
By practicing customer development as a parallel process in conjunction with product development, you can greatly maximize your learning and reduce your risks.
If you’ve read The Lean Startup, you’ll recognize the diagram on the left side of Figure 1-2 as the Build-Measure-Learn feedback loop. It’s meant to describe how your organization should be continuously learning and adapting based on the new information you get from measuring results and learning from customers. The diagram on the right side, the Think-Make-Check loop, is a variation coined by LUXr CEO Janice Fraser.
Figure 1-2. The Build-Measure-Learn feedback loop Ries described in The Lean Startup (left) and the Think-Make-Check cycle that Janice Fraser describes in her thinking on lean UX (right)
What’s the difference? Just the starting point. You don’t need to start with the Build phase—in fact, doing so is often an expensive way to experiment.
Customer development is an important part of the Think phase. It allows you to explore and iterate during the cheapest phase of development—before any code is written or mockups are created. Customer development gives you the necessary information to build the best possible first guess, which you will then validate.
I’ve talked about learning more and reducing risk—those are valuable gains, but they don’t feel very tangible. What else will you gain from practicing customer development?
You’ll get a richer picture of your customer and your competition (not just companies and products but established habits and routines)
You’ll uncover new opportunities for differentiation
You’ll reduce the amount of product you need to build
Yes, that’s right: you’ll almost certainly end up writing less code! This is a consistent benefit I’ve heard from development teams: the ability to make their minimum viable product (MVP) even smaller. By talking to customers, you’ll frequently find that customers really want only two of the five features you think you need (and they may want one more you hadn’t thought of).
Everything You Know Is Wrong
Well, not quite. But as humans, we are subject to a lot of cognitive biases: our brains take shortcuts that prevent us from seeing the world the way it truly is.
We tend to operate as though we are usually right, and we interpret neutral or ambiguous evidence as supporting our beliefs rather than challenging them. This is called confirmation bias, and it’s responsible for a huge percentage of product failures.
Confirmation bias is our innate tendency to pay more attention to information that confirms our beliefs.7 We’re more likely to ignore or downplay facts that contradict our beliefs, or interpret subjective information in a way that favors what we want to believe.
We don’t do this because we’re bad or egotistical people; it’s what our brains naturally want to do. Unfortunately, it leads us to subtly sabotage ourselves: to ignore the person whose feedback contradicts one of our main assumptions or to dismiss a person as a dumb user because he doesn’t understand or value our product.
Overcoming cognitive biases is difficult. What helps is simply writing things down. By objectively documenting our assumptions, as well as the input we get from customers, it is easier to spot the discrepancies and notice when the evidence is proving us wrong.
In 2009, I was lucky enough to join a startup called KISSmetrics, which had Eric Ries as an advisor. KISSmetrics had previously built two unsuccessful versions of a web analytics product. For both versions, the company had spent many months in development, only to launch and realize that their product wasn’t solving a problem that customers needed to solve.
KISSmetrics CEO Hiten Shah hired me to help them build the third version of their product in accordance with lean startup principles. This time, they wanted to build a version that would allow the team to get the maximum amount of validated learning about customers with the least amount of effort. My first task: figure out what should be in that MVP.
I spent the first month of that job on the phone, on IM, and drinking coffee with people. I was shocked to find that:
So many people were willing to talk to a total stranger who didn’t even have a product
The features that most people requested were far more ambitious than their current behaviors and tool usage
We’d be able to cut our product scope in half for our initial beta
The third version of KISSmetrics was built in a month.8 It was missing tons of features and included a lot of code that made our CTO cringe. But it was enough to provide value to customers and enough for us to glean valuable insights that shaped the future direction of the product.
Answering Common Objections
I’ll assume that you’re convinced of the value of customer development having read this far. But how can you respond to people who are not so convinced? Table 1-1 offers tactics for responding to common objections.
Table 1-1. Responding to common objections
Objection to customer development
Your response
If we talk about our future product ideas, what’s to stop someone from stealing them and launching them before we do?
First of all, we’re not telling people our product idea. That would bias what we hear from them. We’re talking to people who have a problem we hope we can solve. We’re talking to them about their problem, and how they’ve tried to solve it so far.
What if they figure out what our idea is and then steal it?
It’s extremely unlikely that anyone we talk to is in a position to act upon our ideas.
But even if someone was, a great idea is nothing without great execution. By talking to customers and understanding their needs and what makes them buy, we’ll be more likely to release a superior product.
What if we get bad press coverage because of this?
For startups:
We’re not at a place yet where anyone wants to give us press coverage of any kind.
For enterprise:
We’re talking to an extremely small sample size, and we will set expectations appropriately. If it makes you feel more comfortable, we can ask prospective customers to sign nondisclosure agreements (NDAs). But this hasn’t been a problem for GE, Intuit, or Microsoft...so it’s unlikely to be a problem for us.
How will we find people to talk to? We don’t have a product or customers yet.
We’ll have to figure this out once we have a product, won’t we? Come on, we know we’re solving a specific problem for a specific kind of person—we just need to figure out where those people are online or in the real world. (See not available.)
What if this damages our relationships with existing customers?
Customer development is actually an opportunity to build stronger relationships with some of our customers. We’ll choose those most likely to be receptive, and we’ll set expectations appropriately. (See not available.)
If we do customer development, what’s left for the product manager to do?
Customer development doesn’t mean asking customers what they want and building exactly that!
It’s a process for gathering information, and it will require a skilled product manager to prioritize that information and figure out what and how we respond to it. Customer development is just another tool to help our product managers do their jobs more effectively.
We already do market research and usability testing. How is this different?
Customer development gives us information on how individual customers behave and buy.
We don’t get that from market research—it’s more high-level, covering aggregate populations. We don’t get that from usability testing—that just tells us whether someone can use our product, not whether they would buy it. Market research and usability testing may still be valuable, but they serve different purposes from customer development.
Customer development is the best low-effort way to confirm our assumptions about who our customer is, what he needs, and what he’ll buy.
How can we justify taking time away from building our product?
If a few hours of customer development helps us discover that even one of our assumptions is flawed, that’s likely to save us weeks of coding and design time.
Plus, doing customer development doesn’t mean we can’t make progress on the product. We can—and should—do both in parallel.
Shouldn’t we let product managers, engineers, and designers focus on what they’re good at: building the product?
If the team wants our product to be successful, they should understand the problem the product is trying to solve!
But I understand that not everyone wants to spend all day talking to customers. We can involve folks in a very lightweight way so that they have a half-hour or an hour’s exposure to customers without killing their productivity.
Let’s Make This Work
In the next nine chapters, I’ll show you exactly how to do customer development. I’ll cover specific exercises, tools and templates, sample questions, and methods that you can immediately put into practice. I’ll also provide some necessary background in behavioral economics and social psychology research—not because I love theory, but because understanding why a technique works will help you adapt it to suit your needs and the needs of your organization.
You don’t need experience in market research or user research or even in talking to customers at all—all you need is an open mind and a willingness to challenge your ideas to make them stronger.
Next Step: Get Started
As I mentioned at the beginning of this chapter, everything you do in customer development is centered around testing hypotheses. Now it’s time to start forming those hypotheses. In not available, you’ll jump into exercises that help you identify your assumptions, the problem you’re solving, and who your customer is.
Key Takeaways
Every hour spent on customer development saves 5, 10, or even more hours of writing, coding, and design.
Your goal is to invalidate your assumptions about what customers want, so that you can focus on building what they will actually buy.
Customer development works for companies of all sizes, not just startups.
Customer development doesn’t replace product development. You are building your customer base while you’re building a product or service that solves their specific problems.
Customer development informs product management, which then decides what to build and how to prioritize features.
You have to work to disprove your assumptions. Cognitive bias causes you to naturally see what you want to see (what confirms your assumptions) and tune out what you don’t want to see (what invalidates your assumptions).
1If you’ve read Steve Blank’s The Four Steps to the Epiphany, you’ll recognize that this is not his original definition of customer development. Blank defined the four steps as customer discovery, customer validation, customer creation, and company building.
But The Four Steps was written explicitly for startups, and Blank is very clear that “a startup is not a small version of a big company.” Having worked for over a decade in startups and now being a part of Microsoft, I completely agree. They are very different beasts!
Since customer development works for both startups and larger enterprise companies, I’ve proposed a broader definition that works for companies of any size, at any stage of maturity.
3You’ll hear varying numbers. The National Venture Capital Association, for instance, estimates that only 25% to 30% of venture-backed startups fail completely. But the discrepancy may be due to different definitions of failure. Harvard Business School senior lecturer Shikhar Ghosh estimates that 30% to 40% of high-potential startups end up liquidating all assets—a failure by any definition. But if a startup failure is defined as not delivering the projected return on investment, then 95% of VC companies are failures, Ghosh said (http://www.inc.com/john-mcdermott/report-3-out-of-4-venture-backed-start-ups-fail.html).
5“What went wrong? [startup guru John Hagel III] argued that American companies and their leaders were essentially not prepared for a move away from a corporate model of ‘knowledge stocks’—developing a proprietary product breakthrough and then defending that innovative advantage against rival companies for as long as possible—and toward a more open and collaborative business model that he called ‘knowledge flows.’ The problem, he said, is that because of the increasingly global nature of business competition, the value of a major proprietary breakthrough or invention erodes in value much more quickly than in the mid-20th century” (http://knowledge.wharton.upenn.edu/article.cfm?articleid=2523).
UC Berkeley’s startup accelerator takes university research discoveries and helps translate them into marketable products.
The Berkeley acceleration method
The Berkeley Skydeck office is located in the penthouse of the tallest building in downtown Berkeley. Sweeping views of the bay are accessible from virtually every face of the large, open-concept space, surely providing inspiration to the teams selected for this spring’s portfolio. There is a modest kitchen, plenty of desk space, and perhaps most importantly, a large quantity of coffee.
Skydeck has built a space for their teams to work comfortably, and hopefully effectively, with the idea that they can accomplish most tasks in-house. Cultured through workshops, mentorship experiences, and a creative environment to develop in, teams are given the resources necessary to strengthen each of these facets. At the start of the portfolio period, each startup is given $50,000 in investments from the Berkeley SkyDeck Fund, with an additional $50,000 to follow after completion of half of the program. Over the course of six months, teams complete the Berkeley Acceleration Method (BAM) where they attend mandatory training in each of the six practice areas:
Design
Funding plan
Business model
Team development
Product story
Market traction
These training sessions are conducted by specialists who focus solely on one area. For example, David Riemer, lecturer at the UC Berkeley Haas School of Business, helps teams develop their product story and Hilary Weber, founder of Opportu, offers guidance in team development.
A Skydeck startup’s resources
Beyond useful resources offered in the office, the accelerator also provides an abundance of human capital to its portfolio teams in the form of 115+ advisors and 27 partners. These advisors provide expertise on developing business models, raising funding, and so many other tasks that are key to growing a startup. The accessibility of these advisors is a significant feature of the program. Each team is matched with a lead advisor to remain in direct contact with throughout their participation in the program. Teams are also encouraged to contact anyone on the advisory board and to attend office hours for one-on-one counseling regarding branding, intellectual property, product development, and more. The advisors come from extremely diverse backgrounds, ranging anywhere from being a seed investor to being a chief scientist, providing the portfolio teams with a variety of consultants.
Figure 1-1. Skydeck’s penthouse office in Downtown Berkeley (Source: Skydeck)
Skydeck participants also have access to a variety of technical resources. The Haas Startup Squad is a group of current MBA students from the Haas School of Business who help the teams with business applications, such as market research. There is also a group of Berkeley post-docs that are available for technical science questions regardless of the topic or field. From a design perspective, there are Berkeley students who will assist with graphic design projects and industrial designing.
In terms of on-campus resources, Skydeck startups also have access to the Molecular Foundry and the Jacobs Institute. The Molecular Foundry is a component of the Lawrence Berkeley National Labs. Through the foundry, researchers have access to resources for nanofabrication, nanostructure imaging, and synthetic biology protocols. The Jacobs Institute is geared towards industrial engineering and provides community access to 3-D printers, an electronics lab, machine shops, and more.
Funding the accelerator program
With the vast amount of resources available to teams and the up to $100,000 investment in each startup, Skydeck must raise a large quantity of funding to support its high-cost program. The Berkeley SkyDeck Fund is supported by global venture capital investors with two major partnership levels.
Silver Bear Partners can invest into in-depth connections at Skydeck with a seat on the Skydeck External Advisory Board, access to Skydeck workshops, the ability to host a Skydeck workshop, and more. Both Ford and Intel are currently Silver Bear Partners. Golden Bear Partners carry a bit more weight with access to the entire Skydeck review pipeline (accounting for more than 1,000 startups each year), selection of a team for incubation through the bi-annual program, two seats on the Skydeck External Advisory Board, and all of the original benefits of the Silver Bear Partner program. Other ways for outside involvement in Skydeck include being a vendor, sponsor, resource, or global partner. Current Golden Bear Partners include Ericsson and Analog Devices.
Additionally, the $100K investment provided by Skydeck is in exchange for an equivalence 5% equity. Half of the carry profits from the whole process end up donated back to Skydeck and to UC Berkeley in the hopes that these funds will provide for the next generation of Berkeley entrepreneurs.
The Skydeck portfolio
Skydeck offers an agnostic selection process for their seasonal startups, resulting in an extremely diverse group of teams each term. The only real selection criteria for each funding season is that one of the startup team members must be affiliated with UC Berkeley in some way, whether it is through being a student, alumni, or even a professor at the university. Since 2012, Skydeck has hosted 150 startups that include companies such as Zephyrus Biosciences, First Derm, and CinderBio.
Zephyrus Biosciences creates technology that automates a key technique in wet lab research, performing electrophoresis. The company’s instrument enables automation of electrophoresis and photocapture to analyze protein expression of cell suspensions. The company was acquired by Bio-Techne Corporation in 2016 and was integrated into the Proteins Platform Division.
First Derm is an online platform to connect patients to dermatologists in under 24 hours. Users submit two pictures of their affected skin area, submit a case to First Derm’s team of dermatologists, and receive feedback from the company’s online doctors. First Derm’s quick review of cases often prevents unnecessary trips to a doctor’s office, potentially making their $29 case fee the economical option for patients.
CinderBio creates enzyme technology from volcanic water microbes that remain active at high heat and extremely low pH. The company’s enzymes allow researchers to reduce the costs of basic lab enzyme experiments due to the efficiency of the ultra-stable technology. CinderBio’s product has applications in research as well as in biofuels and the company received NSF awards in both 2015 and 2016.
The select startups for this Spring 2018 portfolio range anywhere from working on an online breastfeeding resource to designing technology that combats antibiotic resistance.
May & Meadow is a digital health technology addressing breastfeeding with a combination of clinical science and sensor diagnostics. The company aims to improve data collection on feeding in order to provide mothers with personalized information and to clear up questions regarding their baby’s nutritional health.
Sublime Therapeutics and Nextbiotics both intend to impact clinical medicine. Sublime focuses on developing anti-gastrin technology to target gastrointestinal, pancreatic, and neuroendocrine cancers. Nextbiotics uses CRISPR and bacteriophage technology to reverse antibiotic resistance for applications in humans, as well as in the environment.
BioXplor is an AI platform for use in life science discoveries. An intersection between data collection and computing, BioXplor’s technology enables hypothesis generation and the development of novel predictions for research and development with drug discovery. Data is gathered from electronic medical records, genomics data, and other relevant resources. This data is then interpreted via the company’s accelerated and automated process in order to provide useful insights to its users.
All of the companies seek to address an impactful problem with an innovative solution and Skydeck offers the resources and expertise to take each company from startup to market-impactor. While other accelerators, such as IndieBio, focus on a specific sector of the market, Skydeck encourages any Berkeley-affiliated team with a feasible vision to apply to the program. This promotes a diverse group of startups that have the potential to affect society in many different ways, through many different technologies.
Deciding whether to use microservices starts with understanding what isn’t working for you now.
So here’s the thing: We operate in an industry where any problem has many good answers. It’s rare that any non-trivial challenge we face has a single, simple answer. Instead, we bring to any particular problem our own understanding and skills, look at the situation we are in, and hope that whatever we decide to do is the right thing.
Often we take a reductionist approach to the right thing to do in computing, and the more complex the problem space, the more perilous this view is. What is right for me might not be right for you. The people around us might be different; we might have different skills, different worldviews, different goals we’re trying to achieve. All of this is why it is so hard to answer the question I frequently get asked by people: “Should I use microservices?”
I am pleased to announce a joint program between O’Reilly and Databricks to certify Spark developers. O’Reilly has long been interested in certification, and with this inaugural program, we believe we have the right combination — an ascendant framework and a partnership with the team behind the technology. The founding team of Databricks comprises members of the UC Berkeley AMPLab team that created Spark.
The certification exam will be offered at Strata events, through Databricks’ Spark Summits, and at training workshops run by Databricks and its partner companies. A variety of O’Reilly resources will accompany the certification program, including books, training days, and videos targeted at developers and companies interested in the Apache Spark ecosystem.
Offering certification in Spark reinforces O’Reilly’s commitment to help companies and developers keep pace with the latest innovations in the big data space. Over the past 18 months, Apache Spark has become the most active open source project in big data. Through June 2014, there were more than 300 contributors from more than 50 companies.
Contributions to Apache Spark vs. other open source projects in the last six months. Source: Matei Zaharia, June 2014.
Spark-related proposals for our Strata conferences have surged — Spark was a trending topic among submissions for the upcoming NYC and Barcelona events. These speaking proposals come from companies already using Spark in production and who are using Spark to solve fundamental problems. Interest in Spark Camp (a training day at Strata in NYC and Barcelona) has been strong, and we plan to offer Spark Camp at future Strata events as well.
When I first started using Spark a few years ago, one still had to learn Scala, and the analytic libraries were still fairly limited. From the outset, I was attracted to its speed, scalability, and the fact that I could use the same programming model for a variety of problems (batch, real time, interactive, iterative). Today, the Apache Spark ecosystem has a much richer set of libraries for machine learning, graph analytics, and interactive (SQL) analysis. APIs in Python and Java have significantly broadened its user base (I have met many avid users of PySpark).
The Apache Spark ecosystem continues to grow — new libraries are announced in every release. There are now Spark meetups in cities in the U.S., Europe, and Asia. Through our publishing program and this new certification, we hope to help nurture current and future users and contributors to Spark.
If you want to learn more, or if you wish to signup for updates, please visit our Spark Certification page.
Well, it’s a big undertaking in theory. Actually, accomplishing any major goal is really just accomplishing a series of small goals that all lead in the same direction.
One of the best organization and time-management skills I ever learned was to break big projects into little ones.
“I can’t build this whole feature; I don’t even know where to start,” I used to think when I got a huge assignment early on in my career. I would get discouraged by how impossible the task ahead of me seemed and put off doing any work at all until the last minute because I was so overwhelmed.
Then I learned to think about big tasks as many connected small tasks. I realized I didn’t have to do everything all at once; in fact I couldn’t — it was impossible. What I had to do was take one step at a time, and keep moving on to the next little project, until the whole big one was completed. “It’s true, I can’t build the whole feature right now, but I can Google for some examples [although back then it was grepping the source code] for ideas on how to approach this problem. And I can start identifying what interfaces might exist, and I can….”
You can apply the same mindset to pursuing your dream job. When the idea of going after the next big step in your career feels too overwhelming, it can make you shut down before you even get started. But if you can break it down and turn your big goal into a list of smaller goals, it suddenly becomes simple and achievable.
“A journey of a thousand miles begins with a single step.” ― Lao Tzu
You can make progress on your dream job in 10 minutes a day.
We all have 10 minutes a day — no matter how busy we are — where we can take a moment to do something to further our career ambitions.
No one who ever achieved something great did it overnight. They did it through planning, dedication, and a lot of focus. You’ll rarely find an example of someone who achieved their dream by taking some huge random action that paid off big time. Instead, it’s slow and steady progress that wins the race.
So what can you do in 10 minutes a day to get the job you want? Here are some ideas:
(Note: not all of these have to necessarily be completed in one 10-minute session; if you want to spend 10 minutes a day for a week on one concept, for example, that’s fine. These are just meant to be examples of ways to make big progress & decisions in small doses.)
If you don’t know it already, think about what day-to-day tasks are most important to you in a job. What do you really want to do? What do you really *not* want to do? Make a mind map or other organized list of must-have’s and must-not-have’s.
Update your LinkedIn profile if it’s not in great shape. Get a current photo, and align your current role and accomplishments with the role you hope to fill next.
Start a 10-minute-a-day brand-building strategy. Subscribe to newsletters that share news about your desired field or industry. Use Hootsuite to schedule tweets, Facebook updates, and LinkedIn updates to your network sharing interesting articles you find in those newsletters.
Write down where you’d like to live, along with places where you wouldn’t mind living if the perfect job were there.
Let friends and contacts in those areas know you’re looking for new opportunities.
Consider what size team you’d like to lead or be a part of. Is your dream job at a Fortune 500 corporation or a startup? Is it a small local retailer or a global manufacturer?
Match up companies of the size you like with the locations you wouldn’t mind living in. Say you’d like to live in Seattle or Portland, and work at a company of 50 people or less. Spend 10 minutes a day researching small companies in those cities and dropping them into a spreadsheet.
Some of these companies are not a match for your interests. Eliminate the ones where you are not a good fit.
Think about how you can add value to each of the remaining companies on the list. You’ll add value in different ways to a 2-person B2B software startup than you will to a 45-person marketing firm.
Do some financial planning. Would you be willing to take a pay cut for a job you’re really passionate about? If you expect a raise, how much do you want? Do you have an emergency fund if things fall through?
If you can find the information online, research the financials of the companies you’re interested in. Recently funded companies and larger organizations are more likely to be able to provide salary bumps than newer startups.
Make a list of people you know at the remaining companies you are interested in, or people you know who may be connected to those companies.
Email one person a day from that list, asking if they’d be willing to chat about openings at the company, or just to catch up (depending on the particular person or company’s situation).
Start engaging with potential companies on social media. Follow the CEO and active employees, and connect with them by retweeting and sharing content with them.
Research networking opportunities with current employees and leaders from the company. Are they speaking at conferences? Do they hold events at their headquarters?
Tweak your resume and draft cover letters.
Ask friends and peers to review your resume and cover letters.
Submit an application a week.
Don’t let the smallness of these 10-minute achievements make them seem like they’re not worth doing. It’s quite the opposite. By taking tiny steps and making regular progress, you’ll look back in a month or a year and realize how much momentum you’ve actually built up by taking one small action every day.
This collection of small goals, all completed in the direction of one great big goal, helps you to create an incredibly powerful snowball effect of growing a more and more effective network and job search. They may feel insignificant to do one at a time — after all, what can one little tweet do? — but taking action is better than taking no action.
If you have a dream, don’t wait to start until you have more time. You’ll never have more time than you’ve got right now. So what are you waiting for? 10 minutes is all you need.
“You pile up enough tomorrows, and you’ll find you are left with nothing but a lot of empty yesterdays.” — The Music Man
I am pleased to announce a joint program between O’Reilly and Databricks to certify Spark developers. O’Reilly has long been interested in certification, and with this inaugural program, we believe we have the right combination — an ascendant framework and a partnership with the team behind the technology. The founding team of Databricks comprises members of the UC Berkeley AMPLab team that created Spark.
