Rolandt

Imagine someone you’ve never met walks up to you, shows you a copy of a magazine article, and says:

“I’ve recently read this article about [widget]. What do you think about [widget]? Do you think [widget] will be the future of our industry?”

If you’re cornered, you might give a quick reply and then try to move on. The approach just feels a little weird.

The obvious catch with an online community is people aren’t cornered. If the approach is off and there’s no real benefit of replying, people don’t reply.

A lot of newcomer community managers, in their haste to initiate activity, will try to create discussions like this. They will share a relevant article (or raise a topical issue) and ask “what do you think?”

These are the worst types of discussions for almost every community.

People reply to discussions for an emotional payoff. They want to feel useful and know they’ve helped someone. Giving opinions on random topics doesn’t create that feeling. This is why initiating a bunch of discussions to ‘get activity going’ usually fails.

It’s also why in the very early days you need to be very careful how you structure the question.

For example, if you ask the question as:

Subject: Would you set aside a budget for [widget] in 2022?

“Next week I need to submit my budget for the coming year. I’m trying to work out whether it’s worthwhile setting some money aside for [widget] as this article seems to recommend.

Does anyone have any experiences or insights into whether [widget] is likely to be important without our industry? Any examples or case studies would be really useful”

Now we have a question which lets members both give an opinion and feel they’re helping. It’s clear why the person is asking the question and the value they will get from the answers.

The challenge in the early stages isn’t to start discussions to ‘get activity going’. It’s to find people who need help and persuade them to ask questions in the community (or, last resort, ask them on behalf of the person).

The post Stop Creating “What Do You Think?” Discussions first appeared on FeverBee.

Marc Andreessen has changed his tone over the past year; there is a cynical interpretation, but I think the shift is justified.

If the Internet Archive is to be believed, Marc Andreessen’s April 2020 coronavirus-driven exhortation It’s Time to Build never actually included that futuristic picture of skyscrapers; it’s in the metadata so that it shows up on social media like Twitter:

What is even more intriguing is when, exactly, the building happened; after all, Andreessen wrote last week in Technology Saves the World:

Last April, I issued a call to our technology industry that it was time to build — and I am so proud of how we delivered. Please join me in an enthusiastic — virtual! — round of applause for all of the amazing workers in our spectacular technology industry who made all this possible. The experience of COVID has made crystal clear both how important our technology is to human flourishing, and how well we can deliver. Technology helped save the world.

Mission accomplished?

Future Business Models

Andreessen’s new essay was published on Andreessen Horowitz’s (a16z) much-ballyhooed new media property Future; you will recall much of the media establishment losing its mind over the announcement in January; I argued in Publishing is Back to the Future that the announcement seemed to be part of a much larger shift in media from a world of scarcity, where newspapers were profitable thanks to geographic monopolies, to one of abundance, where publications succeeded based on their ability to attract audiences who could visit any website on the Internet:

To put it another way, what the New York Times has become is not so different from what Andreessen Horowitz is proposing to build. Margit Wennmachers said in that introductory post that “Our lens is rational optimism about technology and the future”; as a long time subscriber of the New York Times, I think it is fair to call their lens rational skepticism about technology and its effects. What is notable about both is that their lenses are perfectly aligned with their business models (and, I would note, both claim to be motivated to change the world).

a16z’s business model is, of course, venture capital, and Wennmachers, in an interview with Bloomberg’s Emily Chang, was clear that driving a16z’s venture business was the primary focus of Future:

I’m trying to accomplish a business goal. Our firm wants to advance the future and thinks that technology is a good force in the world, and by implication we think that will make us attractive to entrepreneurs, and that’s what our business is all about. The three things that matter in venture capital are seeing the deals, picking the deals, and very importantly, winning the deals. If my function can help us see the deals then I’m making a contribution to advancing the future.

Wennmachers added that Future wasn’t going to be focused on traditional news reporting, and, for the record, none of this is in conflict with the original announcement; that’s not to say the media’s overreaction wasn’t warranted: I can only imagine how many page views and subscription dollars were driven by said overreaction. Understanding business models has always been one of the most reliable ways to understand the behavior of organizations.

Technology Saves the World

One could certainly make a similar argument about the striking differences between It’s Time to Build and Technology Saves the World; in How Tech Can Build I noted that the venture capital business model, which is biased towards zero marginal cost business models, wasn’t particularly well-suited to the sort of industrial policy that Andreessen seemed to be espousing:

I agree with Andreessen that much of the software revolution is inevitable; I also agree that tech’s seeming exclusivity on innovation has also been about the online space being the one place without the inertia and regulatory capture Andreessen decries. If you are talented and ambitious, what better place to be?

What I also sense in Andreessen’s essay, though, is the acknowledgment that tech too has chosen the easier path. Instead of fighting inertia or regulatory capture, it has been easier to retreat to Silicon Valley, justify the massive costs of doing so by pursuing infinite-upside outcomes predicated on zero marginal costs, which means relying almost exclusively on software as the means of innovation.

Technology Saves the World seems to imply that is sufficient; Andreessen cites a number of ways that technology has excelled during the pandemic:

• Vaccines, particularly those developed using mRNA, were created, tested, and delivered at scale within a year.
• Telemedicine was enabled at scale.
• The majority of businesses continued to function thanks to technology platforms that enabled remote work.
• Huge numbers of small businesses moved online, thanks to platforms like Facebook, Instacart, Doordash, and more.
• Schools figured out online learnings, laying the groundwork for a huge expansion in educational opportunities.
• Online entertainment kept people entertained, and online networks kept people connected.
• The realization that people can work remotely separated the link between geography and economic opportunity.

This leads to the cynical argument: all of the pieces for these success stories were built before the pandemic hit — that’s why the U.S. was able to navigate the pandemic as well as it did. Which, conveniently enough, means that venture capital was already getting things right. It’s not-so-much “Time to Build” as it is “Keep Building the Things We Were Building All Along.” The business model is safe!

Revisiting Compaq and Coronavirus

Your ears may have perked up at the phrase “that’s why the U.S. was able to navigate the pandemic as well as it did”; wasn’t the American response an abject disaster? It certainly seemed so a year ago; Andreessen opened It’s Time to Build by writing:

Every Western institution was unprepared for the coronavirus pandemic, despite many prior warnings. This monumental failure of institutional effectiveness will reverberate for the rest of the decade, but it’s not too early to ask why, and what we need to do about it. Many of us would like to pin the cause on one political party or another, on one government or another. But the harsh reality is that it all failed — no Western country, or state, or city was prepared — and despite hard work and often extraordinary sacrifice by many people within these institutions. So the problem runs deeper than your favorite political opponent or your home nation.

I had struck a similarly despondent tone a couple of weeks earlier in Compaq and Coronavirus:

The fact of the matter is that we do make tradeoffs between human lives and economic activity all the time — speed limits are perhaps the most banal example. What is truly tragic is the utter lack of resolve and lack of a bias for action in this so-called tradeoff. The only options are to give up the economy or give in to the virus: the possibility of actually beating the damn thing is completely missing from the conversation. To put it another way, the West feels like Compaq in the 1990s, relying on its brand name and partnerships with other entities to do the actual work, forgetting that it was hard work and determination that made it great in the first place.

I drew a contrast to Taiwan, which responded rapidly to limit the spread of the coronavirus, and then kept it at bay for over a year, allowing life to operate normally. Today, though, the tables have turned: the U.S. is almost completely open, thanks to vaccines, while Taiwan, like many other Asian countries, is struggling with outbreaks and imposing lockdowns, and pinning their hopes in part on U.S. vaccine exports.

That’s not to minimize the massive suffering that occurred over the last year: over 600,000 deaths in the U.S., and a fatality rate of 183/100,000 people, 20th in the world (Taiwan, even with the recent outbreak, is at a mere 2/100,000 people). It is, though, a reminder that making grand pronouncements in the first inning is often a mistake. I reflected earlier this year in a Daily Update:

If one were to have presented you with a hypothetical in which the U.S. population was impossible to coordinate during a crisis, yet it was the U.S. that the led the way technologically and logistically in ending the crisis, that would make total sense, right? Moreover, it seems clear that the failure in the beginning is related to the triumph in the end: the U.S. remains a dynamic place with more variance than anywhere in the West, which is why you should expect both the highest highs (when there is a clear goal with an uncertain route to success) and the lowest lows (when there is an unclear goal with top down control).

Everything is indeed a trade-off, but what is important to remember is that the trade-off extends beyond a single pandemic as well. I don’t think it’s an accident that China both crushed the pandemic and also was the place where it originally spiraled out of control, thanks in part to the suppression of the spread of information; the country is also nowhere near opening up even as its vaccines are of questionable efficacy. Is it a stretch to wonder if a bias towards top-down control might be better for mass coordination problems, and worse for innovative and dynamic responses?

The Promise of Remote Work

I suspect this sort of reflection is just as much of a driver of Andreessen’s change of tone as is the a16z business model; after all, the first non-pandemic example he gave in It’s Time to Build of American sclerosis was housing:

You see it in housing and the physical footprint of our cities. We can’t build nearly enough housing in our cities with surging economic potential — which results in crazily skyrocketing housing prices in places like San Francisco, making it nearly impossible for regular people to move in and take the jobs of the future. We also can’t build the cities themselves anymore. When the producers of HBO’s “Westworld” wanted to portray the American city of the future, they didn’t film in Seattle or Los Angeles or Austin — they went to Singapore. We should have gleaming skyscrapers and spectacular living environments in all our best cities at levels way beyond what we have now; where are they?

However Technology Saves the World, as noted, highlights remote work:

For thousands of years, until the time of COVID, the dominant fact of every productive economy has been that people need to live where we work. The best jobs have always been in the bigger cities, where quality of life is inevitably impaired by the practical constraints of colocation and density. This has also meant that governance of bigger cities can be truly terrible, since people have no choice but to live there if they want the good jobs.

What we have learned — what we were forced to learn — during the COVID lockdowns has permanently shattered these assumptions. It turns out many of the best jobs really can be performed from anywhere, through screens and the Internet. It turns out people really can live in a smaller city or a small town or in rural nowhere and still be just as productive as if they lived in a tiny one-room walk-up in a big city. It turns out companies really are capable of organizing and sustaining remote work even — perhaps especially — in the most sophisticated and complex fields.

This is, I believe, a permanent civilizational shift. It is perhaps the most important thing that’s happened in my lifetime, a consequence of the Internet that’s maybe even more important than the Internet. Permanently divorcing physical location from economic opportunity gives us a real shot at radically expanding the number of good jobs in the world while also dramatically improving quality of life for millions, or billions, of people. We may, at long last, shatter the geographic lottery, opening up opportunity to countless people who weren’t lucky enough to be born in the right place. And people are leaping at the opportunities this shift is already creating, moving both homes and jobs at furious rates. It will take years to understand where this leads, but I am extremely optimistic.

There are echoes here of the self-serving arguments proffered by business executives who profited massively from moving jobs abroad: look at how much life improved for billions of people (particularly in China)! It’s a complicated argument because it is, objectively speaking, true; the human race as a whole is in a far better place today than it was forty years ago, thanks to globalization. At the same time, who believes that human betterment was the goal, as opposed to corporate profits? And what costs were incurred, both to middle class Americans and to America’s industrial capacity, in the meantime?

Still, the unbundling of work and geography seems like the only way to cut the Gordian Knot that is the U.S. housing crisis; NIMBY housing policies are a perhaps unavoidable outcomes of any democratic system that inherently favors those who live in a particular location over those that wish they did. It seems far more compatible with our ideals to overcome those problems with competition than top-down fiat, and technology has created the conditions for that sort of competition to occur. Perhaps it is appropriate that It’s Time to Build only had skyscrapers in metadata; the actual solution may be small towns and suburbs.

Progress

“Progress” is an interesting term nowadays; self-described Progressives like Ezra Klein, for example, responded to It’s Time to Build by bemoaning our inability to pass new laws:

The question, then, is why don’t we build? What’s stopping us? Here’s my answer: The institutions through which Americans build have become biased against action rather than toward it. They’ve become, in political scientist Francis Fukuyama’s term, “vetocracies,” in which too many actors have veto rights over what gets built. That’s true in the federal government. It’s true in state and local governments. It’s even true in the private sector.

I’m not against soliciting more ideas of what to build. But what we need is sustained funding, focus, and organizing to make building in America possible again. And that requires patiently engaging with the kinds of institutions that frustrate builders.

Klein is very sharp on the changes in U.S. politics that have driven increased polarization and the difficulty in passing new laws; at the same time, it is notable that the solution to an array of progressive priorities have come from competitive impulses. Take privacy, as an example: Apple’s iOS 14 changes, which are great for Apple’s business, have done far more to change the advertising industry than Europe’s GDPR, which only entrenched the biggest players. It’s the same thing with climate change: Tesla has driven a wholesale shift in the automative industry first-and-foremost by being cool, while solar prices have plummeted far faster than expectations; both are poised to succeed not by telling customers what they can’t do, but by making it more attractive to do what is better for the environment.

This for me is one of the biggest lessons from the last year: solutions that give customers what they want, from cooler cars to the ability to work from anywhere, are the way out of intractable problems, particularly if we want to remain true to Western values of self-determination and individual choice. That’s exactly what happened in the case of the pandemic: the U.S. didn’t beat the coronavirus by locking people up in their homes against their will, but by inventing new technology that let them live their lives as they wish. The ultimate governor on progress is the human condition, and catering to that reality, both in terms of what we build as well as why we build it, is a feature, not a bug.

One more point: it is notable, I think, that Technology Saves the World — along with most of Future’s initial slate of posts, I would add — didn’t really move the needle, particularly in contrast to the viral sensation that was It’s Time to Build. Even for Andreessen bad news is popular news!

That, though, is another feature-disguised-as-a-bug. Give me a world of invention and dynamism with media eagle-eyed for where things go wrong, over a world of conformism and stagnation with declarations that everything is going great. The value of what is built is borne out by reality on the ground — including the value of technology in the pandemic — while narratives are only as good as the restrictions on freedom necessary to make them unquestioned.

Subscription Information

Member: Roland Tanglao

Manage your account

If the Internet Archive is to be believed, Marc Andreessen’s April 2020 coronavirus-driven exhortation It’s Time to Build never actually included that futuristic picture of skyscrapers; it’s in the metadata so that it shows up on social media like Twitter:

What is even more intriguing is when, exactly, the building happened; after all, Andreessen wrote last week in Technology Saves the World:

Last April, I issued a call to our technology industry that it was time to build — and I am so proud of how we delivered. Please join me in an enthusiastic — virtual! — round of applause for all of the amazing workers in our spectacular technology industry who made all this possible. The experience of COVID has made crystal clear both how important our technology is to human flourishing, and how well we can deliver. Technology helped save the world.

Mission accomplished?

Future Business Models

Andreessen’s new essay was published on Andreessen Horowitz’s (a16z) much-ballyhooed new media property Future; you will recall much of the media establishment losing its mind over the announcement in January; I argued in Publishing is Back to the Future that the announcement seemed to be part of a much larger shift in media from a world of scarcity, where newspapers were profitable thanks to geographic monopolies, to one of abundance, where publications succeeded based on their ability to attract audiences who could visit any website on the Internet:

To put it another way, what the New York Times has become is not so different from what Andreessen Horowitz is proposing to build. Margit Wennmachers said in that introductory post that “Our lens is rational optimism about technology and the future”; as a long time subscriber of the New York Times, I think it is fair to call their lens rational skepticism about technology and its effects. What is notable about both is that their lenses are perfectly aligned with their business models (and, I would note, both claim to be motivated to change the world).

a16z’s business model is, of course, venture capital, and Wennmachers, in an interview with Bloomberg’s Emily Chang, was clear that driving a16z’s venture business was the primary focus of Future:

I’m trying to accomplish a business goal. Our firm wants to advance the future and thinks that technology is a good force in the world, and by implication we think that will make us attractive to entrepreneurs, and that’s what our business is all about. The three things that matter in venture capital are seeing the deals, picking the deals, and very importantly, winning the deals. If my function can help us see the deals then I’m making a contribution to advancing the future.

Wennmachers added that Future wasn’t going to be focused on traditional news reporting, and, for the record, none of this is in conflict with the original announcement; that’s not to say the media’s overreaction wasn’t warranted: I can only imagine how many page views and subscription dollars were driven by said overreaction. Understanding business models has always been one of the most reliable ways to understand the behavior of organizations.

Technology Saves the World

One could certainly make a similar argument about the striking differences between It’s Time to Build and Technology Saves the World; in How Tech Can Build I noted that the venture capital business model, which is biased towards zero marginal cost business models, wasn’t particularly well-suited to the sort of industrial policy that Andreessen seemed to be espousing:

I agree with Andreessen that much of the software revolution is inevitable; I also agree that tech’s seeming exclusivity on innovation has also been about the online space being the one place without the inertia and regulatory capture Andreessen decries. If you are talented and ambitious, what better place to be?

What I also sense in Andreessen’s essay, though, is the acknowledgment that tech too has chosen the easier path. Instead of fighting inertia or regulatory capture, it has been easier to retreat to Silicon Valley, justify the massive costs of doing so by pursuing infinite-upside outcomes predicated on zero marginal costs, which means relying almost exclusively on software as the means of innovation.

Technology Saves the World seems to imply that is sufficient; Andreessen cites a number of ways that technology has excelled during the pandemic:

• Vaccines, particularly those developed using mRNA, were created, tested, and delivered at scale within a year.
• Telemedicine was enabled at scale.
• The majority of businesses continued to function thanks to technology platforms that enabled remote work.
• Huge numbers of small businesses moved online, thanks to platforms like Facebook, Instacart, Doordash, and more.
• Schools figured out online learnings, laying the groundwork for a huge expansion in educational opportunities.
• Online entertainment kept people entertained, and online networks kept people connected.
• The realization that people can work remotely separated the link between geography and economic opportunity.

This leads to the cynical argument: all of the pieces for these success stories were built before the pandemic hit — that’s why the U.S. was able to navigate the pandemic as well as it did. Which, conveniently enough, means that venture capital was already getting things right. It’s not-so-much “Time to Build” as it is “Keep Building the Things We Were Building All Along.” The business model is safe!

Revisiting Compaq and Coronavirus

Your ears may have perked up at the phrase “that’s why the U.S. was able to navigate the pandemic as well as it did”; wasn’t the American response an abject disaster? It certainly seemed so a year ago; Andreessen opened It’s Time to Build by writing:

Every Western institution was unprepared for the coronavirus pandemic, despite many prior warnings. This monumental failure of institutional effectiveness will reverberate for the rest of the decade, but it’s not too early to ask why, and what we need to do about it. Many of us would like to pin the cause on one political party or another, on one government or another. But the harsh reality is that it all failed — no Western country, or state, or city was prepared — and despite hard work and often extraordinary sacrifice by many people within these institutions. So the problem runs deeper than your favorite political opponent or your home nation.

I had struck a similarly despondent tone a couple of weeks earlier in Compaq and Coronavirus:

The fact of the matter is that we do make tradeoffs between human lives and economic activity all the time — speed limits are perhaps the most banal example. What is truly tragic is the utter lack of resolve and lack of a bias for action in this so-called tradeoff. The only options are to give up the economy or give in to the virus: the possibility of actually beating the damn thing is completely missing from the conversation. To put it another way, the West feels like Compaq in the 1990s, relying on its brand name and partnerships with other entities to do the actual work, forgetting that it was hard work and determination that made it great in the first place.

I drew a contrast to Taiwan, which responded rapidly to limit the spread of the coronavirus, and then kept it at bay for over a year, allowing life to operate normally. Today, though, the tables have turned: the U.S. is almost completely open, thanks to vaccines, while Taiwan, like many other Asian countries, is struggling with outbreaks and imposing lockdowns, and pinning their hopes in part on U.S. vaccine exports.

That’s not to minimize the massive suffering that occurred over the last year: over 600,000 deaths in the U.S., and a fatality rate of 183/100,000 people, 20th in the world (Taiwan, even with the recent outbreak, is at a mere 2/100,000 people). It is, though, a reminder that making grand pronouncements in the first inning is often a mistake. I reflected earlier this year in a Daily Update:

If one were to have presented you with a hypothetical in which the U.S. population was impossible to coordinate during a crisis, yet it was the U.S. that the led the way technologically and logistically in ending the crisis, that would make total sense, right? Moreover, it seems clear that the failure in the beginning is related to the triumph in the end: the U.S. remains a dynamic place with more variance than anywhere in the West, which is why you should expect both the highest highs (when there is a clear goal with an uncertain route to success) and the lowest lows (when there is an unclear goal with top down control).

Everything is indeed a trade-off, but what is important to remember is that the trade-off extends beyond a single pandemic as well. I don’t think it’s an accident that China both crushed the pandemic and also was the place where it originally spiraled out of control, thanks in part to the suppression of the spread of information; the country is also nowhere near opening up even as its vaccines are of questionable efficacy. Is it a stretch to wonder if a bias towards top-down control might be better for mass coordination problems, and worse for innovative and dynamic responses?

The Promise of Remote Work

I suspect this sort of reflection is just as much of a driver of Andreessen’s change of tone as is the a16z business model; after all, the first non-pandemic example he gave in It’s Time to Build of American sclerosis was housing:

You see it in housing and the physical footprint of our cities. We can’t build nearly enough housing in our cities with surging economic potential — which results in crazily skyrocketing housing prices in places like San Francisco, making it nearly impossible for regular people to move in and take the jobs of the future. We also can’t build the cities themselves anymore. When the producers of HBO’s “Westworld” wanted to portray the American city of the future, they didn’t film in Seattle or Los Angeles or Austin — they went to Singapore. We should have gleaming skyscrapers and spectacular living environments in all our best cities at levels way beyond what we have now; where are they?

However Technology Saves the World, as noted, highlights remote work:

For thousands of years, until the time of COVID, the dominant fact of every productive economy has been that people need to live where we work. The best jobs have always been in the bigger cities, where quality of life is inevitably impaired by the practical constraints of colocation and density. This has also meant that governance of bigger cities can be truly terrible, since people have no choice but to live there if they want the good jobs.

What we have learned — what we were forced to learn — during the COVID lockdowns has permanently shattered these assumptions. It turns out many of the best jobs really can be performed from anywhere, through screens and the Internet. It turns out people really can live in a smaller city or a small town or in rural nowhere and still be just as productive as if they lived in a tiny one-room walk-up in a big city. It turns out companies really are capable of organizing and sustaining remote work even — perhaps especially — in the most sophisticated and complex fields.

This is, I believe, a permanent civilizational shift. It is perhaps the most important thing that’s happened in my lifetime, a consequence of the Internet that’s maybe even more important than the Internet. Permanently divorcing physical location from economic opportunity gives us a real shot at radically expanding the number of good jobs in the world while also dramatically improving quality of life for millions, or billions, of people. We may, at long last, shatter the geographic lottery, opening up opportunity to countless people who weren’t lucky enough to be born in the right place. And people are leaping at the opportunities this shift is already creating, moving both homes and jobs at furious rates. It will take years to understand where this leads, but I am extremely optimistic.

There are echoes here of the self-serving arguments proffered by business executives who profited massively from moving jobs abroad: look at how much life improved for billions of people (particularly in China)! It’s a complicated argument because it is, objectively speaking, true; the human race as a whole is in a far better place today than it was forty years ago, thanks to globalization. At the same time, who believes that human betterment was the goal, as opposed to corporate profits? And what costs were incurred, both to middle class Americans and to America’s industrial capacity, in the meantime?

Still, the unbundling of work and geography seems like the only way to cut the Gordian Knot that is the U.S. housing crisis; NIMBY housing policies are a perhaps unavoidable outcomes of any democratic system that inherently favors those who live in a particular location over those that wish they did. It seems far more compatible with our ideals to overcome those problems with competition than top-down fiat, and technology has created the conditions for that sort of competition to occur. Perhaps it is appropriate that It’s Time to Build only had skyscrapers in metadata; the actual solution may be small towns and suburbs.

Progress

“Progress” is an interesting term nowadays; self-described Progressives like Ezra Klein, for example, responded to It’s Time to Build by bemoaning our inability to pass new laws:

The question, then, is why don’t we build? What’s stopping us? Here’s my answer: The institutions through which Americans build have become biased against action rather than toward it. They’ve become, in political scientist Francis Fukuyama’s term, “vetocracies,” in which too many actors have veto rights over what gets built. That’s true in the federal government. It’s true in state and local governments. It’s even true in the private sector.

I’m not against soliciting more ideas of what to build. But what we need is sustained funding, focus, and organizing to make building in America possible again. And that requires patiently engaging with the kinds of institutions that frustrate builders.

Klein is very sharp on the changes in U.S. politics that have driven increased polarization and the difficulty in passing new laws; at the same time, it is notable that the solution to an array of progressive priorities have come from competitive impulses. Take privacy, as an example: Apple’s iOS 14 changes, which are great for Apple’s business, have done far more to change the advertising industry than Europe’s GDPR, which only entrenched the biggest players. It’s the same thing with climate change: Tesla has driven a wholesale shift in the automative industry first-and-foremost by being cool, while solar prices have plummeted far faster than expectations; both are poised to succeed not by telling customers what they can’t do, but by making it more attractive to do what is better for the environment.

This for me is one of the biggest lessons from the last year: solutions that give customers what they want, from cooler cars to the ability to work from anywhere, are the way out of intractable problems, particularly if we want to remain true to Western values of self-determination and individual choice. That’s exactly what happened in the case of the pandemic: the U.S. didn’t beat the coronavirus by locking people up in their homes against their will, but by inventing new technology that let them live their lives as they wish. The ultimate governor on progress is the human condition, and catering to that reality, both in terms of what we build as well as why we build it, is a feature, not a bug.

One more point: it is notable, I think, that Technology Saves the World — along with most of Future’s initial slate of posts, I would add — didn’t really move the needle, particularly in contrast to the viral sensation that was It’s Time to Build. Even for Andreessen bad news is popular news!

That, though, is another feature-disguised-as-a-bug. Give me a world of invention and dynamism with media eagle-eyed for where things go wrong, over a world of conformism and stagnation with declarations that everything is going great. The value of what is built is borne out by reality on the ground — including the value of technology in the pandemic — while narratives are only as good as the restrictions on freedom necessary to make them unquestioned.

There is some real dissonance in this article in which a charter school teacher describes her experiences during the pandemic. When Covid hit, she writes, the supports provided by the school evaporated. "The wealthiest parents snapped up teachers for 'microschools,'... others left for private school... middle-class parents who could work remotely toughed it out at home, checking in on school between their own virtual meetings. Those with younger kids or in-person jobs scraped together education and child care." Bad, right? Right. Lelac Almagor writes, "I am still bewildered and horrified that our society walked away from this responsibility, that we called school inessential and left each family to fend for itself." Well, yeah, because that's what people wanted when they set up a charter school. In the real public school system, meanwhile, teachers and staff performed miracles making sure everyone was included.

Whenever my family attempts a major buying decision (TV, couch, computer, remodel, shed, dining table, etc) we’re inevitably confronted with a choice between two options:

1. The perfect, more expensive, but out-of-stock option
2. The less perfect, less expensive, in-stock option

This spins us out in a quandary of self-doubt. From the soul-seeking “Do we love it $700 more?”, to the hopefully naïve “What if they update it this year and it’s perfect?”, to the downright existential “Are we $700 more people?” We hem and haw, spinning around the issue like two black holes hoping to collide on a single opinion.

Almost always, we go with option two; the less perfect, less expensive, but in-stock option. Perhaps it’s us giving into impulsiveness, but there’s a small joy in walking away with a physical, tangible box or a tracking link instead of a lousy 12-digit order confirmation number. Even though we have enough datapoints to predict our behavior, we find ourselves in the same predicament over and over. Which path to take? Which path brings the most happiness? Which path is the secret to a happy lifestyle?

I know people who do the perfect, more expensive, but out-of-stock so they have to wait option and they seem happy with their perfect, expensive goods. But they must have a level of patience or perfection I do not. Did they perform better at the marshmallow experiment? Not that we’re unhappy with our on-time, lower grade wares. But I wonder what would happen if we were the people who went all out, exhausted the budget, and then waited weeks or months for the Platonic ideal to arrive. Eight more gigabytes. Eight more horsepower. Eight more threads per inch. Eight more florps. It must be worth it. It must be.

mkalus shared this story from jwz.

The thing that struck me most about our Friday night re-opening party, Turbo Drive, was that it wasn't the usual Turbo Drive crowd. I got the impression that a lot of people said to themselves, "I don't know what synthwave is, but I'm going anyway." And they stuck around, too! Also they all dressed up, but everyone dressed up for a different party. Here's a girl dressed for the symphony, here's a guy dressed for a punk show in 1977, here's Lara Croft swing dancing with Snake Plissken.

I very much hope that this wasn't a one-off and that people hold on to that attitude: "I don't know what this is, but let's check it out anyway". Nature is healing and you are allowed to try new things.

Then Saturday, we had three events: the early show was Hubba Hubba Revue, followed by Bootie in the main room and Lower Underground in Above DNA. The audiences for these parties were more traditional; it was the usual Hubba, Bootie and dubstep/rave crowds, but, there were nearly twice as many of them as in the Before Times, which was awesome.

I had wondered whether people would stil be skittish about being in crowds, and the answer to that is, NOPE. They were happy being absolutely packed in, sweating all over each other. Though all our staff are still wearing masks, I'd say that less than 1% of the customers were. No soft re-entry here: they have embraced the Full Florida.

I also very much expected to see some people getting seriously sloppy by about 1am, since everyone is out of practice at drinking in public, but nope again: they mostly held their shit together pretty well! Good form, SF.

Our crew were all pretty exhausted by the end. It's going to take a little while to get our sea legs back. And also we are still hiring -- bartenders, security, restaurant and floor. Email jobs@dnalounge.com.

Tonight: The Return of Death Guild! Bereft (I am reliably informed) in deathly bloom.

Reddit have long had an unofficial (I think) API where you can add .json to the end of any URL to get back the data for that page as JSON.

I wanted to track new posts on Reddit that mention my domain simonwillison.net.

https://www.reddit.com/domain/simonwillison.net/new/ shows recent posts from a specific domain.

https://www.reddit.com/domain/simonwillison.net/new.json is that data as JSON, which looks like this:

{
  "kind": "Listing",
  "data": {
    "modhash": "la6xmexs8u301d6d105d24f94cdaa4457a00a1ea042c95f6e2",
    "dist": 25,
    "children": [
      {
        "kind": "t3",
        "data": {
          "approved_at_utc": null,
          "subreddit": "programming",
          "selftext": "",
          "author_fullname": "t2_2ks9",
          "saved": false,
          "mod_reason_title": null,
          "gilded": 0,
          "clicked": false,
          "title": "Joining CSV and JSON data with an in-memory SQLite database",
          "link_flair_richtext": [],
          "subreddit_name_prefixed": "r/programming"

Attempting to fetch this data with curl shows an error:

$ curl 'https://www.reddit.com/domain/simonwillison.net/new.json'
{"message": "Too Many Requests", "error": 429}

Turns out this rate limiting is based on user-agent - so to avoid it, set a custom user-agent:

$ curl --user-agent 'simonw/fetch-reddit' 'https://www.reddit.com/domain/simonwillison.net/new.json'
{"kind": "Listing", "data": ...

I used jq to tidy this up like so:

[.data.children[] | .data |  {
  id: .id,
  subreddit: .subreddit,
  url: .url,
  created_utc: .created_utc | todate,
  permalink: .permalink,
  num_comments: .num_comments
}]

Combined:

$ curl \
  --user-agent 'simonw/fetch-reddit' \
  'https://www.reddit.com/domain/simonwillison.net/new.json' \
  | jq '[.data.children[] | .data |  {
    id: .id,
    subreddit: .subreddit,
    url: .url,
    created_utc: .created_utc | todate,
    permalink: .permalink,
    num_comments: .num_comments
  }]' > simonwillison-net.json

Output looks like this:

[
  {
    "id": "o3tjsx",
    "subreddit": "programming",
    "url": "https://simonwillison.net/2021/Jun/19/sqlite-utils-memory/",
    "created_utc": "2021-06-20T00:25:51Z",
    "permalink": "/r/programming/comments/o3tjsx/joining_csv_and_json_data_with_an_inmemory_sqlite/",
    "num_comments": 10
  },
  {
    "id": "nnsww6",
    "subreddit": "patient_hackernews",
    "url": "https://til.simonwillison.net/bash/finding-bom-csv-files-with-ripgrep",
    "created_utc": "2021-05-29T18:04:38Z",
    "permalink": "/r/patient_hackernews/comments/nnsww6/finding_csv_files_that_start_with_a_bom_using/",
    "num_comments": 1
  }
]

My latest challenge project was selected already last year when I set the challenge. As this had to do with fireflies, I knew I had to have it ready for early June, when they’d be all over the place, doing the flashing they’ve been doing for millions of years.

Needless to say, I’ve been waiting to do this for a long time.

Though, by the time I finally got to this project, I had wizened up in terms of what I can do in the one month I had, including all the logistical constraints due to some things going on in my life.

I had originally wanted to make a fancy PCB, inspired by Boldport bugs (tho with no expectation of being anywhere as brilliant haha). But I realized that I didn’t really have the unbroken time to do a design. And then there’s the crazy chipocalypse of parts shortages and price hikes. Also, a comment on being thoughtful when making physical devices, made me, well, be thoughtful regarding this project. And, lastly, a recent project spiked my component buying burn rate, so I’ve been trying to do my projects with as many components from my stash as possible (c’mon, constraints can be liberating for creativity).

With all that, I decided to build this with what I had on hand. I did eventually buy some parts: a super capacitor and a solar panel, tho they were well below my usual project budget (in other words, helped get my average monthly expenditure down).

The build
Since this is the ultimate blinky project, the key component is an LED (green). I wasn’t sure what the local blink pattern would be, so I programmed a trusty ATtiny45 (undervalued in an world of ATtiny85 snobbery) with multiple blink patterns (see image right), and connected a momentary button to switch between blink patterns (state remembered between power-downs via EEPROM).

For power, I realized that I didn’t want to use a CR2032, so I hopped on Amazon and purchased a 5V solar panel and some super capacitors.

An aside on buying components on Amazon
Normally, I try to support my favorite vendors or smaller vendors I encounter – anything to avoid Amazon. Not only that, when I go to other vendors, I buy the number of components I need. Usually, when buying at Amazon, even when the per part price is cheaper, you end up getting more of them than you need. I can’t count the number of times I got a ton of extra LEDs, LDRs, header pins, resistors I didn’t want, but that were shoved into the purchase.*

So, to get one solar panel and two caps, I got eight more caps, and nine more panels. Ok, I partly thought I’d be making a few copies, but stopped at one. Now I have even more extra components around.

Back to the build – many firsts
I had been wanting to make a project with solar panels and super capacitors for the longest time. As the project would be out on multiple nights, I figured it was perfect for a solar panel to charge some caps, which would then discharge during the night, to be recharged in the morning.

I had a Schottky diode to make sure the solar panels wouldn’t suck charge from the cap at night. I could not find a 5V-ish super cap, nor a super cap with significant charge (F). To get the voltage up, I put two 2.7V/3.3F super caps in series, which gave me 5.4V but 1.65F (that’s how it works out). The voltage of the super caps in series was actually what I needed, as you charge them 10-20% below their total V (or that’s what I read). I could put two pairs in parallel to get back up to 3.3F, but 1.65F was more than enough.

As I kinda wanted this to be off by day and blink by night, I wanted to make a light sensing circuit. To do this I used a CdS light detecting resistor and a PNP MOSFET as a high-side switch to power or not the ATtiny. I had just built a high-side MOSFET- and LDR-driven switch that also had an NPN transistor (off when dark), inspired by having to do the high-side switch for this build. Doing these high-side transistor switches was a first for me.

Another first for me, I didn’t program the blinks with the usual delay() nor with millis(), both which would keep the chip powered on. I wanted to stretch the power by sleeping the ATtiny as much as possible. I found a good article that set me on my way to using the watchdog timer to sleep the chip during the pauses in the blink. Also, the button press to select blink patterns was also an interrupt, so the sleep intervals were not holding the chip hostage, as using the delay() would have. In the end these techniques economize power (or so I think) by spending more time asleep than on.

That completes the circuit. I threw in a 0.1µF cap on the ATtiny. Because. And mounted the ATtiny in a DIP socket, so that I could pull it out to program.

Origami-ish
I sure wish I knew how to lay out a circuit for a protoboard in KiCAD. I’ve seen lots of hacks, but I’m surprised there isn’t any KiCAD plug-in for this.

In any case, I dove into the origami of twisting and connecting the components on the protoboard. Nerve-racking, but actually a fun chestnut to crack (a multi-meter helps).

There were some connections I bridged with lacquered winding wire (I so like using it for bodges, once I got the hang of it).

You can image how pleased I was when this assembly worked on the first try.

In the field
The whole point of the project was to put it out at night and attract fireflies.

The first night I finally saw fireflies, the blink pattern was NOT any of the ones I had. I had to then program the flash I saw (three quick pulses) and then put the contraption back out in the night.

OK, so it charged well by day. Blinked when the night came. But I think these particular fireflies prefer to be up in the trees, which is where they all were. I’m used to fireflies that hang out in the grass. So I suspect the hot-to-trot fireflies in the treetops were looking down and wondering why there was one in the wrong place missing all the action. Haha.

Alas, I really didn’t have luck attracting them. Though last night there were a few fireflies that came down near my contraption, flying around or down on the grass below the deck banister I had the contraption on.

Nonetheless, this was a fun build. Got me out at night to marvel at the fireflies and their biology.** And learn a few new things about electronics (especially that I need to learn the math around transistors).

Oh, and I happened to complete this challenge project on time!

*Well, extra LEDs actually led to this crazy thing
**In high school we actually did the chemical reaction for the bioluminescence – what a nerd!

Fastmail is the latest email provider to be blocked by Russia.

We strive to make Fastmail's privacy-first email service available to as many people as possible. We received an order from the Russian Government to comply with their data laws, and we challenged that order in Russian courts. Unfortunately, like many other email and internet service providers, we lost our case. As a result, new Fastmail subscriptions will no longer be available for purchase in Russia.

Russian requirements

In July 2020, the Russian media regulation body, the Roskomnadzor, demanded that Fastmail comply with Russian data laws. This is because Russia has a national goal of controlling the flow of information within its borders. Towards that goal, they want our business to place a server physically within their country, register a local business entity, and become subject to their data access regime.

If we didn't comply, they stated they had the right to block our website (and possibly all data transmission) between us and anyone within Russian borders.

Our fight over the past year

We engaged a legal team to represent us and challenged Russia's original demands.

In late 2020, we lost the initial case, with the Russian state police also joining the case alongside the Roskomnadzor. We appealed. Unfortunately, in May 2021, the court upheld the original ruling, denying our appeal.

Having lost the appeal, we examined if we could do what was being asked of us. We concluded that it would not be possible for us to comply from a technical, business, or financial perspective.

Meeting the Russian requirements would mean: finding and placing servers (primary and backup) in a Russian data center, making the technical changes required to ensure that only Russian customer data was hosted there, and keeping Russian data from relying on other worldwide servers. In addition, we might have the cost of setting up a corporate subsidiary in Russia to comply with their registration process. Doing this could create additional jurisdictional exposure because subjecting any of our customers to their data access laws could create unacceptable privacy risks.

Given our minimal number of customers in Russia, the impact of meeting these requirements acceptably was more than we could justify.

We have not been the only target

Many email and digital companies worldwide have had to deal with this situation over the past few years, with similar impacts and outcomes, such as NordVPN, ProtonMail, Tutanota, Mailfence, and StartMail.

What this means for Fastmail customers

We have made the decision to block signups for all IP addresses we identify as coming from within Russia. We have also removed our app from the Apple and Google app stores for customers in Russia.

As of the time of publication, there are no changes to other accounts, and email flow in and out of Russia has not been blocked, but we will continue to monitor the situation.

What you can do if you're affected

• Talk to Roskomsvoboda: a digital rights advocacy organization.
• Further questions? Talk to our friendly support team on the web or by email to support@fastmail.com.

Note: This post has been edited to reduce confusion about the specifics of the court case.

The new sqlite-utils memory command can import CSV and JSON data directly into an in-memory SQLite database, combine and query it using SQL and output the results as CSV, JSON or various other formats of plain text tables.

sqlite-utils memory

The new feature is part of sqlite-utils 3.10, which I released this morning.

I've recorded this video demonstrating the new feature - with full accompanying notes below.

sqlite-utils already offers a mechanism for importing CSV and JSON data into a SQLite database file, in the form of the sqlite-utils insert command. Processing data with this involves two steps: first import it into a temp.db file, then use sqlite-utils query to run queries and output the results.

Using SQL to re-shape data is really useful - since sqlite-utils can output in multiple different formats, I frequently find myself loading in a CSV file and exporting it back out as JSON, or vice-versa.

This week I realized that I had most of the pieces in place to reduce this to a single step. The new sqlite-utils memory command (full documentation here) operates against a temporary, in-memory SQLite database. It can import data, execute SQL and output the result in a one-liner, without needing any temporary database files along the way.

Here's an example. My Dogsheep GitHub organization has a number of repositories. GitHub make those available via an authentication-optional API endpoint at https://api.github.com/users/dogsheep/repos - which returns JSON that looks like this (simplified):

[
  {
    "id": 197431109,
    "name": "dogsheep-beta",
    "full_name": "dogsheep/dogsheep-beta",
    "size": 61,
    "stargazers_count": 79,
    "watchers_count": 79,
    "forks": 0,
    "open_issues": 11
  },
  {
    "id": 256834907,
    "name": "dogsheep-photos",
    "full_name": "dogsheep/dogsheep-photos",
    "size": 64,
    "stargazers_count": 116,
    "watchers_count": 116,
    "forks": 5,
    "open_issues": 18
  }
]

With sqlite-utils memory we can see the 3 most popular repos by number of stars like this:

$ curl -s 'https://api.github.com/users/dogsheep/repos' \
  | sqlite-utils memory - '
      select full_name, forks_count, stargazers_count as stars
      from stdin order by stars desc limit 3
    ' -t
full_name                     forks_count    stars
--------------------------  -------------  -------
dogsheep/twitter-to-sqlite             12      225
dogsheep/github-to-sqlite              14      139
dogsheep/dogsheep-photos                5      116

We're using curl to fetch the JSON and pipe it into sqlite-utils memory - the - means "read from standard input". Then we pass the following SQL query:

select full_name, forks_count, stargazers_count as stars
from stdin order by stars desc limit 3

stdin is the temporary table created for the data piped in to the tool. The query selects three of the JSON properties, renames stargazers_count to stars, sorts by stars and return the first three.

The -t option here means "output as a formatted table" - without that option we get JSON:

$ curl -s 'https://api.github.com/users/dogsheep/repos' \
  | sqlite-utils memory - '
      select full_name, forks_count, stargazers_count as stars
      from stdin order by stars desc limit 3
    '  
[{"full_name": "dogsheep/twitter-to-sqlite", "forks_count": 12, "stars": 225},
 {"full_name": "dogsheep/github-to-sqlite", "forks_count": 14, "stars": 139},
 {"full_name": "dogsheep/dogsheep-photos", "forks_count": 5, "stars": 116}]

Or we can use --csv to get back CSV:

$ curl -s 'https://api.github.com/users/dogsheep/repos' \
  | sqlite-utils memory - '
      select full_name, forks_count, stargazers_count as stars
      from stdin order by stars desc limit 3
    ' --csv
full_name,forks_count,stars
dogsheep/twitter-to-sqlite,12,225
dogsheep/github-to-sqlite,14,139
dogsheep/dogsheep-photos,5,116

The -t option supports a number of different formats, specified using --fmt. If I wanted to generate a LaTeX table of the top repos by stars I could do this:

$ curl -s 'https://api.github.com/users/dogsheep/repos' \
  | sqlite-utils memory - '
      select full_name, forks_count, stargazers_count as stars
      from stdin order by stars desc limit 3
    ' -t --fmt=latex
\begin{tabular}{lrr}
\hline
 full\_name                  &   forks\_count &   stars \\
\hline
 dogsheep/twitter-to-sqlite &            12 &     225 \\
 dogsheep/github-to-sqlite  &            14 &     139 \\
 dogsheep/dogsheep-photos   &             5 &     116 \\
\hline
\end{tabular}

We can run aggregate queries too - let's add up the total size and total number of stars across all of those repositories:

$ curl -s 'https://api.github.com/users/dogsheep/repos' \
| sqlite-utils memory - '
    select sum(size), sum(stargazers_count) from stdin
' -t
  sum(size)    sum(stargazers_count)
-----------  -----------------------
        843                      934

(I believe size here is measured in kilobytes: the GitHub API documentation isn't clear on this point.)

Joining across different files

All of these examples have worked with JSON data piped into the tool - but you can also pass one or more files, of different formats, in a way that lets you execute joins against them.

As an example, let's combine two sources of data.

The New York Times publish a us-states.csv file with Covid cases and deaths by state over time.

The CDC have an undocumented JSON endpoint (which I've been archiving here) tracking the progress of vaccination across different states.

We're going to run a join from that CSV data to that JSON data, and output a table of results.

First, we need to download the files. The CDC JSON data isn't quite in the right shape for our purposes:

{
  "runid": 2023,
  "vaccination_data": [
    {
      "Date": "2021-06-19",
      "Location": "US",
      "ShortName": "USA",
      ...

sqlite-utils expects a flat JSON array of objects - we can use jq to re-shape the data like so:

$ curl https://covid.cdc.gov/covid-data-tracker/COVIDData/getAjaxData?id=vaccination_data \
  | jq .vaccination_data > vaccination_data.json

The New York Times data is good as is:

$ wget 'https://github.com/nytimes/covid-19-data/raw/master/us-states.csv'

Now that we have the data locally, we can run a join to combine it using the following command:

$ sqlite-utils memory us-states.csv vaccination_data.json "
  select
    max(t1.date),
    t1.state,
    t1.cases,
    t1.deaths,
    t2.Census2019,
    t2.Dist_Per_100K
  from
    t1
      join t2 on t1.state = replace(t2.LongName, 'New York State', 'New York')
  group by
    t1.state
  order by
    Dist_Per_100K desc
" -t
max(t1.date)    state                       cases    deaths    Census2019    Dist_Per_100K
--------------  ------------------------  -------  --------  ------------  ---------------
2021-06-18      District of Columbia        49243      1141        705749           149248
2021-06-18      Vermont                     24360       256        623989           146257
2021-06-18      Rhode Island               152383      2724       1059361           141291
2021-06-18      Massachusetts              709263     17960       6892503           139692
2021-06-18      Maryland                   461852      9703       6045680           138193
2021-06-18      Maine                       68753       854       1344212           136894
2021-06-18      Hawaii                      35903       507       1415872           136024
...

I'm using automatically created numeric aliases t1 and t2 for the files here, but I can also use their full table names "us-states" (quotes needed due to the hyphen) and vaccination_data instead.

The replace() operation there is needed because the vaccination_data.json file calls New York "New York State" while the us-states.csv file just calls it "New York".

The max(t1.date) and group by t1.state is a useful SQLite trick: if you perform a group by and then ask for the max() of a value, the other columns returned from that table will be the columns for the row that contains that maximum value.

This demo is a bit of a stretch - once I reach this level of complexity I'm more likely to load the files into a SQLite database file on disk and open them up in Datasette - but it's a fun example of a more complex join in action.

Also in sqlite-utils 3.10

The sqlite-utils memory command has another new trick up its sleeve: it automatically detects which columns in a CSV or TSV file contain integer or float values and creates the corresponding in-memory SQLite table with the correct types. This ensures max() and sum() and order by work in a predictable manner, without accidentally sorting 1 as higher than 11.

I didn't want to break backwards compatibility for existing users of the sqlite-utils insert command so I've added type detection there as a new option, --detect-types or -d for short:

$ sqlite-utils insert my.db us_states us-states.csv --csv -d
  [####################################]  100%
$ sqlite-utils schema my.db
CREATE TABLE "us_states" (
   [date] TEXT,
   [state] TEXT,
   [fips] INTEGER,
   [cases] INTEGER,
   [deaths] INTEGER
);

There's more in the changelog.

Releases this week

• sqlite-utils: 3.10 - (78 releases total) - 2021-06-19
  Python CLI utility and library for manipulating SQLite databases
• dogsheep-beta: 0.10.2 - (20 releases total) - 2021-06-13
  Build a search index across content from multiple SQLite database tables and run faceted searches against it using Datasette
• yaml-to-sqlite: 1.0 - (5 releases total) - 2021-06-13
  Utility for converting YAML files to SQLite
• markdown-to-sqlite: 1.0 - (2 releases total) - 2021-06-13
  CLI tool for loading markdown files into a SQLite database

TIL this week

• Mouse support in vim

Treatments

This preprint might be important. It describes a novel treatment for COVID-19 (and possibly for all viruses).

The active ingredient (SLR) activates a molecule called RIG-I, which in turn gooses the production of interleukin-1, which is a cytokine, part of the immune system associated with inflammation. Basically, SLR revs up the innate immune system to kill viruses off (or at least knock their forces down) before the adaptive immune system can get its act in gear and send in the heavy troops.

The paper shows really good results in mice (if you get SLR to the mice quickly enough after they get infected with COVID-19). It also works against all the scariants, though not as well against Beta or Delta.

Vaccines

This article isn’t exactly about COVID-19, but talks about a support player in the immune system, and is quite entertaining!

This Twitter thread (longish) is a masterclass on the tropes of anti-vaxxism.

This article (long) explains why COVAX failed to get doses to low-income countries.

Immunocompromised patients frequently don’t mount a good immune response to vaccines. Should they get a third dose? This article looks at the question. (Pro: sometimes a third dose gives a response. Con: sometimes it doesn’t.)

It’s been a few months since I moved this site to Azure static storage, and I think a few notes are in order.

The site is still powered by my Wiki engine and I still keep it updated by tossing Markdown files into a git repository, but it now has even more unique twists.

TL;DR: Instead of rebuilding the entire site whenever a single page changes (or even syncing generated content), my current engine does delta updates directly onto an Azure storage container upon git push.

Or, to put it another way, it:

• Takes the deltas off that push.
• Matches them against what is currently published in $web.
• Figures out what (if any) images, related pages and back-links need updating via a small SQLite database that holds all those references.
• POSTs the generated HTML for any updated pages directly to a $web public container (which it does wickedly fast thanks to my own asyncio library for talking to Azure Storage).

This contrasts with the approach of many static site generators of rebuilding the entire site, saving the HTML to a folder and then syncing all of that to a public bucket/container, which can be pretty I/O intensive.

Or (like many people do these days with GitHub Actions et al) spawning and provisioning an entire VM to do so.

I don’t like either, and find the “pseudo-CI” approach a tad lazy (and wasteful).

While it’s true that the site generator is on an “always on” (and admittedly pokey) Azure B1ls instance (1 VCPU, 512MB RAM), that machine is shared with a bunch of other small services, and the builder service is actually removed from RAM when idle (thanks to Piku/uwsgi magic), so overall costs are negligible.

The process has minimal storage and I/O impact (it only needs to work with a copy of the raw content and a SQLite database with page relationships), and takes less than 10 seconds to generate and publish a new page (or even a few hundred of them) from the moment I type git push, which compares well with my age-old, Dropbox-driven “live” syncing approach.

I could run the whole thing on a Pi Zero (and yes, I’ve tested it on a 3A+, just because the Zero is quite slow for full rebuilds), so that’s fine.

There are a few distinct advantages with this setup:

• No more worrying about HTTP handling and the constant flurry of botnets trying to hack their way into something that was never a Wordpress site in a rather heavy-handed way.
• Much better availability as there would occasionally be some OS weirdness, VM update or configuration tweak that would take the site offline until I noticed.
• Maybe a smidgeon more performance (even considering I had optimized HTTP request handling and have been using Cloudflare for years, the site does feel a tad snappier).

There are a few things I’m not happy about, though:

• My “magical” Wiki URL redirection feature was replaced by a gnarly bit of JavaScript that I’m not exactly enamoured with either. I might push that back to a Cloudflare worker, but am wary of relying too much on any single service.
• I’m not particularly impressed with DuckDuckGo site search. Even after fiddling with various things like Bing Webmaster tools (since DuckDuckGo gets part of its catalog from it), search is just not good enough. I publish a complete sitemap and have OpenGraph everywhere, but their crawler simply doesn’t do a good a service as my old SQLite-powered full text indexer, and it’s getting very annoying.
• The engine is a bit slow when rendering the full working set of 8000+ pages (when I do layout changes, for instance). Python can only do so much, and even though I pre-bake a lot of stuff, I’d like it to be a bit faster when parsing and rendering the actual pages.

I’ve been putting off using Azure Search because I don’t want to have to maintain a search endpoint, but I might end up doing it regardless.

There are a few minor things I’d like to have done, though:

• I still haven’t pushed out the code to render archives, which has an annoying bug I haven’t had the time (or willpower) to fix because it will eventually force me to break a couple of longstanding abstractions. Update: I decided to not overthink it and just use my old Wiki code to generate archive pages.
• OpenGraph can be improved little further, especially where images are concerned. Adding better image descriptions and the like feels like something I might have some fun doing, so will probably be adding Azure Cognitive Services to my rendering pipeline
• I would love to have the builder run as an Azure Function, but there is simply no way I can get git to run inside one, and Piku has rendered this kind of service so simple and easy to maintain that I don’t feel the urge to tackle the amount of extra complexity required to work around that.

Finally, the current codebase could do with a little cleaning up, since it has suffered a bit of feature creep over the years.

It’s relatively small at ~4000 LOC (discounting templating and HTML), but the actual static generator is a ~1000 LOC bolt-on that could be streamlined and blended in a little better, so I’d like to trim the whole thing back into the ~3000 LOC range.

And, in general, I just wish the entire codebase was simpler, for Python is a wonderful language but still lacks the conciseness of LISP–and I would very much like to go back to something like the Hy implementation I had a few years back¹.

And yet, replacing Python would be a tall order indeed. There are a few interesting candidates (Janet, Fennel, Clojure, Kotlin and even F#), but none of them have the full range of wonderful libraries I currently rely on.

I do, however, hear the siren call of building something leaner and meaner, so I might do some experiments over summer break.

There are a few things I most definitely won’t be doing, though:

• Building a Docker container (again, Piku has made things so easy to deploy and update that setting up a private registry, a full-blown CI/CD pipeline and whatever else would feel like a chore).
• Getting this to run on Kubernetes (ha!).
• Porting the engine to Rust (I gave it a go and realized that implementing the XML transformation pipelines I have would be a bit of a chore).

So, in essence, things are stable, reliable and with enough headroom and small things to tweak to keep being interesting, but definitely low impact.

I might do a redesign, though. It’s about time to see if there is something else I can do layout-wise (although I do like Georgia‘s readability and universal reach, and the lack of any unnecessary frills).

This post describes an unusual migration that took place in the world of Inter-Relay Chat (IRC) recently. An IRC server managed by volunteers to help with Linux support went through a series of ownership changes and ended up in private hands. As a result, most of the volunteers resigned, accusing the new owner of executing a hostile takeover and  set up Libera as an alternative. Because IRC is open source, they were able to create an exact clone of the original site, including the history, groups, and all the other data. But while the incident points to the advantage of open source and open networks, it also points to the fragility of the network as a whole, because where possible private owners will try to take over and monetize public infrastructure.

There's only one source for this story - DigiDay, which limits access (here's a screen shot) - though you can see it echoed around the technosphere - Reddit, Slashdot, Hacker News - and copied in various cut-and-paste publications. One person on Reddit also added, "Amazon also stopped including in their order-confirmation emails the details of what you ordered, on the grounds that webmail was reading that and leaking it back to Google or ISPs for their own marketing." It does raise the question of how much control websites have over being tracked themselves by browsers like Chrome that send website data to the competition. Previously.

Vaccination Clinics

I got my second shot today (wooohoo!) at the Convention Centre, escorted by my beloved spouse. This is what I observed:

• There were about three greeters at the door to the outside of Canada Place. One asked me if anyone in my bubble had been sick, but I think that was the only question I got asked.
• There were a pair of stations that asked us to take off our masks and take their procedure mask, then sanitize our hands. I was wearing an N95 particulate (i.e. with a vent) underneath a procedure mask, but they made me take that off as well. That was odd, but whatever.
• The person at the sanitization station then pointed us at a woman waving a sign that said something like “COME HERE NEXT” at the entrance to a rope maze (the kind that snakes around to turn a 2d line into a 3d area). There were probably 20-30 people in the maze ahead of us. I think we stopped moving once or twice very briefly, but basically walked continuously.
• In the next area, I could see about forty stations on the left, A1 to D5. On the right, there were tables with 20 yellow bins which I presume were bins for discarding needles. I saw one sign that said “Pfizer”, so I assume those tables were also where they filled syringes.
• A man at the end of the rope maze directed me to a nearby table (A1, as it turned out).
• This table was for verifying information. A woman at that table scanned the printed barcode which I had brought with me and filled out the second-dose date on a vaccination card. I asked if she could also fill in when I’d gotten my first dose, and she said she could not, that she didn’t have that information. I was surprised but pleased to see that very little of my personal information was available to her.
• Someone came over and told me to go over to the woman waving the sign saying “4”.
• At the indicated table, there was a very friendly vaccinator. He said that he retired from being a family doctor in 2007 to go run a nursing home, then he retired from that in 2020. He unretired to help out with vaccinating. We thanked him for his service.
• He confirmed my information against what his laptop was showing him, and then he gave me a shot. “Think of something pleasant!” he told me before pricking my deltoid.
• The vaccinator then filled out the type (Pfizer) the lot, and the location (CC) plus his initials (HR), explaining as he went. I was glad that he explained, otherwise I would have wondered why he was carbon-copying Human Resources!
• He also said that he couldn’t fill out my vax card with the first dose information because he didn’t have information about when I had gotten my first shot either. Yay for data privacy!
• I asked him how many vaccination stations there were, and he said about 40. He said that the convention centre was supposed to get 4000 doses per day, but that it varied a lot. I think he said that the most he’d seen was 5000 in one day. He said it was kind of light when we were there.
• I asked him which day of the week was busiest and he said it varied wildly.
• We then got pointed at the waiting area, where a very friendly man pointed us to some seats. We asked him if it was a fun job. He said that he normally worked shepherding tourists onto cruise liners, so in his regular job, he was used to dealing with happy people. “Everybody’s happy to be going on a cruise!” He said that VCH hired the cruise liner loading crew en masse “because we’re people movers, that’s what we do!” We agreed that it was an inspired choice.
• I asked if he could tell if we were skewing so heavily to second doses because public health got scared about Delta or because there just wasn’t demand for first doses any more. He didn’t have a good answer.
• We waited our 15 minutes, and then left. Thank you, Convention Centre! Thank you, VCH! Thank you, BC taxpayers!

Statistics

Today: +109 cases, +1 death, +8,917 first doses, +55,363 second doses.

Currently 128 in hospital / 48 in ICU, 1,389 active cases, 143,748 recovered.

We have 317,109 doses in fridges; we’ll use it up in 5.5 days at last week’s rate. We’ve given more doses than we’d received by 4 days ago.

We have 225,088 mRNA doses in fridges; we’ll use it up in 3.9 days at last week’s rate. We’ve given more mRNA doses than we’d received by 4 days ago.

We have 92,021 AZ doses in fridges; we’ll use it in 16.8 days at last week’s rate.

Charts

From Blake Shaffer:

From Justin McElroy’s tweetstream:

Note that different people have different figures for percent vaccinated because everybody uses a different denominator and nobody every says which one they have chosen grumble grumble grumble…

Yesterday we were walking an incredible cliff top trail in East Sooke Park, in Scia’new territory on Vancouver Island. The Coast Trail there is rugged along the Juan de Fuca side of the park and although it is well travelled, there are sections across bare rock cliff top when the path is all but invisible. It requires a deeper kind of seeing to discern where the path is, especially if you follow what looks to be an obvious route which can take you to some dangerous places. As an experienced trail walker, I find myself in moments like this looking for evidence that I am NOT on the path. Is there broken foliage? Is the soil compressed and eroded by boots rather than hoofs or water? Are the roots underfoot rubbed clean of bark? Are there any trail markers about? When I find myself answering “no” to these questions I move slower, until the evidence is overwhelming, and I stop and track back to find out where I went wrong.

You can see why looking for evidence to DISPROVE your belief creates a safe to fail situation. If I find a single piece of evidence that confirms my belief that I am on the right track, and I follow it unquestioningly, the results become increasingly dangerous, and failure becomes unsafe.

A lot of my life and work is about paying attention to these weak signals. Whether it is making music with others, facilitating groups, helping organizations with strategy, playing and watching sports like soccer, rugby and hockey, it all comes down to paying attention in a way that challenges your beliefs.

The other day I offered a pithy comment on facebook to the question of “what is the difference between critical thinking and buying conspiracy theories?” and it really came down to this: critical thinkers look for evidence to disprove their beliefs and conspiracy theorists look for evidence to confirm their beliefs.

I think the latter is quite the norm in our current mainstream organizational cultures, even if it doesn’t lead to conspiracy theory. The pressure for accountability and getting it right leaves very little space to see what’s going wrong in the organization. The desire to build on what is working – while being an important part of the strategic toolkit – is not served without a critical look at the fact that we might be doing it wrong.

This is why sensemaking has become a critical part of my practice. And by sensemaking I mean collecting large numbers of small anecdotes about a situation and having large numbers of people look at them together. The idea is that with a diverse set of data points and a diverse number of perspectives, you get a truer picture of the actual culture of an organization, and you can act with more capacity to find multiple ways forward, including those which both challenge your assumptions about what is right and good now and those which discover what is better and better.

Recently in Canada we have been having a little debate about whether celebrating Canada Day on July 1 is appropriate given that fact that this month – National Indigenous Peoples Month, as it turns out – has been marked by a reckoning with the visible evidence of the genocide that has been committed here. While hundreds of thousands of people here are in mourning or grief, and are reliving the trauma that has travelled through their families as a result of the genocidal policies of residential school and the non-consensual adoption of children, many others are predictably coming out with a counter reaction that goes something like this “yeah, well let’s get over it. Canada is still the best country of the world to live in.”

And that makes sense for many people – like me – who live here and have a great life. But as I have been saying elsewhere on Twitter: don’t confuse you having a great life with this being a great country. There is nothing wrong with people having a great life. That is what we should want for all people. But Canada is not a place where that happens for everyone. The story is very different for lots of people who struggle to find contentment and acceptance inside this nation-state. Canada’s very existence is owed to broken treaties, environmental destruction, relational treachery, economic injustice, and genocide.

Paying attention to the weak signals is important here. If all you can see is how great your own life is, and you think we just need to keep doing whatever it is that we are doing that assures that continuity, then we are headed for a precipice. We are headed off an environmental cliff, into a quagmire of injustice and economic inequality that destabilizes everything you have in a catastrophic way.

Listening to First Nations – really paying attention to possibilities – is mutually beneficial to everyone. If one wants all lives to matter, then one has to ensure that every life matters, which means taking the lead from those whose lives have been considered dispensable in the project called “Canada.” And it’s not like they haven’t been out here for the past 250 years calling for a better way. It’s just that the mainstream, largely led by commercial interests who have hungered for and exploited natural resources that never belonged to them, have cheered on the idea that if Canada is good for me, it must be good period.

Let seeing be disbelieving. This country is not an inherently GOOD place. But it could be. It could be great. It could be safe, healthy, prosperous, balanced, creative and monumentally amazing. But it requires us to first question the limiting beliefs we have that it could never be better than this and second to pay attention to the weak signals that help guide us onto a path that takes us there.

It is far too early to celebrate Canada Day. We haven’t yet fulfilled the promise of the treaties and the vision with which indigenous Nations entered into relationships with Europeans oh so long ago, and that vision which is continually offered up to settlers through reciprocity and relationship. If there is anything to celebrate, perhaps it is the fact that we do have the resources to make this country work for all and we have the intelligence and creativity and willingness to do it, but you won’t find that in the Board rooms and the Parliamentary lobbies and the Cabinet offices and the global markets.

It is in the weak signals, the stories and small pathways of promise out there that are born in struggle and resilience and survival and generate connection, sustainability and the promise of well-being for all.

I set up my GPT-3 account so I can text with it over SMS. It’s pretty fun.

To build this, I combined Airtable and Twilio. Here’s how it works:

1. I text my query to a Twilio phone number. E.g. “What is a good thing to reflect on as I fall asleep?”
2. Twilio receives the text and sends a webhook with the query to Airtable.
3. Airtable receives the webhook and this triggers an automation.
4. The automation stores the query.
5. The automation sends the query to the GPT-3 API.
6. The automation gets the response from the GPT-3 API.
7. The automation stores the response.
8. The automation texts the response back to me.

Airtable serves two purposes.

First, it sends the query to the GPT-3 API.

Second, it provides a way to store queries and responses, which allows it to build up context in an exchange. For example, imagine I send this initial message to GPT-3:

The following is a conversation with an AI movie buff.
The movie buff has excellent taste in movies and is
great at recommending obscure but high quality movies.

Human: Hello, can you recommend me a movie?
AI: Sure, how about Hoop Dreams?
Human: I loved that.  Can you recommend me a movie like it?
AI:

It replied with:

Sure, how about Beautiful Girls.

I replied:

...Human: I actually didn't like that movie at all.  Can you
recommend me another?

GPT-3:

AI: Sure, how about Boyhood?

Me:

...Human: I did like that one.  I really loved Drinking
Buddies which felt more truthy.  Can you recommend a movie
like that?

The ...s are what create the context. GPT-3 only operates based on your query. It has no memory of what you sent it previously. To retain context, each query you send must recapitulate earlier messages.

So, in step 5 above, Airtable gathers up all earlier consecutive queries that start with ..., plus one more (the query that started the exchange e.g. “The following [etc]”), plus the intervening replies from GPT-3. It sends this whole chunk to GPT-3 as a query.

This enables you to steer the algorithm towards more and more useful responses.

In the early weeks and months of getting Diamond off the ground, we were intentional about the kind of organizational culture we wanted to create. Within the founding group we had decades of cumulative experience, mostly in organizations known for the strength of their culture together with a sprinkling of less pleasant experiences. 

In those early days, we spoke of “getting the band back together.” We were based in Chicago after all, so the Blues Brothers reference was a natural. 

Of course, what we were trying was more complicated. We weren’t getting _a_ band back together so much as we were trying to create a super group combining and mixing the talents of people who had been stars in their own groups. 

Consulting egos are rarely small; possibly never. Everyone had an opinion, often several. Whatever status or authority we might have had in prior organizations was politely acknowledged and promptly ignored. How things were done at McKinsey or Accenture or Booz or wherever was evaluated on the merits not the pedigree. 

More often than not, the differences were more cosmetic than substantive. Over time we were cobbling together a creole consulting language of our own. 

Sometimes the noise levels got out of hand. After one incident, one of my partners ordered up and distributed “No Whining” buttons to all of us. 

It was good for a laugh and a boost in morale. And it did help lower the temperature on some conversations. While I had a fraught relationship with this partner, I had also learned that their instincts routinely beat my analyses. Disrupting a conversational cycle before it degenerated into a shouting match helped. 

Unfortunately, “no whining” too often defaulted to no conversation at all. Labeling inquiry or pushback as “whining” was equivalent to “shut up and do it my way” or “if you can’t handle this, we’ll find someone who can.”

These sentiments might be marginally acceptable in some circumstances. If the work is sufficiently routine, problem solving can devolve into selecting a workable answer from a menu of known responses (I was about to suggest that you Google “the bedbug letter” but Snopes.com has a more interesting take—FACT CHECK: The Bedbug Letter.

I don’t live in that world. Neither did Diamond. And, if your work is knowledge work, neither do you. You get the problems that don’t have clear solutions. Which means you have to listen more carefully; to those you work with and to yourself. 

You must learn to distinguish between “whining”— a vague complaint that something isn’t fair—and the spark of discomfort signaling learning that needs to happen. Forbidding “whining” makes it harder to recognize the learning signals embedded in discomfort.

The post Whining or Learning appeared first on McGee's Musings.

It’s been a few months since I moved this site to Azure static storage, and I think a few notes are in order.

The site is still powered by my Wiki engine and I still keep it updated by tossing Markdown files into a git repository, but it now has even more unique twists.

TL;DR: Instead of rebuilding the entire site whenever a single page changes (or even syncing generated content), my current engine does delta updates directly onto an Azure storage container upon git push.

Or, to put it another way, it:

• Takes the deltas off that push.
• Matches them against what is currently published in $web.
• Figures out what (if any) images, related pages and back-links need updating via a small SQLite database that holds all those references.
• POSTs the generated HTML for any updated pages directly to a $web public container (which it does wickedly fast thanks to my own asyncio library for talking to Azure Storage).

This contrasts with the approach of many static site generators of rebuilding the entire site, saving the HTML to a folder and then syncing all of that to a public bucket/container, which can be pretty I/O intensive.

Or (like many people do these days with GitHub Actions et al) spawning and provisioning an entire VM to do so.

I don’t like either, and find the “pseudo-CI” approach a tad lazy (and wasteful).

While it’s true that the site generator is on an “always on” (and admittedly pokey) Azure B1ls instance (1 VCPU, 512MB RAM), that machine is shared with a bunch of other small services, and the builder service is actually removed from RAM when idle (thanks to Piku/uwsgi magic), so overall costs are negligible.

The process has minimal storage and I/O impact (it only needs to work with a copy of the raw content and a SQLite database with page relationships), and takes less than 10 seconds to generate and publish a new page (or even a few hundred of them) from the moment I type git push, which compares well with my age-old, Dropbox-driven “live” syncing approach.

I could run the whole thing on a Pi Zero (and yes, I’ve tested it on a 3A+, just because the Zero is quite slow for full rebuilds), so that’s fine.

There are a few distinct advantages with this setup:

• No more worrying about HTTP handling and the constant flurry of botnets trying to hack their way into something that was never a Wordpress site in a rather heavy-handed way.
• Much better availability as there would occasionally be some OS weirdness, VM update or configuration tweak that would take the site offline until I noticed.
• Maybe a smidgeon more performance (even considering I had optimized HTTP request handling and have been using Cloudflare for years, the site does feel a tad snappier).

There are a few things I’m not happy about, though:

• My “magical” Wiki URL redirection feature was replaced by a gnarly bit of JavaScript that I’m not exactly enamoured with either. I might push that back to a Cloudflare worker, but am wary of relying too much on any single service.
• I’m not particularly impressed with DuckDuckGo site search. Even after fiddling with various things like Bing Webmaster tools (since DuckDuckGo gets part of its catalog from it), search is just not good enough. I publish a complete sitemap and have OpenGraph everywhere, but their crawler simply doesn’t do a good a service as my old SQLite-powered full text indexer, and it’s getting very annoying.
• The engine is a bit slow when rendering the full working set of 8000+ pages (when I do layout changes, for instance). Python can only do so much, and even though I pre-bake a lot of stuff, I’d like it to be a bit faster when parsing and rendering the actual pages.

I’ve been putting off using Azure Search because I don’t want to have to maintain a search endpoint, but I might end up doing it regardless.

There are a few minor things I’d like to have done, though:

• I still haven’t pushed out the code to render archives, which has an annoying bug I haven’t had the time (or willpower) to fix because it will eventually force me to break a couple of longstanding abstractions. Update: I decided to not overthink it and just use my old Wiki code to generate archive pages.
• OpenGraph can be improved little further, especially where images are concerned. Adding better image descriptions and the like feels like something I might have some fun doing, so will probably be adding Azure Cognitive Services to my rendering pipeline
• I would love to have the builder run as an Azure Function, but there is simply no way I can get git to run inside one, and Piku has rendered this kind of service so simple and easy to maintain that I don’t feel the urge to tackle the amount of extra complexity required to work around that.

Finally, the current codebase could do with a little cleaning up, since it has suffered a bit of feature creep over the years.

It’s relatively small at ~4000 LOC (discounting templating and HTML), but the actual static generator is a ~1000 LOC bolt-on that could be streamlined and blended in a little better, so I’d like to trim the whole thing back into the ~3000 LOC range.

And, in general, I just wish the entire codebase was simpler, for Python is a wonderful language but still lacks the conciseness of LISP–and I would very much like to go back to something like the Hy implementation I had a few years back¹.

And yet, replacing Python would be a tall order indeed. There are a few interesting candidates (Janet, Fennel, Clojure, Kotlin and even F#), but none of them have the full range of wonderful libraries I currently rely on.

I do, however, hear the siren call of building something leaner and meaner, so I might do some experiments over summer break.

There are a few things I most definitely won’t be doing, though:

• Building a Docker container (again, Piku has made things so easy to deploy and update that setting up a private registry, a full-blown CI/CD pipeline and whatever else would feel like a chore).
• Getting this to run on Kubernetes (ha!).
• Porting the engine to Rust (I gave it a go and realized that implementing the XML transformation pipelines I have would be a bit of a chore).

So, in essence, things are stable, reliable and with enough headroom and small things to tweak to keep being interesting, but definitely low impact.

I might do a redesign, though. It’s about time to see if there is something else I can do layout-wise (although I do like Georgia‘s readability and universal reach, and the lack of any unnecessary frills).

Today I learned that if you hit Esc in vim and then type :set mouse=a and hit enter... vim grows mouse support! In your terminal!

You can use the mouse to select blocks of text and move the insertion cursor around, then hit del to delete it or type to replace it.

I learned this after tweeting a demo video of Will McGugan's brilliant new textual Python library for building TUIs - terminal user interfaces - and marveling at how his demo application can already respond to mouseover events and scroll wheel activation while running in the terminal.