Rolandt

Many of the examples in Software Design by Example are too long to show comfortably in one block of code on a printed page, so I needed a way to break them up. As an experiment, I wrote a custom module loader that reads a source file that has specially-formatted comments containing the names of other files, and then reads and inserts those files before running the code. Modern programming languages don’t work this way, but it’s how C and C++ handle header files and how static site generators do snippet inclusion.

I decided to use a different approach in the end because ESLint didn’t know what to make of file inclusions. Despite being a dead end, I thought the inclusion tool was worth a chapter because it shows how programs turn source code into something executable. I also thought it was important to have a least one example of a working dead end so that readers would be comfortable when their ideas did what they were supposed to but still shouldn’t be put into production.

Terms defined: header file, literate programming, loader, sandbox, search path, shell variable.

Servo to Advance in 2023

This is excellent news: Serve, the browser-in-Rust project started by Mozilla in 2012 that produced the Rust programming language, is getting re-activated with four new full-time developers provided by Igalia.

Igalia are a fascinating organization - I hadn't realized quite how influential they've been until I read their Wikipedia page just now

They've been around since 2001, and "in 2019 they were the #2 committers to both the WebKit and Chromium codebases and in the top 10 contributors to Gecko/Servo" - including implementing and maintaining CSS Grid Layout!

Via Hacker News

This post started as a discussion on Mastodon where a new blogger asked for feedback. The discussion turned to one of Miguel Guhlin's recent posts, which was a great example of a quality blog post. Guhlin mentioned that he had a formula, someone asked for it, and here it is. "If you start out with a blank page, writing can be hard," he writes. "But if you're following a formula, you're filling in the empty spots with words. When you're done, you edit and revise until you have something." This post mentions only a few of the many formulae that are available to authors: the list, the inverted pyramid, visual storytelling. There are so many more: the chronology, past present future, thesis antithesis synthesis, strength weakness opportunity threat (SWOT), four-corners analysis, etc., etc. etc. Here's a whole list of them.

"Where is the humanities toolbox for creating or making a better world?" asks Robert Danisch. "My pitch to reverse this trend," he writes. "Let's teach students to be makers, builders and creators, the architects of the future, and not just demolition crews." I completely agree; this was my thinking behind "the joy of ethics". We are given an opportunity not only to identify what's wrong, but to build something that's good and right. "The humanities aren't just instruments of criticism," says Danisch. "They also hold the keys to how we might imagine a better world, particularly through their emphasis on the public good and the skills necessary for constructively managing difference."

I finished this audiobook last week, and I’m taking a few minutes to write down my thoughts before I forget them. It’s a hard thing to remember to do! The book is called Gangsters of Capitalism, by Jonathan M. Katz. Superficially, it’s a biography of the American Marine officer Smedley Butler, best known, I think, for his part in the Business Plot of the 1930s. But more substantially, it’s an examination of America’s practice of military imperialism between about the Spanish American War and World War II — a series of invasions of countries around Latin America, the Pacific, and Asia.

Butler was a Quaker from Philadelphia who volunteered for the Spanish American War as an underage teen. His father was a congressional representative, so Butler became an officer. From his first battle experience in Cuba, he went on to participate in almost every important military invasion of the first half of the 1900s — the Philippines, China, Nicaragua, Honduras, Haiti, Mexico, the Dominican Republic, on and on.

The structure of the book has a consistent rhythm. First, we see the high-level politics that use extremely flimsy excuses to incite a military invasion of yet another country. Then, we focus down on Butler himself, as he’s shipped from his previous post to a new combat post in the new country. There’s a discussion of the military campaign, both in macro form and in the specific fighting Butler is part of. As invasion becomes occupation, Butler’s wife and children often join him in the occupied zone. There’s then a recounting of the invaded country’s history from the invasion until today, and finally the author’s first-hand account of visiting the country in the 2010s or 2020s for the book.

I’m not a scholar of American imperialist history, so I’d mostly heard of these campaigns as isolated trivia, like, “Did you know the US invaded and occupied Haiti for a decade?” It’s interesting to see the stories connected and presented together, which gives more of an understanding of the process as whole, rather than isolated parts. The focus on Butler’s thoughts, experiences, and ambition gives a human perspective into the process. He was a man trying to live an ambitious 20th-century life, whose career was primarily about overthrowing governments to help shady US corporate interests.

Butler retired in the late 1920s, and was supportive of the Bonus Army movement of the early 1930s to get support for American veterans. According to Butler, a cabal of fascist-leaning corporate entities read his populism as being anti-Roosevelt, and contacted him to lead a military takeover of Washington D.C. He testified to Congress about this effort, called the Business Plot. The named plotters denied the charges, and Butler was decried as a kook. He turned against the imperialism of his time, and wrote the book War is a Racket in 1935, comparing military expeditions to organised crime. He died just before the US entered World War II.

The book is wonderfully done, zooming up and down in perspective from national to individual, back and forth in time from the 1900s to 2020s, and around the globe from China to the Americas. It’s hard to capture the tenor of those times and their effects on our own; it was a really good idea to look at it through the lens of Butler’s life. Katz’s modern day research — traveling to remote parts of each country to visit the places Butler lived and fought — is thorough, amusing, and really interesting. It’s amazing to think of how quickly our colonialist history disappears into ruins. A consistent theme of the modern era sections of the book is how important the invasions were to Filipinos and Mexicans, for example, whereas Americans have largely forgotten them.

The audiobook was a treat; well read by Adam Barr. I’d recommend it to anyone interested in modern American history.

Igalia: the Open Source Powerhouse You’ve Never Heard of

An in-depth article about Igalia from July 2022. I had no idea how much stuff they had worked on: arrow functions, generators, async/await, MathML, CSS Grid and a whole bunch more.

Via Brian Kardell

Recently my friends at Crisp invited me to participate in a special virtual conference where a set of technology speakers, each with distinctly different points of view, would discuss how teams and companies can best deal with the challenges of complexity. My views on formal process, and how Agile has been taken over by process...

The post Principles over Process appeared first on Silicon Valley Product Group.

mkalus shared this story from The Breach.

For years, Mountain Equipment Co-op (MEC) was a giant in Canada’s cooperative landscape, and something of a business success story. At its peak, it had about 5 million members, and around $700 million in annual revenues. 

That was all possible because MEC was a cooperative, owned and controlled by its members, for their needs. The cooperative was freed from making profit, because its only mandate was to serve the membership.

For years, MEC operated in a way that broke in significant ways from profit seeking, prioritizing world-leading environmental practices and a cooperative right-to-repair mindset. In the early 2000s, for example, a member could buy hiking boots made from recycled materials at a reasonable price. If the boots wore out after a few years, they could go back to the same energy-efficient, passive-solar store and get them repaired or replaced for free. 

But in the 2010s, the board made big moves to distance itself from member input, putting up barriers to participation and requiring approval to run for the board. 

And one October morning in 2020, members woke up to shocking news: MEC had, in secret, declared bankruptcy and sold all of its assets to a California-based private equity firm. It is now the “Mountain Equipment Company,” run for profit and no longer owned by its members.

In response to the news, thousands participated in a last-ditch effort to save the cooperative. Kevin Harding, a Vancouver-based public policy professional who works with cooperatives and community enterprises, was at the centre of the spontaneous mobilization which occurred online during the height of the pandemic.

The Breach’s Dru Oja Jay spoke to Kevin via video conference from Quebec. This transcript has been edited for length and clarity.

The Breach: So what happened at MEC, from your perspective?

Kevin Harding: Mountain Equipment Co Op—MEC—was formed in 1971 by a group of students at the University of British Columbia who were members of a hiking club. They couldn’t buy climbing gear in Canada, and the easiest way to get it was to drive down to Seattle and buy it from REI, the American cooperative. They built a buyer’s club, essentially—they’d collect order sheets, go across the border, and probably smuggle it back, to be perfectly honest.  

Over the years, they built up this organization into Canada’s largest consumer cooperative, a multi-hundred-million-dollar business. When they started in 1971, they very consciously built it as a co-op, with a solid intention to meet needs and not put profit above access. 

They started by selling out of a VW van, they would park and people would come and shop from there. The first tents that they sold were made by members—it was cooperative labor and production. They started out in an attic in someone’s house in Kitsilano. And over time, they bought a store, and put in a workshop, still making equipment. 

As they grew over time, they put principles into their purchasing—eliminating waste, sourcing sustainable materials—but they also looked at human rights. There were ups and downs with those policies throughout their history, but that was core to their business. As they grew, they connected with other businesses and other provinces, they merged with other cooperatives and grew themselves into a pan-Canadian organization. 

Democracy was in the heart of the business. At certain points, members had actually a fair bit of influence in the way that the co-op was run. There were debates over whether or not things like products from occupied Palestine should be sold in stores, or whether products connected to arms manufacturers should be sold in stores and, and members made their voices heard. 

It was fascinating—an alternative way of doing business. But around the 2010s, the business took a really significant turn in how it managed itself, its principles and its commitment to cooperativism. And it ended in 2020, in that really heartbreaking manner.

So there’s a typical story in cooperatives, where expansion leads the co-op to look for experienced staff, who of course are likely to come from the mainstream business sector and are really geared toward maximizing profit, so you have a shift in culture. Is that how the shift in MEC that led to the selloff happened, or was there something else?

The expansion of MEC was a bit of an adventure. You often had examples of growing pains. In the late 1990s, when MEC started carrying things like kayaks or canoes, or bicycles, some of the earlier members were really upset at that point, because they saw that as a diversion from the core principle of the cooperative, the first word in the name. Members had a debate, the board had a debate. 

The idea that prevailed in these discussions was that the core purpose of a cooperative is to meet members’ needs, not to generate profit. If members of the co-op have a problem—an access-to-market problem, an access to resources or finance problem—the best way to solve it is by working together and supporting each other. 

So when MEC went in the direction of carrying things like kayaks, it was because members couldn’t access something and decided to provide those services and those products to help each other out. But it seems to have been around the 2010s that there was a shift away from growing to meet members’ needs and toward growing the membership to meet the business’s needs. 

It’s a nuanced shift, but it was fundamental, because you go from “what do our members need” to “how many members do we need to grow our business?” At that same time, they started hiring from the private sector. They hired executives from large retail businesses to drive their product strategy and their marketing strategy. 

There were little signals that people saw from the outside. One was the abandonment of the name Mountain Equipment Co-op for the logo MEC. Some saw that as a hint of problems to come. There was a really significant restructuring of the internal democracy of the cooperative.

I once considered running for the board, and I had experience on the board of university, a multi-hundred-million-dollar public institution. And even then the people assessing board memberships determined I didn’t have the background that was the minimum criteria to be considered for election to the board. These little things added up to start to create problems.

Once you narrow the range of perspectives and opinions that sit around the board table, your members are not being heard. The direction of the business goes in one direction while members want it to go in another. That’s the beginning of that chasm.

I appreciate how structural that analysis is. It’s not like an individual failing, but more of a systemic look at what happened there. Even from that perspective, it’s hard not to look at what happened and think that there’s a nefarious thing that happened—like somebody made a decision at some point: “we’re gonna keep this secret and sell it off, we’re gonna hoodwink the members.”

Yeah in 2020, we’re able to look back and say, “okay, in 2016, they did this and in 2013, they hired this person.” They hired some people from mass market jewelry and clothing chains to be senior executives. You can point to that, but those individuals and their values were, I think, symptoms of the bigger thing. 

There were bigger strategic issues, a change from their initial strategy as an organization to “let’s grow the members as much as we can by throwing everything we can at them.” So one vision is mass market retail, where you can walk into a big box store and not walk out for a week, because it’s full of products that you never knew you needed. 

The root business problem here was that MEC went into this expansion phase, like you’ll see with a lot of other businesses. They thought, “you know, Calgary needs three MEC stores, Saskatoon needs two, Vancouver needs four.” 

And the alternative is fewer smaller, focused stores with a curated selection of products founded on understanding the member-owners rather than growing the customer base. 

It’s that shift. Individuals enabled that shift, and then a narrowing of perspectives cemented that shift.

You were on the frontlines of this sort of battle—when the cooperative announced it was going bankrupt and selling all its assets to a private equity firm in California, there was a legal challenge. Thousands of people gave time and money to help with the effort to stop the sale. What did that effort look like?

I still look back and think about how, at the height of the COVID pandemic, we managed to somehow pull together tens of thousands of Canadians who connected the only way that we could online—through WhatsApp and Facebook—to oppose this giant corporate sale. It was bizarre, but it was also the spirit of the co-op trying to assert itself. 

It started with this surprise announcement on CBC radio that the co-op was being sold. I remember being heartbroken because cooperatives are core to where I see society needs to go to build a better possibility. I wanted to get out there and just share with people that this was probably a business failure—you know, numbers on a spreadsheet were wrong, somebody made a poor strategic decision—it wasn’t a failure of the co-op model. 

And it turned out that there were hundreds of people talking about that when I woke up in the morning because the news had already hit in the east. There was a Zoom meeting where there were organizers with lots of experience in political party organizing or union labor organizing. A great group of people was convened and, and really everyone pulled together. I’d had experience with co-ops in the past, helping with business development. We had communications experts, people were translating everything into French, we had an online campaigner who gave us the tools and the framework to run a campaign. People behind the scenes were pulling together experts and lawyers—everything came together. 

It was really the heart of what a co-op is supposed to be. We had a problem that we wanted to solve—this organization that we owned, and that was ours, was being sold out without notice, without consultation—and we needed to organize. It was just an astounding thing to experience.

Can you just sketch out like how that played out? Do you feel like you ever came close to being successful?

It’s such such a good question—how close did we get? It’s fascinating. This group of people existed on Slack and Zoom meetings for weeks. We pulled together tens of thousands of people, received advice from large cooperative financial institutions. We were remarkably close to being able to save the co-op. 

What ended up hitting us was the legal system. 

MEC was in a position where the money that it owed was bigger than the money that it had. The pandemic had hit them really hard, so they weren’t generating a huge amount of revenue. They had been secretly sort of shopping around to get a lease or a loan, and it was hard to do that. 

They were talking to banks, and banks didn’t want to do this for them. Their financial advisors told them, “you really need to start looking around to sell.” And then this US capital firm showed up and said, “we’ll buy it.”

Part of that agreement to sell the business assets was that they’d go under the Canadian Companies’ Creditors Arrangement Act—federal bankruptcy legislation dating back to the Great Depression—that allowed them to circumvent all the bylaws of the co-op and all the provincial laws. So you have a judge stamp a piece of paper that says, despite everything else, you can sell the business.

MEC executives then went to the Supreme Court in British Columbia to get approval to start the sale. That was when members first heard of it, and started organizing—we started a petition, started an email list. We were just outraged people meeting together. But some law firms stepped up, with pro bono support. 

We felt it was important to pay them for some of the hours, and members across the country and around the world stepped up and put money into a GoFundMe. 

Lawyers told us they had a lot of experience with these kinds of bankruptcy arrangements, which allow companies to walk away from pension obligations, from unpaid wages and deliver assets to other corporate buyers. 

They told us that the only way we’d succeed is if we pitched another way to satisfy the loans, allowing the cooperative to continue. So we pulled together advisors from the co-op and credit union financial sector from across the country. 

There was a very real possibility that had there even been five business days worth of time to look at the books of MEC, that the credit unions and the financial institutions would have been able to meet their due diligence obligations. A credit union is owned by members—it can’t just risk money, because that’s people’s homes and savings. 

But if they’d gotten enough time to look at the books—the company still exists, it’s still doing business. There was a valid case to be made. They could have been able to put up sufficient funds to bridge the co-op through the challenging time, and we would have been able to pull together a board of experts that could have stepped in and run the co-op and rearrange things keeping jobs and activities going. 

But this law from the 1940s is not written to allow that. It’s designed to satisfy banks and lenders. It’s not designed to prioritize social capital or communities. And because of the way that law is written, and because the judge is duty-bound to follow the law, that reasonable five day pause to look at the books wasn’t permitted. 

There are obviously critiques of the big cooperative financial institutions. But the fact that all of this cooperative infrastructure could have been mobilized to keep MEC alive, if only the co-op’s executive had any inkling or intention of operating in a cooperative context, as opposed doing a fire sale is pretty remarkable.

Yeah it was eye-opening! 

Their loan was from the big five banks in Canada. The credit union sector had always been a little bit upset that MEC—Canada’s largest retail co-op—hadn’t done loans through the credit union sector, because there would have been aligned interests and principles. The banks just want their bills paid. 

Credit unions want to support communities in their members in a different direction. So that was another opportunity, in the 80s, or the 90s, or the early 2000s, that should have been taken because you can build the system in a different direction. 

At some point MEC people were saying “well, you can’t just stop growing when you’re that big, because of various financial imperatives.” That was the justification for them opening up all these urban boutique yoga shops. What do you make of that case that was being made at the time?

This is a core question to a lot of co-ops right now, in the country. There’s always going to be some degree of growth that a cooperative business needs. Members may resign their membership, they may leave the country, they might not need that business any longer. So you need to have that base of core members interacting with you to sustain your business. 

But one of the core differences between a co-op and a private company is that the members or the owners and members are mainly interested in having a place they can buy their hiking boots or get their bike repaired. They’re not investors that are looking for a 3.5% annual dividend. When you’re in an investment-oriented firm, you need to generate growth to satisfy the investors. 

But when you have a leadership team at a retail co-op that doesn’t understand that growth isn’t actually the imperative for a co-op as it is for a private business, then you get into the zone where you need aggressive expansion and expand your product selection massively. Growth is something that you can plan for carefully, but that’s not your core imperative, as a cooperative business.

So coming back to the Supreme Court—B.C. has an NDP government at that moment, and David Eby is Attorney General.  So was there nothing he could do, or is there a lack of motivation to do it?

This is one of these moments of utter heartache. When the members came together and started to organize, there were immediate connections to the provincial government in British Columbia. A number of cabinet ministers and elected MLAs had been on boards of credit unions or boards of co-ops. They know the history of the NDP as the Cooperative Commonwealth Federation, which had a core philosophy of replacing the economy with cooperatives. So they get the idea. 

But unfortunately, three days after this started happening, the Premier called an election. And so at that point, there was literally nothing the province could do. The government goes into this zone of no commitments, just maintain the ship of state. 

There was something the province could have done. We were at the height of COVID, all levels of government were throwing billions of dollars at businesses to sustain economies. This was like a $300 million loan that MEC needed to solve itself. And the products that MEC makes, a lot of them that could be made in British Columbia. This would have made a perfectly reasonable economic case at the height of COVID. We had lines of communication with cabinet ministers, but they all had to stop because the election was called. The timing was unfortunate.

That’s a great point. Why can’t a consumer co-op—like Seikatsu did in Japan—go upstream and say “okay, let’s start using this $700 million in revenue that we have, and channel some portion of that to worker coops”? Can you say more about those possibilities?

From 2013 to 2016, I was working in the co-op sector, and I actually did a fair bit of work with MEC. It was always an adventure, because I’d hear from members and old people organizing around the democracy in the co-op. 

And I can remember being at the MEC store in Quebec City and the staff members there had shirts that said “membre et équipe”—member and staff. On the English side, there was none of this dual terminology—that connection to cooperativism. 

One of the criticisms that members had organized around, is that a lot of MEC’s production had been moved overseas into potentially quite exploitative environments. There had been a push to either improve working conditions or to meet human rights standards. MEC had been working on that—if you believe their reports. There was also a lot of research into innovative product design—new ways of designing clothing, new fabrics that meet outdoor conditions. 

There’s a government program to support that kind of innovation in B.C., and in other provinces and federally there are tax benefits and other incentives for bringing production back to Canada—to employ people who reside in Canada to design and build these products and then sell them through a cooperative. There was a lot of potential there. That was something that people in government had been interested in. 

But timing was our enemy here, because the board did all of this in secret, and didn’t ask anyone their opinion before they sold.

The post Mountain Equipment Chaos appeared first on The Breach.

2023 update to this 2019 poster; right click and open in a separate tab to view or print full-size
Dire warnings about imminent or ongoing economic or ecological collapse always used to be followed by obligatory “It might not be too late; here’s what we need to do” conclusions. But recently, many of the leading writers about collapse have ceased proffering ‘solutions’ or even ‘preparations’ for it, because we can’t possibly know precisely how collapse will unfold, or its timeline.

Instead, writers about collapse are acknowledging it as a natural and inevitable consequence of large-scale systems that fall out of sustainable equilibrium. But this is cold comfort to those of us who, naturally, want to “do something” about it.

Collapse is not a new phenomenon, and it occurs at many levels and to many different types of unsustainable systems. So I’ve begun to think about how people have coped in the past with the collapse of other systems — ecosystems, businesses, military campaigns, nations, and even systems of thought. If it’s all going sideways and you can’t fix it, mitigate its effects, or prepare for it, what can you do?

If your army is facing certain defeat, how do you deal with that reality? An orderly retreat makes sense, rather than denial or panic. Same thing if your business is insolvent and facing bankruptcy, or if your product is becoming obsolete because of competitor innovations.

If you’re a gatherer or hunter, and epidemics or floods or fires or other ecological changes have suddenly depleted available resources, what do you do? Yell at the gods, or move on to another area? Pray for salvation, or adapt in place? If your country is being balkanized because it no longer meets the needs of its diverse citizens, do you embark on a civil war, or renegotiate a confederation of nations that makes more sense for all? If your Theory of Everything suddenly comes apart because of new scientific evidence that totally undermines it, do you try to kill the messengers, or do you start over, with a new theory that the evidence supports?

The answers to these questions depend, of course, on the context. But generally, denial, rage, panic, blaming others, and doubling down are pretty bad strategies.

I wrote about collapse preparation a couple of years ago, before CoVid-19, suggesting we might be better off learning to be more adaptable so that we’re ready for anything, rather than trying to prepare for a specific scenario that might never arise.

In the intervening period, we have seen enormous fragility and vulnerability of many of the collapsing systems of our culture, political, economic, ecological and social. There’s also a growing consensus that dealing with collapse is going to require strong communities, and in much of the world people now live in huge cities, or in isolated or polarized areas, where there is often little or no sense of community at all.

So I’ve updated my “Being Adaptable: Reminders to Myself” poster to include more on community-building, autonomy, creativity, and dealing with loss, since it may be time to start devoting some more attention and energy to these important aspects of adaptability.

Here are a few notes on the changes:

• I’ve reworded the “reminders” so they’re in the first person instead of the second
• I’ve added cooperation and acceptance to the 3rd reminder; I’ve found it helps to keep reminding myself, again and again: We’re all doing our best; no one is ‘to blame’ for the mess we find ourselves in.
• I’ve added the suggestion that I rationalize all the disparate ‘communities’ I am a part of, and home in on trying to move them from virtual to face-to-face physical communities, as the timeline for the end of air travel looms closer. I know who I’d love to have on my ‘team’ helping my local community fend with widespread system collapse, and right now they’re all over the place, in more senses than one.
• I’ve also acknowledged that the community I am in when all hell breaks loose is going to have to consist of people who have skills and passions I lack, and vice versa, and that’s a good thing, though it will take some adjustment to learn to “love people [in my community] I don’t particularly like”.
• I’ve added a clause on helping to build sustainable, truly collaborative, cooperative local enterprises that care about meeting real needs, so that my community will be a little better able to deal with the collapse of large corporations, large systems (health care, energy, education, food, infrastructure), and the disappearance of most global trade.
• I’ve added a clause on creativity and imagination, because unless we encourage and enable more of this (the trend is in the opposite direction), we will be unable to come up with necessary innovations to deal with what are now likely unimaginable and novel new challenges.
• And I’ve added a clause on letting go, which I am currently very incompetent at. I hate admitting I was wrong, I hate losing things and ways of living I’m attached to. I am going to have to be more agile in changing my beliefs, and changing my ways of doing things, than I have been in past, and not take such losses so personally.

I hope you find the updated poster useful, or at least interesting, as the challenges of coping with collapse deepen. These “reminders” remain mostly about how to be (when TSHTF), rather than what to do, because it’s still too early to know with any certainty what we will have to do.

If there are other “reminders” that are helping you deal with the accelerating crises of our time, I’d be interested in hearing them.

Semantic search answers: Q&A against documentation with GPT3 + OpenAI embeddings shows how Datasette can be used to implement semantic search and build a system for answering questions against an existing corpus of text, using two new plugins: datasette-openai and datasette-faiss, and a new tool: openai-to-sqlite.

I wrote several posts last year about why teaching or learning programming is much harder than actually programming ([1], [2]) and about the tools I wish we had to make this easier ([3], [4], [5]). Looking at what’s supposed to be the final draft of the Python version of Software Design by Example, I don’t think I’m going to be happy shipping it until I can incorporate at least one new explanatory technology. Before I start building anything, I’d like to know if there’s already a tool (preferably written in Python or JavaScript) that will take two text files as input and produce the sort of side-by-side diff that many version control GUIs display:

The catch is that the result must still be text that can be copied and pasted (and that can be accessed by screen readers and other accessibility aids), not an image or an SVG with characters placed one by one. The connecting lines can be drawn using SVG, though, or with a canvas element or whatever else clever kids are using these days. And yes, this only solves half the problem—I’ll need to figure out some way to produce LaTeX in order to create the PDF my publisher needs—and yes, I’ll need to embed this in a pipeline that selects small portions from larger before-and-after files so that I can show the differences between selected sections rather than the differences between entire files, but I know how to figure out all that stuff.

Similarly, along with memory diagram snapshots, I’d like a tool that illustrates the execution order of a small snippet of code:

I can create all these diagrams by hand, but (a) it takes a lot of time and (b) maintenance is a killer: if I make even a small change in an example I have to check and re-do all the diagrams, and if I decide to change fonts or line styles I have to re-do all of them. If you have solutions, I’d be grateful if you would reach out.

Introducing sqlite-xsv: The Fastest CSV Parser for SQLite

Alex Garcia continues to push the boundaries of SQLite extensions. This new extension in Rust wraps the lightning fast Rust csv crate and provides a new csv_reader() virtual table that can handle regular, gzipped and zstd compressed files.

By Jennifer Elrick, McGill University

The illusion of science that surrounds these numbers and their emotive force make them powerful political tools that need to be better understood by the public to avoid provoking anti-immigration sentiment.

Immigration targets are immigration bureaucrats’ best guess, based on institutional experience and analyses, of how many people can join a society and economy without threatening social cohesion.

In other words, they are estimates of what Canadian immigration bureaucrats have historically referred to the country’s “absorptive capacity.”

Writing in 1948, a high-ranking civil servant called the concept “difficult if not impossible to measure,” while noting that it includes factors such as the population to land ratio (accounting for expected standards of living), demographic trends, employment opportunities and immigrants’ economic, social and human capital.

Based on these considerations, the federal government has aimed since the 1950s — and, as shown below, mostly failed — to bring in the equivalent of one per cent of Canada’s population annually through immigration.

Immigration intake

Ottawa’s announcement late last year of plans to raise its immigration target to 500,000 a year by 2025 is therefore unremarkable from a policy perspective.

Statistics Canada’s low-growth scenario for Canada’s population in 2025 is 39,861,100, which would make the planned immigrant intake equivalent to 1.25 per cent of the population.

The problem with immigration targets entering public debate is that they eliminate nuance while raising anxieties. Like governments, the general public is concerned about “absorptive capacity,” but it seldom has access to the kinds of detailed academic research on the economic and social integration of immigrants that civil servants do.

Integration successes and challenges can vary by someone’s human capital (for example, education and linguistic ability); how they enter the country (for example, as a skilled worker or a spouse); their age and macroeconomic conditions upon arrival; broad institutional contexts (including the structure of labour markets); racial or ethnic discrimination; and the quality of their social ties in Canada.

None of the nuance needed to answer the question of how many and what kind of immigrants have a positive or negative effect on society is reflected in a single, large number.

Absent a more complete picture of immigrant integration dynamics and outcomes, immigration targets can easily activate public anxieties about immigrants threatening social cohesion by increasing competition for resources like health care, housing, education and desirable jobs, or by creating what some might regard as too much socio-cultural diversity.

Scapegoating the stranger

This is evident in Québec Premier François Legault’s declaration last year that admitting more than 50,000 immigrants a year to the province would be “a bit suicidal.”

This statement followed former Québec immigration minister Jean Boulet’s claim that “80 per cent of immigrants (who) go to Montréal don’t work, don’t speak French or don’t adhere to the values of Québec society.”

Such statements are neither new nor exclusive to Québec.

Immigrants have long been the scapegoats for broader political failures. Just ask Georg Simmel, a Jewish sociologist in early 20th-century Germany, who identified “the stranger” as a key position in modern societies and one often held by immigrants.

Economic insiders but perpetual social outsiders, “strangers” are at once appreciated for their economic utility and easily declared an “inner enemy” when troubles arise.

The scapegoating of “the stranger” is politically expedient, as it allows both political leaders and dissatisfied citizens to avoid painful questions about the real sources of their troubles, including long-term trends in social, health and housing policies, which require sustained political efforts to fix.

Fuelling polarization

When immigration targets cause public anxiety, they can fuel political polarization and be used by politicians to justify harsh immigration policies.

Take the case in the United Kingdom. In 2010, Conservative Prime Minister David Cameron announced that immigration needed to be limited to “tens of thousands” rather than “hundreds of thousands” of people per year to reduce pressure on public services.

Plucked out of thin air during a television appearance, the “tens of thousands” guesstimate, which became known as the “net migration target,” has had a large impact on policymaking and public perceptions of British immigration over the past decade.

The idea that there were too many immigrants being admitted by a factor of 10 helped to justify a new, restrictive immigration policy — as well as Brexit — as a means of limiting immigration from Europe under the European Union’s freedom of movement clause.

The target also became a justification for creating a hostile environment for immigrants in the United Kingdom, whereby controls on legal status have been integrated into everyday settings like classrooms, health-care facilities and workplaces.

Guesstimates like immigration targets can be useful and expedient for policymaking. But in the public arena, they need to be more fully understood if immigration debates are to be grounded in evidence.

Jennifer Elrick, Associate Professor of Sociology, McGill University

This article is republished from The Conversation under a Creative Commons license. Read the original article.

The post The problem with immigration targets: they’re “guesstimates” easily misunderstood by the public appeared first on Pancouver.

The first work week of this year was a filled one. As is implied by sparse blogging (like the last few months which were very busy too). However it looks like this was the last peak, and the coming weeks allow more time to actually focus on a few things, not just running to the next deadline.

This week I

• Prepared some talking points and had a conversation with the new board member of a client, about the work I do for them
• Had a meeting with my businesspartners discussing the final financial numbers of 2022 and determining the yearly bonus for the team. We had about 10% more turnover than last year.
• Later in the week had another meeting to finish the 2023 budgeting. Salaries are rising significantly, as we’re indexing in line with the statistics office inflation figure for 2022, and there are the regular yearly increases for our team members. Our portfolio is well filled however and if we do the work, in terms of turnover should be more than ok to cover those salaries.
• Did a lot of small things and several meetings to do the final preparations for a meeting with a minister next week. Quite amazing the level of preparation and detailed planning that goes into such working visits.
• Had the weekly client meetings
• Started the week with a empty-my-head exercise to create an overview of all the things going on in the various projects I’m involved in, and to choose the starting and focus points after returning from holidays.
• Had a pleasant conversation with Frank Meeuwsen about networked agency and personal professional development of teams. He booked a conversation with me through my ‘open hours‘. Fun to catch up.
• Took Y to her weekly swimming lesson. This was her first swimming lesson where she was fully clothed. It means she’s getting closer to getting her swimming diploma. The Dutch benchmark for a swimming certificate is being able to manage situations where you accidentally end up in the water, meaning you need to be able to swim distances, also under water, with clothes and shoes. Not merely a hypothetical situation, given the many surface waters in our part of town.
• Helped E set-up her own VPS and Yunohost.
• Spent Sunday morning with E and Y visiting the Rijksmuseum Amsterdam. Y wanted to see the Nightwatch (currently being restored in situ), so we did.
• Visited a few new years meet-ups, from clients, and today the meet-up of the open source geo community, which took place in Hilversum, on the way back from visiting the Rijks in Amsterdam.

Portrait of Ramses Shaffy (1933-2009) as ‘metro map’ of the life lines of people important to him (24 by 13 meters). By Marjan Laaper. At Vijzelgracht metro station, near the Rijksmuseum

Neighbourhood police station 1919-1938 Hilversum, designed by Dudok

Late yesterday, The Information reported that it had seen internal Twitter Slack communications confirming that the company had intentionally cut off third-party Twitter app access to its APIs. The shut-down, which happened Thursday night US time, hasn’t affected all apps and services that use the API but instead appears targeted at the most popular third-party Twitter clients, including Tweetbot by Tapbots and Twitterrific by The Iconfactory. More than two days later, there’s still no official explanation from Twitter about why it chose to cut off access to its APIs with no warning whatsoever.

To say that Twitter’s actions are disgraceful is an understatement. Whether or not they comply with Twitter’s API terms of service, the lack of any advanced notice or explanation to developers is unprofessional and an unrecoverable breach of trust between it and its developers and users.

Twitter’s actions also show a total lack of respect for the role that third-party apps have played in the development and success of the service from its earliest days. Twitter was founded in 2006, but it wasn’t until the iPhone launched about a year later that it really took off, thanks to the developers who built the first mobile apps for the service.

Twitterrific’s website in 2009. Source: The Wayback Machine.

By 2007 when the iPhone launched, The Iconfactory’s Twitter client, Twitterrific, was already a hit on the Mac and had played a role in coining the term ‘tweet’. The app was first to use a bird icon and character counter too. And, although the iPhone SDK was still months off, The Iconfactory was already experimenting with bringing Twitterrific to the iPhone with the help of a jailbroken iPhone and class dumps of iPhone OS.

By the time Twitter bought Tweetie, Twitterrific was already on version 3.0.

Despite the popularity of the iPhone, Twitter didn’t build its own mobile app. Instead, the company purchased Loren Brichter’s Tweetie in 2010. Not only was Tweetie a beautifully designed app that performed better on early iPhones than many of its competitors, but it also introduced the world to ‘pull-to-refresh, a UI detail that was later folded into iOS itself. Twitter re-skinned Tweetie as the official Twitter app, followed up not long after by a flawed iPad app, and later, a Mac version in early 2011 that Brichter had been working on at the time of the acquisition.

Tweetbot 1.0.

Even after Twitter had its own suite of apps, the third-party app market flourished. Tweetbot by Tapbots came along in 2011 and quickly became a favorite of many users, distinguishing itself with its steady stream of new power-user features and thoughtful design. But it wasn’t long before Twitter’s relationships with third-party developers began to sour. It started with a vague set of rules introduced in 2012 that preferred CRM and analytics apps over clients like Twitterrific and Tweetbot. The ups and downs over the years that followed are too numerous to count, but the consequence was that for many years few new Twitter clients were developed.

With the introduction of version 2.0 of its API, third-party Twitter apps, like Spring, gained new, innovative features.

However, relations began to thaw with the announcement of version 2.0 of the Twitter API, which went into effect in 2021. Not only did the API update make new features available, but Twitter promised to loosen restrictions on third-party developers. That led to renewed interest in third-party client development, resulting in innovative new features in apps like Spring, which was a runner-up in the Best App Update category of the 2022 MacStories Selects awards.

As it turns out, Twitter’s developer detente was short-lived. The fact that Elon Musk’s Twitter has cut off third-party developers isn’t surprising. Franky, I expected it sooner, but I didn’t expect it to be done with an utter lack of respect for the developers who played such a critical role in the service’s success for more than a decade. The number of people who use third-party Twitter apps may be small in comparison to the service’s overall user base, but the role of the developers of those apps and the value of those power-users to Twitter’s success is outsized by comparison. The developers of Twitterrific, Tweetbot, and every other app that has lost access to Twitter’s APIs deserved better than a silent flip of the switch late one Thursday night.

There have been many moments over the past decade when we worried that third-party Twitter apps might have met their end and didn’t. Unfortunately, I think this time, those past fears have been realized.

The MacStories team moved its social media presence to Mastodon in mid-December. So, if you’ve lost the use of your favorite third-party Twitter app and are looking for an alternative place to keep up on everything Apple, you can follow us there:

• MacStories Accounts
  • MacStories
  • AppStories
  • Club MacStories
• MacStories Team Accounts.
  • Federico
  • John
  • Alex
  • Silvia

Tapbots’ Ivory beta.

Also, Tweetbot users will be happy to know that Tapbots is working on a Mastodon client called Ivory that is currently in beta and should be released soon. We’ll have coverage of Ivory on MacStories as soon as it’s released publicly.

Support MacStories and Unlock Extras

Founded in 2015, Club MacStories has delivered exclusive content every week for over six years.

In that time, members have enjoyed nearly 400 weekly and monthly newsletters packed with more of your favorite MacStories writing as well as Club-only podcasts, eBooks, discounts on apps, icons, and services. Join today, and you’ll get everything new that we publish every week, plus access to our entire archive of back issues and downloadable perks.

The Club expanded in 2021 with Club MacStories+ and Club Premier. Club MacStories+ members enjoy even more exclusive stories, a vibrant Discord community, a rotating roster of app discounts, and more. And, with Club Premier, you get everything we offer at every Club level plus an extended, ad-free version of our podcast AppStories that is delivered early each week in high-bitrate audio.

Here’s an example from about a month ago, showing the fogging that can occur when it is cold, and your breath is directed upwards by a covering over your mouth. When is is really cold, the fog can also start freezing to your glasses.

North45 is a company that sells balaclavas that claim to eliminate fogging of either ski goggles or glasses when you otherwise need full facial protection from the cold. I got one of their balaclavas for Xmas, but until yesterday it really wasn’t cold enough to put it to the test.

The model that I got is meant for glasses rather than ski goggles. Here is what it looks like from the side. It is a medium weight merino wool balaclava with an extra band of cloth.

When you bring this band down to the correct position, then it covers the lower half of your face, and the top edge of the band has a bit of wire or metal that gives a good fit around your nose.

This is what it looks like in operation. Bottom line: no fogging of glasses.

It is gets too warm, it is easy to pull the band down to your chin. Also, the fact that it is merino means that if it gets a big soggy from your breath, it will still keep you warm, and in fact the flap stays further from your face than a regular balaclava.

The reason it works is that the band is open at the bottom so that your breath is all directed downwards away from your glasses. I can imagine rigging up something by taking an N45 mask, cutting big vents in the bottom, and combining it with a balaclava to get much the same affect, but it wouldn’t quite provide the same level of protection when it is really cold.

At any rate, it is a high quality item. It is made in China, but buying it supports a small company in the Canadian Rockies. From the postal markings on the package I see that mine passed through both Lake Louise and Canmore AB.

Here’s to hoping that I’ll actually not using it much this winter. At -9°C yesterday and this morning, it was too warm.

This morning on the exercise bike, I read a big chunk of Daniel Gross and Tyler Talk Talent, from the Conversations with Tyler series. The focus of this conversation is how to identify talent, as prelude to the release of their book on that topic.

The bit I've read so far has been like most Conversations with Tyler: interesting ideas with Cowen occasionally offering an offbeat idea seemingly for the sake of being offbeat. For example, if the person he is interviewing has read Shakespeare, he might say,

"Well, my hypothesis is that in Romeo and Juliet, Romeo and Juliet don't actually love each other at all. Does the play still make sense?" Just see what they have to say. It's a way of testing their second-order understanding of situations, diversity of characters.

This is a bit much for my taste, but the motivating idea behind talking to people about drama or literature is sound:

It's not something they can prepare for. They can't really fake it. If they don't understand the topic, well, you can switch to something else. But if you can't find anything they can understand, you figure, well, maybe they don't have that much depth or understanding of other people's characters.

It seems to me that this style of interviewing runs a risk of not being equitable to all candidates, and at the very least places high demands on both the interviewee and the interviewer. That said, Gross summarizes the potential value of talking to people about movies, music, and other popular culture in interviews:

I think that works because you can learn a lot from what someone says -- they're not likely to make up a story -- but it's also fun, and it is a common thing many people share, even in this era of HBO and Netflix.

This exchange reminded me of perhaps my favorite interview of all time, one in which I occupied the hot seat.

I was a senior in high school, hoping to study architecture at Ball State University. (Actual architecture... the software thing would come later.) I was a pretty good student, so I applied for Ball State's Whitinger Scholarship, one of the university's top awards. My initial application resulted in me being invited to campus for a personal interview. First, I sat to write an essay over the course of an hour, or perhaps half an hour. To be honest, I don't remember many details from that part of the day, only sitting in a room by myself for a while with a blue book and writing away. I wrote a lot of essays in those days.

Then I met with Dr. Warren Vander Hill, the director of the Honors College, for an interview. I'd had a few experiences on college campuses in the previous couple of years, but I still felt a little out of my element. Though I came from a home that valued reading and learning, my family background was not academic.

On a shelf behind Dr. Vander Hill, I noticed a picture of him in a Hope College basketball jersey, dribbling during a college game. I casually asked him about it and learned that he had played Division III varsity ball as an undergrad. I just now searched online in hopes of confirming my memory and learned that he is still #8 on the list of Hope's all-time career scoring leaders. I don't recall him slipping that fact into our chat... (Back then, he would have been #2!)

Anyway, we started talking basketball. Somehow, the conversation turned to Oscar Robertson, one of the NBA's all-time great players. He starred at Indianapolis's all-black Crispus Attucks High School and led the school to a state championship in 1955. From there, we talked about a lot of things -- the integration of college athletics, the civil rights movement, the state of the world in 1982 -- but it all seemed to revolve around basketball.

The time flew. Suddenly, the interview period was over, and I headed home. I enjoyed the conversation quite a bit, but on the hour drive, I wondered if I'd squandered my chances at the scholarship by using my interview time to talk sports. A few weeks later, though, I received a letter saying that I had been selected as one of the recipients.

That was the beginning of four very good years for me. Maybe I can trace some of that fortune to a conversation about sports. I certainly owe a debt to the skill of the person who interviewed me.

I got to know Dr. Vander Hill better over the next four years and slowly realized that he had probably known exactly what he was doing in that interview. He had found a common interest we shared and used it to start a conversation that opened up into bigger ideas. I couldn't have prepared answers for this conversation. He could see that I wasn't making up a story, that I was genuinely interested in the issues we were discussing and was, perhaps, genuinely interesting. The interview was a lot of fun, for both of us, I think, and he learned a lot about me from just talking.

I learned a lot from Dr. Vander Hill over the years, though what I learned from him that day took a few years to sink in.

Short weeknotes this time because the key things I worked on have already been covered here:

• How to implement Q&A against your documentation with GPT3, embeddings and Datasette described my experiment using custom SQL functions in Datasette to provide a Q&A interface against an existing corpus of documents. This resulted in a flurry of software releases:
  • datasette-openai - custom SQL functions for querying the OpenAI APIs from Datasette
  • datasette-faiss - a plugin that builds a FAISS index for fast vector similarity search
  • datasette-cookies-for-magic-parameters - a supporting plugin that helps store secrets such as API keys in cookies so they won't be transmitted in query strings
  • openai-to-sqlite - a command-line tool for storing OpenAI API responses in a SQLite database - useful for calculating embeddings
• Datasette 0.64, with a warning about SpatiaLite - I released Datasette 0.64 with a new --setting default_allow_sql off setting, due to the discovery that some of the SQL functions added by the SpatiaLite module can be used to crash the Datasette instance.
• Building a location to time zone API with SpatiaLite is the latest in the official series of tutorials for Datasette. It demonstrates how SpatiaLite can be used to safely build and deploy a geospatial API.

Releases this week

• datasette-openai: 0.2 - (4 releases total) - 2023-01-14
  SQL functions for calling OpenAI APIs
• openai-to-sqlite: 0.2 - (2 releases total) - 2023-01-13
  Save OpenAI API results to a SQLite database
• datasette-cookies-for-magic-parameters: 0.1.2 - (3 releases total) - 2023-01-12
  UI for setting cookies to populate magic parameters
• git-history: 0.7a0 - (10 releases total) - 2023-01-11
  Tools for analyzing Git history using SQLite
• datasette: 0.64.1 - (124 releases total) - 2023-01-11
  An open source multi-tool for exploring and publishing data
• datasette-faiss: 0.1a0 - 2023-01-11
  Maintain a FAISS index for specified Datasette tables
• json-to-files: 0.1 - 2023-01-10
  Create separate files on disk based on a JSON object
• datasette-auth-passwords: 1.1 - (12 releases total) - 2023-01-09
  Datasette plugin for authentication using passwords
• datasette-publish-fly: 1.3 - (9 releases total) - 2023-01-09
  Datasette plugin for publishing data using Fly
• shapefile-to-sqlite: 0.4.2 - (6 releases total) - 2023-01-08
  Load shapefiles into a SQLite (optionally SpatiaLite) database

TIL this week

• Querying the GitHub archive with the ClickHouse playground
• Breakfast tacos with eggs, vegetarian sausages and avocado sauce
• Geopoly in SQLite
• Loading SQLite extensions in Python on macOS
• Scraping the Sky News Westminster Accounts, a Flourish application
• Upgrading a pipx application to an alpha version

The programs generated by Openai‘s chat-bot ChatGPT, sometimes referred to as GPT-3, may have the requested behavior, but often contain mistakes that illustrate the fact that ChatGPT does not know anything about programming.

ChatGPT does not understand anything, it is a token prediction engine driven by the corpus used to train its language model. The likelihood of the output being correct is dependent on the quantity of text used during training (the more text used, the more likely that it will contain a relevant response). GTP-3, the language model behind ChatGPT, was trained on the deduplicated contents of the Internet (the language model has 175 billion parameters, the GTP-2 model contains 1.5 billion parameters). The practicality of its implementation is the result of various trade-offs, along with technical breakthroughs in language model building, e.g., transformers.

ChatGPT output appears to be that of a next generation search engine, one that returns answers rather than links to pages.

Readers who believe that any ‘understanding’ pixie dust is involved can disabuse themselves of this notion by reading an overview of GTP-3’s architecture, and how the transformer algorithm converts text to matrices of values.

If matrix algebra is not your thing, check out the 257 lines of python that nanoGTP uses for training, and the 75 lines used to generate responses from a trained model.

As a prediction engine, ChatGTP sometimes gets things wrong. What kind of mistakes appear in ChatGPT generated source code?

The exchanges below occurred with ChatGPT Jan 9 version, with the text typed by me appearing in bold, and the ChatGPT output appearing in the following boxed text.

Some of my questions were inspired by the list of undefined behaviors in the C Standard.

Write a program that adds apples to oranges.

I'm sorry, I don't understand what you mean by "adding apples to oranges." Can you please provide more information or context?

A suitable response can only be provided when an appropriate match is found in the trained model.

Write a regular expression that matches all prime numbers.

It is not possible to write a regular expression that can match all prime numbers, as the set of prime numbers is infinite and a regular expression is a finite representation of a pattern. Additionally, regular expressions are used to match strings of characters, while prime numbers are numerical values.
It is possible to use a regex to match a number in a string but it's not capable of determining if the number is prime or not.

The training data may include the information that the request cannot be met.

Write a regular expression that matches all even numbers.

A regular expression that matches all even numbers is:

^(0|[2468]*[048]|[13579][26])$

Explanation:

    ^ matches the start of the string
    (0|[2468]*[048]|[13579][26]) matches any of the following:
        0
        any number of even digits followed by a 0 or 8 (for example, 200, 2468000)
        any odd digit followed by a 6 (for example, 1357926)
    $ matches the end of the string

This regular expression will match strings that contain only even numbers, but not those that contain other characters.

The generated regular expression does not do what I asked, or what the last sentence from ChatGTP claims.

What is going on?

The tokens “Write a regular expression that matches” were enough for ChatGTP to figure out the task.

The tokens “all even numbers” was not specific enough for ChatGTP to focus on the appropriate training data. There is a random component to the selection of output, which will play a larger role when there are many possibilities.

The method used to tokenized/encode character sequences during training (i.e., byte-pairs), along with the relatively limited number of distinct tokens (i.e., 50,257) means that only one-/two-digit numbers are likely to be represented as discrete entities; longer digit sequences are treated as co-occurrences of one-/two-digit tokens. So it is not a surprise to see ChatGPT applying co-occurrences characteristics of long digit sequences to its component subsequences.

Write a regular expression that matches all odd numbers.

A regular expression that matches all odd numbers is:

^(1|[13579]*[13579])$

Explanation:

    ^ matches the start of the string
    (1|[13579]*[13579]) matches any of the following:
        1
        any number of odd digits followed by another odd digit (for example, 1357913579)
    $ matches the end of the string

This regular expression will match strings that contain only odd numbers, but not those that contain other characters.

The output for odd numbers is more consistent, in that the regular expression does what the last sentence of the output claims, but not what I asked.

During a chat session, it is possible to provide more (information in human terminology; character sequences in ChatGTP terminology). The following prompt text, from me, has nothing to do with teaching ChatGTP to do maths, but providing tokens that enable a more focused match against training data.

Even numbers are defined as divisible by two with no reminder. If a number is not even then it is odd.

You are correct, even numbers are defined as divisible by two with no remainder, and if a number is not even, it is considered odd. I apologize for any confusion caused by my previous statement.

To match all even numbers, we can use the regular expression:
^([0-9]*[02468])$

To match all odd numbers, we can use the regular expression:
^([0-9]*[13579])$

Both of this regular expression will match numbers that are divisible by 2 and not divisible by 2 respectively.

Both generated regular expressions are correct.

ChatGPT can learn to translate from an invented language to English (and presumably other languages).

Give an example of C source code assigning a character constant that contains an invalid multibyte character.

The following is an example of C source code assigning a character constant that contains an invalid multibyte character:

#include 

int main() {
    char invalid_char = '\xF0\x90\x80\x80';  // This is an invalid multibyte character
    printf("%c", invalid_char);
    return 0;
}

In this example, the character constant '\xF0\x90\x80\x80' is assigned to the variable invalid_char. This particular sequence of bytes is not a valid Unicode character, and it will produce undefined behavior when attempting to print it.

There was a time when '\xF0\x90\x80\x80' was an invalid multibyte character.

Things changed around 20 years ago, but the training for the model used by ChatGTP does not tag information with valid-between dates.

ChatGTP searches for, and finds, an invalid multibyte character, and uses it to provide an answer.

Give an example of C source code where an object is referred to outside of its lifetime.

The following is an example of C source code where an object is referred to outside of its lifetime:

#include 

int main() {
    int x;
    {
        int x = 5;
    }
    printf("x: %d\n", x);
    return 0;
}

In this example, the variable x is defined twice in the same function main. The first definition int x; is a global variable that has a lifetime throughout the entire program execution. The second definition int x = 5; is a local variable that has a lifetime that is limited to the inner block {...} it is defined in.

After the inner block is executed and the inner variable x goes out of scope, the program attempts to access the inner variable x by using the global variable x in the printf statement, which refers to an object (the inner variable x) that is no longer in its lifetime. This behavior is undefined according to the C standard and can lead to unexpected results.

The generated code is incorrect. The human interpretation of this code is that ChatGTP has confused the name of an object (what the C Standard calls a variable) and the storage it represents. There are two different objects defined, both having the name x (beginners often make the same mistake).

A common example of the requested behavior is freeing the dynamic storage pointed to by some pointer, and then accessing that storage.

Write a C program containing a conversion of an integer type that produces a value outside the range that can be represented in the integer type.

Here is an example of a C program that demonstrates the conversion of an integer type that produces a value outside the range that can be represented in the integer type:

#include 

int main() {
    int x = 2147483647;  // largest value representable in a signed 32-bit integer
    unsigned int y = (unsigned int) x + 1;  // overflow occurs here
    printf("%u\n", y);
    return 0;
}

In this program, the value of x is initialized to the largest value that can be represented by a signed 32-bit integer (2147483647). Then, x is cast to an unsigned 32-bit integer and 1 is added to it. Since the result of this operation is larger than the largest representable value in an unsigned 32-bit integer, an overflow occurs and the value of y becomes 0.
This program will produce an output of 0.
It is worth mentioning that undefined behavior (UB) can occur when an integer overflow happens, so this should be avoided in production code.

The detailed explanation makes the output look like its authoritative. However, it is wrong.

There is no overflow in this code.

The integer type ‘unsigned int’ supports a larger range of positive values, which means that after conversion, adding one will not produce a value that is outside the range that can be represented.

You are correct, I apologize for the confusion. Unsigned integers have a larger range of positive values and do not have negative values. When an operation like an addition exceeds the maximum value that can be represented by an unsigned integer, it wraps around to the minimum value representable by that type, which is typically zero, instead of an overflow.

For example, if you use an unsigned int and the maximum value it can represent is 4,294,967,295, if you add 1 to that value, it will wrap around to 0.

Here is an example of a C program that demonstrates the conversion of an integer type that produces a value outside the range that can be represented in the integer type:

#include 

int main() {
    int x = -2147483648; // lowest value representable in a signed 32-bit integer
    unsigned int y = (unsigned int) x -1; // wrap around occurs here
    printf("%u\n", y);
    return 0;
}

In this program, the value of x is initialized to the lowest value that can be represented by a signed 32-bit integer (-2147483648) which is lower than 0. Then, x is cast to an unsigned 32-bit integer and 1 is subtracted from it. Since the result of this operation is lower than 0, the value wraps around to the maximum representable value in an unsigned 32-bit integer (4,294,967,295) . This program will produce an output of 4,294,967,295.
It is worth mentioning that this wrapping around behavior is defined for unsigned integers in C and C++, but it is not always the case in other programming languages.

This output contained the requested behavior, although subtracting one is unnecessary code.

ChatGTP can also be used to create documentation

Write a poem in Iambic pentameter about the benefit of an evidence based approach to software engineering in the style of a party political broadcast.

With evidence-based approach, our work shall shine,
For in software engineering, it's the only way to dine.
We'll test and measure, as we go along,
To make our code robust, and errors gone.

Our methodologies, backed by data and proof,
Will bring success and progress, that's the truth.
We'll avoid the pitfalls, of guesses and guesses,
And build systems, that pass all tests.

We'll listen to the data, and the users' voice,
To make our software, the best by choice.
We'll iterate and improve, with every release,
And build a product, that truly will please.

So join us now, and embrace the way,
Of evidence-based approach, for brighter day.
With data and research, we'll conquer all,
In software engineering, we'll never fall.

This is something I keep reinventing from scratch, and that I thought was worthy of posting a note about given that the Functions runtime is now in v4 and function proxies are deprecated (even though you can enable them again).

To be perfectly candid, it bugs me a lot that a small function app doesn’t have the bare minimum to serve up one HTML file and a handful of bundled resources by itself instead of having to resort to Azure CDN or a storage account, which just add to the amount of moving parts and add far too much complexity for small things.

So here’s what I did today while building a small PoC app:

1. Set AzureWebJobsDisableHomepage: true in settings.json

2. Created an index function with these bindings in function.json, so it has effectively no route and spits out binary data:

# These are shorthand for the JSON entries
bindings[0].route: "{default:maxlength(0)?}"
bindings[0].dataType: "binary"

3. Created a static function that responds to bindings[0].route: "static/{*file}", with the same dataType.

4. Set extensions.http.routePrefix: "" in host.json to remove the /api prefix.

5. Fished out an old function I had that essentially uses fs and mime-types to send out a context.res.body: fs.readFileSync(file, null) after sorting out the full path and the content-type, and added that to both index and static.

Bam, you can now serve index.html and static assets that get deployed with your functions.

I then carried on to write the rest mostly unhindered, although it was JavaScript and I keep forgetting to await some of the APIs I’m calling.

The gist of things is that I’m only doing this in this way because JavaScript, for all its foibles, is still relatively quick to iterate upon.

Actually, let me rephrase that. JavaScript is actually much slower than Python to iterate on for API development due to things not failing and spitting out undefined all over the place, but I just can’t get the confusingly named v2 Python programming model of the v4 Azure Functions runtime to work for me, and I really wanted to do this test I just did.

Also, my usual default of building a container and deploying it would be overkill for this scenario.

Something that saddens me a bit is that I am really losing faith in Azure Functions as a “rapid” programming model because every time I go back to it after a while everything I had done needs updating (editor extensions, runtime, code, dependencies, ways to configure basic settings, the works).

It’s just easier to go build and slap a binary behind caddy, or pack it into an ersatz container and push it to an Azure Container Instance. I can even pack all my static assets into the Go binary¹.

Which is what I will probably be doing from now on for prototypes², as long as they don’t involve too much JSON handling (which is still easier to do in dynamic languages).

The Internet Transition, @robinberjon:

“we’re trying to run a planetary society that needs to solarpunk the fuck out of itself in a hurry on the collective intelligence of an 18th century principality that’s heard of the Enlightenment from some guy at the pub.”

I decided long ago (for better or for worse) to stick to the first-party Twitter app because I really thought it wouldn’t get any worse (ha!) but it’s been really annoying to keep jumping through hoops to go back to the chronological view, and I completely get that many people would prefer a crud-free, more polished experience (also, not being in the US the ads I see are very different).

But what Twitter is doing goes way beyond negligence and borders on abuse of dominant position (even though it is their network and they’re not a utility, etc). API downtime would prevent third-party clients from logging in, but deregistering them from user accounts cannot be an availability issue–they removed the application keys, period.

I don’t expect anything good to come of this, and even if the situation is resolved I suspect many people will finally understand Twitter is not to be trusted. Mastodon may have its problems (including arbitrary banning by ultra-sensitive, overly political server admins), but nothing anywhere as vicious as this.

I stand by my predictions for 2023 (which never took this scenario into consideration): Twitter is now visibly evil.

Update: They are now claiming to have blocked those applications for violating API rules, which is complete and utter bullshit, and added a new clause explicitly banning third-party clients. Yep, evil.

Wastewater

From Jeff’s spreadsheet:

Jeff also just made individual charts, with nine-day rolling averages. (Why nine-day? It worked better.) From those charts, you can see that there was a definite rising trend for a while in Vancouver and the north shore, but levels now falling. He suspects that we’re seeing the end of a post-holiday spike.

Non-COVID Respiratory Diseases

It looks to me like the non-COVID respiratory diseases have peaked. This will probably be the last time I report on non-COVID respiratory diseases again in this blog unless something dramatic happens.

From BC CDC Pathogens Characterization with data through 7 Jan, adult non-COVID respiratory illness cases are continuing to fall nicely:

For Vancouver-Richmond-North Shore kids, everything sure looks like it has peaked.

Respiratory illnesses are definitely on their way down in the US Pacific Northwest. From the US CDC’s FluView Interactive:

I spent most of yesterday exploring some different options to create a non-static blog. I’m currently building this with Jekyll and hosting it on GitHub. I love that I can focus on the text without character limits. 280 characters is great for keeping me from getting long-winded, but it’s no substitute for an editor. Very few of us read past the headlines anymore, and I think it is worth slowing down a bit sometimes.

I also love the simplicity of a static site, but I want to publish more than just text. There is some great software for the IndieWeb out there that’s evolved over the years. I read the MicroPub spec, I read about IndieAuth, and I made some pretty good progress with IndieKit; it’s excellent, but the complexity is a bit more than I want to deal with. I don’t want a complex server environment to manage, and I certainly don’t want to maintain a database.

Nevertheless, I want to be more focused on writing this year, mostly because I’ve been reading a lot, and I want to improve my understanding of what I read.

And I still want images, so I’m going to build something that is a bit more distributed. My phone and my laptop are powerful computers. I’ll let them do the work, and upload it to a simple static site. Later if it gets more complicated, it will all be locally processed.

I’ll make my work public, and I’ll talk about it here. Hopefully you will find it useful.

Snowy trail © 2023, Steven Wart

If you've spent any time with GPT-3 or ChatGPT, you've likely thought about how useful it would be if you could point them at a specific, current collection of text or documentation and have it use that as part of its input for answering questions.

It turns out there is a neat trick for doing exactly that. I've been experimenting with it using my Datasette web application as a rapid prototyping platform for trying out new AI techniques using custom SQL functions.

Here's how to do this:

• Run a text search (or a semantic search, described later) against your documentation to find content that looks like it could be relevant to the user's question
• Grab extracts of that content and glue them all together into a blob of text
• Construct a prompt consisting of that text followed by "Given the above content, answer the following question: " and the user's question
• Send the whole thing through the GPT-3 API and see what comes back

I've been calling this the semantic search answers pattern.

This is a form of prompt engineering (and hence is prone to prompt injection attacks, so bear that in mind). You need to fit your overall prompt into the token limit for GPT-3, which is currently 4,000 tokens. A token is more-or-less the same thing as a word.

Since publishing this post I've been pointed to the Question Answering using Embeddings notebook in the OpenAI cookbook which describes this exact same technique.

In this post I'll show how I implemented this pattern using SQLite, Datasette and some custom SQL functions and tools written in Python.

First, some demos

I've implemented this pattern against my blog - 3,000 posts going back over twenty years. I built it using some custom SQLite SQL functions that call the OpenAI APIs.

First, some examples of questions and answers I ran against my blog:

What does shot scraper do?

Shot-scraper is a command-line tool that helps automate the process of taking screenshots of web pages and scraping data from them using JavaScript. It can be used to take one-off screenshots or take multiple screenshots in a repeatable way by defining them in a YAML file. It can also be used to execute JavaScript on a page and return the resulting value.

shot-scraper is a tool I released in 2022 - after the 2021 GPT-3 training cut-off date. This is a very good summary - better I think than anything I've written about shot-scraper myself.

What museums has Simon been to?

Simon has been to the Pitt Rivers museum, the British Museum, the Science Museum, the Natural History Museum, the V&A, the Victoria and Albert Museum, and the Pioneer Museum in Paso Robles. He has also seen the Giant Squid at the Natural History Museum and Charles Darwin's collection at the spirit building.

This is entirely correct, though I've talked about a lot more museums than that on my blog.

What are some unconventional things you can do with GitHub Actions?

1. Create dynamic content for GitHub repository templates using cookiecutter and GitHub Actions
2. Automate repetitive tasks like code linting, testing, and deployment using GitHub Actions
3. Run scheduled scrapers in GitHub Actions
4. Use GitHub Actions to trigger a webhook call when a repository is forked
5. Create an automated issue-tracking system with GitHub Issues and GitHub Actions
6. Automatically extract captions and transcripts from online videos using Whisper and GitHub Actions.

Some of my relevant posts:

• Dynamic content for GitHub repository templates using cookiecutter and GitHub Actions
• Git scraping: track changes over time by scraping to a Git repository
• A tool to run caption extraction against online videos using Whisper and GitHub Issues/Actions

I've never written anything about triggering webhook calls or issue-tracking systems though, so 4 and 5 in that list are a little suspicious.

What is the big grey box at the top of the utility pole? What does it do?

The big grey box at the top of the utility pole is likely a transformer. Transformers are used to step down the voltage of electricity from the high-voltage lines that run along the pole to the lower-voltage lines that are used to provide electricity to homes and businesses.

I'm including this example because I've never written anything even remotely related to transformers and utility poles on my blog. This demonstrates that in the lack of useful context GPT-3 will answer the question entirely by itself - which may or may not be what you want from this system.

Try this out yourself

If you want to try this out yourself you'll need to get your own API key from OpenAI. I don't want to foot the bill for people using my blog as a free source of GPT-3 prompt answers!

You can sign up for one here. I believe they are still running a free trial period.

Now head over to this page:

https://datasette.simonwillison.net/simonwillisonblog/answer_question?_hide_sql=1

You'll need to paste in your OpenAI key. I am not logging these anywhere, and the form stores these in a cookie in order to avoid transmitting it over a GET query string where it could be accidentally logged somewhere.

Then type in your question and see what comes back!

Let's talk about how this all works - in a whole lot of detail.

Semantic search using embeddings

You can implement the first step of this sequence using any search engine you like - but there's a catch: we are encouraging users here to ask questions, which increases the chance that they might include text in their prompt which doesn't exactly match documents in our index.

"What are the key features of Datasette?" for example might miss blog entries that don't include the word "feature" even though they describe functionality of the software in detail.

What we want here is semantic search - we want to find documents that match the meaning of the user's search term, even if the matching keywords are not present.

OpenAI have a less well-known API that can help here, which had a big upgrade (and major price reduction) back in December: their embedding model.

An embedding is a list of floating point numbers.

As an example, consider a latitude/longitude location: it's a list of two floating point numbers. You can use those numbers to find other nearby points by calculating distances between them.

Add a third number and now you can plot locations in three dimensional space - and still calculate distances between them to find the closest points.

This idea keeps on working even as we go beyond three dimensions: you can calculate distances between vectors of any length, no matter how many dimensions they have.

So if we can represent some text in a many-multi-dimensional vector space, we can calculate distances between those vectors to find the closest matches.

The OpenAI embedding model lets you take any string of text (up to a ~8,000 word length limit) and turn that into a list of 1,536 floating point numbers. We'll call this list the "embedding" for the text.

These numbers are derived from a sophisticated language model. They take a vast amount of knowledge of human language and flatten that down to a list of floating point numbers - at 4 bytes per floating point number that's 4*1,536 = 6,144 bytes per embedding - 6KiB.

The distance between two embeddings represents how semantically similar the text is to each other.

The two most obvious applications of this are search and similarity scores.

Take a user's search term. Calculate its embedding. Now find the distance between that embedding and every pre-calculated embedding in a corpus and return the 10 closest results.

Or for document similarity: calculate embeddings for every document in a collection, then look at each one in turn and find the closest other embeddings: those are the documents that are most similar to it.

For my semantic search answers implementation, I use an embedding-based semantic search as the first step to find the best matches for the question. I then assemble these top 5 matches into the prompt to pass to GPT-3.

Calculating embeddings

Embeddings can be calculated from text using the OpenAI embeddings API. It's really easy to use:

curl https://api.openai.com/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{"input": "Your text string goes here",
       "model":"text-embedding-ada-002"}'

The documentation doesn't mention this, but you can pass a list of strings (up to 2048 according to the official Python library source code) as "input" to run embeddings in bulk:

curl https://api.openai.com/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{"input": ["First string", "Second string", "Third string"],
       "model":"text-embedding-ada-002"}'

The returned data from this API looks like this:

{
  "data": [
    {
      "embedding": [
        -0.006929283495992422,
        -0.005336422007530928,
        ...
        -4.547132266452536e-05,
        -0.024047505110502243
      ],
      "index": 0,
      "object": "embedding"
    }
  ]

As expected, it's a list of 1,536 floating point numbers.

I've been storing embeddings as a binary string that appends all of the floating point numbers together, using their 4-byte representation.

Here are the tiny Python functions I've been using for doing that:

import struct

def decode(blob):
    return struct.unpack("f" * 1536, blob)

def encode(values):
    return struct.pack("f" * 1536, *values)

I then store them in SQLite blob columns in my database.

I wrote a custom tool for doing this, called openai-to-sqlite. I can run it like this:

openai-to-sqlite embeddings simonwillisonblog.db \
  --sql 'select id, title, body from blog_entry' \
  --table blog_entry_embeddings

This concatenates together the title and body columns from that table, runs them through the OpenAI embeddings API and stores the results in a new table called blog_entry_embeddings with the following schema:

CREATE TABLE [blog_entry_embeddings] (
   [id] INTEGER PRIMARY KEY,
   [embedding] BLOB
)

I can join this against the blog_entry table by ID later on.

Finding the closest matches

The easiest way to calculate similarity between two embedding arrays is to use cosine similarity. A simple Python function for that looks like this:

def cosine_similarity(a, b):
    dot_product = sum(x * y for x, y in zip(a, b))
    magnitude_a = sum(x * x for x in a) ** 0.5
    magnitude_b = sum(x * x for x in b) ** 0.5
    return dot_product / (magnitude_a * magnitude_b)

You can brute-force find the top matches for a table by executing that comparison for every row and returning the ones with the highest score.

I added this to my datasette-openai Datasette plugin as a custom SQL function called openai_embedding_similarity(). Here's a query that uses it:

with input as (
  select
    embedding
  from
    blog_entry_embeddings
  where
    id = :entry_id
),
top_n as (
  select
    id,
    openai_embedding_similarity(
      blog_entry_embeddings.embedding,
      input.embedding
    ) as score
  from
    blog_entry_embeddings,
    input
  order by
    score desc
  limit
    20
)
select
  score,
  blog_entry.id,
  blog_entry.title
from
  blog_entry
  join top_n on blog_entry.id = top_n.id

Try that out here.

This takes as input the ID of one of my blog entries and returns a list of the other entries, ordered by their similarity score.

Unfortunately this is pretty slow! It takes over 1.3s to run against all 3,000 embeddings in my blog.

I did some research and found that a highly regarded solutions for fast vector similarity calculations is FAISS, by Facebook AI research. It has neat Python bindings and can be installed using pip install faiss-cpu (the -gpu version requires a GPU).

FAISS works against an in-memory index. My blog's Datasette instance uses the baked data pattern which means the entire thing is re-deployed any time the data changes - as such, I can spin up an in-memory index once on startup without needing to worry about updating the index continually as rows in the database change.

So I built another plugin to do that: datasette-faiss - which can be configured to build an in-memory FAISS index against a configured table on startup, and can then be queried using another custom SQL function.

Here's the related entries query from above rewritten to use the FAISS index:

with input as (
  select
    embedding
  from
    blog_entry_embeddings
  where
    id = :entry_id
),
top_n as (
  select value as id from json_each(
    faiss_search(
      'simonwillisonblog',
      'blog_entry_embeddings',
      input.embedding,
      20
    )
  ), input
)
select
  blog_entry.id,
  blog_entry.title
from
  blog_entry
  join top_n on blog_entry.id = top_n.id

This one runs in 4.8ms!

faiss_search(database_name, table_name, embedding, n) returns a JSON array of the top n IDs from the specified embeddings table, based on distance scores from the provided embedding.

The json_each() trick here is a workaround for the fact that Python's SQLite driver doesn't yet provide an easy way to write table-valued functions - SQL functions that return something in the shape of a table.

Instead, I use json_each() to turn the string JSON array of IDs from datasette_faiss() into a table that I can run further joins against.

Implementing semantic search with embeddings

So far we've just seen embeddings used for finding similar items. Let's implement semantic search, using a user-provided query.

This is going to need an API key again, because it involves a call to OpenAI to run embeddings against the user's search query.

Here's the SQL query:

select
  value,
  blog_entry.title,
  substr(blog_entry.body, 0, 500)
from
  json_each(
    faiss_search(
      'simonwillisonblog',
      'blog_entry_embeddings',
      (
        select
          openai_embedding(:query, :_cookie_openai_api_key)
      ),
      10
    )
  )
  join blog_entry on value = blog_entry.id
  where length(coalesce(:query, '')) > 0

Try that here (with extra some cosmetic tricks.)

We're using a new function here: openai_embedding() - which takes some text and an API key and returns an embedding for that text.

The API key comes from :_cookie_openai_api_key - this is a special Datasette mechanism called magic parameters which can read variables from cookies.

The datasette-cookies-for-magic-parameters plugin notices these and turns them into an interface for the user to populate the cookies with, decsribed earlier.

One last trick: adding where length(coalesce(:query, '')) > 0 to the query means that the query won't run if the user hasn't entered any text into the search box.

Constructing a prompt from semantic search query results

Getting back to our semantic search answers pattern.

We need a way to construct a prompt for GPT-3 using the results of our semantic search query.

There's one big catch: GPT-3 has a length limit, and it's strictly enforced. If you pass even one token over that limit you'll get an error.

We want to use as much material from the top five search results as possible, leaving enough space for the rest of the prompt (the user's question and our own text) and the prompt response.

I ended up solving this with another custom SQL function:

select openai_build_prompt(content, 'Context:
------------
', '
------------
Given the above context, answer the following question: ' || :question,
  500
  ) from search_results

This function works as an aggregate function - it takes a table of results and returns a single string.

It takes the column to aggregate - in this case content - as the first argument. Then it takes a prefix and a suffix, which are concatenated together with the aggregated content in the middle.

The third argument is the number of tokens to allow for the response.

The function then attempts to truncate each of the input values to the maximum length that will still allow them all to be concatenated together while staying inside that 4,000 token limit.

Adding it all together

With all of the above in place, the following query is my full implementation of semantic search answers against my blog:

with query as (
  select
    openai_embedding(:question, :_cookie_openai_api_key) as q
),
top_n as (
  select
    value
  from json_each(
    faiss_search(
      'simonwillisonblog',
      'blog_entry_embeddings',
      (select q from query),
      5
    )
  )
  where length(coalesce(:question, '')) > 0
),
texts as (
  select 'Created: ' || created || ', Title: ' || title || 
  ', Body: ' || openai_strip_tags(body) as text
  from blog_entry where id in (select value from top_n)
),
prompt as (
  select openai_build_prompt(text, 'Context:
------------
', '
------------
Given the above context, answer the following question: ' || :question,
  500
  ) as prompt from texts
)
select
  'Response' as title,
  openai_davinci(
    prompt,
    500,
    0.7,
    :_cookie_openai_api_key
  ) as value
  from prompt
  where length(coalesce(:question, '')) > 0
union all
select
  'Prompt' as title,
  prompt from prompt

As you can see, I really like using CTEs (the with name as (...) pattern) to assemble complex queries like this.

The texts as ... CTE is where I strip HTML tags from my content (using another custom function from the datasete-openai plugin called openai_strip_tags()) and assemble it along with the Created and Title metadata. Adding these gave the system a better chance of answering questions like "When did Natalie and Simon get married?" with the correct year.

The last part of this query uses a handy debugging trick: it returns two rows via a union all - the first has a Response label and shows the response from GPT-3, while the second has a Prompt label and shows the prompt that I passed to the model.

Next steps

There are so many ways to improve this system.

• Smarter prompt design. My prompt here is the first thing that I got to work - I'm certain there are all kinds of tricks that could be used to make this more effective.
• Better selection of the content to include in the prompt. I'm using embedding search but then truncating to the first portion: a smarter implementation would attempt to crop out the most relevant parts of each entry, maybe by using embeddings against smaller chunks of text.
  • Yoz tipped me off to GPT Index, a project which aims to solve this exact problem by using a pre-trained LLM to help summarize text to better fit in a prompt used for these kinds of queries.
  • Spotted this idea from Hassan Hayat: "don't embed the question when searching. Ask GPT-3 to generate a fake answer, embed this answer, and use this to search". See also this paper about Hypothetical Document Embeddings, via Jay Hack.
• Hold out for GPT-4: I've heard rumours that the next version of the model will have a significantly larger token limit, which should result in much better results from this mechanism.

Want my help implementing this?

I plan to use this pattern to add semantic search and semantic search answers to my Datasette Cloud SaaS platform. Please get in touch if this sounds like a feature that could be relevant to your organization.

Mark Gurman recently reported that Apple’s much-rumored headset will combine AR and VR technologies, which Brendon Bigley argues could be the wrong approach:

… I don’t think the road to mass adoption of virtual reality actually starts with virtual reality, it starts instead with augmented reality — a technology that can quickly prove its function if presented in a frictionless way. While even the best VR headsets demand isolation and escapism, a hypothetical product focused first and foremost on augmented reality would be all about enhancing the world around you rather than hiding from it.

Brendon’s story nails something that has been nagging me about recent headset rumors. The iPhone was a hit because it took things we already did at a desk with a computer and put them on a device we could take with us everywhere we go, expanding the contexts where those activities could be done. As Brendon observes, the Apple Watch did something similar with notifications. AR feels like something that fits in the same category – an enhancement of things we already do – while VR is inherently limiting, shutting you off from the physical world.

Like Brendon, it’s not that I’m not excited about the prospect of an Apple headset or the long-term prospects for virtual reality as a technology, but given where the technology is today, it does seem as though jumping into VR alongside AR could muddy the waters for both technologies. Of course, we’re all still working off of speculation and rumors. I have so many questions still and can’t wait to see what Apple has in store for us, hopefully later this year.

→ Source: wavelengths.online