Lambda Snapstart, and snapshots as a tool for system builders

Clones.

Yesterday, AWS announced Lambda Snapstart, which uses VM snapshots to reduce cold start times for Lambda functions that need to do a lot of work on start (starting up a language runtime³, loading classes, running static code, initializing caches, etc). Here’s a short 1 minute video about it:

Or, for a lot more context on Lambda and how we got here⁵:

Snapstart is a super useful capability for Lambda customers. I’m extremely proud of the work the team did to make it a reality. We’ve been talking about this work for over two years, and working on it for longer. The team did some truly excellent engineering work to make Snapstart a reality.

Beyond Snapstart in Lambda, I’m also excited about the underlying technology (microVM snapshots), and the way they give us (as system builders and researchers) a powerful tool for building new kinds of secure and scalable systems. In this post, I talk about some interesting aspects of Snapstart, and how they point to interesting possible areas for research on systems, storage, and even cryptography.

What is Snapstart?

Without snapstart, the cold start time of a Lambda function is a combination of time time taken to download the function (or container), to start the language runtime³, and to run any initialization code inside the function (including any static code, doing class loading, and even JIT compilation). The cold start time doesn’t include MicroVM boot, because that time is not specialized to the application, and can be done in advance. A cold start looks like this¹:

With snapstart, a snapshot of the microVM is taken after these initialization steps (including all the memory, device state, and CPU registers). This snapshot can be used multiple times (also called cloned), to create multiple sandboxes that are initialized in the same way. Cloning like this has two benefits:

• The work of initialization is amortized over many sandboxes, rather than being done once for every sandbox. Lambda runs one sandbox for ever concurrent function execution, so for a function with N concurrent executions this reduces initialization work from O(N) to O(1).
• The time taken by execution no longer accrues to the cold-start time, significantly reducing cold start latency for applications that do a lot of work on startup (which is typically applications that use large language runtimes like the JVM, or large frameworks and libraries).
• JIT compilation and other warmup tasks can be done at initialization time, often avoiding the uneven performance that can be introduced by JITing soon after language runtime² startup.

In diagram form, the snapstart startup regime looks like this¹:

The Problems of Uniqueness

Perhaps the biggest challenge with clones is that they’re, well, clones. They contain the same CPU, the same memory, and even the same CPU registers. Being too alike can cause problems. For example, as we say in Restoring Uniqueness in MicroVM Snapshots:

Most modern distributed systems depend on the ability of nodes to generate unique or random values. RFC4122 version 4 UUIDs are widely used as unique database keys, and as request IDs used for log correlation and distributed tracing. …

and

Many common distributed systems protocols, including consensus protocols like Paxos, and ordering protocols like vector clocks, rely on the fact that participants can uniquely identify themselves.

and

Cryptography is the most critical application of unique data. Any predictability in the data used to generate cryptographic keys—whether long-lived keys for applications like storage encryption or ephemeral keys for protocols like TLS fundamentally compromise the confidentiality and authentication properties offered by cryptography.

In other words, its really important for systems to be able to generate random numbers, and MicroVM clones might find that difficult to do. If they rely on hardware features like rdrand then there’s no problem, but any software-based PRNGs will simply create the same stream of numbers unless action is taken. To solve this problem, our team has been working with the OpenSSL, Linux, and Java open source communities to make sure that common PRNGs like java.security.SecureRandom and /dev/urandom reseed correctly when snapshots are cloned.

State

Another challenge with working with clones is connection state in protocols like TCP. There are actually two problems here:

• Time. If a connection is established during initialization and the clone is used later, it’s likely that the remote end has given up on the connection.
• State. Protocols like TCP provide reliable delivery using state at each end of a connection (like a sequence number), with the assumption that there is one client for the lifetime of the connection. If that one client suddenly becomes two clients, the protocol is broken and the connection must be dropped.

The simple solution is to reestablish connections after snapshots are restored. As with reseeding PRNGs, this requires time and work, especially for secure protocols like TLS, which somewhat dilutes cold start benefit. There’s a significant research and development opportunity here, focusing on fast reestablishment of secure protocols, clone-aware protocols, clone-aware proxies, and even deeply protocol-aware session managers (like RDS proxy)⁷.

Moving Data

Let’s take another look at this snapstart diagram:

If the clones are running on the same machine the snapshot was created on, then this diagram is reasonably accurate. However, if we want to scale snapshot restores out over a system the size of AWS Lambda, then we need to make sure the snapshot data is available where and when it is needed. This hidden work - both in distributing data when it’s needed and in sending restores to the right places to meet the data - was the largest challenge of building Snapstart. Turning memory reads into network storage accesses, as would happen with a naive demand-loading system, would very quickly cancel out the latency benefits of Snapstart.

We’re going to say more about our particular solution to this problem in the near future, but I believe that there are interesting general challenges here for systems and storage researchers. Loading memory on demand can be done if the data layer offers low-enough latency, close enough to the latency of a local memory read. We can also avoid loading memory on demand by predicting memory access patterns, loading memory contents ahead of when they are needed. This seems hard to do in general, but is significantly simplified by the ability to learn from the behavior of multiple MicroVM clones.

Layers Upon Layers

There are multiple places where we can snapshot a MicroVM: just after its booted, after a language runtime has been started, or after customer code initialization. In fact, we don’t have to choose just one of these. Instead, we can snapshot after boot, restore that snapshot to start the language runtime, and store just the changes. Then restore that snapshot, run customer initialization, and store just those changes. This has the big benefit of creating a tree of snapshots, where the same Linux kernel or runtime chunks don’t need to be downloaded again and again.

Unlike traditional approaches to memory deduplication (like Kernel Samepage Merging) there’s no memory vs. CPU tradeoff here: identical pages are shared based on their provenance rather than on post-hoc scanning⁶. This significantly reduces the scope of the data movement challenge, by requiring widely-used components to be downloaded infrequently, reducing data movement by as much as 90% for common workloads.

Incremental snapshotting also allows us to use seperate cryptographic keys for different levels of data, including service keys for common components, and per-customer and customer-controlled keys for customer data.

Hungry, Hungry Hippos

Linux, rather famously, loves to eat all the memory it can lay its hands on. In the traditional single-system setting this is the right thing to do: the marginal cost of making an empty memory page full is very nearly zero, so there’s no need to think much about the marginal benefit of keeping around a disk-backed page of memory (whether its an mmap mapping, or an IO buffer, or whatever). However, in cloud settings like Lambda Snapstart, this calculus is significantly different: keeping around disk-backed pages that are unlikely to be used again makes snapshots bigger, with little benefit. The same applies to caches at all layers, whether they’re in the language runtime, in application code, or in libraries.

Tools like DAMON provide a good ability to monitor and control the kernel’s behavior. I think, though, that there will be a major change in thinking required as systems like Snapstart become more popular. There seems to be an open area of research here, in adapting caching behaviors (perhaps dynamically) to handle changing marginal costs of full and empty memory pages. Linux’s behavior - and the one most programmers build into their applications - is one behavior on a larger spectrum, that is only optimal at the point of zero marginal cost.

Snapshots Beyond Snapstart

I can’t say anything here about future plans for using MicroVM snapshots at AWS. But I do believe that they are a powerful tool for system designers and researchers, which I think are currently under-used. Firecracker has the ability to restore a MicroVM snapshot in as little as 4ms (or about 10ms for a full decent-sized Linux system), and it’s no doubt possible to optimize this further. I expect that sub-millisecond restore times are possible, as are restore times with a CPU cost not much higher than a traditional fork (or even a traditional thread start). This reality changes the way we think about what VMs can be used for - making them useful for much smaller, shorter-lived, and transient applications than most would assume.

Firecracker’s full and incremental snapshot support is already open source, and already offers great performance and density⁴. But Firecracker is far from the last word in restore-optimized VMMs. I would love to see more research in this area, exploring what is possible from user space and kernel space, and even how hardware virtualization support can be optimized for fast restores.

In Video Form

Most of this post covers material I also covered in this talk, if you’d prefer to consume it in video form.

Footnotes

1. Diagram from Brooker et al, Restoring Uniqueness in MicroVM Snapshots, 2021.
2. I particularly enjoyed Laurence Tratt’s recent look at VM startup in More Evidence for Problems in VM Warmup.
3. In this post I’ve used the words language runtime to refer to language VMs like the JVM, to avoid confusion with virtualization VMs like Firecracker MicroVMs. This isn’t quite the right word, but it seemed worth avoiding the potential for confusion.
4. As we covered in detail in Firecracker: Lightweight Virtualization for Serverless Applications at NSDI’20.
5. I really like the compute cache framing that Peter uses in this keynote (from 1:00:30 onwards). It’s different from the one that I use in this post, but very clearly explains why cold starts exist, and why they matter to customers of systems like Lambda. The discussion of being unwilling to compromise on security is also important, and has been a driving force behind our work in this space for the last seven years.
6. Granted, this does miss some merge opportunities for pages that end up becoming identical over time, but in a world of crypto and ASLR that happens infrequently anyway.
7. One interesting approach is the one described by Erika Hunhoff and Eric Rozner in Network Connection Optimization for Serverless Workloads. Unfortunately, they don’t seem to have taken this work further.

Regular readers will know that I find the tools we have for authoring long-form explanations of code very frustrating. I’ve used LaTeX (both desktop and online), RMarkdown, Jupyter Notebooks, Microsoft Word, Google Docs, and a variety of lesser-known Markdown-based tools, and they all make simple things hard.

For example, suppose I want to include a snippet of code in an explanation—not an entire class or function, but just a few lines or a method. I can’t do this with computational notebooks: so far as I can tell, they all require cells to be runnable as-is, which means I can’t have just a few key lines from the middle of a function or one method from the middle of a class. Similarly, suppose I want to show a class with placeholder ellipses instead of the method bodies so that readers can get an overview of what’s where before we dive into specifics. Again, the tools that let me run code in situ don’t allow this.

Authors therefore wind up hacking together code scrapers that (for example) look for markers and directives in source code files and copy ranges of code during document compilation. This approach leads to code like that shown below, which is unpleasant to read if you’re trying to single-step through it in a debugger, and which is usually made even more complicated by extra directives to suppress warning messages from linting tools:

# [range key="class" elide=true]
class Example:
    MARKERS = {"[", "]"}

    def __init__(self, filler):
        self._filler = filler

    # [range key="revise"]
    def revise(self, contents):
        end = None
        for (i, val) in enumerate(contents):
            if self.spans(val):
                if (end is None) or (i > end):
                    end = i
        return end
    # [/range]

    def spans(self, value):
        return self._filler(value) in self.MARKERS
# [/range]

So what do I want?

1. I want a plugin for [pick your favorite IDE—I’ll switch to it if you offer this feature] that allows me to select a range of lines in a source code file and add an annotation on them that is stored outside the source file in the same way that reviewers’ comments in Microsoft Word or Google Docs are stored beside the main text.

2. I also want a counterpart plugin for the IDE that allows me to put a marker in a text file that says, “Find the annotation with this label and display the selected range of lines right here, right now.” I don’t want to have to go through a compilation step in order to see those lines (not even one that runs continuously in the background); I want actual fragment transclusion in real time.

3. And yes, users should be able to jump from the transcluded fragment to the original source file and run things, or better yet, click a button to open the entire runnable program beside the text that displays the fragment so that they can step through that bit of code with the explanatory text beside it.

Once we have that, I want to be able to apply it to diagrams to transclude a portion of a larger diagram so that I can explain parts of the whole without trying to keep half-a-dozen SVG’s in sync with each other. Before then, though, I’d like to be able to drive this in the opposite direction. If I’m viewing the code, whatever’s displaying it should automatically point to the relevant explanations, because that’s what I’d do if I was coding in a classroom with some explanations on slides on another screen or some diagrams on a whiteboard beside me:

It’s all technically feasible, particularly if someone wants to leverage an existing platform like VSCode. I don’t expect to see it in my working lifetime, but I do still hope that programming will some day free itself from backward compatibility with punchcards and all the misery that antiquated constraint causes.

Kartik was a 20 year old Sheridan College student from India who was killed by a pickup driver at Yonge and St. Clair on November 23. Many aspects of the tragedy are especially infuriating. Kartik was in the crosswalk crossing Yonge St on his way back from work. The driver who was westbound on St. Clair made an illegal right turn, hitting Kartik, and then dragging him under the truck for a further 30 meters. Eyewitnesses describe screaming at the driver to stop while this was happening. At the present time, the details are still under investigation and no charges have been laid.

Today was the ghost bike ride in his memory.

The crowd gathers at Bloor and Spadina.

Geoffrey talking to the media.

Many of the usual suspects in attendance. Here from left to right, Albert, John, Ingrid, and Faraz.

Bicycle Mayor Lanrick Bennett Jr., Tom Flood, Matthew and Dave.

Newly elected councillor Diane Saxe pledges to push for safety for cyclists and pedestrians at City Hall. She rides a bike to work.

Geoffrey thanks everyone for coming out on a cold and windy evening, announces the route, and gets us ready to line up in preparation for riding.

The ghost bike.

Lining up along Bloor, Joey and some Marshalls from the Bike Brigade in front.

Of we go.

Pause at the ROM.

At Bloor and Yonge.

Someone with more patience than I can count the number of cyclists. I think I missed about the first 25 in this video.

Now headed north on Yonge.

Approaching the crash site.

There is already a memorial here.

Installing the ghost bike.

Joey calls for a minute of silence.

Tino is writing a message.

The banner. We are thankful for Kasia’s attendance (to the right). Her husband, Tom Samson, was killed ten years to the day earlier than Kartik. We are still putting up ghost bikes.

Thanks to Geoffrey for making the ghost bike, Joey for leading the ceremony, and the Bike Brigade for marshalling tonight. Thanks also to everyone who either rode with us or met us at the crash site.

Deepest condolences to the family and friends of Kartik.

Updates:

• The ride was the first item in this news broadcast on CBC
• “Driver charged in death of cyclist at Yonge and St. Clair” CBC news

John Scalzi's version of ethical is this: "In all other circumstances, and especially when there'™s a commercial intent or application, or when I would otherwise hire an artist, I will seek out artists and commission art from them." Why would this be the most ethical approach? Do we purchase only hand-crafted furniture, meals created by our personal chef, or books written by hand? Of course not. The role of ethics in addressing the replacement of work by artificial intelligence is to ensure that the wealth and benefits produced by the AI are shared equally across society, so that being out of work for the artist isn't equivalent to being sentenced to poverty.

I’m proud to announce that we have acquired Pulse, an incredible team that has developed some truly novel machine learning approaches to help streamline the digital workplace. The products that Raj, Jag, Rolf, and team have built are a great demonstration of their creativity and skill, and we’re incredibly excited to bring their expertise into our organization. They will spearhead our efforts in applied ethical machine learning, as we invest to make Mozilla products more personal, starting with Pocket. 

Machine learning (ML) has become a powerful driver of product experience. At its best, it helps all of us to have better, richer experiences across the web. Building ML models to drive these experiences requires data on people’s preferences, behaviors, and actions online, and that’s why Mozilla has taken a very cautious approach in applying ML in our own product experiences. It is possible to build machine learning models that act in service of the people on the internet, transparently, respectful of privacy, and built from the start with a focus on equity and inclusion. In short, Mozilla will continue its tradition of DOING: building products that serve as examples of a better way forward for the industry, a way forward that puts people first.

Which explains why we were so excited when we began talking to the Pulse team. It became immediately obvious that we both fundamentally agree that the world needs a model where automated systems are built from day one with individual people as the primary beneficiary. Mozilla, with an almost 25 year history of building products with people and privacy at their core, is the right organization to do that. And with Pulse as part of our team, we can move even more quickly to set a new example for the industry. 

One of the things that makes this marriage such a great fit is Pulse’s history of building products that optimize for the preference of each individual customer. They know how to take things from theory and design and turn them into real product experiences that address actual needs and preferences. That kind of background is going to be critical as we work to enhance the experience across our existing and new products in the coming years. I’m particularly excited to enhance our machine learning capabilities, including personalization, in Pocket, a fantastic product that has only just scratched the surface of its ultimate potential.

We have big plans for the Pulse team’s skills and know-how, and are thrilled to have their contributions to our growing entire portfolio of products. 

So, Raj, Jag, Rolf, and team, welcome aboard! We are energized by the chance to work together, and I can’t wait to see what we build.

The post Pulse joins the Mozilla family to help develop a new approach to machine learning appeared first on The Mozilla Blog.

Zur CES 2022 kündigte Lenovo zwei neue ThinkPad Z13/Z16 an, die mit vielen gewohnten ThinkPad-Designs brachen und dazu noch den von mir seit 20 Jahren nicht mehr genutzten TrackPoint weniger betonten. Und ich fand die Geräte schlicht scheußlich. Der Deckel war mit “veganem Leder” bezogen und von einem messingfarbenen Rahmen umgeben.

Dabei hatte ich übersehen, dass es diesen ThinkPad auch in grau oder in schwarz gibt. Nachdem sich Marco positiv über ein Testgerät geäußert hatte, habe ich dann diese Woche noch einmal genauer hingesehen und ein ThinkPad Z13 Gen 1 (21D20029GE) bestellt. Ausgepackt, eingeschaltet, und rumms: Editor-refuses-to-give-it-back Award.

Ich habe sofort beschlossen, mein geliebtes X1 Yoga gen 7 zurückzugeben. Dabei ist der Z13 der totale Gegenentwurf zum Y1 Yoga. Der Bildschirm lässt sich nur bis 135 Grad aufklappen. Immerhin hat er einen Touchscreen und funktioniert auch mit meinem Lenovo Active Pen 2. Für eine Unterschrift reicht es also in jedem Fall.

Ich muss also Tablet-Nutzung aufgeben und habe eine ganz Menge Ports weniger. Da ich ein Dock habe und auch mit den beiden Thunderbolt Ports des Surface Pro gut auskomme, sind das für mich keine Nachteile. Dafür bekomme ich ein wesentlich kompakteres Gerät. Weil der Deckel hinter die Basis klappt, scheint der Bildschirm beinahe randlos zu sein.

Der ThinkPad Z16 ist im Prinzip gleich aufgebaut, hat also nicht wie in meinem Tiktok-Video behauptet ein Numpad. Ich brauche so einen riesigen Rechner nicht, aber so schaut er aus:

Auf diesem PR-Bild von Lenovo sieht man auch das sehr eckige Design, das sich bis in die Tastatur fortsetzt. Auch diese Tasten haben den typischen Smile Shape mit abgerundeter Unterseite, aber er ist weniger ausgeprägt. Ebenfalls eine Herausforderung für Menschen mit ThinkPad-Vergangenheit: Die Tastenreihenfolge ist Strg-Fn-Win-Alt und nicht Fn-Strg-Win-Alt. Ich mag das. Wer es nicht mag, kann es über Vantage umschalten. Und hat dann halt die falsche Beschriftung. Vielleicht kann man sogar die Kappen tauschen? Ich werde es nicht probieren, denn mir gefällt das so besser.

Im Video heißt es auch, das Gerät habe zwei Thunderbolt-Ports. Das ist richtig und falsch. Richtig ist die Leistung, falsch ist der Name. Der gehört nämlich Intel und dieses ThinkPad hat einen AMD Ryzen 7 PRO 6850U mit integrierter Radeon 680 M Grafik. Die Leistung im JetStream2 Browser Benchmark ist mit der aktuellen Edge-Version entsprechend ordentlich.

Also eine tolle Kiste und meine erste mit AMD. Ich bin gespannt, wie wir uns vertragen werden.

It’s Giving Tuesday and if you are in Canada and looking for places to donate money, I encourage you to head over to a new website launched and hosted by The Circle on Philanthropy which connects Indigenous communities and funders, foundations and donors.

The new website is called The Feast House and it is a place where you can donate directly, abundantly and without restriction to Indigenous-led organizations and projects across the country. It also contains links to articles, podcasts and videos to hep you learn more about giving and philanthropy in an Indigenous context

Donating money to Indigenous-led work is the bare minimum next move in what The Circle calls “Active Reciprocity.” What has been known as “reconciliation”in Canada should be a set of practices that develop relationship, return resources to Indigenous community and enable Indigenous-led organizations, projects and Nations themselves to lead the work.

For many years now, I have given locally to organizations and Nations in whos territory I am working. Whenever I am paid to run a meeting and the responsibility to acknowledge Indigenous territories falls to me, I donate to a local cause that requires unrestricted funds to do it’s work. This means that I have to research and make a connection with local people and local change efforts and so that becomes a beautiful part of this responsibility.

The Feast House is a great resource to help you do this too. So as you ponder how to spend your Giving Tuesday and how to put active reciprocity in your personal commitment to reconciliation, spend some time there today.

If posts in a social media app do not have URLs that can be linked to and viewed in an unauthenticated browser, or if there is no way to make a new post from a browser, then that program is not a part of the World Wide Web in any meaningful way.

Consign that app to oblivion.

— JWZ

How to Build a Happy Life from The Atlantic is a podcast on finding happiness:

In our pursuit of a happy life, we build, we structure, and we plan. Often, we follow conventional wisdom and strategize. But what happens when our plans fall through and expectations don’t meet reality—when the things that should make us happy don’t?

In season 3 of our How To series, Atlantic happiness correspondent Arthur Brooks and producer Rebecca Rashid seek to navigate the unexpected curves on the path to personal happiness—with data-driven insights and a healthy dose of introspection.

I’m late to this, but I had some downtime during the Thanksgiving break and liked the data- and research-centric episodes. As you might expect, there’s a lot of fuzziness in the numbers and there’s more than one way to find happiness.

Tags: Atlantic, happiness, podcast, uncertainty

Over on Mastodon, there are many people who enjoy not being in the grip of software like Facebook or Twitter that single-mindedly tries to maximize “engagement”, which means the amount of time you stare at the screen so they can show you ads. These algorithms don’t care what they’re showing you and if it turns out that showing you exclusively stories vilifying or praising Donald Trump (depending) maximizes engagement, then that’s what you’ll see. So the chant over there is “No algorithms on Mastodon!” This chant is wrong, and the discussion around it teaches us that we need clarity on what algorithms are, what moral weight they can carry, and whether they can be avoided. (Spoiler: They can’t.)

This all started when I interjected here, and the longest and most twisted Mastodon thread I have so far seen ensued. Let’s start with my initial remarks:

I disagree. An algorithm is not intrinsically bad. As long as we understand that it represents the interests of whoever paid to have it constructed. I think an algorithm with human values that simply wanted to enrich experience is perfectly possible.

I haven't seen one, probably because nobody has ever had a financial incentive to construct it.

Mastodon would be a good place to try to make one.

In the following discussion I’m going to use Mastodon terminology: “Boost” is a synonym for “Retweet” and “Favorite” for “Like”.

What an algorithm is and why there will always be one

Consider a Mastodon instance that is engaged in creating a feed for, well, you, because you’ve just opened that tab or app. It gets the list of who you follow, probably finds some of those posts already in its memory, and reaches out to other Mastodon instances for the rest. It ends up with a big jumbled unsorted list of what could go in your feed. “Good,” you say, “just sort them in reverse chronological order and we’re done. Don’t talk to me about algorithms!”

Well, first of all, sorting is an algorithm, and not a simple one at that.

But it’s not that simple. Suppose one post from someone you don’t follow has been boosted by 5 people that you do, all at different times. Does it appear just once in your feed (and using which boost timestamp?) or five times? Neither answer is obviously right, but when you make that choice, you’re designing an algorithm.

Some of the posts are replies to each other, and to you. Do they appear right next to each other or in strict chronological order, mixed up with other posts? If you’re going to thread them together, do you do that in chronological not reverse-chronological order? Welcome to the world of algorithm design.

I guarantee the algorithm that generates “simple chronological order” isn’t simple at all. Among other things, it handles lots of corner cases other than the ones I just mentioned, that nobody is smart enough to think of until they actually start writing the code to present the posts to the people.

What “engagement” algorithms are like

Engagement maximization, a la Facebook or Twitter, is a special and very interesting case. It begins innocently; let’s include a few posts in your feed from people you don’t follow if those people are super-popular, or are followed by nearly all the people you do follow, or contain hashtags that you’ve been searching for a lot. Mostly harmless, right?

But then it gets deep. These things involve the application of large-scale Machine Learning (ML) technology. The big operators have billions of data points; they know what appears in people’s posts, and they know how long the people were “engaged”. So, by processing those billions of data points, they produce an “ML Model” which is then exercised to answer one question: “What selection of posts should we feed this person to keep them engaged?”

Modern ML models aren’t simple at all, and also are generally not comprehensible by humans. The people who built the model can’t answer questions like “Why is the model feeding a lot of posts by Ben Shapiro to people who seem only mildly conservative?” But they can prove that doing what the model says maximizes “engagement”.

A lot of people are now reacting viscerally, saying “I never want that kind of algorithm in my life.” Well, that kind of algorithm is being used every day to filter spam out of your email. And being used on Twitter to combat Nazis and incels, to the extent that when oppressed groups migrate to Mastodon, they get a lot more abusive bigotry. (I think we can fix that.)

Protect me from what I want

And anyhow, those algorithms are just showing you what you want. Don’t try to deny it, if it wasn’t what you wanted you wouldn’t be doomscrolling so much, would you? These ML models know what you want and that’s what they show you.

(Jenny Holzer is wonderful. One time many years ago I was walking across Times Square in New York and on a huge otherwise-blank billboard were her words: “Protect me from what I want.” It felt like someone sticking a knife into my brain; that phrase explains so very much.)

In whose interest?

In my Mastodon-post outtake above, I said that an algorithm “represents the interests of whoever paid to have it constructed”. That’s true and, in the context of a capitalist enterprise, a fairly complete answer. But in the world of software, things can happen outside the control of capitalist enterprises.

Open-source, for example; the algorithms in Linux that make computers useful, or in Postgres that reliably store and retrieve data, or in Nginx that efficiently respond to Web traffic, were mostly written by people who found the work interesting and had a problem they needed to solve for themselves.

With the advent of the Fediverse generally and Mastodon specifically, for the first time we have a large-scale opportunity to experiment with algorithms that are written for people by people just because they’re cool, or because they produce feeds that the programmer likes for herself, or that her Dad likes, or that she notices causes her kids to be less obsessive about screen time.

So let’s stop saying “No algorithms!” because that’s just wrong, and figure out how to get nice algorithms built, ones that primarily are there to serve humanity’s best interests.

One thing I think we can all agree on is this: We want algorithms that (unlike every commercial social-media algorithm) don’t tell anyone else what we’re watching!

Write your own algorithm?

I’ll be honest: I want the algorithm both to give me, and protect me from, what I want. And I want some control over it. I can think of a couple of ways this could happen:

1. Someone figures out a “feed algorithm construction kit” that has a bunch of knobs on it you can twist, with labels like “tennis” and “activism” and “Christianity” and “keto diet” and “baroque music” and “surprise” and “pretty” and “outrage” and so on, and you fool with the settings until you get a feed you like.

   I think this is plausible, but very difficult.

2. Mastodon introduces a feature where you can download and install algorithms, which can be posted by anyone. They are given the raw unsorted list of posts from people you follow and use that produce a coherent feed. You might have to pay for them. They could be free. They could involve elaborate ML, or not. They might sometimes pull in posts from people you don’t follow. They could be open-source, or not.

   I like this idea a lot, although the technology would require careful design. The algorithm would have to be able to store information about what you’ve seen and how you’ve reacted, and some ability to follow the social network around the Fediverse, but simultaneously would have to be restricted so it could never reveal what it knows about you.

Your friendly local algorithm shop

I’m simultaneously a pretty extreme leftist and convinced that free marketplaces, deployed in our service, can produce useful results. I’d like there to be multiple feed-construction algorithms that are competing for my business, that people talk about at the pub while watching the hockey game or at the toddler drop-in or in Mastodon threads.

I’m pretty sure there are surprises waiting, and some of them are going to be good.

I used to use a combination of actions/setup-python and actions/cache in all of my Python GitHub Actions projects in order to install Python dependencies via a cache, rather than hitting PyPI to download copies every time.

actions/setup-python added built-in caching a while ago, but it wasn't obvious to me from the documentation how I could use that with setup.py based projects (as opposed to projects that install dependencies via Pipfile or requirements.txt).

The trick is to use cache: pip and cache-dependency-path: setup.py as arguments to the actions/setup-python action.

Here's the pattern I found that works:

    - name: Set up Python 3.11
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'
        cache: pip
        cache-dependency-path: setup.py
    - name: Install dependencies
      run: |
        pip install '.[test]'

And here's a full .github/workflows/test.yml configuration that uses a matrix to run tests against five currently supported Python versions:

name: Test

on: [push, pull_request]

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
    steps:
    - uses: actions/checkout@v3
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}
        cache: pip
        cache-dependency-path: setup.py
    - name: Install dependencies
      run: |
        pip install '.[test]'
    - name: Run tests
      run: |
        pytest

I updated my various cookiecutter templates to use this new pattern in this issue.

Stable Diffusion 2.0 and the Importance of Negative Prompts for Good Results

Stable Diffusion 2.0 is out, and it's a very different model from 1.4/1.5. It's trained using a new text encoder (OpenCLIP, in place of OpenAI's CLIP) which means a lot of the old tricks - notably using "Greg Rutkowski" to get high quality fantasy art - no longer work. What DOES work, incredibly well, is negative prompting - saying things like "cyberpunk forest by Salvador Dali" but negative on "trees, green". Max Woolf explores negative prompting in depth in this article, including how to combine it with textual inversion.

There’s so much to say about what’s happening at Twitter, but I’m going to start with one word: “Hardcore.” That’s what Elon said he wants from all his employees going forward – a “hardcore” mentality, a coder-first culture, a sleep-at-the-office-and-pound-Red-Bull kind of sensibility.

I’m pretty familiar with this culture – an earlier, less toxic version of it pervaded the pre-Elon tech world, a culture I reported on at Wired, the Standard, and in coverage of Google and similar companies in the early 2000s. While it had its charms – most of us have pulled an all nighter trying to get a product out – memorializing “hardcore” as a work ethos is a deeply flawed management technique. Not only does it foster unhealthy relationships to work, it also celebrates a toxic brand of male-dominated power – the kind of power that many of tech’s current titans, including Musk, Andreessen, Thiel, and their ilk – seem to believe is threatened. In their writings, investments, and political lobbying, it’s clear that “hardcore” is a philosophy this group of Valley troll-bullies seem desperate to entrench.

In fact, I think it’s fair to postulate that Musk’s takeover of Twitter, supported as it is by Andreessen’s a16, Oracle’s Larry Ellison, a boatload of Saudi money, and crypto mystery man Changpeng Zhao, is at its core driven by a  neo-reactionary response to the ongoing decline of the Great Man narrative across our society and in what they deride as “woke” corporate culture. The myth of the hero founder has always dominated Silicon Valley: The energy-drink fueled “great man” of tech who solves the world’s problems not with diplomacy and tact, but through singular genius, movie star grit, an assholic temperament, and raw coding prowess. Elon is, of course, this myth’s current favored son, its most treasured proof. This is the man that took on the entire auto industry, that landed rockets on barges, that married rock stars and inspired countless Iron Man sequels. Never mind the government handouts, the endless lawsuits, the fatherless children or the constant promotion of hate under the guise of “free speech.”

You’ll probably not be surprised to hear that I think the Great Man founder myth is stupid and facile, worthy of ridicule. And far better writers than me have torn it down. But I can’t get the pre-Elon Twitter out of my mind – because I knew the company so well. This was a culture committed to building relationships, that admitted its role in the world was complicated, nuanced, and required endless patience and diplomacy. Its people were not dedicated to profit above all else. Instead, they were dedicated to fostering the site’s unique role in the online world, to telling that story, to listening to criticism, to supporting those who needed support. Yeah, it was often disfunctional, but they worked on that disfunction tirelessly, and with humanity. And because of their work, Twitter’s corporate culture was truly special.

And who was at the core of that culture? I have a theory: It was the women.

Twitter was probably the most intentionally open, accommodating, and thoughtful work culture the Valley has ever produced at scale. And it’s not a coincidence that a healthy percentage of Twitter’s senior executives were women. Nor is it a coincidence that nearly all of them have left. I started keeping a list of the extraordinary women I worked with over the past few years who have recently departed the company. And just for posterity, and perhaps for you all to add to, I present it here. Think about all the men cheering on Elon’s “Hardcore” philosophy, who agree with him that the people below, and countless others, are unnecessary. Read through these names, click on their profiles, and ponder the roles they played in the nuanced ecosystem Twitter once was.

And then, let that sink in.

Leslie Berland – CMO

Sarah Personette – CCO

Dalana Brand – Chief People and Diversity Officer

Vijaya Gadde – Legal, Public Policy & Trust and Safety Lead

Lea Kissner (They, Them) – CISO

Robin Wheeler – Head of Sales

Sonya Penn – GM of Data and Developer Platform

Rebecca Hahn – VP Communications

Lara Cohen – VP, Global Head of Partners

Meg Haley – VP Revenue Product

Maggie Suniewick – VP Partnerships

Stephanie Prager – VP Global Business

Melissa Barnes – VP, Canada and Latin America

Julianna Hayes – VP Finance

Nola Weinstein – Global Head of Brand Experience & Engagement

Sarah Rosen – Sr. Director & Head of US Content Partnerships

Joanna Geary – Sr. Director, Curation

Of course there are, literally, thousands more. These are just the first 15 or so names of senior women executives who I interacted with at one time or another, people who helped make Twitter the amazing place it once was. Please add yours in comments or email me, I’d be happy to add to this list. Oh, and the photos – top is a picture Elon posted to prove he and his engineers were “hardcore.” Love the gender diversity (and the Trumpian thumbs up pose). And below, inexplicably but in perfect troll baiting form, is a picture of his night stand. 

Scaling Mastodon: The Compendium

Hazel Weakly's collection of notes on scaling Mastodon, covering PostgreSQL, Sidekiq, Redis, object storage and more.

Via hachyderm.io/@nova

The answer to this question is "yes", or at least, "it's complicated". Every job will be impacted, and a lot of work will disappear entirely. But the point of this article is to deny deny deny. "Whoa, now. Time for a reality check," writes Matt Crosslin. "There is still an element to creativity that involves blending those influences into something else that transcends being strictly derivative of existing styles." Quite so. But why would anyone suppose AI is limited to being strictly derivative of existing styles? Techniques such as transfer learning allow an AI to combine input from any number of sources, as illustrated, just as a human does.

I don't know what the point would be of wrecking Athabasca University, but this seems to be the plan, at least as seen from the outside. Says one observer, "I cannot think of another situation where government is mandating where employees live." It would likely result in a loss of staff, not merely senior executive, leaving the university. The real loss would be of faculty and staff, without whom the university could not operate.

Disruptive innovation requires viewing the world differently from conventional wisdom. Disruptors who succeed manage to change the conventional wisdom. Disruptors who fail end up victims of the conventional wisdom. And that includes the conventional wisdom inherent in government regulation Disruptors who ignored the rules YouTube was built on pirated, copyrighted content. It violated rules and … Continued

The post Disruption wins — except when it loses appeared first on without bullshit.

I enjoyed this story about Last.fm’s 20th (!) anniversary by Jacob Kastrenakes, writing for The Verge:

I was a little surprised to see that Last.fm was still around when I first started writing this story, let alone that it had new communities flourishing around its data. (The company didn’t respond to a request for an interview.) But I suppose in a world where most services close off and hide your data, there’ll always be people looking for a way to track it and analyze it themselves. And in exchange, they get the joy of arguing about music stats every day — and not just once a year when Wrapped comes out.

My co-hosts on Connected like to make fun of me for being One of Those People Who Still Scrobbles, but I can honestly say it’s one of the best things I’ve done for my music consumption in the past few years. (That, plus having an offline library with albums I own that I can enjoy with my favorite headphones and amp – which I also scrobble via Roon.) Ever since I started scrobbling again last year thanks to Marvis Pro on iPhone and iPad (and NepTunes on the Mac), I’ve been able to enjoy some fascinating monthly and annual breakdowns of my music listening habits that go much more in depth than Apple Music or Spotify would ever want to.

An example of a monthly Last.fm report.

In Internet years, it’s pretty wild for anything to turn 20 – let alone a service that faces competition from the likes of Apple and Spotify. And yet Last.fm has been able to carve a niche for itself by appealing to people like me, who want to know more about the music they listen to. Maybe it’s a weird thing to say in 2022, but if you listen to a lot of music every day, I can’t recommend dusting off your old Last.fm account enough.

→ Source: theverge.com

When NASA’s Webb Space Telescope team and artist Ashley Zelinskie wanted to bring space exploration to everyone, they chose Mozilla Hubs, our open source platform for creating 3D virtual spaces right from your browser. 

Ashley told us that they “didn’t want to cut people out that didn’t have fancy VR headsets or little experience in VR. … If we were going to invite the world to experience the Webb Telescope we wanted everyone to be able to attend.” 

That’s exactly why Mozilla has been investing in the immersive web: We believe that virtual worlds are part of the future of the internet, and we want them to be accessible and safe for all. 

That means each Hubs user controls access to the virtual world they created, which is only discoverable to the people they share it with. Hubs users and their guests can also immerse themselves in this world right from their desktop or mobile browser – no downloads or installations required. And while you can use a VR headset, you can access the same spaces through your phone, tablet or desktop computer.

If you’re curious, take a look at a few ways people have been creating immersive worlds with Mozilla Hubs: 

To create art galleries and portfolios

It’s not just space art. A virtual museum of art prints put together by the creative agency Paradowski helped raise money for a COVID-19 response fund by the World Health Organization. In St. Louis, Missouri, the American Institute of Graphic Arts showcased artists’ work during the school’s annual design show. In the U.K., the Royal Institute of British Architects presented an exhibition that immersed visitors in architectural milestones over the last five centuries. 

While Mozilla Hubs can host projects on the grandest scale, you can use it for personal projects too: Whether you’re an artist, photographer or a 3D modeler, you can create an immersive portfolio that’s easy to use and accessible to anybody with a browser.

To build spaces for hobbies (and meet new people)

The website Meetup lets people find local events based on their interests – from pet poultry to coding to learning a new language. In addition to in-person gatherings, the platform allows people to organize online. Those who wish to meet up virtually can do so in a 3D space through Hubs. 

You can create your own immersive space and invite others. You can also just grab an existing room made available by another creator, remix it and make it your own.

To teach and learn 

NYU Langone Health, one of the largest healthcare systems in the northeast U.S., uses Hubs to teach anatomy. Hubs helps instructors immerse medical students in the coursework, including 3D vascular stereoscopic models.

Oklahoma State University’s Emerging Technologies and Creativity Research Lab created a virtual science expo that showcased different Earth environments and hosted Q&A sessions with scientists.

Olympic medalist Sofía Toro, along with professors from the Universidad Católica San Antonio in Spain, even taught a windsurfing class online using Hubs.

Virtual spaces offer new opportunities for connections and innovation. Through Hubs, Mozilla wants to make those opportunities available to everyone. Learn more and join the Hubs community here.  

The post 3 ways to use Mozilla Hubs, a VR platform that’s accessible and private by design appeared first on The Mozilla Blog.

It feels like a day of transitions. The weather is clear today, and a strong westerly pummeled Vancouver overnight. It is sunny now, but the cold air and sea level snow that is a hallmark of an El Niña winter is upon us for later in the week. So time to chop some more wood and harvest the last of my salad greens from the garden.

Canada’s men’s national soccer team lost to Croatia this morning our Alphonso Davies scored the first goal for Canada’s men’s team in world cup history and it was a beauty. The above photo is from his last game as a Vancouver Whitecap in 2018 before he headed to Bayern Munich where he has since set the world on fire. We have one game remaining and then this team will transition into the next cycle as we get ready to host the 2026 men’s World Cup without the likes of Atiba Hutchinson and Milan Borjan and some of those veterans that carried us for so long as we languished in obscurity. Today’s loss was tough, but we need this learning and tempering in the cauldron of global competition if we are to stay at this level. So one more game against Morocco and then after this tournament is over, attention transitions away to follow the women’s team who will be playing in the 2023 World Cup. I am keen to see how we do as the only major women’s soccer power in the world without a domestic professional league.

And it is the beginning of Advent today, a season I very much appreciate. The waiting for something to materialize, for the light to return…in all its physical and spiritual manifestations, this is a powerful season of transition into deep darkness and then out again. As if to embody it, Friday I went for a cliff top hike along the south shore of our Island, in a place known as Nicháych Nexwlélexwm, which is the very edge of the world in so many ways. I was looking for the humpback whales that have been hanging out there and after an hour of watching and waiting finally there were three, breaching and splashing and diving and feeding. The Sound is full of anchovies and herring at the moment and there is lots to eat. Even this morning, watching from the ferry as sea lions and gulls filled their bellies.

mkalus shared this story from jwz.

tl;dr avoid Hive and Post.

If posts in a social media app do not have URLs that can be linked to and viewed in an unauthenticated browser, or if there is no way to make a new post from a browser, then that program is not a part of the World Wide Web in any meaningful way.

Consign that app to oblivion.

Most social media services want to lock you in. They love their walled gardens and they think that so long as they tightly control their users and make it hard for them to escape, they will rule the world forever.

This was the business model of Compuserve. And AOL. And then a little thing called The Internet got popular for a minute in the mid 1990s, and that plan suddenly didn't work out so well for those captains of industry.

The thing that makes the Internet useful is interoperability. These companies hate that. The thing that makes the Internet become more useful is the open source notion that there will always be more smart people who don't work for your company than that do, and some of those people will find ways to expand on your work in ways you never anticipated. These companies hate that, too. They'd rather you have nothing than that you have something they don't own.

Instagram started this trend: they didn't even have a web site until 2012. It was phone-app-onlly. They were dragged kicking and screaming onto the World Wide Web by, ironically, Facebook, who bought them to eliminate them as competition.

Hive Social is exactly this app-only experience. Do not use Hive. Anyone letting that app -- or anything like it -- get its hooks into them is making a familiar and terrible mistake. We've been here before. Don't let it happen again.

John Ripley:

So many people, who should know better, blogging about their switch to Hive on the basis of user experience or some other vacuous crap, and not fundamentals like, "Is this monetized, and if not yet, when how and who?" or "who runs this?" or "is it sane to choose another set of castle walls to live as a peasant within?"

Post Dot News also seems absolutely vile.

First of all, Marc Andreessen is an investor, and there is no redder red flag than that. "How much more red? None. None more red", as Spinal Tap would say. He's a right wing reactionary whose idea of "free speech" is in line with Musk, Trump, Thiel and the rest of the Klept.

Second, it appears to be focused on "micropayments", which these days means "cryptocurrency Ponzi schemes", another of marca's favorite grifts.

They call themselves "a platform for real people, civil conversations". So, Real Names Policy and tone policing by rich white dudes is how I translate that. But hey, at least their TOS says they won't discriminate against billionaires:

life, liberty, and the pursuit of happiness, regardless of their gender, religion, ethnicity, race, sexual orientation, net worth, or beliefs.

Mastodon is kind of a mess right now, and maybe it will not turn out to be what you or I are looking for. But to its credit, interoperability is at its core, rather than being something that the VCs will just take away when it no longer serves their growth or onboarding projections.

There is a long history of these data silos (and very specifically Facebook, Google and Twitter) being interoperable, federating, providing APIs and allowing others to build alternate interfaces -- until they don't. They keep up that charade while they are small and growing, and drop it as soon as they think they can get away with it, locking you inside.

Incidentally, and tangentially relatedly, Signal is not a messaging program but rather is a sketchy-as-fuck growth-at-any-cost social network. Fuck Signal too.

Previously, previously, previously, previously, previously, previously, previously, previously.

Christine Sinclair, photo by Ray Terrill, CC AT-SA 2.0

I have been loving watching Canada back in the men’s World Cup for the first time since 1986. And while this has been going on on social media and in the broadcasts, a number of people have been insisting that we remember that this is the Men’s World Cup and that Canada has been long known as a dominant force in international soccer on the women’s side. There are two FIFA run World Cups for senior teams. And it’s mind boggling how invisible our achievements on the global stage have been because of the neglect of women’s soccer in Canada.

I’m a bit salty about this, because our women’s team has been AMAZING. Here are some interesting facts.

• The Canadian National Women’s team is often referred to on social media as the “CanXNT” because we have a non-binary trans player, Quinn, who became the first trans athlete to win a gold medal last year when we won the Olympics.
• The top scorer in world soccer of any gender is Christine Sinclair who has scored 190 goals in 310 performances for Canada. She eclipsed American Abby Wambach last year and has played and scored in five World Cups one of only three players ever to do that (the others being Marta and Ronaldo). Next year, she will likely play in, and hopefully score in, her sixth.
• International women’s soccer in North and Central American and the Caribbean (the CONCACAF Confederation) is generally played in tournament settings, often all in one place,and hardly ever in Canada. THis is unlike men’s soccer where qualifying and Nations League games are played home and away. So despite appearing 310 times for Canada since 2000, Sinclair has only ever appeared in 13 competitive games on Canadian soil for Canada and the last time was seven and half years ago at the 2015 World Cup. The greatest international soccer goalscorer of all time has not played a competitive match in her own country since 2015. Just friendlies.
• The USA has played 63 competitive matches at home during that same time.
• And that isn’t just international matches too. Sinclair currently plays for the Portland Thorns and hasn’t played for a Canadian club team since she left the Vancouver Whitecaps in 2008.
• And even if she wanted too, she couldn’t play a competitive club match in Canada because Canada does not have a domestic women’s soccer league. Of all the countries appearing in the 2023 women’s World Cup, Canada will be the only one without a national women’s league, and this is despite the fact that Canada hosted the 2015 Women’s World Cup, an event which usually results in the establishment of professional infrastructure. We have had seven years of not much happening. Well, we did finally start a men’s professional league in Canada, the Canadian Premier League. Yes our domestic men’s soccer league was only started in 2019, despite some failed attempts earlier in the century.
• The highest level of women’s soccer is currently one of three regional semi-professional summer leagues in Ontario, Quebec and BC federated in League 1 Canada. League 1 BC features nine teams including TSS Rovers, the supporter-owned club I have written about before. We go into the World Cup behind in our development to Zambia, Vietnam, Nigeria, Morocco, and every other European, Asian, South and Central American country who has qualified.
• I THINK we will be the only team at the 2023 World Cup that has no players playing domestically for their country. All of our national team players play professionally abroad, in the USA and Europe.

People are excited to start talking about Canada as a finally arriving as a soccer country, and while lots has happened to get us to this point, we cannot make that claim. Not while we have the greatest goal scoring in international history only playing a handful of friendly matches at home. We will be the last country in the world at the 2023 World Cup to be without a league.

This largely falls at the feet of the Canadian Soccer Association, but I’m not going to list all the ways that the CSA has screwed this up. Suffice to say that the best accomplishments of a generation of Canadian global sport personalities have been hidden, squandered, and wasted. We have built no legacy and every year, even as we propo up the illusion with an appearance in a World Cup here or there, with a big name signing to a European juggernaut here or there, we fall behind the rest of the world because we coasted on the talents of a century-level player whose achievements were lauded, but whose appeals for a legacy fell upon deaf ears.

My involvement with TSS Rovers is a small way that I can be a part of investing in the sustainability of long term development of soccer talent in Canada. As the saying goes, the best time to have done this was 20 years ago. The second best time is today. So let’s enjoy the Morocco game on Thursday and look forward to the Women’s World Cup in July and maybe, just maybe, we’ll get around to taking the “We’re a Soccer Country!” idea seriously.