This TIL is mainly a rehash of these two articles by Jacob and Andrew:

• Setting up a personal Fediverse ID / Mastodon instance - Jacob Kaplan-Moss
• The Fediverse, And Custom Domains - Andrew Godwin

I decided if I was going to get into Mastodon I wanted it on a domain that I controlled.

Here's my research issue where I first figured this all out.

masto.host

Both Andrew and Jacob chose masto.host as a managed host for their instances. Mastodon is open source and runs on Ruby, PostgreSQL and Redis, but managing those is enough of a hassle that I'd much rather have someone else do it for me.

Since Andrew and Jacob had done the due diligence on this already I just went with the one they are using.

I see this as a pleasantly low-risk vendor, because since I'm pointing my own domain at it I can move elsewhere pretty easily if I need to.

My impressions of it so far have been excellent - especially the speed of their customer support (see later note).

The Mastodon docs have a list of other companies that provide hosted instances like this.

Pay for an account, then set up a CNAME

I paid for a subscription, then added the following configuration to my Cloudflare DNS:

masto.host provided detailed instructions for this, including making sure to turn off the Cloudflare caching proxy.

This started working within less than a minute - and https://fedi.simonwillison.net/ was live.

Creating an account, promoting it to admin

Once my instance was live I used the default account creation flow to create myself an account.

I then used the https://my.masto.host/hosting interface to find the "Change User Role" option and used that to upgrade my new user account to administrator status.

Having done that I gained access to the https://fedi.simonwillison.net/admin/settings/edit interface, where I blocked anyone else from creating an account and changed the site theme to "Mastodon (Light)" (which I like better).

Frustratingly these settings require an email address, which is shown publicly on a page on the site. I used iCloud in Mobile Safari on my phone to create a disposable email address to use here.

Getting a vanity address

I wanted @simon@simonwillison.net as my ID, but it started out as @simon@fedi.simonwillison.net.

To do this, you need to set up some /.well-known/... URLs on your core domain.

I exactly copied how Andrew did this. Here's my commit to my Django blog.

There's one last step here: as explained in Mastodon usernames different from the domain used for installation you need to update the LOCAL_DOMAIN and WEB_DOMAIN settings. These aren't currently available for masto.host customers to change, but you can email their support team about it.

I emailed them and they fixed it for me six minutes later! And now @simon@simonwillison.net both works and is displayed on https://fedi.simonwillison.net/@simon

Finding people to follow

I started by following @jacob@jacobian.org and @andrew@aeracode.org. Then I looked at who they were following. Then I tweeted about my new account and started following-back people who followed me.

The iPhone app is really good

I installed the official Mastodon app on my iPhone. It's really good! I signed in by giving it my https://fedi.simonwillison.net/ address, and now it's sending me notifications and generally working exactly how I would want it to.

Veritasium: How An Algorithm Could Have Stopped The Nuclear Arms Race. (YouTube, 26:32min)

“The Fast Fourier Transform is used everywhere but it has a fascinating origin story that could have ended the nuclear arms race. […]
Thanks to Grant Sanderson of 3Blue1Brown for his helpful feedback on the script. His great video on the Fourier Transform is here”

3Blue1Brown: But what is the Fourier Transform? A visual introduction. (YouTube, 21min)

It blew my mind that Carl Friedrich Gauß found the discrete Fourier transform – 150 years earlier that Fourier.

Vaccines

This paper from Qatar says that 2 doses of Pfizer doesn’t actually give kids much protection against Omicron infection, and the protection wanes quite quickly.

In last week’s General blog post, I gave the disappointing news that two papers found that the bivalent vaccines weren’t that much better than the monovalent boosters. There are three new papers which give confusing and conflicting information.

I dug into this paper from the USA which says that a BA.5 bivalent booster actually does quite a bit better against some of the newer strains than the monovalent booster. (Yay!) It uses a more faithful measuring technique (live virus instead of pseudoviruses), so maybe it is a better study. However, when I dug into the supplemental info of the paper, the data looked slightly flakey. (I choose to believe that this is sloppiness (from trying to get the data out fast) and not malice.)

There were twelve people in each of the three groups (1 dose of monovalent (i.e. original formulation), 2 doses of monovalent, 1 dose BA.5 bivalent) which is kind of small. The groups had significant differences in their composition and treatment. The bivalent group had more prior infections than either monovalent group, was significantly younger (40 years old) than the double-monovalent (65.7 years old), and the serum collection was sooner on average since the last booster (22 days) than the double-monovalent group (82.7 days).

The differences between the single-monovalent and the single-bivalent groups were not as stark: the average time between booster and collection for the single-monovalent samples was 21 days (although all were either 1 week or 4 weeks; the standard deviation of the bivalent group was much lower); the average age for the single-monovalent group was 33 years old.

I thus think that the improvement in antibody levels going from the double-monovalent to the single-bivalent is probably an overestimate. However, the difference between the single-monovalent and the single-bivalent probably is meaningful. In this small study, one dose of the monovalent (where 2/3 of the group got Pfizer, 1/3 Moderna) was not as good as one dose of the bivalent (with 1/4 Moderna, 1/12 Pfizer, and 2/3 unreported).

This paper from the USA says that one booster of bivalent BA.5 is better than one booster of monovalent, but they are both pretty shit against XBB.1:

This paper from Israel, like last week’s papers, says that the bivalent is not much better than the monovalent:

This preprint from Qatar says that people with three doses were more likely to get reinfected with an Omicron infection that those who only had two doses. (Two or three doses was still better than zero doses.)

This is a surprising result, and suggests that maybe there is some imprinting going on.

Also, I keep thinking about the US CDC graph (from this page) which shows that one booster (solid dark blue) doesn’t give much survival advantage over two (dashed light blue), while two boosters (solid purple) really does, at least in people over 50:

(It might be, however, that people who get double-boosted are just more careful than the single-boost people.)

This preprint from South Africa says that Novavax doesn’t do as well against BA.1 or BA.5 as it does against earlier strains (no surprise). It says that Novavax’ effectiveness is comparable to mRNAs and better than AZ:

This paper from Germany says that boosters bump up antibody levels but not the number or effectiveness of T-cells.

Pathology

This tweet thread interprets this report from Australia on excess deaths. (Yes, yes, it’s Australia and not Canada, but I suspect there are things that are common between the two countries.) There were a ton of excess deaths in 2022 — +16% in July 2022:

Some were listed as COVID-19, but a ton of the excess deaths came from heart problems of various types:

and unassigned causes of death:

This graph of excess deaths by year relative to 2019 shows that deaths are way higher in Australia than they were in 2020 and 2021:

This paper from the USA says that the vast majority of COVID-19 patients develop antibodies to PF4 (a totally normal element of human blood which inhibits clotting), and the level of anti-PF4 antibodies correlates with the severity of the acute disease.

The alert reader will recognize PF4 being implicated in the (rare) clotting problems that the AZ vaccine caused. This paper, for example, says that the adenovirus used in the AZ vaccine can bind to PF4 — acting much like an anti-PF4 antibody.

I thought about putting this in the Long COVID section because while it is not quite a smoking gun Long COVID, it is certainly suspicious. (One of the leading theories on what causes Long COVID is micro-clots.)

This paper from the USA says that a combination of antibiotics and COVID-19 can mess up your gut biome, and that it can help microbes from your gut get loose into your bloodstream. It’s not clear how much is from COVID-19 and how much from antibiotics, though this tweet says that they see some leaky gut in COVID-19-infected mice (who did not get antibiotics).

This paper from the UK found a few genes which made people more susceptible to COVID-19. These genes were also associated with lupus. Some of the genes which had similar effects for lupus and COVID-19 (i.e. were correlated), but there were some genes which had opposite effects for lupus and COVID-19 (i.e. were anti-correlated).

This paper with an international team looked at genetics of people all over the world, and found that toll-like receptor TLR7 (on the X chromosome) was associated with a 5.3-fold increase in severe disease, and was not sex-linked.

This paper from Mexico (from January 2022) found that probiotics reduced viral COVID-19 loads, infiltrates and the duration of both digestive and non-digestive symptoms!

Treatments

This paper from Belgiam looked at using throat sprays of lactobacillaceae (frequently used in probiotics) shortly after COVID-19 infection. The study was small, self-selected, and partially survey-based, but there seemed to be small but significant benefit from the throat spray. Symptom severity seemed to be inversely correlated with the amount of lactobacillaceae measured in the throat; 6.7% of the treated group tested positive after three weeks, versus 26% of the placebo group. This study was by no means a slam dunk, but I think it hints at interesting future explorations.

Long COVID

There are two books for long haulers which just came out. I haven’t read either book, so I can’t recommend (or dis-recommend), but if you or someone you care about has Long COVID, maybe they will like one of these books. (Let me know what you/they think!)

This paper from Finland gave remdesivir to hospitalized patients and followed up a year later. They found that whether you got remdesivir or not did not change the number of people with Long COVID after a year. About 15% in the remdesivir group and 15% in the standard care group had Long COVID a year later.

Recommended Reading

This long general-audience article is on what’s up with nasal vaccines.

This article talks about prophylactic nasal treatments.

This article talks about how Evusheld — a monoclonal antibody that immunocompromized people could take to functionally act sort of like a vaccine — doesn’t work against the new variants. Ulp.

This article from the USA talks about the crisis they are having at pediatric hospitals with flu, COVID-19, RSV, rhinoviruses, and enteroviruses. Is Canada facing the same? I don’t know.

Spotted by: Liesbeth den Toom

After teaching many data professionals about bullet graphs and using them in many dashboards, I started to notice that they had a fair number of downsides. A few years ago, I started using an alternative called “action dots” that, I believe, are more informative, easier to understand, faster to visually scan, more compact, easier to implement, and don’t have any of the downsides of bullet graphs.

For the full story, check out my latest article in Nightingale, the Journal of the Data Visualization Society.

By the way...

If you’re interested in attending my Practical Charts or Practical Dashboards course, here’s a list of my upcoming open-registration workshops.

Following up on “How to federate like our business ecosystem” I went ahead and created a Mastodon-instance for my company. It’s at m.tgl.eu. Next to me and a generic ‘team’, three colleagues have created an account, but it remains to be seen whether they’ll be active or not.

Some observations:

• The distinction between separate accounts can cause confusion. On Twitter one is conditioned to see one account as ‘all of me’. On e-mail however we have different mail addresses for different contexts, of which work and private mail addresses are the most common two. To make the distinction visible between my personal and company account I used different avatars.
• Some of our team have been hesitant about the context collapse between work and private life. E.g. sharing phone numbers, or posting work related material on your personal social media accounts. This is completely understandable (even if for me personally all work has come from personal interests so there’s an almost complete overlap between work and private, even if not the other way around: almost all of my work would fit in my private context, not all of my private context would fit in my work context.)

Having company accounts could help, as having different accounts for different contexts works as a sort-of category filter for the types of content that are being shared. I notice that from my work account I now do follow organisational accounts, whereas from private accounts I usually avoid doing that.

That’s all on the individual level.

The actual experiment here is to see a) what occurs if everyone in the team has an individual voice in their professional context and b) how that works out across organisations in our ecosystem. Does it lead to different types of interaction, more low threshold casual interaction, between people from organisations we regard as part of our ‘scene’? It’s a type of collectiveness that is impossible on global platforms like Twitter. It’s based on a locally more dense network between specific groups of people. As yet, this is still entirely theoretical, as it depends on other organisations also having such instances, from which people connect to us. Having at least our instance makes it possible to start the experiment if there’s at least one other. Maybe having ours helps that second one to start.

I’ve previoulsy written about WASM support for in-browser SQLite databases, as well as using the DuckDB query engine to run queries over various data sources from a browser based termianl (for example, SQL Databases in the Browser, via WASM: SQLite and DuckDB and Noting Python Apps, and Datasette, Running Purely in the Browser), but now it seems we can also get access to a fully blown PostgreSQL database in the browser via snaplet/postgres-wasm (announcement; demo; another demo). At the moment, I don’t think there’s an option to save and reload the database as a browser app, so you’ll need to initially load it into a tab from either a local or remote webserver (so it’s not completely serverless yet…).

Key points that jump out at me from the full demo:

• you get a psql terminal in the bowser that lets you run psql commands as well as SQL queries;

• you can save and load the database state into browser storage:
  

You can also save and load the database state to/from a desktop file.

• a web proxy service is available that lets you query the database from a remote connection; that is, the db running in your browser can be exposed via a web proxy and you can connect to it over the network. For example, I connected to the proxy from a Python Jupyter kernel running in a Docker container on my local machine; the database was running in a browser on the same machine.

From an educational perspective, having access to a fully blown DBMS engine, rather than just a simple SQLite database, for example, is that you get access to both the psql command line line, but also database management tools such as roles and role based permissions. Which means you can teach a lot more purely within the browser.

Note that I think a webserver is still required to load the environment until such a time as a PWA/progressive web app version is available (I don’t think datasette-lite is available as a PWA yet either? [issue]).

In terms of managing a learning environment, one quick and easy way would b to run two open browser windows side by side: one containing the instructional material, the other containing the terminal.

Questions that immediately come to mind:

• What’s the easiest way is to be able to run the proxy service on localhost?
• Is it in-principle possible for the database server to run as a browser app and publish its service from there onto the localhost network?
• Is there in-principle way for the database server to run in one browser tab and expose itself to a client running in another browser tab?
• Can you have multiple connections onto the same browser storage persisted database from clients open in different/multiple tabs, or would you have to hack the common storgae by repeatedly saving and loading state from each tab?
• At the moment, we can’t connect to postgres running wheresoever from a in-browser Python/Pyodide environment (issue), which means we can’t connect to it from a JupyterLite Python kernel. Would it be possible to create some sort of JupyterLite shim so that you could load a combined JupyterLite+postgres-wasm environment to give you access to in-browser postgres db storage via JupyterLite notebook scripts?
• How easy would it be to fork the jupyterlite/xeus-sqlite-kernel to create a xeus-postgres-wasm kernel? How easy would it be to also bundle in pandas and some sql magic for a pandas/postgres hybrid, (even if you have access to no other python commands than pd. methods (and what would that feel like to use?!), along with support for pandas plots/charts?
• How easy would be to wire in some custom visual chart generating Postgres functions?!
• With a python-postgres-wasm, could you support the creation of Postgres/Python custom functions?

It could be so much fun to work on a computing course that tried to deliver everything via the browser using a progressive web app, or at most a web-server…

Today at 14:07 twenty years ago, I posted my first blog post. Well over 3000 posts later, this blog has been an integral part of quite a stretch of my life, to the point where it is unavoidable that if you’ve read along you now probably know more about me than I think I’ve actually shared in writing.

In the past few years I’ve taken this blog’s anniversary as a moment to reflect on some of my blogging practices. That yearly reflection started 5 years ago when I was just leaving Facebook. This time it coincides with #twittermigration, where many people are exploring federated options now that Musk has taken over Twitter. Whether that is something that will stick is uncertain of course, but it is interesting to watch playing out. Other earlier such reflections: 2018, 2019, 2020, 2021.

Last year I wrote:

For the coming time this note-to-blog pipeline, and making it easier for myself to post, will be my area of attention I think. Let’s see next year around this time, when I hit the two decade mark with this blog, how that went.

Indeed, that is exactly what I did from early this year: ensure that I could post directly to this blog in different ways. The key to that was create a Micropub client, which posts to this site. Once I had that I could create different paths to feed a post to that Micropub client. From inside a feedreader, directly from my notes in Obsidian, or through a simple web form. More recently I created different versions of that web form, to also post check-ins, and announce travel plans. In all fairness, my habits in how I post things haven’t fundamentally changed yet: I’m writing this in the WordPress back-end. But increasingly I am using those other paths to get content into this site.
Making it easier to post, puts the friction of blogging where it needs to be: wanting to write something.

Connecting things up into flows, blurring the lines between my site, online interaction and my notes for instance, stays an interesting thing to experiment with. In the past months I started using Hypothes.is more intensively, to annotate things I read on the web. Already all those annotations seamlessly end up in my local notes, from where I can work with them, and where they concern my own site I’ve made them visible here.

But most of all, aside from all the more nerdy things of tweaking this site and my information flows, this blog has been a source of conversation for twenty years now. It was my original hope, and my ongoing motivation to keep blogging.

Which brings me back to the earlier mentioned #twittermigration. Musk declared the bird is freed, but it seems quite a few people think the bird was caught and rather take wing on their own. Quite a few of those are the people I early on conversed with through their blogs too. If there’s a key difference between ActivityPub/Mastodon and Twitter, it’s that the federated version only ‘works’ if you actually interact with other people. Likes don’t matter in highlighting a message. Boosts do only share a message with your own followers, and has no other effect. It doesn’t mean it will be put higher in the timeline of others, it’s all in the now. There’s no amplification. Conversation is the key, if you interact then others may also see it and join the conversation. Twitter used to be like that too.

Conversation is key, and that is why I blog.
Here’s to another year of blogging and conversation.

Charts

MetroVan COVID-19 levels in wastewater, from Jeff’s spreadsheet:

I can actually see a very sloooooow decline in the Vancouver levels, but everywhere else seems pretty stable.

I've been playing around with the Mastodon timelines API. It's pretty fun!

I'm running my own instance which means I don't feel limited by politeness in terms of rate limits - if I'm just using my own instance's resources I feel fine hammering it.

Many instances (including mine) expose a public timeline API. You can use that to request all "local" posts on the server - since my server only has one user (me) that's all of my stuff.

That API starts here: https://fedi.simonwillison.net/api/v1/timelines/public?local=1

If you fetch it through curl -i to include headers you'll spot some interesting headers:

$ curl -s -i 'https://fedi.simonwillison.net/api/v1/timelines/public?local=1'
...
X-RateLimit-Limit: 300
X-RateLimit-Remaining: 294
X-RateLimit-Reset: 2022-11-05T00:40:00.971772Z
Cache-Control: no-store
Vary: Accept, Accept-Encoding, Origin
Link: <https://fedi.simonwillison.net/api/v1/timelines/public?local=1&max_id=109277272598366124>; rel="next", <https://fedi.simonwillison.net/api/v1/timelines/public?local=1&min_id=109288139761544382>; rel="prev"

The rate limit stuff is cool, but the thing that instantly caught my eye was the Link: header. It looks like Mastodon implements cursor-based pagination in that header.

I just so happen to have written a tool for working with that kind of header! paginate-json, which fetches every page by following those rel="next" links until they run out.

Here's how to download everything from that feed and save it to a single newline-delimited JSON file (the --nl option):

paginate-json 'https://fedi.simonwillison.net/api/v1/timelines/public?local=1' --nl > /tmp/simon.json

Output:

https://fedi.simonwillison.net/api/v1/timelines/public?local=1
20
https://fedi.simonwillison.net/api/v1/timelines/public?local=1&max_id=109277272598366124
20
https://fedi.simonwillison.net/api/v1/timelines/public?local=1&max_id=109276505589113334
20
https://fedi.simonwillison.net/api/v1/timelines/public?local=1&max_id=109276498877771086
13
https://fedi.simonwillison.net/api/v1/timelines/public?local=1&max_id=109276387855048126
0

And here's how to load the resulting file into a SQLite database using sqlite-utils, which also supports newline-delimited JSON:

cat /tmp/simon.json | sqlite-utils insert /tmp/simon.db posts - --pk id --nl --replace

The --pk id option sets the primary key on the newly created posts table to id. The --replace option means any records that clash with an existing ID will be replaced - this means I can safely run the command multiple times.

I can even combine the two commands like this:

paginate-json 'https://fedi.simonwillison.net/api/v1/timelines/public?local=1' --nl \
  | sqlite-utils insert /tmp/simon.db posts - --pk id --nl --replace

The result is a SQLite database!

Here it is in Datasette Lite as a demo.

Elon Musk laid off about half of Twitter this morning. There are many terrible stories emerging about how this went down, but one that particularly struck me was that he laid off the entire accessibility team. For me this feels like a microcosm of the whole situation. Twitter's priorities are no longer even remotely aligned with my own.

I've been using Twitter since November 2006 - wow, that's 16 years! I've accumulated 42,804 followers there. It's been really good to me, and I've invested a lot of work generating content there to feed the machine.

I can't see myself putting the same work in to help the world's (current) richest man pay the billion dollar annual interest on the loans he took out to buy the place on a weird narcissistic whim.

So I've started to explore Mastodon - and so far it's exceeding all of my expectations.

My new profile is at https://fedi.simonwillison.net/@simon - you can follow @simon@simonwillison.net in your Mastodon client of choice.

Not ready to sign up for Mastodon? It turns out RSS support is baked in too - you can subscribe to https://fedi.simonwillison.net/@simon.rss in your feed reader (I really like NetNewsWire for macOS and iOS these days).

Why Mastodon?

The lesson I have learned from Twitter is that, even if a service you trust makes it past an IPO and becomes a public company, there's always a risk that it can be bought by someone who very much doesn't share your values.

Mastodon has been designed to avoid this from the start. It operates as a federated network of independent servers, each of which is run by a different person or organization with the ability to set their own rules and standards.

You can also host your own instance on your own domain.

My initial nudge to try this out was from Jacob and Andrew, who figured out how to do exactly that:

• The Fediverse, And Custom Domains - Andrew Godwin
• Setting up a personal Fediverse ID / Mastodon instance - Jacob Kaplan-Moss

Andrew and Jacob both opted to pay masto.host to run their instance for them. I've decided to do the same. It's on my domain, which means if I ever want to run it myself I can do so without any visible disruption.

I'm paying $9/month. I find it darkly amusing that this is a dollar more than Elon has been planning to charge for users to keep their verified status on Twitter!

If you don't want to use your own domain there are plenty of good free options, though I recommend reading Ash Furrow's post about his shutdown of mastodon.technology to help understand how much of a commitment it is for the admins who run a free instance.

This post by @klillington@mastodon.ie has some good links for getting started understanding the system. I particularly enjoyed Nikodemus’ Guide to Mastodon as it matched most closely the questions I had at first.

Initial impressions

Despite taking the second hardest route to joining Mastodon (the hardest route is spinning up a new server from scratch) it took me just less than an hour to get started. I wrote up a TIL describing what I did - more or less directly following the steps described by Andrew and Jacob.

I signed into my new account and started following people, by pasting in their full Mastodon names (mine is @simon@simonwillison.net). I was initially surprised that this did nothing: your timeline won't be populated until the people you follow have said something.

And then people started to toot, and my timeline slowly kicked into life.

And it was really, really pleasant.

My fear was that everyone on Mastodon would spend all of their time talking about Mastodon - especially given the current news. And sure, there's some of that. (I'm obviously guilty here.)

But there's lots of stuff that isn't that. The 500 character limit gives people a bit more space, and replies work much like they do on Twitter. I followed a bunch of people, replied to a few things, posted some pelican photos and it all worked pretty much exactly as I hoped it would.

It's also attracting very much the kind of people I want to hang out with. Mastodon is, unsurprisingly, entirely populated by nerds. But the variety of nerds is highly pleasing to me.

I've been checking in on the #introduction hashtag and I'm seeing artists, academics, writers, historians. It's not just programmers. The variety of interest areas on Twitter is the thing I'll miss most about it, so seeing that start to become true on Mastodon too is a huge relief.

Considering how complicated a federated network is, the fact that it's this smooth to use is really impressive. It helps that they've had six years to iron out the wrinkles - the network seems to be coping with the massive influx of new users over the past few days really well.

I'm also appreciating how much thought has been put into the design of the system. Quote tweeting isn't supported, for reasons explained by Eugen Rochko in this 2018 post:

Another feature that has been requested almost since the start, and which I keep rejecting is quoting messages. Coming back to my disclaimer, of course it’s impossible to prevent people from sharing screenshots or linking to public resources, but quoting messages is immediately actionable. It makes it a lot easier for people to immediately engage with the quoted content… and it usually doesn’t lead to anything good. When people use quotes to reply to other people, conversations become performative power plays. “Heed, my followers, how I dunk on this fool!” When you use the reply function, your message is broadcast only to people who happen to follow you both. It means one person’s follower count doesn’t play a massive role in the conversation. A quote, on the other hand, very often invites the followers to join in on the conversation, and whoever has got more of them ends up having the upper hand and massively stressing out the other person.

Mastodon so far feels much more chilled out than Twitter. I get the impression this is by design. When there's no profit motive to "maximize engagement" you can design features to optimize for a different set of goals.

And there's an API

Unsurprisingly, Mastodon has a powerful API. It's necessary for the system itself to work - those toots aren't going to federate themselves!

Poking around with it is really fun.

First, a friendly note. @pamela@bsd.network wrote the following:

Hacky folks, please resist finding ways to scrape the fediverse, build archives, automate tools and connect to people via bot without their consent.

[...]

Whatever your thing is, make it 100% opt-in. Make it appropriate for a significantly more at-risk user than you are. Make sure it forgets things, purges info about servers it can't contact, can't operate in any sort of logged-in mode where consent is an issue.

We will straight up help advertise your cool thing if it respects users properly and takes the time to consider the safety and preferences of every person involved. There are a lot of fun, thoughtfully-designed toys! And there are a lot of people really tired of having to come and tell you off when you wanted to help, honestly. Help yourself and ask around before you flip on your cool new thing, let folks point out what you're missing.

(Read the whole thing, it's great.)

So far I've done a couple of things.

I built a Git scraper to track the list of peer instances that various servers have picked up. This feels like a reasonable piece of public information to track, and it's a fun way to get a feel for how the network is growing.

I also figured out how to Export a Mastodon timeline to SQLite using the timelines API and my paginate-json and sqlite-utils CLI tools, so I could explore it in Datasette.

Running my own instance means I have no ethical qualms at all about hammering away at my own API endpoint as fast as I like!

I like to follow a lot of different people, and I don't like to feel committed to reading everything that crosses my timeline - so I expect that the feature I'll miss most from Twitter will be the algorithmic timeline! This is very much not in the spirit of Mastodon, which is firmly committed to a reverse chronological sort order.

But with access to the raw data I can start experimenting with alternative timeline solutions myself.

I'm somewhat intrigued by the idea of iterating on my own algorithmic timeline, to try and keep the variety of content high while hopefully ensuring I'm most likely to catch the highlights (whatever that means.)

Past experience building recommendation systems has taught me that one of the smartest seeming things you can do is pick the top 100 most interesting looking things based on very loose criteria and then apply random.shuffle() to produce a final feed!

I have a hunch that this is going to be a lot of fun.

The decades long quagmire of encapsulated HTML

This is part of a series of posts I'm writing about creating an HTML Document format: 

• We need an HTML Document standard
• The Hypertext HTML Document Editor
• My simple blog generator: Hypertext Blogmaker

So my first thought when I started down the path of creating Hypertext HTML Document Editor was that I wanted an "all in one" file format which would include the CSS, images and other assets that a web page usually needs. I figured, "How hard can it be to throw together a zip file with an index page and a folder of files that an editor can read?" Silly me! It turns out that during the past 25 years or so, lots and lots of other developers and organizations have thought the same thing. Yet apparently it's one of those things which seems to never get any sort of traction.

After taking a look at what's been done already in order to not recreate the wheel, I came to realize a few things: First, I decided that nothing that's been done so far would have the simplicity nor widespread support that I was looking for. Secondly, trying to create Yet Another Bundled Web File Format was a really bad idea. But most importantly, since my goal was to replace Markdown as a simple, easily manageable rich text document format, trying to also tackle the bundling issue was besides the point. Markdown doesn't include files, doesn't care about them and will never have them. There's no reason for an HTML Document standard to deal with encapsulated files either, as essential as it might first seem. 

All that said, I'm sure someone, somewhere might wonder why my Hypertext editor doesn't have some sort of option to save as a zipped file format of some sort, I thought I'd share what I discovered reading up on the topic. Then maybe someone can get the W3C, Mozilla, Google and Apple to finally make up their minds and decide on a standard that we can use in the future. (I won't hold my breath.

So, let's take a look at some of what's out there. But first, a look back in time.

Back to the future... .htmld 

You won't be surprised to find out that way back in 1995, ideas for the web encapsulation problem had already been batted around, and some had actually been implemented. I found a message from someone named Dan Grillo at NeXT to a CalTech email list about the .htmld file format used on NeXT servers. (See the original here.): 

From: Dan Grillo <Dan_Grillo@NeXT.COM>
Subject: What's in a .htmld or .htmd?
To: webstep@xent.caltech.edu, khare@CALTECH.EDU

> This one is a raging open question, but probably the easiest settled
> Generally, we want such a format to be self-contained, so the intention is that
> the wrapper can include other files, directories, symlinks; and that such
> structure should be preserved by cooperating applications (eText, for example,
> garbage collects). Here are a few of the design choices:
>
> .htmd (HyperText Markup Document) vs .htmld
> index.html vs. _____ (TXT.rtf, multiple topics, a la sanguish)
> multi-format representations (index-ascii, index-mono, index-color)

I'll continue with this one.

First, .htmld is already registered with NeXT's type registry;
I did it a long time ago. I don't think .htmd is.

What should be in a .htm[l]d? Right know I know of 3 forms.

1. foo.htmld/index.html
2. foo.htmld/TXT.html
3. foo.htmld/foo.html

I think forms 1 & 3 are useful.

Pages, StepWise, and NeXTanswers all currently serve files saved as .htmld
Right now NeXTanswers uses form #3, so the internal .html can be FTP'ed
or saved from a web browser and used on it's own. This is hard to
do if they all are index.html.

--Dan

Of course the thread goes on with a bunch of questions about the format, and as we know, it disappeared from existence. It just goes to show... 

MIME: .mhtml 

In 1999 the MIME encapsulated HTML format was standardized. Here's the Wikipedia description: 

MHTML, an initialism of "MIME encapsulation of aggregate HTML documents", is a web page archive format used to combine, in a single computer file, the HTML code and its companion resources (such as images, audio and video files) that are represented by external hyperlinks in the web page's HTML code.

The content of an MHTML file is encoded using the same techniques that were first developed for HTML email messages, using the MIME content type multipart/related.

MHTML files use a .mhtml or .mht filename extension. [Also commonly .eml as well. -R]

This is still used for emails formatted in HTML and you can still export pages from Chromium browsers ("Save As -> Web Page, complete") where a page is saved literally as a fake email, complete with a "<Saved By Blink>" in the from: header. (That special from: tag actually turns on some restrictions in Chromium, so it's not just a random entry, without it the browser apparently goes into a less secure Internet Explorer mode.) I had never actually looked closely at that page before, so I was pretty surprised to see what a hack it is, and yet it's the way Chrome still works 20+ years after the format was defined, with little change. 

In terms of using it as a basis for a cross platform document format, it's not supported by Firefox any more, nor Safari. Even though it's all nominally text, it's not a particularly user-friendly format, using as it does a mess of MIME-encoded HTML text with tons of embedded MIME-encoded data chunks. Though it is nice that the page has everything it needs, and is separated into sections (embedding a data:// url is much more disruptive in terms of content flow), it's not compressed in any way, and not something you'd want to store an .mp4 video in and send it to anyone. 

Amusingly, on my Mac, the .mhtml file format is registered to Microsoft Word, not the default browser. If I had Outlook installed, it would probably claim first dibs. Opening the file in Word ignores all the embedded files - for security reasons I presume - making it a bit less than useful in that app. Opening an .mhtml file in Chrome also has a bunch of odd restrictions as well. And examining the page in DevTools doesn't really expose all the magic that's happening - like the process of decoding the file attachments nor the explicit security issues. It's very weird.

A Bugzilla entry from 23 years ago(!), showed some enthusiasm for using .mhtml in the then upcoming Mozilla Editor, though I'm not sure if that ever happened. Just shows again how long this topic has been floating out there. Apparently, XUL even had an <editor> tag. 

Side note: MIME Types vs. File Extensions

Slightly off topic - but since I'm already talking about MIME. Linux is the only OS that actually pays attention to a text-file's MIME type, like "text/html", etc. Windows and macOS both depend on the file extension to decide what type of file it is. Thus, if you save an HTML5 file as .xhtml it will be treated as XML file - with the strict XML validation rules - regardless of the DOCTYPE or other meta tags in the header. If you name it as a .mhtml file, it will add extra security to the page, preventing it from opening images and other links from external sources. Using MIME-encoded HTML, though possible and even considered as a solution at one point to encapsulated HTML, really isn't really a good solution today. 

Mozilla's MAFF

From Paolo Amadini's website : 

MAFF  files are standard ZIP files containing one or more web pages, images, or other downloadable content. Additional metadata, like the original page address, is saved along with the content. Unlike the related MHTML format, MAFF is compressed and particularly suited for large media files.

Support for reading and writing MAFF archives was provided in the Mozilla Application Suite, Firefox, and SeaMonkey thanks to the Mozilla Archive Format add-on from 2004 to 2018. While the original add-on is no longer maintained, the file format specification  is still available and can be referenced by third-party software to provide better interoperability.

Seems pretty simple! Too bad it's dead.

Checking out the file format spec, you can immediately see it's a product of its 2000's era conception: The meta-data file which specifies which files are included in the zipped bundle has to be .rdf: An oddball XML spec which doesn't make any sense to anyone not really into the "semantic web", though it's not particularly horribly done in this instance. Paolo's site has a few examples, and it seems pretty simple. My best guess for why it died is browser security concerns about pulling in some random zip file into the browser. 

A MAFF spec v2 could easily solve some of its shortcomings by using a JSON file as the meta, requiring a CSP header on the index page to prevent malicious scripts from running, and adding in a signature of some sort to guarantee that the file matches what the manifest says, similar to signed JAR files. But I guess that would be a totally different spec.

"Web Page, complete"

Nowadays, Firefox doesn't bother with a single-file format at all, not even MHTML. Saving a page as "webpage, complete" simply creates a folder with all the page's assets and it rewrites the URLs to point at it, just like Chromium does using the same save as option. (By the way, is it "Webpage" or "Web Page"? Chrome uses the first, Firefox uses the latter.) I find both browsers defaults to save the folder with the same name as the web page's title, complete with spaces, incredibly annoying, as it ends up having to put %20 everywhere in the URLs, or worse (Chrome) just including the spaces in the URLs, making what would be a relatively clean export into a complete mess. Also, rather than separating out the various media types - css, images, icons, etc., the browsers just dump them all into a single folder. 

Apple's .webarchive

Of course, Apple has to be "different", so Safari doesn't have an option to download a web page and all the assets into a separate folder (in other words, "complete"), instead it only has the option to save in .webarchive format, which (according to Wikipedia, again) is, " a concatenation of source files with filenames saved in the binary "p-l ist" format using NSKeyedArchiver". If you ever want to see what's inside, you can convert the binary p-list into an text plist by using the command  plutil -convert xml1 foo.webarchive -o foo.plist   and you'll see that Apple simply just saved all the page's assets in a bog-standard Mac XML property file. 

This format is supported by macOS Safari as well as iOS, though I can't imagine many people actually know about it, let alone use it much. There doesn't seem to be any projects out there that open or create a .webarchive using JavaScript, though there are some written in Python, Go and Ruby. I can't imagine it would be too hard to figure out, but then only Safari has support for reading the format natively, which isn't the idea. 

Apparently, in addition to being Apple-only, .webarchive files aren't particularly safe to share. The way it stores scripts allows it execute in the context of the opener. It's basically as simple as MAFF above, but done using Macs/iOS in mind. 

While I'm here, I'll express my long held belief that whoever came up with Apple's property list format didn't really understand how XML and/or SGML-like tags actually worked. Does it make sense to you that p-lists have stuff like <key>WebResourceData</key> instead of simply just <WebResourceData> ? It's like they were confused.

The eBook File Format - EPUB

So the ePub format seems like it might be an ideal web document container, but it's got quite a few limitations, the biggest of which is that it requires XHTML, rather than HTML5. It's also focused quite specifically on eBooks, not on documents, which are sort of the same, given that the former is made up of the latter.

Using Apple's Pages word processing app, you can export your doc as an ePub, which is pretty cool. It's just a zip file, so you can open it up and see what sort of decisions they made to create it, and how they include CSS, images, etc. The problem, of course, is that it's a one-way export. One would think that since the ePub nominally contains all the parts that make up the document - and the possibility to include as much metadata that may be missing - that they might create some sort of "editable ePub", but sadly they didn't. 

A .pages document, just for those who are curious, is a zip file filled with some XML meta data and .iwa "iWork Archive" binary files which are compressed with "Zippy" - a Google originated compression format. So weird. 

Web ARChive - WARC

Just to be confusing, there's another format out there called "web archive" as well. Did you know that the Internet Archive uses its own HTML bundling format - ISO 28500:2017  - which is supported by the United States Library of Congress?? I had no idea...

A WARC format file is the concatenation of one or more WARC records. A WARC record consists of a record header followed by a record content block and two newlines; the header has mandatory named fields that document the date, type, and length of the record and support the convenient retrieval of each harvested resource (file). There are eight types of WARC record: 'warcinfo', 'response', 'resource', 'request', 'metadata', 'revisit', 'conversion', and 'continuation'. The content blocks in a WARC file may contain resources in any format; examples include the binary image or audiovisual files that may be embedded or linked to in HTML pages. 

It's not meant for an individual file as much as it's meant for entire websites. But in case you were ever wondering how archive.org stores snapshots of the entire Internet, now you know. Pretty cool.

Miscellaneous other formats

Compiled HTML - Microsoft created their proprietary .chm format in the 1990s for its Windows Help system to use. It's actually still supported in Windows 11, though the format hasn't been updated since the early 2000s. 

WACZ - Web Archive Collection Zipped - is used by the WebRecorder project, which seems to be an active effort to create an open standard for web archiving, though you wouldn't realize it by the design of their website. I almost thought it was also a dead 1990s effort until I saw the August 2022 update. 

SingleFile and SingleFileZ is an actively developed web extension for archiving web pages. It takes all the binary assets and converts them into data:// urls and then embeds them into the page automatically, so the end result is a (very large) plain-text .html file. Basically it's sort of like MIME, but using data urls. It works, but I wouldn't want to try to post-process the file. 

Current efforts

So I wrote all the above relatively quickly, as I had a familiarity with most of the stuff I listed, or it wasn't too hard to look up and get familiar with the details. Then over a week has gone by while I tried to wrap my head around what's going on with the latest efforts by the W3C Web Incubator Community Group (WICG) and browsers, specifically the Chrome team. I can't promise I still have a handle on it, so if you see something wrong or see something missing, please let me know. There's a Github repo for the W3C Web Package community group which has an overview of what they're trying to do. 

From what it looks like, it's a concerted effort to finally figure out a standard way to bundle up various bits of a web site or web app into a safe, secure single file archive for a few different use cases. The first is to save web pages, like the various solutions I wrote about above. Another is so that browser clients - especially on mobile - wouldn't have to request so many tiny files over the network. Another is that a standard archive could also be used to package web apps or whole web sites for installing or sharing offline. All well and good. However, concerns about privacy, safety and security has caused a lot of headaches, which they're trying to address in various ways. Here's their explainer. There was also an effort by the W3C Digital Publishing Interest Group to define the use cases for web publications. How all these groups inter-relate besides linking to each other in their docs, I have zero idea. 

Side Note: The Concise Binary Object Representation - CBOR

There is a new, open-spec archiving file format called CBOR. It seems to underpin the web packaging efforts currently going on by the browsers and W3C. CBOR is basically like Apple's binary p-lists, except that it uses JSON to organize all the meta data and compressed binary data contained within the file, which can be addressed and extracted individually without decompressing the whole file. Signing and Encryption has been built in (though covered by another spec called COSE), so it's apparently good solution for a variety of problem domains, including messaging and apps. 

Web Bundles - .wbn

So the latest proposed bundling format is called Web Bundles, which is still in draft stage and available behind a flag in Chrome. Though it nominally addresses the problems listed above, its announcement immediately got blowback because of the idea that web servers could potentially start to serve all their content using bundles. The problem being that if sites started only serving bundles, it would essentially break the web as we know it, because each of the individual files would be opaquely contained within the archive. This would mean that ad-blockers would stop working, and it could potentially have security and privacy issues as well, as who knows what sort of stuff could be contained inside the bundle. The guys at Brave were among many who lost their proverbial shit over the proposal. 

The one and only post from Google about it was back in 2019... so I think the pushback may have been enough to stall this spec for a bit. As of right now, none of the other browser makers seem interested. Also, despite being available via a flag in Chrome, I've yet to get any of the .wbn bundles provided in that post to work on my computer. 

This doesn't mean web bundles are dead and gone, however, as now it seems the focus has been changed from a general, all-purpose solution for bundling web assets, to being a way of bundling PWAs. This is definitely something that's needed, so I can't see it maybe getting a less harsh reception. Web Bundles may also contain "Isolated Web Apps" which are archived PWAs with strict CSP limitations which help with security. 

Summary

I think that's about all the relevant specs, formats, and proposals for web archiving out there. This whole post started as an attempt to learn about what sort of solutions were available which would be a "standard" that could be used for a self-contained HTML Document, including CSS and Images, mostly. Like I thought in the beginning, how hard could it be to throw a bunch of files into a zip and call it a day? Apparently, quite a lot. 

Sadly, not even the latest efforts really provide a solution, as they're focused on web apps and thus have to figure out how to make the bundles safe and secure, and as a result, aren't something that would be easily editable like an encapsulated document - ala .docx or .pages. The one format, ePub, might be a solution, but it's focus on books and its use of XHTML means that it's not suited at all for something easy and simple. 

My next post will be about a proposal for an HTML Document format that I think could be implemented in browsers within a reasonable time, and some ideas as to how that might happen. You can probably guess from the above: It will be text-only, so a MIME-style solution with embedded base64 blobs similar to SingleFile's solution, plus a strict Content Security Policy ruleset which disallows scripting for safety and security. In order for this to work, it would need to have its own file extension - like .mhtml or .xhtml - so that browsers could automatically add those restrictions to HTML Documents opened from the file system. 

I'll flesh it out in the next post. 

-Russ

Twitter and the rebirth of the long form weblog

Long form writing contained in blogs served as static web pages seems to be making a comeback. In addition to consumer sites like Substack, which are making it easy for people to build both a following and an income from their writing, the latest trend among techies is a simple text-oriented blog written on their computer in text, converted to HTML locally and posted to a static server. No backend blog software needed. It's a fantastic trend, of which I am a happy participant.

This week, Twitter started going through serious corporate drama, culminating in today's layoffs. I hate to be mean, but I'm all for it. Twitter - among other social media platforms - helped killed rational thought on the web. Not just the popularity of individual blogs, but the expectations of both the writer and reader of the web in general. When Twitter launched, it was described as a “micro-blog”, because that’s what it was. But quick posts about random subjects soon morphed into its own thing: A tweet.

We all know what a tweet is: A short polemic statement, usually emotional, biased or opinionated, but expressed as self-evident without any justification, explanation or thorough analysis. It became the norm for the web as we know it. Blogs became a thing of the past - almost a cliche. They were something to be embarrassed about. "I used to have a blog" was spoken in the same hushed tone of admission along with having a MySpace page. The blogs that did remain were written by die-hards, or more often by commercial content farms whose main focus was driving clicks or spreading disinformation under the guise of a "blog".

I was an avid blogger at the time Twitter started, with thousands of daily readers. I had every reason to continue. And yet after I and others started tweeting regularly, my blogging slowed and then stopped. The reason is pretty simple: When you blog regularly, there’s a certain itch to express an idea or an opinion that builds up and you want to scratch it by writing a post. But before you wrote, you’d ruminate on the idea, and as you wrote you expanded and clarified it, like I am now. The end result was usually a decently sized essay with both a thought and the rationale behind it.

But tweeting could scratch that itch instantly - no need to save up enough of an idea to write a full blog post of a few paragraphs, you could just write, “Wow, foo sucks. Totally prefer bar instead.” And the feeling was satisfied. I remember thinking back then that I'd save up some ideas that I had tweeted about and write a longer post later. But later never seemed to come. And then, when it did, I'd be so used to culling my opinion down to the essential idea for a pithy tweet, fully expressing a thought became harder. Besides, the more open-ended a tweet was, the more opinionated, the more it would get reactions. In fact, expressing fully formed ideas wasn’t worth as much.

Twitter infected the entire blogging world like this. It was a vicious cycle, you’d blog less, and others who used to blog their ideas or leave comments wrote less as well, and that great feedback loop stalled as everything moved to tweets. Intelligent analysis was replaced by emotional exclamations and hard opinion. There’s no room for nuance in a tweet.

This was pretty obvious, even back then - I blogged about it at the time - but by the time it became clear how bad tweets were for intelligent discourse, the damage had already been done. But worse, Twitter embraced their role in the chaos - taking years to increase the size of tweets even a little, and doing nothing to encourage longer, more well thought out ideas. The fact that even in 2022, I regularly see "tweet storms" of connected tweets rather than a simple single page post shows how little they want their users to think about anything. Controversy and chaos is good for their platform, and they have encouraged it throughout their existence, regardless of the effect on society as a whole. 

I never really tweeted all that much, but stopped for good in January 2016 in protest against the company for ignoring their own community guidelines and allowing a demagogue to use the platform to spread lies and disinformation. I didn't delete my account, but I did write a script to go through and delete all my old tweets. Later, I learned about the vast Russian disinformation campaign which created thousands of fake AI bots which engaged users in order sow discord among the Western democracies. And again, Twitter did nothing. 

So to say I have little sympathy for the company and their employees now that they have Musk in charge, doesn't really express my feelings about it. But it's not schadenfreude or even glee - it's just a feeling that maybe justice is being served. Though really, I doubt Twitter will collapse because of the new management, and all those laid off employees are just going to flood the job market making my own job search all that much harder. So really, it's just more pain all around. The world would truly be a better place if Twitter never existed, or if it ever had the right employees who could have admitted the negative effect it was having on public discourse and dialogue, and did the right thing from the beginning. But they didn't. And we're all paying the price for it. 

What I hope though is that today marks a crest of a wave for the age of tweets, and that the trend towards a more thought-out web, filled with complete thoughts and fully expressed ideas will again become the norm.

-Russ

Anyone saying “Mastodon should do X” should understand a couple of things.

1) There are 4 staff listed as working for the German non-profit, incorporated in 2021

2) ActivityPub-the-protocol is the bigger ecosystem, also without a “lead”

'Fossil activists' is a very useful framing.

Fossil activists destroying a 19th century Romanesque Revival church designed by Erasmus Schüller. . https://t.co/Ru6gjpCD03

— Open Climate Data (@openclimatedata) October 27, 2022

If twitter is going to fall apart as a place for recommendation/discovery we're all going to have to work harder at pointing out the good stuff. So here are a couple of newsletters well worth some of your subscribing calories:

Converge Guy Moorhouse on design and stuff he likes, he has such great taste. There's bound to be stuff in here you don't know about but will love.

Checksies Anna and Rod on "Ways to get started and keep going on your money journey. Savings, pensions and other money whatnot." Complex things made simple. But not too simple.

Feminist Friday "A manageable number of links (2-3), about or around women and feminism, every Friday. Leans towards culture and history. Usually light on commentary." Always fascinating stuff from Alex.

ustwoeuropenewsletter (is that what it's called?) Nicki writes from ustwoland. Always interesting and properly opinionated, like a leader should be.

If you don't actually want to read this whole post (no shame at all) the tl;dr is that I'm going to try using a GitHub project to organize and hold myself accountable for open source maintenance.

Mental health is a weird thing, sometimes.

Like, oh I don't know, everyone, I've struggled over the past couple of years. One thing that has fallen by the wayside has been open source maintenance. (I think my GitHub profile still has a status I set in March 2020 about not committing to much open source work.) For a long time, the very last thing I wanted to do was think about code outside of work.

But in the meantime, a lot of things have changed. (We are still piling on collective trauma after collective trauma. That hasn't.) I'm in a different kind of role (warning: linkedin, I never wrote a blog post about this) at work, which has allowed me a lot more capacity to dig into, well, myself. I've shifted a lot of my focus in therapy, started and changed meds, have tried to invest in me-outside-of-work in a way I never really have before, and even took some time off (I am eternally grateful for Dave and the leadership at Policygenius for supporting me in doing that) to try to put myself on a more solid mental foundation.

All of that internal work is murky and messy and complicated. I like to think these investments have put me on an ultimately better path, but it certainly hasn't been a fun one all of the time. Especially recently.

And so, recently, I have found a certain comfort in code, again. A calmness in holding technical systems in my head.

To be totally honest, I was a bad maintainer well before the pandemic started. Part of it was my work role—I was a manager and then director for a long time—leaving me drained. Part of it was that my open source projects are nearly all Python and for Django, and I haven't worked in Python day-to-day in 5 or 6 years, or in Django for longer than that. Some of it is guilt, like the shame spiral of not answering an email that only gets worse over time. And some of it is focus, which has been particularly difficult this year. (I intended to dig into this almost 4 hours ago, but I did find a couple of nice floor lamp options for a reading spot!)

But I also don't want to do some giant, unsustainable burst of open source activity and then wait another 4 years to do it again. This work is still important to me, these projects are still important to me, and even though I'm not a particularly active member lately, these communities are still important to me.

So, I'm going to follow a spark of inspiration from Jeff, and create something of a roadmap for myself—and for you to follow along. (Except, I live in NYC, and have happily not owned a car in over a decade, so, a Subway Map, I guess?)

For now, it's going to live as a GitHub Project, and my first priority is to get caught up, i.e.:

• Make sure the active repositories have decent CI and branch protection rules.
• Drop Python 2 support from everything.
• Test against the most recent Python 3 versions (even if it's failing).
• Test against modern Django (even if it's failing) for the Django-based projects.
• Assess the open issues and PRs and decide what goes into the next major release.

After that, I'll probably try to work through those releases in some order. Probably smallest-first to get something shipped.

I don't know exactly how long this will take across the 4 big projects (pystatsd, django-adminplus, django-jsonview, and django-ratelimit). I want to make sure I'm setting myself up to be more consistent going forward. I think organizing this in one open place will help.

I just watched this video about using the Map View plugin in Obsidian, which allows you to visualise the existence of notes grouped by geo-location on a map. The Map View plugin allows viewing and organising notes along a dimension that is currently missing in my notes, geographic location. I do have an overview of all my travel since the late eighties in my notes. The creator of the video, Zsolt is the creator of both the Obisidian Excalidraw and Brain plug-ins, which I both enjoy using and recommend. So when he suggests a plugin by someone else, it piques my curiosity.

Now I am of course thinking about integrating that with my check-in forms for this site enabling individualised Plazes. Specifically it may play a role in determining venue or location.

Would it be doable to, like I’m already posting from my notes to the site, automatically create an optional check-in record here when I create a geolocated note in Obsidian? Or the other way around, to have a check-in made through the webform also create a note in Obsidian?

Well, that was a “fun” week. I was originally planning to clean up another of my long-standing drafts and post about one of my projects, but in between a small trip and the chaos at Twitter, I thought I’d hop on the bandwagon and write about that as well.

Elon’s… “emotionally challenged” approach at Twitter layoffs couldn’t have come at a worse time. Besides all the legal and personal drama that is just starting to play out, it capped off a week (or two) of similar announcements from Lyft, Stripe, OpenDoor and a few others, not to mention the teeth-grinding drop of tech stocks when Meta and others’ tried to tap dance through their earnings calls a little while back.

Being a regular reader of The Economist for a few decades now, I have been drip-fed enough skepticism and awareness that everything based on advertising (or in turning people into the product, one way or another) had been skating on thin ice, so most of my shock was due to the domino-like toppling (and my MSFT stock going along for the ride, even though both my employer and Apple have shown decent results–that’s just market stupidity at work).

Also, there is a war on and the portions of the tech industry that, you know, do actual tech have finally woken up to the fact that TSMC and most of the world’s silicon and precision manufacturing aren’t evenly distributed, although China is relatively quiet–for now. And investors, even though they love playing roulette, only really do so when they know how the game is rigged, so they’re moving their funds to less risky things like security companies and lots of unobtrusive bits of the military manufacturing supply chain.

So localized collapses of social networking and upstart services that were trying to flip the table on payments and transportation are… well, somewhat predictable and pretty reasonable outcomes, although it sure as heck sucks for the actual people involved.

And most people in the tech industry (including myself) are completely and utterly exhausted from the weirdly off-center washing and rinsing cycle the thing’s apparently set to, what with the pandemic, the mad dash for growth that came both before and (surprisingly) during it, and, of course, all the political nonsense that simmered alongside.

Elon, the trickster of our age, is just providing a new trigger for a lot of pent-up frustration. One that quite a few people seem to be acting on–partially because moving to another social network is something they can actually control (which is a welcome change in their lives), and partially because of the “grass is greener” phenomenon.

And at least in this case, you might actually be able to see some grass instead of ads.

I don’t know a lot about the internals of Twitter (technical or otherwise), but it doesn’t seem overly complex to grasp. Jack Dorsey went on at length about their goals even as annual reports painted a pretty factual picture of their risk factors, though:

Our business operations are subject to numerous risks and uncertainties, including those outside of our control, that could cause our business,
financial condition or operating results to be harmed, including risks regarding the following:

Business and Operational Factors

    * our ability to increase our mDAU (monthly Daily Active Users), ad engagement or other general engagement on our platform;
    * the loss of advertising revenue;
    * competition for people to use our platform and for content and data partners;
    * competition for advertising spend;
    * our prioritization of the long-term health of our service;
    * our prioritization of product innovation;
    * our ability to maintain and promote our brand;
    * our ability to hire, retain and motivate highly skilled personnel;
    * the interoperability of our products and services across third-party services and systems;
    * the impact of spam and fake accounts on our platform experience;
    * actual or perceived security breaches or incidents, as well as errors, vulnerabilities or defects in our software and in products of third-party
    providers;
    * our international operations;
    * the impact of the COVID-19 pandemic and responsive measures;
    * our significant past operating losses and any inability to maintain profitability or accurately predict fluctuations in the future;
    * our reliance on assumptions and estimates to calculate certain key metrics;
    * catastrophic events and interruptions by man-made problems

Maybe that last bullet is where Elon fits in. Not sure yet.

More to the point, I do have friends there, and yes, some got sacked. So I guess there goes one of their goals:

A company-wide three-year objective focused on diversity and decentralization

And then there’s the technology, the security, and the politics.

Like everyone else I watched as armchair critics dissected its policies, its internal security stance, the dramas around not banning very stable geniuses that would have been unthinkable in Europe, etc.

I acknowledge there is a certain amount of impedance mismatch between my viewpoint and many of the people pouring a river of pixels onto the topic, but considering that back in the early 2000s a few colleagues of mine were hacking SMS gateways to post to Twitter and it was mostly a nerd-and-burning-man kind of thing, it having become a 7000+ FTE company with global reach is just… insane.

That said, painting it as “the world’s town square” is disingenuous. There are many people who either can’t use it or just don’t care about it in the least, and who are perfectly happy on Meta estates or (increasingly) TikTok. But it did take off amazingly well in some countries (not so much in mine, apparently, although our politicians seem to have realized it exists during the pandemic), and thus has, well, a cross-sectional reach worldwide.

But to techies (politically minded or not, inside or outside the US) it has been a pretty great place (if you’re lucky), and even if you discount all the crypto and web3 idiocy that still runs rampant there, it is a great place for people to connect, keep track of news (especially industry news) and hawk their wares (which is pretty essential for occupations like game development, where indie projects can garner significant following).

And yes, of course you have all the TV shows, media, and mainstream news, and whatnot. But those would always have the means to either pay for mainstream advertising (if it worked–that’s another point coming…)

The thing about Twitter isn’t just about major brands placing sponsored ads in your timeline–that’s the archaic way of thought (doubling down on a captive audience) that harks back to TV commercials and newspaper ads. And yes, of course their withdrawing explicit advertising will hurt Twitter’s bottom line.

The real pain is likely to come from loss of organic reach stemming from people being alienated by knee-jerk platform monetization strategies (yeah, I still can’t believe Elon was haggling with Stephen King about the $8 fee either), losing interest in it due to lack of moderation or political bias, or leaving it altogether.

That is going to hurt Twitter. And let’s face it, the US ambivalence about “free speech” is also going to be put to the test. As an European, the notion of having to share a timeline with extremist jerks who tossed out random passive-aggressive tweets while holding office (and who likely encouraged rebellion after) doesn’t appeal to me.

As to the exodus itself, I honestly don’t think it will amount to much. I’ve been on Mastodon for a while (comparatively not much, but enough to know the difference), and of course I’ve gone back to it again this week and am trying all the cool new apps (right now MetaText seems to be the nicest for me), but what I’m doing right now is looking for the same people I followed on Twitter to make sure I keep enjoying their posts, which just means I’m carrying over my bubble¹.

But that isn’t something most Twitter users will bother with. And the Fediverse’s take on “following” and reach isn’t palatable to heavy Twitter users who thrive on reach and the flatness of its landscape, let alone the corporations who leveraged Twitter for marketing purposes. Content-wise, to anyone landing on Mastodon out of the blue the only thing that seems about the same is the amount of cat pictures, but that might not be enough unless, like many migrants, you’ve been driven there because you just wanted out due to principle, i.e., your current perception of Twitter and what you fear Elon will change.

So the exodus will continue until morale improves, but mostly as far as the exiles are concerned. Many will come back, or post on multiple networks, or just do what normal people do and use different forums for different things.

Corporate-wise, and although it is impossible to predict what kind of learning journey Elon is actually embarking upon (he might be perceived as ruthless and out of touch, but he gets things done, even if they’re what he wants and not what most people wish he did), I think Twitter is likely to carry on ploughing itself into the ground–following the asymptotic curve of the digital advertising market that no matter how many opinionated people continue to eschew, actually still works for social networks.

Until, of course, it doesn’t. But that’s yet to come.

It’s been great to see so many people flood into the Fediverse over the last week or so. It’s mainly been people fleeing Twitter and signing up for Mastodon accounts at mastodon.social or mastodon.online, which are both instances run by Eugen Rochko, the founder, CEO and lead developer of Mastodon. He said earlier today that, since October 27th, there have been almost 200,000 new sign-ups, which is huge.

While it’s fantastic news that people are experimenting with Mastodon as an alternative to Twitter, there’s two concerns here:

1. Newcomers default to the two ‘official’ instances — this leads to increased pressure on the server, which is run by volunteers and donations rather than Big Tech. Also, the experience in terms of the local timeline is vastly different on a server with tens or hundreds of thousands of users, compared to tens or hundreds of people.
2. Mastodon is not the Fediverse — the thing that makes the Fediverse is the underlying standard on which is it is based. Mastodon implements this standard, ActivityPub, in its own particular way, and with additional technologies and approaches. The problem with one platform becoming much larger than others is that it gets to dictate the way that standards are implemented.

I had already set up a Pixelfed instance to run exercise.cafe for people interested in sharing and discussing exercise and fitness-related activities. Yesterday, I set up a Misskey instance at wao.wtf for testing purposes. I’m paying for it myself and haven’t actually got the go-ahead from other WAO members to be running it, to be perfectly honest. It may stay up for days, weeks, months, or years. But if you’d like an account, let me know.

What makes the Fediverse unique, different, and resilient is lots of instances of different sizes (cities/towns/villages) running different software. Why not have a look to see if you can perhaps set up your own instance — it’s easy with managed hosting!

Musk is sinking the Twitter ship much faster than I expected. Lots of people like Mastodon as an alternative, and I don’t believe it’s quite there yet. But it’s a safe way to interact with decent people with a much nicer experience than on Twitter. I will explain how they get rid of the toxic discussions later.

I am not going to use Mastodon terminology but words you understand, like “server” instead of “instance”. I am also not going to explain the Fediverse. But you must know a major difference:

My name on Mastodon is not @vowe as it is on Twitter but rather @vowe@chaos.social – yes, like email addresses with an @ in front of it. It is a federated infrastructure of thousands of servers.

Step 1: Find your server and find your followers

People have started to add their Mastodon names on their Twitter profile, as either @vowe@chaos.social or https://chaos.social/@vowe – you can click this now.

There is a tool that lets you scan all your followers and lists on Twitter. We will use this to find the best server for you. https://fedifinder.glitch.me/ You need to authorize it to read your Twitter data. It’s open source. Check if you don’t trust.

If found 293 friends out of 6843 accounts. It shows them organized by server and it will let you know if the server has open registrations. You download the list as .csv and keep it for later.

Now you pick a server. I recommend a local one where you live (for local content) or a server with likeminded people. Go to that server and register. Take a note of your new profile address and add it to your Twitter profile so that others can find you.

Log into the server and check the browser UI. You can enable an expert mode that shows you multiple columns.

2. Start following people

Remember the .csv file from the last step. Go to Settings and upload the file. Boom. You just requested to follow all people you followed on Twitter that have made their address available.

Mastodon is not exactly like Twitter but you can just start using it like it was. Once you get your feet wet, you can look at two other feeds, next to your Home feed which contains all your follows. “Local timeline” shows you everything going on on your server, and “Federated timeline” shows you everything available to you.

3. Don’t cancel Twitter

Keep your Twitter account. You can still find your old follows once they add their Mastodon names. You also want to leave your Mastodon name on your Twitter profile so that they can find you.

4. Why Mastodon is less toxic

You can report bad people and your local admin might ban them. And you can report bad servers (racism, conspiracy, Covid deniers, holocaust deniers etc.) and your server might stop federating with them. Here is a list of the servers my server blocks.

It is with great pleasure that I am announcing the launch of our new contribute page in SUMO a.k.a SUpport.Mozilla.Org. SUMO is one of the oldest contribution areas in Mozilla, and we want to show you just how easy it is to contribute!

There are many ways you can get involved with SUMO, so getting started can be confusing. However, our new contribute page should help with that, since the pages are now updated with simpler steps to follow and a refreshed design.

We also added two new contribution areas, so now we have five ways to contribute:

1. Answer questions in the support forum
2. Write help articles
3. Localize help articles
4. Provide support on social media channels (newly added)
5. Respond to mobile store reviews (newly added)

The first 3 areas are nothing new for SUMO contributors. You can contribute by replying to forum posts, writing help articles (or Knowledge Base articles as we call them here), or translating the help article’s content to your respective locales.

Providing support on social media channels is also nothing new to SUMO. But with the ease of tools that we have now, we are able to invite more contributors to the program. In 2020, we started the @FirefoxSupport account on Twitter and as of now, we have posted 4115 tweets and gained 3336 followers. If you’re a social media enthusiast, the Social Support program is a perfect contribution area for you.

Responding to user reviews on the mobile store is something relatively new that we started a couple of years ago to support the Firefox for Android transition from Fennec to Fenix. We realize that the mobile ecosystem is a different territory with different behavior. We wanted to make sure that we serve people where they need us the most, which means providing support for those who leave us app reviews. If this sounds more like your thing, you should definitely join the Mobile Store Support program.

And if you still can’t decide, you can always start by saying hi to us in our Matrix room or contributor forums.

Keep on rocking the helpful web,

Kiki

"Everyone knows that an AI revolution is coming,"writes Tyler Cowen in a paywalled Bloomberg link, "but no one seems to realize how profoundly it will change their day-to-day life.... This view is likely to be proven wrong - and soon, as AI is about to revolutionize our entire information architecture. You will have to learn how to use the internet all over again." There are a few more items in this post, all on the theme of how AI is rapidly becoming more capable. "Large language models are not yet as good at responding to my prompts as the readers of my blog. But their capacity is improving fast and the prices are dropping."

OK, I don't think that being a clown itself can improve teaching (certainly not being the clown pictured for this article) but there is an element of truth to Gustavo Espinoza Ramos's observation, though it has nothing to do really with being funny or telling stories. Here's how I would draw out the lesson, following Mike Bullard's school of comedy:

1. Find something you have in common with your audience
2. Bring them around to your point of view
3. Make them laugh at themselves

Now in teaching, obviously, the point isn't to make people laugh at themselves, but through the same process they can learn about themselves, and this maybe just maybe might make them want to do something about it.