Rolandt

Datasette 0.45, out today, features magic parameters for canned queries, a log out feature, improved plugin documentation and four new plugin hooks.

As I did for Datasette 0.44, I'm going to provide an annotated version of the full release notes here on my blog.

Magic parameters for canned queries

Canned queries now support Magic parameters, which can be used to insert or select automatically generated values. For example:

insert into logs
  (user_id, timestamp)
values
  (:_actor_id, :_now_datetime_utc)

This inserts the currently authenticated actor ID and the current datetime. (#842)

This is a fun new feature that extends the capabilities of writable canned queries, introduced in Datasette 0.44.

The key idea here is to make it easy to insert contextual information such as the current timestamp, the authenticated actor or other automatically generated values as part of a writable query.

This means Datasette's canned queries are now powerful enough to build things like simple comment systems or logging endpoints purely by defining a SQL query with the right magic parameters.

There's even a :_random_chars_32 parameter that automatically generates a random text string - useful for things like generating authentication tokens for use with datasette-auth-tokens. More on this below.

Log out

The ds_actor cookie can be used by plugins (or by Datasette's --root mechanism) to authenticate users. The new /-/logout page provides a way to clear that cookie.

A "Log out" button now shows in the global navigation provided the user is authenticated using the ds_actor cookie. (#840)

Out of the box, Datasette's authentication system is quite primitive: the only way to get an authenticated session is to use the --root option to get a special link when the server first starts running. As described in the documentation, the goal is for plugins to fill in the rest.

Even with just that mechanism it still makes sense to let people log out again! The new /-/logout page can do that, and Datasette's navigation now includes a log out button if the user is logged in using that ds_actor cookie.

You can see what this looks like in Datasette's pattern portfolio.

New plugin hooks

register_magic_parameters(datasette) can be used to define new types of magic canned query parameters.

I'm increasingly trying to have Datasette internally use plugin hooks for default behaviour. This hook can define custom magic parameters - you can see the implementation of the default parameters using this hook in default_magic_parameters.py.

startup(datasette) can run custom code when Datasette first starts up. datasette-init is a new plugin that uses this hook to create database tables and views on startup if they have not yet been created. (#834)

Here's an example datasette-init plugin configuration in metadata.yaml. This will create a dogs table when the server starts, but only if one has not yet been created:

plugins:
  datasette-init:
    my_database:
      tables:
        dogs:
          columns:
            id: integer
            name: text
            age: integer
            weight: float
          pk: id

canned_queries(datasette, database, actor) lets plugins provide additional canned queries beyond those defined in Datasette's metadata. See datasette-saved-queries for an example of this hook in action. (#852)

This started out as a feature request from Amjith Ramanujam on Twitter.

Canned queries, like these ones, are usually defined in the increasingly poorly-named metadata.json/yaml.

Letting plugins define them opens up some neat possibilities. datasette-saved-queries is an interesting example: it lets users store new queries in their database, inserting them using a writable canned query that the plugin itself returns from that hook by default.

Here's the code. It also uses the new startup() hook to create its own table.

forbidden(datasette, request, message) is a hook for customizing how Datasette responds to 403 forbidden errors. (#812)

I need this for the next version of datasette-auth-github - it's a way to customize what happens when a user fails a permission check.

Even more plugins

Thanks to the datasette-plugin cookiecutter template I can turn out simple plugins in just a few minutes. Here are my new releases from the past week:

• datasette-init, described above.
• datasette-write provides a /-/write page that can be used to directly execute write SQL queries against a selected database.
• datasette-allow-permissions-debug which is absolutely tiny. All it does is listen for permissions-debug checks and return True for them. This means you can access the /-/permissions debug page on your Datasette instance without authenticating first, which is handy for debugging.
• datasette-glitch is designed for use with Glitch. It outputs a magic one-time use URL to the private Glitch console which you can use to authenticate with your Datasette instance there as the root user.
• I also released a new version of datasette-auth-tokens, which allows users to configure API tokens to be used to access a private Datasette instance. It now lets you store tokens in a database table based on a configured SQL query. The :_random_chars_32 magic parameter mentioned above can be used to help create new tokens.

What's next?

I've already slipped one feature into the Datasette 0.46 milestone, but my focus from here on should really be on getting everything in place for Datasette 1.0.

Crappy Cheapo Architecture @CheapoCrappy
Apparently "Edmonton and Toronto" is trending, here are two ugly buildings from Edmonton and Toronto. pic.twitter.com/zQhEWBNf3n

Hannah McGregor @hkpmcgregor
Reminder: “Canada is three mining companies in a trench coat, wearing a stupid hat and carrying a gun.” theoutline.com/post/8686/cana…

Remembertron @remembertron
Remember eating at a restuarant?

Adrianna Tan @skinnylatte
Didn’t have time to make the telur pindang, opor ayam and tempeh but oh god I love all types of coconut rice

Adrianna Tan @skinnylatte
Nasi and sambal is all I need.

Three Geese Radius @shawnmicallef
Long live Toronto. Long live its people. Long live this lake. pic.twitter.com/uKG3L1eYPq

The New Yorker @NewYorker
Curling, the metric system, coins called "loonies" and "toonies," and other reasons to praise our neighbor to the n… twitter.com/i/web/status/1…

Hogtown 101 @Hogtown101
I sometimes forget how far south Toronto is compared to the rest of Canada - and to the U.S. Compared to the States… twitter.com/i/web/status/1…

Packed into this sign, posted at the head of a trail at Sally’s Beach in eastern PEI, is a potent set of only-in-Canada assumptions: that anyone will read such a wordy sign, that anyone has the vaguest notion of what 1.83 meters apart represents, that we require the paternalistic advice of the state to guide what are ultimately common sense decisions. 

Oliver was distressed this morning that almost all of the usual Canada Day routines had been erased from the schedule, either due Catherine’s death or COVID-19. So there was to be no music in Victoria Park, no Goat Parade in Rustico, no seafood at the Blue Mussel Café, no fireworks. While I might have preferred that his distress had been expressed with less ferocity, I had sympathy for its roots, and resolved that we should carve some new July 1 traditions out of the darkness.

So here’s what we did:

1. Drove to Souris. It was a beautiful day: sunny with a touch of mist, not too cold, not too hot.
2. Had lunch at The Poké Shack on the beach. Excellent.
3. Charged our Kia Soul EV at the new fast charger Irving on Main Street.
4. Takeout cold brew coffee and a smoothie at Evergreen Café. Excellent.
5. A walk on Sally’s Beach on Spry Point.
6. Hot fudge sundaes at Gillis’ Drive-in in Montague.
7. A visit with our old friends Wennie and Kenny in downtown Montague.
8. Takeout fish & chips from Water-Prince Corner Shop.

We will finish up in a few minutes with something called “virtual fireworks” on the CBC, the nature of which I have no idea.

We missed Catherine–we talked about her many times over the day–and Rustico and the goats and the rest. But we managed to enjoy the day as best we could, and, at the very least, ended on a higher note than we started.

So yeah. Don't ride on the sidewalk you numbnut.

For the second year in a row, Apple held a developers conference that should frighten its competitors. Relying on a nearly maniacal obsession with the user experience, Apple is removing oxygen from every market that it plays in. At the same time, the tech landscape is riddled with increasingly bad bets, indifference, and a lack of vision. Apple is pulling away from the competition to a degree that we haven’t ever seen before. Given how we are just now entering the wearables era, implications of this shift will be measured in the coming decades, not years.

WWDC 2020

It speaks volumes that Apple held its strongest WWDC in years during the middle of a pandemic while two of its largest competitors, Google and Facebook, decided to skip their annual developers conferences. Just a few years ago, fortunes were reversed. Apple was coming under fire for WWDCs that appeared to be more reactionary to Google, Facebook, and Samsung. Apple was also struggling to contain growing unrest among its pro users who were tempted by Microsoft Surface hardware. 

What changed?

The last two WWDCs stood out for two reasons: 

A revised Apple product strategy. A few years ago, Apple was most aggressive with products capable of making technology more relevant and personal (iPhone and Apple Watch). As shown in Exhibit 1, in the pull strategy, the Apple Watch and iPhone were Apple’s clear priorities while the iPad, Mac portables, and Mac desktops ended up facing a battle for management attention as if they were located at the end of the rope that was Apple management was pulling.

Apple changed from a “pull” strategy in which some products like the iPad and Mac seemed to be having a hard time keeping up to a push strategy characterized by every major product category moving forward simultaneously. This shift appears to have been born in 2017, which would explain why we are still seeing the initial fruit of the effort. The iPad and Mac product categories have benefited the most from this revised “push” product strategy with more frequent and noteworthy updates. 

Exhibit 1: Apple’s Changing Product Strategy

Apple has doubled down on its unique interpretation of innovation. During his opening remarks at the iPhone and Apple Watch event last September, Tim Cook said that Apple sells tools containing "[i]nnovations that enrich people's lives to help them learn, create, work, play, share, and stay healthy." Instead of defining innovation as either being first or doing something different, Apple looks at innovation as something that improves customers’ lives. A major consequence of this has been software and hardware releases that have prioritized feature quality over quantity. This year’s WWDC came in a full 20% shorter than previous keynotes. While having a digital format helped cut down on the timing due to quicker transitions, no clapping etc., there were also fewer new features announced. However, the features that were announced contained more significance when it comes to pushing the user experience forward. 

A Stronger Apple

Unfortunately for Apple competitors, the combination of a revised product strategy and unique definition of innovation didn’t just make for strong WWDC keynotes. Consumers are noticing and wanting what Apple is selling. Consider the following trends:  

• Apple hasn’t just held its own in the smartphone space but rather is continuing to take share from Android. Of all the smartphone manufacturers, Apple saw the largest sales share increase in the smartphone industry last quarter, and that was during a pandemic.  

• Apple is adding approximately 20 million new iPad users per year despite the iPad being 10 years old and already having an installed base exceeding 300 million users.

• Apple’s oldest major product category, the Mac, is adding 10 million new users per year. 

• Apple Watch and AirPods are quickly approaching 100 million user bases each.

• Apple users are paying for 518 million subscriptions across Apple’s platforms, which is up 126 million in just a year.  

All of the preceding items amount to an Apple ecosystem gaining momentum. A different way of highlighting Apple’s growing ecosystem over the past 10 years is to look at the number of people using at least one Apple device. As shown in Exhibit 2, Apple’s installed base recently surpassed a billion users. 

Exhibit 2: Apple Installed Base (Number of Users)

While new user growth rates have slowed, Apple is still bringing tens of millions of users into the fold. Due to Apple’s views regarding innovation and its focus on the user experience, once someone enters the Apple ecosystem, odds are good that customer will remain in the ecosystem. 

This is why one subtheme from last week’s WWDC keynote flew under the radar. (My complete WWDC 2020 review is available here for Above Avalon members.) It’s not just about Apple pushing multiple product categories forward at the same time. Instead, it’s about adding cohesiveness and commonality between product categories. Apple is making it easier for people to buy multiple Apple devices. As users move deeper into the Apple ecosystem, satisfaction and loyalty rates stand to go even higher. The end result is that Apple’s billion users aren’t just any billion users. Instead, they are a billion users less likely to use non-Apple devices and services going forward. For the competition, this is a highly concerning development. 

More worrying for competitors, Apple is still in the early stages of bringing its users deeper into the ecosystem. According to my estimate, approximately 50% of Apple users still own just one Apple device: an iPhone. This group serves as a prime market for products like the iPad, Apple Watch, AirPods, and various Apple services. In a few years, that percentage may decline to something more like 30%. Such a development will remove much of the remaining oxygen from the markets Apple plays in.

 Competition Is Weakening 

While Apple sails forward with a strengthening ecosystem made possible by a clear product vision and a functioning organizational structure that prioritizes design (i.e. the user experience), the competition is rudderless. 

Apple competitors have been striking out with one bad product bet after another. Few have long-term vision as to where computing is headed. Consider the following events, developments, and observations. By no means is this an inclusive list. 

• Samsung remains rudderless from a product vision perspective. With no clear direction as to where to go, the company aimlessly launches new products and features for no other reason than to say they are first. The strategy is no different than throwing things against the wall and hoping something sticks. Even worse, the products and features that Samsung is announcing aren’t even ready for public usage. 

• Google continues to prioritize technology over design. While new software features may seem compelling on paper, the lack of attention given to the user experience quickly becomes apparent. It has also become difficult to miss the growing enthusiasm gap between Android and iOS. On the hardware front, Google is struggling to match such efforts with its ambient computing future (which doesn’t make much sense to me). 

• Amazon’s massive bet on voice with Alexa and Echo was the wrong one. The stationary smart speaker space was a mirage. Amazon should have instead bet on wearables with voice as a user input. However, the company doesn’t have the corporate culture to excel with computers worn on the body. 

• Microsoft appears to be running into growing trouble with the consumer when it comes to Surface. What had been a genuine chance to rip into the iPad and Mac stronghold due to growing user unrest looks to have been successfully crushed by Apple. Microsoft Surface revenue is increasingly being driven by commercial clients (i.e. Microsoft is taking share from its OEMs rather than Apple).

• Facebook ended up placing the wrong social bet. Instead of going after our closest social network, Facebook evolved to offer a curated version of the web via the News Feed. The company’s pivot back to a privacy-focused social platform built around messaging emphasizes this wrong bet. A message sent through Apple’s Messages is a message not sent through a Facebook property. 

• Snap, the company considered to have the best odds of competing with Apple on AR, botched its first major foray into AR hardware with Spectacles. The company has backed itself in a corner by management’s refusal, and then failure, to appeal to older demographics. This will serve as a headwind for mass market AR successes. 

• Spotify was not able to prevent Apple Music from gaining critical mass despite Apple Music not having a free tier. The same is now taking place with Netflix, which is unable to stop new entrants into paid video streaming from gaining traction. This ends up diffusing near universal praise in the press for first movers. 

For an industry that was expected to put Apple in its place, that sure is a lot of fails, flops, and disappointments. When looking outside the U.S., the overall picture isn’t dramatically different. While some companies still have pockets of strength where Apple is not a major player, in geographies Apple is playing in, the company continues to see growing ecosystem momentum while the competition flounders. The number of paid subscriptions being run through Apple’s platform points to increased services and app adoption outside the U.S.

The never-ending tales of Apple being crushed by the local competition in China have been met with Apple seeing existing users move deeper into the ecosystem as measured by App Store, iPad, and wearables momentum. Huawei’s struggles in Europe appear to be benefiting Apple at the premium end of the market.

Changing Narrative

If there was still doubt about Apple’s momentum in the marketplace, one doesn’t need to look any further than the dramatic change in narrative facing Apple in the press. 

For years, Apple was positioned as one iPhone update away from implosion. Low market and sales share were paraded around as signs of an incompetent product strategy. Simply put, Apple was framed as being weak and vulnerable, dependent on revenue sources that could disappear overnight due to consumers fleeing to the competition. 

The narrative has completely shifted. The press is now infatuated with Apple’s power, its ironclad grip over the App Store, and the idea that Apple users are stuck or imprisoned in a massive walled garden where things like iMessage, Apple Watches, and AirPods force people to remain within Apple’s walls. Government regulators are viewed as the only entity capable of protecting Apple users from Apple.

If competitors actually believe this narrative, they are setting themselves for more failure. Thinking that Apple users are somehow being forced against their will to buy products like Apple Watches and AirPods is nothing more than looking for someone to blame for market failures when the problem is found internally with a bad vision, inadequate corporate culture, and lack of understanding as to what makes Apple unique. 

Risks

On a list of risk factors facing Apple, greater regulation is far from the top. The same can be said about things like App Store policies and employee retention. While these items make for juicy headlines capable of grabbing people’s attention, they won’t play a major role in Apple’s future. Instead, Apple is where it is today by saying “no” more than “yes.” By remaining focused on making technology more personal, which is inherently about using a design-led culture to push the user experience, Apple is able to develop a dynamic, yet nimble, ecosystem of tools that people are willing to pay for. lf it were to lose focus, Apple would move that much closer to its competitors. 

Apple ends up being its toughest competitor as it releases products that surpass the previous version. This is where betting on the user experience and taking a unique stance on innovation is critical. 

Next Ten Years

When the iPhone was unveiled in 2007, Steve Jobs claimed that Apple had a five-year head start against the competition. He ended up being mostly right. By 2012, Samsung and Google were shipping credible iPhone alternatives, thanks partially to ruthless copying that led to time in the courtroom.

With wearables, my thinking has been that Apple has a lead that is closer to 10 years. This estimate reflects not just software or hardware advantages, but also the byproduct of Apple controlling both items and its resulting achievements with custom silicon. 

As time passes, Apple has been facing less competition in wearables. This is remarkable considering how Apple Watch has already ushered in the next paradigm shift in computing. We are seeing the future today. Yet most companies either don’t see it or even worse, see it but are unable to respond. 

Giving Apple a 10-year head start against the competition with wearables may end up giving too much credit to the competition. Excelling in wearables requires a corporate culture, product development process, and business model that few companies other than Apple possess. In many ways, Apple was built to excel in wearables. Apple should probably get used to being its own toughest competitor.  

Listen to the corresponding Above Avalon podcast episode for this article here.

Receive my analysis and perspective on Apple throughout the week via exclusive daily updates (2-3 stories per day, 10-12 stories per week). Available to Above Avalon members. To sign up and for more information on membership, visit the membership page.

Doug Russell, who used to work on accessibility at Apple, writes:

some of the code that powers accessibility on apple platforms is just disgusting to look at and to work on.

most of the code that makes apple software accessible lives in what’s called an accessibility bundle. without diving into the minutia of the thing, bundles are a way to load something akin to a plugin into a cocoa app at runtime if an assistive technology is activated. it involves manipulating the app or framework class hierarchy and using objective-c dynamism to read app state and build up a usable accessibility hierarchy. insert a super class here, read an instance variable there, swizzle in a method and store the state for it in associated objects.

In other words — Objective-C and its runtime play a big role in making Apple’s great accessibility possible.

What happens when that’s not really a thing anymore?

Josh Bersin, Jul 02, 2020

It's definitely worth watching what Microsoft is doing in this space. If you add it all together, it's a considerable investment. "The combination of LinkedIn Learning, GitHub, Microsoft Learn, and Microsoft certifications are a massive investment in this market. In this announcement, the company offers to reduce prices, provide LinkedIn learning paths for free, donate $20 Million to upskilling programs, and new GitHub offerings. The goal is to 'upskill' 25 million people by the end of the year."

Web: [Direct Link] [This Post]

Back in May I wrote a little command-line utility called Topfew (GitHub). It was fun to write, and faster than the shell incantation it replaced. Dirkjan Ochtman dropped in a comment noting that he’d written Topfew along the same lines but in in Rust (GitHub) and that it was 2.86 times faster; at GitHub, the README now says that with a few optimizations it’s now 6.7x faster. I found this puzzling and annoying so I did some optimization too, encountering surprises along the way. You already know whether you’re the kind of person who wants to read the rest of this.

Reminder: What topfew does is replace the sort | uniq -c | sort -rn | head pipeline that you use to do things like find the most popular API call or API caller by plowing through a logfile.

Initial numbers

I ran these tests against a sample of the ongoing Apache access_log, around 2.2GB containing 8,699,657 lines. In each case, I looked at field number 7, which for most entries gives you the URL being fetched. Each number is the average of three runs following a warm-up run. (The variation between runs was very minor.)

	Elapsed	User	System	vs Rust
Rust	4.96	4.45	0.50	1.0
ls/uniq/sort	142.80	149.66	1.29	28.80
topfew 1.0	27.20	28.31	0.54	5.49

Yep, the Rust code (Surprise #1) measures over five times faster than the Go. In this task, the computation is trivial and the whole thing should be pretty well 100% IO-limited. Granted, the Mac I’m running this on has 32G of RAM so after the first run the code is almost certainly coming straight out of RAM cache.

But still, the data access should dominate the compute. I’ve heard plenty of talk about how fast Rust code runs, so I’d be unsurprised if its user CPU was smaller. But elapsed time!?

What Dirkjan did

I glanced over his code. Gripe: Compared to other modern languages, I find Rust suffers from a significant readability gap, and for me, that’s a big deal. But I understand the appeal of memory-safety and performance.

Anyhow, Dirkjan, with some help from others, had added a few optimizations:

1. Changed the regex from \s+ to [ \t].

2. The algorithm has a simple hash-map of keys to occurrence counts. I’d originally done this as just string-to-number. He made it string-to-pointer-to-number, so after you find it in the hash table, you don’t have to update the hash-map, you just increment the pointed-to value.

3. Broke the file up into segments and read them in separate threads, in an attempt to parallelize the I/O.

I’d already thought of #3 based my intuition that this task is I/O-bound, but not #2, even though it’s obvious.

What I did

In his comment, Dirkjan pointed out correctly that I’d built my binary “naively with ‘go build’”. So I went looking for how to make builds that are super-optimized for production and (Surprise #2) there aren’t any. (There’s a way to turn optimizations off to get more deterministic behavior.) Now that I think about it I guess this is good; make all the optimizations safe and just do them, don’t make the developer ask for them.

(Alternatively, maybe there are techniques for optimizing the build, but some combination of poor documentation or me not being smart enough led to me not finding them.)

Next, I tried switching in the simpler regex, as in Dirkjan’s optimization #1.

	Elapsed	User	System	vs Rust
Rust	4.96	4.45	0.50	1.0
ls/uniq/sort	142.80	149.66	1.29	28.80
topfew 1.0	27.20	28.31	0.54	5.49
“[ \\t]”	25.03	26.18	0.53	5.05

That’s a little better. Hmm, now I’m feeling regex anxiety.

Breakthrough

At this point I fat-fingered one of the runs to select on the first rather than the seventh field and that really made topfew run a lot faster. Which strengthened my growing suspicion that I was spending a lot of my time selecting the field out of the record.

At this point I googled “golang regexp slow” and got lots of hits; there is indeed a widely expressed opinion that Go’s regexp implementation is far from the fastest. Your attitude to this probably depends on your attitude toward using regular expressions at all, particularly in performance-critical code.

So I tossed out the regexp and hand-wrangled a primitive brute-force field finder. Which by the way runs on byte-array slices rather than strings, thus skipping at least one conversion step. The code ain’t pretty but passed the unit tests and look what happened!

	Elapsed	User	System	vs Rust
Rust	4.96	4.45	0.50	1.0
ls/uniq/sort	142.80	149.66	1.29	28.80
topfew 1.0	27.20	28.31	0.54	5.49
“[ \\t]”	25.03	26.18	0.53	5.05
no regexp	4.23	4.83	0.66	0.85

There are a few things to note here. Most obviously (Surprise #3), the Go implementation now has less elapsed time. So yeah, the regexp performance was sufficiently bad to make this process CPU-limited.

Less obviously, in the Rust implementation the user CPU time is less than the elapsed; user+system CPU are almost identical to elapsed, all of which suggests it’s really I/O limited. Whereas in all the different Go permutations, the user CPU exceeds the elapsed. So there’s some concurrency happening in there somewhere even though my code was all single-threaded.

I’m wondering if Go’s sequential file-reading code is doing multi-threaded buffer-shuffling voodoo. It seems highly likely, since that could explain the Go implementation’s smaller elapsed time on what seems like an I/O-limited problem.

[Update: Dirkjan, after reading this, also introduced regex avoidance, but reports that it didn’t appear to speed up the program. Interesting.]

Non-breakthroughs

At this point I was pretty happy, but not ready to give up, so I implemented Dirkjan’s optimization #2 - storing pointers to counts to allow increments without hash-table updates.

	Elapsed	User	System	vs Rust
Rust	4.96	4.45	0.50	1.0
ls/uniq/sort	142.80	149.66	1.29	28.80
topfew 1.0	27.20	28.31	0.54	5.49
“[ \\t]”	25.03	26.18	0.53	5.05
no regexp	4.23	4.83	0.66	0.85
hash pointers	4.16	5.10	0.65	0.84

Um well that wasn’t (Surprise #4) what I expected. Avoiding almost nine million hash-table updates had almost no observable effect on latency, while slightly increasing user CPU. At one level, since there’s evidence the code is limited by I/O throughput, the lack of significant change in elapsed time shouldn’t be too surprising. The increase in user CPU is though; possibly it’s just a measurement anomaly?

Well, I thought, if I’m I/O-limited in the filesystem, let’s be a little more modern and instead of reading the file, let’s just pull it into virtual memory with mmap(2). So it should be the VM paging code getting the data, and everyone knows that’s faster, right?

	Elapsed	User	System	vs Rust
Rust	4.96	4.45	0.50	1.0
ls/uniq/sort	142.80	149.66	1.29	28.80
topfew 1.0	27.20	28.31	0.54	5.49
“[ \\t]”	25.03	26.18	0.53	5.05
no regexp	4.23	4.83	0.66	0.85
hash pointers	4.16	5.10	0.65	0.84
mmap	7.17	8.33	0.62	1.45

So by my math, that’s (Surprise #5) 72% slower. I am actually surprised, because if the paging code just calls through to the filesystem, it ought to be a pretty thin layer and not slow things down that much. I have three hypotheses: First of all, Go’s runtime maybe does some sort of super-intelligent buffer wrangling to make the important special case of sequential filesystem I/O run fast. Second, the Go mmap library I picked more or less at random is not that great. Third, the underlying MacOS mmap(2) implementation might be the choke point. More than one of these could be true.

A future research task is to spin up a muscular EC2 instance and run a few of these comparisons there to see how a more typical server-side Linux box would fare.

Parallel?

Finally, I thought about Dirkjan’s parallel I/O implementation. But I decided not to go there, for two reasons. First of all, I didn’t think it would actually be real-world useful, because it requires having a file to seek around in, and most times I’m running this kind of a pipe I’ve got a grep or something in front of it to pull out the lines I’m interested in. Second, the Rust program that does this is already acting I/O limited, while the Go program that doesn’t is getting there faster. So it seems unlikely there’s much upside.

But hey, Go makes concurrency easy, so I refactored slightly. First, I wrapped a mutex around the hash-table access. Then I put the code that extracts the key and calls the counter in a goroutine, throwing a high-fanout high-TPS concurrency problem at the Go runtime without much prior thought. Here’s what happened.

	Elapsed	User	System	vs Rust
Rust	4.96	4.45	0.50	1.0
ls/uniq/sort	142.80	149.66	1.29	28.80
topfew 1.0	27.20	28.31	0.54	5.49
“[ \\t]”	25.03	26.18	0.53	5.05
no regexp	4.23	4.83	0.66	0.85
hash pointers	4.16	5.10	0.65	0.84
goroutines!	6.53	57.84	7.68	1.32

(Note: I turned the mmap off before I turned the goroutines on.)

I’m going to say Surprise #6 at this point even though this was a dumbass brute-force move that I probably wouldn’t consider doing in a normal state of mind. But wow, look at that user time; my Mac claims to be an 8-core machine but I was getting more than 8 times as many CPU-seconds as clock seconds. Uh huh. Pity those poor little mutexes.

I guess a saner approach would be to call runtime.NumCPU() and limit the concurrency to that value, probably by using channels. But once again, given that this smells I/O-limited, the pain and overhead of introducing parallel processing seem unlikely to be cost-effective.

Or, maybe I’m just too lazy.

Lessons

First of all, Rust is not magically and universally faster than Go. Second, using regular expressions can lead to nasty surprises (but we already knew that, didn’t we? DIDN’T WE?!) Third, efficient file handling matters.

Thanks to Dirkjan! I’ll be interested to see if someone else finds a way to push this further forward. And I’ll report back any Linux testing results.

Quotebacks have been mentioned in various corners of the IndieWeb a lot in the past few weeks. As it was launched as a Chrome plugin, I didn’t try it out (Chrome is an unpalatable ad delivery vehicle imo). Now however there is a Firefox Quotebacks plugin, Tom Critchlow announced.

As Tom says, Quotebacks are meant to reduce friction in quoting other blogs/sites/sources, and if that increases the number and length of distributed conversations I’m all for it.

I think of it as smoothing some friction for behaviors we’re interested in encouraging

Tom Critchlow on Twitter responding to https://twitter.com/bixtweets/status/1270754438567858177

How is it different from a block-quote? It isn’t actually, under the hood it is still a block-quote. It’s just styled differently, and the browser plugin makes it very easy to capture everything you need and paste it into your blog-editor. The quoteback you see above is a html block-quote in the source:

While reading you can select text and in Firefox press alt s, and the plugin will pop-up. It allows you to add / edit things, and then copy the html encoded quote to your clipboard, to paste into your blog editor.

I like the easy ‘quote, copy, paste’ flow and having it look nice. I do think that the styling, which mimicks how e.g. Tweets are embedded in websites, may sometimes however actually break the flow of a blogpost, where a block-quote is more like a highlight in the pace and rhythm of a text, while a quoteback is presented as an embed, a different thing separate from the text. In fact I mostly actively dislike the embedded tweets in e.g. ‘news’ articles. There it feels like a way of not having to write an actual article or story, resulting in ‘news’ items along the basic template of “X said something, Twitter wasn’t having it” (With the article often stating the content of a tweet in its text, and then embedding the tweet below it, repeat 12 times. Voila, ‘journalism’ done.) It’s an additional visual amplification, easy on the eyes yes and instantly recognisable as a visual pointer to elsewhere, that probably isn’t always warranted, and may even reduce attention to the post the quote is used in. That would then decrease the level of distributed conversation, not increase it as intended.

Of course it is entirely possible to use the quoteback plugin, and not having the visual style of embedding applied. Below is the exact same quoteback as above, but with the class="quoteback" removed, reverting it back to a regular block-quotes (but keeping the link to the source and comments you may have added). Alternatively you can also delete the script element that provides the styling information for the quoteback. (I do exactly the same with Flickr embeds)

I think of it as smoothing some friction for behaviors we’re interested in encouraging

Tom Critchlow on Twitter responding to https://twitter.com/bixtweets/status/1270754438567858177

I’ll experiment for a while to see how it works for me in practice. I’ve put the script that styles the embed on my own domain, so I can also fiddle a bit with the styling if I want.

It feels like everything is falling to pieces outside, but remember that fundamentally, most people are actually nice to each other and when people aren’t nice to each other, it’s generally out of fear and hurt.

My friend Dan Hon, slinging wisdom in his newsletter today.

---

“out of fear and hurt” was originally published in stating the obvious on Medium, where people are continuing the conversation by highlighting and responding to this story.

This makes for fascinating reading. I was aware of some of the intricacies behind the connector from the very beginning of MFI, but it’s nice to see most of it in one place.

Ian Hanomansing @ianhanomansing
First visit to Granville Island Market in a long time. Lots of masks and lots of space. The slow return to some sor… twitter.com/i/web/status/1…

Of course it’s in direct conflict with FB’s business model but

social networks should reintroduce friction into their sharing mechanisms. Think of it as the digital equivalent of social distancing.

Leo Mirani https://www.theguardian.com/commentisfree/2020/jun/29/social-distancing-social-media-facebook-misinformation

makes a lot of sense otherwise. There’s no viable path of doing only content moderation or filtering. Another option is breaking monopolistic silos up by requiring open API’s for them to be seen as true platforms. That too will reduce amplification, as it puts the selection into the hands of a wider variety of clients built on top of such a true platform. Of course that too is anathema to their business model.

Hi folks,

Here I am again sharing with you the amazing experience of another All Hands.

This time no traveling was involved, and every meeting, coffee, and chat were left online.

Virtuality seems the focus of this 2020 and if on one side we strongly missed the possibility of being together with colleagues and contributors, on the other hand, we were grateful for the possibility of being able to connect.

Virtual All Hands has been running for a week, from the 15th of June to the 18th, and has been full of events and meetups.

As SUMO team we had three events running on Tuesday, Wednesday, and Thursday, along with the plenaries and Demos that were presented on Hubs. Floating in virtual reality space while experiencing and listening to new products and features that will be introduced in the second part of the year has been a super exciting experience and something really enjoyable.

Let’s talk about our schedule, shall we?

On Tuesday we run our Community update meeting in which we focussed around what happened in the last 6 months, the projects that we successfully completed, and the ones that we have left for the next half of the year.

We talked a lot about the community plan, and which are the next steps we need to take to complete everything and release the new onboarding experience before the end of the year.

We did not forget to mention everything that happened to the platform. The new responsive redesign and the ask-a-question flow have greatly changed the face of the support forum, and everything was implemented while the team was working on a solution for the spam flow we have been experiencing in the last month.

If you want to read more about this, here are some forum posts we wrote in the last few weeks you can go through regarding these topics:

• Responsive Redesign
• Community Strategy Project Update
• New AAQ flow

On Wednesday we focused on presenting the campaign for the Respond Tool. For those of you who don’t know what I am talking about, we shared some resources regarding the tool here. The campaign will run up until today, but we still need your intake on many aspects, so join us on the tool!

The main points we went through during the meeting were:

• Introduction about the tool and the announcement on the forum
• Updates on Mozilla Firefox Browser
• Update about the Respond Tool
• Demo (how to reply, moderate, or use canned response) – Teachable course
• Bugs. If you use the Respond Tool, please file bugs here
• German and Spanish speakers needed: we have a high volume of review in Spanish and German that need your help!

On Thursday we took care of Conversocial, the new tool that substitutes Buffer from now on. We have already some contributors joining us on the tool and we are really happy with everyone ‘s excitement in using the tool and finally having a full twitter account dedicated to SUMO. @firefoxsupport is here, please go, share and follow!

The agenda of the meeting was the following:

• Introduction about the tool
• Contributor roles
• Escalation process
• Demo on Conversocial
• @FirefoxSupport overview

If you were invited to the All Hands or you have NDA access you can access to the meetings at this link: https://onlinexperiences.com

Thank you for your participation and your enthusiasm as always, we are missing live interaction but we have the opportunity to use some great tools as well. We are happy that so many people could enjoy those opportunities and created such a nice environment during the few days of the All Hands.

See you really soon!

The SUMO Team

Rands recently wrote about his work-from-home routine. I love the idea of walking around a large wooded yard while doing audio meetings... One of his reasons for feeling so at ease struck a chord with me:

Everyone desperately wants to return to normality. I am a professional optimist, but we are not returning to normal. Ever. This is a different forever situation, and the sooner we realize that and start to plan accordingly, the sooner we will feel unstuck.

I have written or spoken a variation of this advice so many times over my fifteen years as department head, most often in the context of state funding and our university budget.

Almost every year for my first decade as head, we faced a flat or reduced budget, and every time several university colleagues expressed a desire to ride the storm out: make temporary changes to how we operate and wait for our budgets to return to normal. This was usually accompanied by a wistful desire that we could somehow persuade legislators of our deep, abiding value and thus convince them to allocate more dollars to the university or, failing that, that new legislators some future legislature would have different priorities.

Needless to say, the good old days never returned, and our budget remained on a downward slide that began in the late 1990s. This particular form of optimism was really avoidance of reality, and it led to many people living in a state of disappointment and discomfort for years. Fortunately, over the last five or ten years, most everyone has come to realize that what we have now is normal and has begun to plan accordingly. It is psychologically powerful to accept reality and begin acting with agency.

As for the changes brought on by the pandemic, I must admit that I am undecided about how much of what has changed over the last few months will be the normal way of the university going forward.

My department colleagues and I have been discussing how the need for separation among students in the classroom affects how we teach. Our campus doesn't have enough big rooms for everyone to move each class into a room with twice the capacity, so most of us are looking at ways to teach hybrid classes, with only half of our students in the classroom with us on any given day. This makes most of us sad and even a little depressed: how can we teach our courses as well as we always have in the past when new constraints don't allow us to do what we have optimized our teaching to do?

I have started thinking of the coming year in terms of hill climbing, an old idea from AI. After years of hard work and practice, most of us are at a local maximum in our teaching. The pandemic has disoriented us by dropping us at a random point in the environment. The downside of change in position is that we are no longer at our locally-optimal point for teaching our courses. The upside is that we get to search again under new conditions. Perhaps we can find a new local maximum, perhaps even one higher than our old max. If not, at least we have conducted a valuable experiment under trying conditions and can use what we learn going forward.

This analogy helps me approach my new course with more positive energy. A couple of my colleagues tell me it has helped them, too.

As many others have noted, the COVID-19 crisis has accelerated a few changes that were already taking place in our universities, in particular in the use of digital technology to engage students and to replace older processes. Of the other changes we've seen, some will certainly stick, but I'm not sure anyone really knows which ones. Part of the key to living with the uncertainty is not to tie ourselves too closely to what we did before.

A while ago now, I used to teach adults about energy efficiency. One such venture was concerned with buildings. The students were learning to qualify for a national program, based in the United States, and I travelled to Bellingham as a guest speaker. Nearly everyone else they heard would have been talking about building issues – insulation and so on. Hardware mostly, but also management. I talked about the bigger picture. How a building’s use and location was actually much more significant in terms of its greenhouse gas footprint than the energy used by its HVAC system.

The consulting company that ran the course gave me a nice little memento. The notebook, which was part of the kit given to the students. It is a monument to the principles that were being taught. The cover is made from recycled tyres. The paper, of course, was recycled too – every page has a pale grey logo printed on it.

The pages are all punched and the binding is by four small openable rings. Towards the end of the book there is page printed with the contact information of the maker. So that users can order a refill. Today I contacted them by email as, when I went to their web page, I could not find a refill that would fit this format.

Their reply. “Yes, this is a long discontinued item and we do not carry these refills anymore.
I am sorry about it.”

I won’t embarrass them by publishing their name. It is just a sad reality that business is business, and clearly this product, designed to be reusable for much longer than any one pad of paper might be, was not a commercial success. Which says more about us than them.

Small-m mirrors are a staple in practically every gym in America. But the big-M Mirror—a device that streams live and on-demand fitness classes from a companion app via a two-way video feed—is pushing the boundaries of connected fitness by putting a virtual instructor (and your own reflection) on a sleek screen meant for home use. Pre-pandemic, I spent 10 days with the camera- and mic-equipped device, during which I took multiple classes, got acquainted with the company’s offerings, and hosted seven curious volunteers of all fitness levels for Mirror trials at Wirecutter HQ in New York. We found that using the Mirror is fun, and that its thoughtful details and varied, high-quality programming set it apart from typical streaming scenarios. But although it is beautifully made and nearly seamless to operate, the Mirror costs around $2,200 for the first year, including the required monthly subscription. That said, if your intention is to replace your gym membership or pricey fitness-class habit with in-home workouts that suit your schedule and approximate the feel of a boutique studio, the Mirror may be worth your consideration.

By Scott

Bike builds can often revolve around hubs (yes, pun intended). For many bike owners, hubs are the center piece of their bikes. Over the years, I've picked up a couple hubs, either on a deal (like an old Sun Tour XC front hub I snagged for $10 I think) or as a part for a dream build. I bought an early 90's XT rear cassette hub (back in the early 90's) and I have put it aside for a special frame one day.


So let's say that magical day comes. You found some cool rims to go with the hub for your favorite frame. Now, just to order up some spokes and nipples and get started with building a wheel. But, my hubs are old and all the hub data that seems easy to find for a new hub isn't so easy to find. What to do?

First step is to determine what you need to know. Looking at a wheel building web page/spoke calculator is the first place to start. I found my rear hub there:
https://leonard.io/edd/hub/589-shimano-deore-xt-fh-m737-rear

If you're not as lucky and your hub isn't listed on a site, you're not out of luck. With a few measurements, you can get the raw data you need for any spoke calculator and be ready to go.

The main things you need to know for spoke length calculations are:

Hole Count - Your hubs and rims should match each other's hole count.

Lacing Pattern - 2 or 3-cross is the traditional lacing pattern. The number of crosses signifies how many times a spoke crosses other spokes. For other lacing patterns like Radial, 1-cross, and Crow's Foot, consult the manufacturers to confirm whether or not it's ok for that hub/rim.

Traditional 3-cross wheel lacing

Left and Right Centre to Flange Measurement - This is the distance from the center of the hub to the left (non-driveside) and right (driveside) flange. Here's how to measure these distances:

1. Figure out the distance between the lock nuts (where the hub sits in the dropouts) - 100 or 110 mm is typical for the front. Most rear wheels are 126, 130, 135, or 142mm (for thru-axles). This is commonly referred to as the OLD (over locknut dimension).
2. Take that number and divide by 2.
3. Then measure from the flange to the lock nut.
4. Subtract that number from step 2's figure (OLD/2) and you have the left/right centre to flange number. 

Where Your Caliper Is - Digital is great, but analog ones work fine as well. You can do it with a ruler too in a pinch.

It's a good idea to practice with a known, measured hub to make sure you're measuring and calculating things properly.



Hub Flange Diameter - The distance across the hub's flange from hole to hole. A caliper makes measuring this much easier.



Rim Offset - You likely won't need this unless you're building really wide (mid-fat or fat bike) or asymmetrical rims.

Once you have those details, you can put those into the spoke calculator and it will give you some spoke lengths. One thing about spoke calculators is that some will create a spoke length that is slightly longer, others slightly shorter than another calculator. Depending on the brand of spoke you want to use, the spoke length recommended by the calculator may be in between available spoke lengths. So the choice of spoke length is a compromise and one that the person building the wheel should make based on their experience.

If you want to get more in depth into the world of wheel building, you can find it in Jobst Brandt's book, the Bicycle Wheel.  Sheldon Brown's site has a great section about Jobst here. Wheel Fanatyk is another great wheel building site with a ton of info and great pictures. Ric at Wheel Fanatyk is one of the folks behind Wheelsmith and an absolute master wheel builder. Lots of stuff on his site to read through and try and absorb.

Wheel building is a fantastic skill to have. Not only can you build any sort of hub/spoke/rim combination to suit your particular build, it also opens up new bike build ideas that you may not have considered.

mkalus shared this story from Charlie's Diary.

---

On April 16—gosh, it feels like an entire year ago—I made some predictions about where we were going. How do they hold up?

1. Vaccine development will take a flat minimum of 12 months.

Too soon to tell; it's only been two and a half months. A few people have signed up for testing, but we won't know if the prototype vaccines have provided any protection for months yet. (I haven't seen any reports of them killing the guinea pigs though, so there's that.)

Manufacturing of vaccine adjuvants and the little extras required to turn a raw product into a deliverable medical treatment are in progress, though.

2. Lockdown can't be sustained more than 1-2 weeks after peak ICU occupancy passes, so it will be lifted in mid-May in the UK and possibly as early as May 1st in the USA.

I am really glad I got this wrong. Lockdown lasted until the end of May in the USA; it began being relaxed in England in mid-June, and here in Scotland non-essential shops are due to lift in July. The English government (that is, the Westminster parliament governing England: Scotland is run by the Holyrood parliament in respect of domestic affairs such as public health and education) tried to convince schools to re-open, but that mostly flopped; Scottish schools are, I believe, staying shut until the next academic year (the Scottish school year ends earlier than the English one by a couple of weeks).

Incidentally, Scotland is broadly coping better than England per this dashboard: there have been no new COVID19 cases confirmed in the past 4-5 days. However, Westminster's desire to dink with the statistics in order to justify a premature re-opening is making it hard to tell just WTF is going on.

3. Trump is shooting for May 1st because he's been told the economy will take 6 months to recover ... 1-4 weeks later there will be a secondary surge in infections and it'll follow the same exponential growth

Because the US lockdown didn't really begin to lift until June, the USA is hitting this secondary surge right now, and some states are trying to lock down again. Red states with Republican governors who are in complete denial are getting hit badly, though—notably Texas and Florida.

4. Extra Lulz in the UK

Boris clung on to power and is now visible again, busy pulling levers and waffling in public, Cthulhu help us. His main approach to COVID19 is to treat it as a public relations project, so he's going with patriotic rah-rah rhetoric rather than coolly rational choices. Because he's not a planner he delegates everything to Dominic Cummings, who is in the process of purging the top ranks of the civil service and replacing the current leadership with those willing to swear a loyalty oath to Brexit.

It'd be hilarious if I wasn't trapped in the back of the bus these clowns are driving towards a cliff.

5. Wildcards: we might conceivably find a simple and effective medical treatment.

Hydroxychloroquine is a bust (snake oil is about what you can expect from a snake oil salesman). Dexamethasone improving the prognosis for ICU patients is a welcome finding, but it's not a magic bullet—it just reduces the death toll by up to 30% among people who require mechanical ventillation. Trump buying up the global supply of remdesivir (even though it's not very effective) is exactly the kind of Milo Minderbinder act you'd expect of the guy. Prediction: it won't work, but it'll make Trump lots of enemies and a little bit of money.

---

What did I miss?

I missed: the rest of the world, where populist authoritarian leaders everywhere are trying to make Trump and Johnson's handling of COVID19 look magisterial and professional, from Brazil's Bolsonaro to Russia's Putin.

I totally didn't see Black Lives Matter turning into a global protest movement. To be fair, George Floyd was brutally murdered by the Minneapolis police department on May 25th, and the predictive blog entry I'm referring back to was posted more than a month earlier. (I've only posted once since then, prior to this, and that last blog entry was about brainstorming a fiction idea, not current affairs.) In retrospect I should have anticipated that heavy-handed racist policing of lockdown would lead to numerous flashpoints worldwide. There have been other side-effects: illegal raves and parties in the UK, for example. Widespread flouting of social distancing and/or masking guidelines, and the emergence of anti-mask rhetoric among Trump loyalists as a very disturbing kind of political statement.

There's a growing, seemingly global, sense that we can't or shouldn't go back to the status quo ante—in many ways, this sentiment ehoes earlier occasions when pressure for major social change emerged during and after world wars. The long term global economic effects, and possibly the death toll, are clearly in the same order of magnitude as a 20th century global war: it may be that such wars (or in this case a frightening pandemic) provide the trigger for the sort of societal change more normally associated with revolutions. (This certainly happened in 1917-19, and again in 1945-49.)

A new world is being born. I just hope I live to see it, and that there's room in it for someone like me to exist.

I moved to the UK on the 1st of October 2014. I picked that date because it's easy to remember, and the start of a quarter. That's why I picked today, the 1st of July 2020 to release our Museum in a Box post about changing phase. Basically, after a bumper first quarter this year, where the online shop we'd spent 2019 preparing was going great guns and we were looking at each other whispering is

It’s a secret to everyone! This post is for RSS subscribers only. Read more about RSS Club.

Today is the 7th anniversary of the death of Google Reader, I celebrate by sharing this post with you on this ancient channel of syndication which we both enjoy.

About two weeks into the pandemic I was stricken with a keep-you-up-at-night anxiety. The thought was three-fold: What if my wife and I got COVID and needed hospitalization, what happens to the kids? Do I hand contaminated children off to my 68 year old mother? And what if the outbreak gets so bad they’re out of available ICU beds and ventillators by the time I inevitably get sick?

That kept me up at night. As a parent, the pandemic presents some unique challenges. It’s not solely your mental health that you have to manage, it’s your kids’ and partner’s as well. And surprise, you teach school now too. Our house in the suburbs is now a prairie home in isolation. Last week a switch flipped in my head from “Working with the kids in the house is temporary” to “Working with the kids in the house is how it’s going to be for the foreseeable future.”

The situation has, to say the least, gotten worse. Texas’ Governor Greg Abbot and Lt. Governor Dan Patrick have chosen the “Kill ’em all and let God sort ‘em out” strategy to managing the pandemic. From the top-down, an unprecedented level of failure in leadership and consistent, science-based messaging. My bones crave one person of power saying “This is hard. We’ll get through this. Wear a mask. God bless.” but instead I sift through incompetence, hand-wringing, finger-pointing, and failure to address the human needs over business needs.

I attended two funerals last week. One remotely. One outside, socially distanced. Both healthy men. Both mysteriously, similarly, and unexpectedly taken. Not officially the virus, but both were going for a walk on a cool morning and fell… away… forever.

A new fear has struck me; what if the numbers get so bad that nothing is safe? Guaranteed exposure. They, the government, have decided inaction is the most cost-effective pathway forward and will prolong the outbreak until all of us, lungs and hearts, are eugenically sorted by this invisble killer.

These are the dire thoughts I scream as I ponder if my country is dead. Can the slowness of democracy provide any meaningful change or, as Cornell West puts it, “nourishment”? It looks as if the system can’t reform itself. If there is a silver lining, it’s clear to me that public education and universal healthcare are at least as valuable and necessary as the tens of trillions we pump into corporations every decade or so. Perhaps one day we will value human capital stock as much as the stock market.

mkalus shared this story from Uploads from Michael Kalus.

---

Michael Kalus posted a photo: