Rolandt

A few weeks ago I came across a Tweet from the FCC discussing how sim-swapping is fraudulent behavior.

SIM swapping is when someone hijacks your mobile number to gain access to your texts and calls for identity theft scams. If you have a sudden loss of service, contact your phone company and bank immediately. For more info, visit: https://t.co/QaL4moF95S #ProtectingConsumers

— The FCC (@FCC) August 26, 2021

It kind of baffled me because years ago, SIM Swapping was just that.... the swapping of sim cards from and to a device for the sole purpose of being able to use said device.  Prior to the BYOD or "Bring Your Own Device" era which I'm so glad that finally came to fruition, the act of sim-swapping was done in order to either use more than one device and still keep your mobile phone provider.  Travelers also engaged in SIM swapping in order to use the provider of the country that they are visiting, as long as their device was compatible.   This was the best way to not incur roaming charges.  As bloggers, we always had more than just one device being used at a time.  In order to test them with our providers we had to switch the sim card out and put it into the device we were going to use.  That was only done with GSM devices, since CDMA and TDMA devices were already programmed for their respective networks.

What I'm not understanding is how did the term go from a positive to a negative?  While I'm fully away of the seriousness of fraud, I don't fully agree with the FCC's definition nor meaning.  According to the FCC website, SIM-Swapping is explained as:

"Another way to perpetrate this scam is to physically steal the victim’s SIM card, a removable device in some mobile phones that carries a unique ID and stores the consumer’s personal data. The scammer can then use the stolen SIM card in their own mobile device.

In either case, the scammer can gain control over the victim's private texts and calls, and may then try to reset credentials for the victim's financial data and social media accounts.  If successful, the scammer can drain the victim’s bank accounts and sell or ransom their social media data."

Back in the day we swapped sim cards for the very purposes that I've explained.  But reading the information on the website still leaves me a bit baffled.  

• Firstly, according to the above, if one physically removes the victim's SIM card what are they replacing it with?  In order for it to be a "swap" another SIM card would be needed to replace the one removed, no?  
• Secondly, a SIM card is NOT a removable device.  A SIM card or Subscriber Identification Module card is a memory chip that assists in connecting you to your providers network.  It does also contain information about you as a customer, but you can control some of the information that is stored on your SIM card.  Depending on the device you have there are ways to select what is stored on the card and what is stored on the phone.  
• Lastly, the scammer can only use the stolen SIM card IF the victim has not put a SIM lock on it.  Because we always swapped SIM's for good reasons, we almost never saved or stored information such as Contacts/Address Books to our SIM's. Instead we stored them to the devices because before the cloud we stored all content to our computers.  I guess that became just the normal practice for me.  

Again, not to deflect from the severity of fraud in general I just think that maybe a better terminology should be used.  But like everything else... that's just my opinion.

This blog has been quiet, but that’s not because I have been slacking off. For the last year or so, I have been hard at work bringing a new vision to life. On this day of Thanksgiving, I’m grateful for the opportunity to work with a smart, focused and kind team at Tandem Vault to make something really great.

In many ways, The DAM Book 3.0 was a blueprint for the software we need, not necessarily what is currently available. As web-hosted visual media becomes a primary method of communication, we need a new platform to match. (And by “institution”, I mean everything from a family to a non-profit, small business, museum or corporation.)

All the pieces for the next generation are sitting there on the shelf. Of course the most important part has become utterly commonplace – behavior. We now use visual media and rich media for nearly every purpose, and it’s increasing daily.

We have built our new software, TV3, to be the Grand Central Station of rich media communication, with incoming and outgoing tracks, and easily operated switch gear. We’ve taken the features from marketing DAMs and adapted them to a much broader use.

I’ve been offering sneak peeks of the new software over that the Tandem Vault blog. These include videos that introduce some of the key features. Here’s a sample.

We’re going into a much larger beta phase in a week or so. If you are interested in participating, sign up here.

The post Introducing the next generation of DAM software. appeared first on The DAM Book.

One VPN technology I wanted to try out for quite some time now is Wireguard. Everybody I talk to about the topic speaks very highly of it but unfortunately, the stable version was just released a bit too late for proper NetworkManager support to be included in Ubuntu 20.04 last year. While I could live with a couple of scripts to establish and tear down a tunnel to my home network, that’s unfortunately not an option for others I work with. But then I remembered that OpenWrt, the open source Linux operating system for embedded network devices such as Wi-Fi access points, has support for the protocol. And as I just needed to update a Linksys WRT-1200 AC Wi-Fi Router to the latest OpenWrt version anyway, I used the opportunity to gain some experience with the protocol along the way.

As I don’t have a Wireguard server at home so far, I decided to use a public VPN provider that has good Wireguard support and a step by step ‘howto’ to get Wireguard working with the Wi-Fi router. And indeed, getting Wireguard working on OpenWrt is not that difficult.

In essence, one downloads and installs the Wireguard packages over the OpenWrt GUI and then creates a new Wireguard tunnel interface. There are only a few parameters that need to be configured for this interface, the main ones being the public and private Wireguard keys that were generated by the VPN provider and the IP addresses for the tunnel endpoints. I won’t go into the details here, as the VPN provider of your choice should have a ‘howto’ for this. Once done, the Wi-Fi access point has a new interface that can be selected as backhaul interface instead of the plain WAN connection.

To see how stable the tunnel is in practice I connected my devices to the Wi-Fi of the Linksys router and ran my Internet connectivity over the tunnel for two days. I didn’t experience a single failure, despite working the connection quite hard and transferring at least 50 GB over it. Also resetting the router to see if the tunnel would come up automatically worked as advertised. No manual intervention necessary and no leakage over the plain WAN connection after the reboot before the tunnel was up again. O.k., I’m convinced, this is the VPN protocol I will go forward with!

"The Enlightenment philosopher Anthony Ashley-Cooper was the first to use the word ‘politeness’ in the modern sense. He took a term associated with jewellery – with polished stones – and elevated it to a social virtue: ‘We polish one another, and rub off our Corners and rough Sides, by a sort of amicable collision.’"

From Conflicted by Ian Leslie

Somehow relates to Joyous Exploration (via Storythings)

This story initiall appeared on TNL.net and was written by Tristan Louis. Accept no substitute for the original.

Reflections on 9/11, 20 years later. For many, it's gone but for those of us who were there, the gap it left is still present.

The post <span class='p-name'>20</span> appeared first on TNL.net.

Looking back over the enthusiasms, arguments and panics around tech in the past few years, I sometimes think that Apple is the $2tn elephant in the corner, mostly silent and serenely indifferent to the news cycle. It doesn’t worry about the “metaverse”, content moderation or hacked elections, and newspaper companies haven’t worked out how to shake it down. It just ships.

Every year, with metronomic precision, it delivers another new set of hardware and software, and another set of technology building blocks that fit into a decade-long strategic plan. Never mind Apple in the 1990s — Microsoft in the 1990s could never manage this. Every year a whole new phone arrives, exactly on schedule, keeping or leading the pace for the entire industry, and then ships in the hundreds of millions of units, machined out of aluminium and stainless steel, at a 40 per cent gross margin. This is very hard. Apple pioneered PCs, and then lost the market and barely survived. But now it’s bigger than PCs — there are more iPhones and iPads in use today than all PCs combined, and they run chips created by Apple that are ahead of anything from Intel or Qualcomm. In Microsoft’s era everything in tech was a PC accessory, but now PCs are smartphone accessories, and Apple does smartphones very well.

Apple’s scale comes with a business model that sets it aside from many of the more difficult choices in tech. If you don’t have a search engine or a social network and messed up your attempt at an ad business, it’s easier to say you won’t try to work out what people are interested in. You’re also free to treat privacy as merely an engineering challenge, much like security or performance, and to sell it as a feature. That’s not to say that Apple isn’t sincere when it says that “privacy is a fundamental human right” — indeed, Tim Cook, despite his public persona as a bland supply-chain engineer whose secret pleasure is an extra energy bar, has used Apple’s voice for social causes far more than Steve Jobs ever did. But if you’ve spent a decade making Apple the privacy company, that makes it easier to sell a credit card, or (one day) a pair of glasses with built-in AI-powered cameras. Privacy is another building block. 

Apple’s sometimes rather pious public stance on privacy can raise hackles in parts of Silicon Valley, especially when those privacy features are not available to its Chinese customers (a choice it did have to make). But the fight in most minds this autumn is the App Store, where Apple insists on a 30 per cent fee for many kinds of content, and on controlling what apps you can install. There are two sides to this, both strong, but it seems clear that Apple will lose the 30 per cent, in some form. Last year that commission was perhaps $15bn (and 80-90 per cent games), but not all of that will go, and $15bn was only 5 per cent of Apple’s revenue. Meanwhile, Google probably paid Apple $10bn last year to be the default search engine on the iPhone. There’s a lawsuit here as well, from the US Department of Justice, but combined, that $25bn just happens to match Netflix’s entire business. Apple is a big company, and the legal challenges, so far, can look small. 

So, Tim Cook has built a machine to make more Apple. But the day that Steve Jobs died, people started asking if the Cook machine was enough, and if Apple could follow the iPhone with another world-changing product without its founding creative genius. The iPhones keep coming, with an unassailable lock on their part of the market, and Apple has spent a decade building cross-leveraged accessories and services that drive revenue and retention. And sure, AirPods are cool. But where’s the next Jesusphone? 

One answer is that Apple now has two very big projects — glasses and cars. A pair of glasses that become a display, and add things to the world that look real, seems worth trying — it might be the next universal device after smartphones, and Apple’s skills should put it at the forefront. But people have been trying for a long time: there are basic, unsolved optics problems, and we don’t know if Apple has the answer, or when it might. Apple Glasses might launch next week, or next summer, or never. Cars are a puzzle as well. Apple might design a better Tesla (and build one, with its $207bn of cash), but what problem would that solve? The iPhone wasn’t a better BlackBerry. Autonomous driving is a big problem, but is it an Apple kind of problem? Why would Apple solve a primary machine learning problem when Alphabet can’t?

Maybe these are the wrong questions. Maybe Apple can find and solve another huge problem (or fail to solve it, as happened with TV and a few other things). But meanwhile it will carry on making a certain kind of product for a certain kind of customer. That’s been the plan ever since the original Macintosh, and in some ways all that’s changed is how many more of those customers there are. The original Mac sold a few hundred thousand units in 1984, but Apple now sells half a million iPhones every day. Apple and the market grew into each other.

I was astounded to read a Twitter thread from a go-getter named John Plumstead, who harassed 58,408 people to get one or two deals. He can do this because there is no cost to harassing people. Let’s change that. Plumstead’s business is finding property owners in financial distress and buying their properties for less than … Continued

The post Annoying people at scale is profitable. It shouldn’t be. appeared first on without bullshit.

This is a quick trick for creating a cumulative chart of the total number of items created over time based just on their creation date.

Try it here

select
  created_at,
  (
    select
      count(*)
    from
      repos repos2
    where
      repos2.owner = 9599
      and repos2.created_at <= repos.created_at
  ) as cumulative
from
  repos
where
  "owner" = 9599
order by
  created_at desc

I imagine there's a more elegant way to do this using a window function but this works fine.

1. Please introduce yourself. Who are you? What do you do? How long have you been into photography?
My name is Regan Hinton. I fly for a U.S. based airline. I’ve been into photography for about 20 years, but I’ve only taken it serious for the past 10 years.

2. In one sentence, please describe what you captured in this shot.
An older gentleman was having a conversation with two young women in the hallway of a building in Havana.

3. Why did you select this photo to share?
I selected this photograph because it is one of the images I’ve captured over the years that I’m most fond of. I also selected it because it was the first time I was able to spot something and set my camera’s settings by eye rather than using a light meter.

4. What style of photography would you describe this as and do you typically take photographs in this style?
I’m not really sure what style I would describe it as. I guess documentary. I normally do photography to document my life and the environment I’m in at the time, so my photographs seem to take on this style.

5. When and where was this photo taken? 
This photo was taken in Havana, Cuba, about three or four years ago in [the month of] December.

6. Was anyone with you when you took this photo?
I was traveling with a good friend and fellow photographer.

7. What equipment (hardware and software) did you use?
The camera used was a Fuji GW690III with a roll of Ilford Delta 400. I developed the image at home in Kodak XTOL developer. I scanned the negative on an Epson V550 using Epson scan software.

8. What drew you to take this photo?
I walked away from my friend who was in the act of photographing something. I walked down a side street and just happened to glance in the open doorway of a large apartment building. I immediately stopped, did a quick guess of what my camera settings needed to be, set the camera to my desired aperture and shutter speed, focused, and took the photograph. I didn’t take time to compose the image the way I would have liked. The one young woman looked at me as soon as I took the photo so I knew a second photograph was out of the question.

9. How many attempts did it take to get this shot? How long did it take you to get one that you were satisfied with?
I made one attempt to take this photograph. I was satisfied after the first attempt.

10. Did you edit (or do any post-processing/production on) this photo?
The photo was edited in Epson Scan for the black and white point. I then did some dodging and burning on the three people as well as the steps in the background.

11. What encouraged you to share this photo online and with others?
I decided to share this image because it is my favorite image that I’ve taken so far. Sure, some would say it has flaws, but for me it was my first time trusting myself to decide on the aperture and shutter speed in order to capture a moment in which I knew I only had one chance. It was the moment when I finally felt like everything was coming together with my photography.

12. Did you learn anything in the process of taking, editing, or sharing this photo?
Yes. I learned that I was becoming less reliant on in camera or handheld light meters and that I didn’t need to take every photograph wide. I knew I wanted as much in focus in this image in order to capture the feel and the environment, but I also knew that given the dim light, I didn’t have much room to play with the shutter speeds before they would become too slow for a hand held shot.

13. Do you remember what you had for breakfast the day you took this photo?
For breakfast I had a croissant and coffee which had become our daily routine before heading out to shoot.

14. What would you like people to take away from this photo?
I want people to realize that in the same way you may have different experiences throughout your day such as a conversation in a hallway with a neighbor, there are billions of people around the world who are each having their own daily experiences. In a strange way, that makes me feel somewhat closer to humanity as I realize most of us are just going about our days and filling them with many different experiences which add to who we are.

15. Is there any feedback that you’d like to get on this shot?
This is one of the few images I’m not interested in getting feedback on. I guess I just want to be happy with it.

16. How can anyone reading this support your work?
You can support my work by visiting my website at Flying4Film.com, following my Flickr account, or following my Instagram account. I would also like to start showing my images in exhibitions or galleries so if you’re interested in any of my images, please feel free to contact me.

Kim Moran-Jones quilted temperature minima and maxima in the UK, along with Covid-19 deaths on the perimeter in grayscale. Data and the physical fit well together.

Tags: coronavirus, Kim Moran-Jones, physical, quilt

Although at times the content of this blog may come across as somewhat technical, as anyone who has looked at any of my code would tell you, I am not a developer (actually, you could interpret that phrase in a lot of ways!). This post represents a stream of consciousness about some of the stumbling blocks that I perceive as preventing me from getting started building my own extensions for JupyterLab.

See also this related JupyterLab issue: Getting Started Docs for Non-Developers.

The code I write is generally written to get things done, not to form part of some production application. It is a means to an end. It’s poorly structured, and eclectically commented. There’s no linting. My repo commits are random collections of files with often vacuous commit messages. You would not want me committing code to your code base.

I typically categorise my code outputs into various classes:

• code fragments, which are simple tricks or hacks for performing a particular effect, often something I’ve picked up from somewhere else. One fragment I need to record in a post somewhere is how to densify points along a geojson linestring, a trick I picked up here ; a recent fragment of my own shows how we might be able to style a JSON fragement that identifies the location of a typo in a text string: there may be better ways, “approved ways”, of doing this, but I didn’t find one when I looked so I made a really simple thing to try to do it myself;
• code sketches, which often take the form of notebooks that describe a mini-project, one way of doing things. The notebooks in my Conversations With Data: Unistats repo are sketches as much as anything; this are often free-form, as I explore a particular topic;
• code recipes start off as sketches, but then I try to tease out and explain some sort of task or process and perhaps tidy up the code a bit; producing a recipe involves a bit of iteration, trying to identify each step and the reason for it, and ensure that everything is complete: things like Visualising WRC Rally Stages With rayshader and R and Visualising WRC Rally Timing and Results Data are packed fill of recipes;
• code hacks are the closest I get to production code, not in the sense of it being properly linted, commented and test but in the sense of something I install and use. My notebook extensions are all hacks.

You’ll note that I don’t write tests: I write a line of code at a time, and look at its output; if it doesn’t look right, or it breaks, I try to fix it. I rerun all the cells in a notebook with a fresh kernel a lot to check that things keep working; if the’re broken, I check to see why the broken cell has broken, then read back up to check each previous step is doing what I wanted and has fed forward what I intended it to feed forward.

When I first started using Jupyter notebooks, the classic notebook, they were still called IPython notebooks. (We started developing a course using notebooks in 2014 that went live, after a 6 month deleay, in 2016B (which is to say, February 2016).) To try to make the experience a bit more like the VLE, I hacked together an extension to augment the notebook with coloured cells to represent activities and to allow tutors to highlight cells they had commented in assessment feedback to students. That extension continues (though updated) as nb_extension_empinken.

Since then we have added various other extensions, such as a riff on empinken that styles cell based on bootstrap-like tags (nb_extension_tagstyler), the ipython_magic_sqlalchemy_schemadisplay extension to display (sort of!) database schemas for a connected database or an extension I liked but no-one else did to pop out cells into a floating widget so you could easily refer back to them (nb_cell_dialog).

Over the years, the repos have added clutter, bits of of automation, more elaborate approaches to packaging, but in the beginning, they were very simple, essentially just a README file, a setup.py file, a directory containing a simple __init__.py file and a static file containing the actual extension code in the form of an index.js file. The packaging structure was cribbed directly from other extensions (typically, the simplest one I could find, a minimum viable extension in other words), the setup for the extension was cribbed directly from other (minimally viable) extensions and much of the code was cribbed from other extensions. The empinken extension is essentially a toolbar button that added metadata to a cell, and a routine that iterated each notebook cell, checked the metadata and updated the css. There were other extensions that in whole or in part demonstrated how to do each of those tasks, which I then pinched and reassmbled to my own purposes.

The code was limited to py packaging (cribbed) and some simple js (largely cribbed).

The development environment was a text editor.

The testing was to install the package, refresh the notebook page and (with the help of browser developer tools) see where it was breaking until it didn’t.

The on-ramp was achievable.

So now we come to JupyterLab, which appears to me as a hostile architecture.

I’ll try to pick out what I mean by that. Please note that the following is not intended as a personal attack on the JupyterLab teams or the docs, it’s a pardoy as much as anything.

From the docs, the getting started is to install a development environment, and I’m already lost. When ever I try to use node it seems to download the internet and the instructions typically say “build the package” without saying what words of incantation I need to type into the command line to actually build the package (why should I just know how to do that?)

The next step is to install a cookiecutter. This doesn’t necessarily help because I have no idea what all the files are for, whether they or necessary, or what changes can be made to each one to perfrom a particular task. I’d rather be interested to a minimally viable set of files one at a time with an explanation of what each one does and containing nothing uncommented that is not essential. (Some “useful but optional” fragments may also be handy so I can uncomment them and try them out to see what they do, but not too many.)

When it comes to trying out some example code, I need to learn a new language, .ts (which is to say, TypeScript). I have no idea what TypeScript is or how to run it.

I also need to import load of things from @ things, whatever they are. If I’m trying to figure out how to do a thing by cribbing code from seeing what files are are loaded to support a working extension in my browser (which I suspect is way harder to do with JupyterLab than it was from classic notebook) I’m not sure if thre’s an obvious way to re-engineer the TypeScript code from the Javascript in the browser that does something like what I want to do.

I’m not totally sure what “locate the extension“ means? Is that something I have to do or is it something the code exampe is doing? (I am getting less than rational at this point because I already know that I am at the point of blindly clicking run at things I don’t understand.)

Before we can try out the extension, it needs building. In the classic notebook extensions I could simply install a package, but now I need to build one:

This is a step back to the old, pre-REPL days because there is a level of indirection here: I don’t get to try the code I’ve written, I have to convert it to somethig else. When it doesn’t work, where did I go wrong?

• with the logic?
• with the javascript?
• with the typescript?
• with the build process?
• can the TypeScript be “right” and the javascript “wrong”? I have no idea…

I think things have improved with JupyterLab now that installing an extension doesn’t require an length delay as the JupyterLab environment rebuilds itself (which was a block in itself in earlier days).

Okay, so skimming the docs doesn’t give me a sense that I’d be able to do anything other than follow the steps, click the buttons and create the example extension.

How about checking a repo to see if I can make sense of a pre-existing extension that does something close to what I what?

The classic notebook collapsible headings extension allows you to click on a heading and collapse all the cells beneath it to the next heading of the same or higher level. It works by setting a piece of metadata on the cell containing the heading you want to collapse. A community contributed extension does the same thing, but uses a different tag, "heading_collapsed": "true" rather than "heading_collapsed": true (related issue). Either that or the JupyterLab extension is broken for some other reason.

Here’s what the repo looks like (again, I’m not intending to mock or attack the repo creator, this is just a typical, parodied, example):

Based on nothing at all except my own personal prejudices, I reckon every file halves the number of folk who think they can make sense of what’s going on… (prove me wrong ;-)

Here’s the Jupyter classic notebook extension repo:

So what do I see as the “hostile architecture” elements?

• there is not an immediately obvious direct way to write some javascript, install it into JupyterLab and check the extension works;
• in many example repos, a lot of the files relate to packaging the project; for a novice, it’s not clear what the extension files are, what the exntension files are, and whether all the project files are necessary
• using TypeScript introduces a level of indirection: the user is now developing for JupyterLab, not for the end user environment they can view source from in the browser. (I think this is something I hadn’t articulated to myself before: in classic notebook extensions, you hack the final code; in JupyterLab, you write code in application land, and magic voodoo converts it to things that run in the browser.)
• in developing for JupyterLab, you need to know what bit o f Juyterlab to hook into. There’s a lot of hooks, and it’s not clear how a (non-developer) novice can find the ones they need to hook into, let alone how to use them.

And finally:

• there isn’t a “constructive” tutorial that builds up a minimally viable extension from a blank sheet an explained step at a time.

As I recall from years and years ago, if you ever see or hear a developer say “just”, or you can add a silent “just” to a statement ((just) build the node thing) you know the explanation is far from complete and is not followable.

Faced with these challenges of having to step up and go and do a developer course, learn about project tools, pick up TypeScript, andd try to familiarise myself with a complex application framework, I would probably opt to learn how to develop a VS Code extension on the grounds that the application is more general, runs by default as an desktop application rather than browser accessed service, has increasingly rich support for Jupyter notebooks, and has a wide range of other extensions to use, and crib from, and that can be easily discovered from the VS Code extensions marketplace.

PS I think followable is missing term in the reproducibility lexicon, in both a weak sense and a strong sense. In the weak sense, if you follow the instructions, does it work? In a strong sense, does the reader come away feeling that they could create their own extension.

PPS In passing, I note this from my Twitter timeline yesterday…

Over the years, I’ve sketched various tools to support the release of notebooks to students, but as I’m not the person who prepares and distributes the releases, they never get used (“Tony hacking crap again” etc.;-).

Anyway, on the basis that the tools aren’t completely crap, and may be of use to others, perhaps even folk working on other modules internally that make use of notebooks and are using them for the first time this presentation, I’ll post a quick summary of some of them here. (And if they are broken, a little use and testing by not-me could well provide the bug reports and motivation I need to fix them to a level of slightly less possible brokenness.)

The package that bundles the tools can be found here: innovationOUtside/nb_workflow_tools.

First up, tm351nbtest is a tool that helps check whether the notebooks run correctly in the latest environment.

The notebooks we save to the private module team repo all have their cells run, in part so that we can review what the expected outputs are. (When checking in notebooks, the tm351nbrun --file-processor runWithErrors . command can be used to ensure all noebooks in the specified path have their cells run.) The nbval package is a handy package that runs the notebooks in the current environment and checks that the contents of the new output cell match those of the previous, saved output cell. (I keep thinking that jupyter-cache might also be handy here?) Cells that are known to generated an error can be ignored by tagging them with the raises-exception tag and cells you want to ignore the output of can be tagged with the nbval-ignore-output tag. Running the tool generates a report identifying each notebook and each cell where the outputs don’t match.

The next tool, nb_collapse_activities, checks that out activity blocks all have their answers precollapsed. Activities are tagged and coloured using the innovationOUtside/nb_extension_empinken extension; activities with answers use the classic notebook collapsible headings extension to collapse the cells beneath an activity answer heading block (all cells are collapsed to the the cell with a header at the same level or higher as the collapsed answer cell header). The nb_collapse_activities utility tries to identify answer head cells and whenever it finds one, adds heading_collapsed: true metadata.

The third tool also processes the notebooks for release: tm351nbrun --file-processor clearOutput clears the outputs of every code cell and essentially resets each notebook to an unrun state.

A fourth tool, nbzip, can be used to zip required notebook folders for release to students.

A sort of release process could then work soemthing like this. In the environment you want to test in:

# Install package
pip3 install --upgrade git+https://github.com/innovationOUtside/nb_workflow_tools

# When checking in notebooks, ensure cells are run
# Ensure that all cells are run even in presence of errors
tm351nbrun --file-processor runWithErrors .

# Test notebooks
tm351nbtest .

# Quality reports
## Whatever...

# Clear outputs
tm351nbrun --file-processor clearOutput .

# Collapse acvitity answers
nb_collapse_activities .

# Spell check
## However... Or run earlier before output cells cleared

# Zip files
# Whichever...

In passing, the nb_workflow_tools package also includes some other utilities not directly relevant to release, but occasionally handy during production: nb_merge to merge two or more notebooks, and nb_split to split a notebook into to or more notebooks.

I’ve also been exploring various approaches to spell-checking notebooks. These are currently being collected in innovationOUtside/nb_spellchecker and the various issues attached to that repo. When I have something reliable, I’ll add it to innovationOUtside/nb_workflow_tools. Another set of quality tools I had been working on but halted due to a universal “why would we want to know anything comparative about the contents of our notebooks” can be found in innovationOUtside/nb_quality_profile. At some point I’ll revisit this and then try to bundle them up into a simple CLI tool I can also add to nb_workflow_tools.

In passing, and for completeness, I’ve also started sketching some innovationOUtside/ou-jupyter-book-tools. The idea of these is that they can provide an intermediate publishing step, where necessary, that maps from cell tags, for example, to complementary Jupyter Book / MyST / restructured text markdown.

This morning, Benjamin and I had to get a blood test. I guess this makes us blood brothers.

It’s been about 8 years since my last test. Now that Benjamin is one year old, his pediatrician wants him tested for lead as a precaution since housing in San Francisco can date back to when paint and pipes had lead.

When getting him ready for the visit, I got him dressed. He started repeating “sosh” when I was putting on his socks. I mistakenly thought the lab could get a sample from his foot, so I skipped putting on his shoes.

He saw me pick up his sandals, and ran at me yelling “shoosh!” But when I dropped them in his diaper bag, he wasn’t having it at all. He associates his “shoosh” with going out, thought this meant I was leaving without him, and started crying.

It took all of five seconds, but that was five seconds too long for my son.

…

I was three or four years old when, for the first time, my dad offered to cook my brother and me eggs.

“How would you like your eggs,” my father asked us?

My brother instantly replied, “Sunny-side up!”

After some thought, I shouted, “Sunny-side down!”

Dad told me, “There is no such thing as sunny-side down.”

“They do! If there is sunny-side up, there must be sunny-side down!”

Pulling out a frying pan from the cupboard, “No, they don’t. Choose something else,”

“I want sunny-side down.”

“How about I make you sunny-side up,” my dad reached into the refrigerator.

I was super pissed to having to eat a sunny-side up egg (which I flipped over and made a mess of). So pissed that to this day that I can recount this story an almost half-century later.

…

The small change of possibly going outside without his shoes was a clear violation of Benjamin’s sense of order.

It’s fascinating that he only started being able to repeat words in the last week and it’s been less than a month since he started stacking my Coke cans instead of just pulling them out of the box and rolling them into random places in the apartment.

Soon everything will need to be done a certain way.

I, however, plan to learn how to cook an egg over-easy before Benjamin gets old enough to ask his daddy for a fried egg.

…

Of course, in hindsight, it was obvious that a pin prick wouldn’t draw enough blood to check his lead levels. Mommy went with him while I waited in the waiting room. I could hear him crying a lot longer than the few seconds this morning between me putting away his shoes and when he, realizing that I just went into the foyer to put on my shoes, ran into my arms to be picked up.

In the car after we dropped Benjamin off in day care, M— said, “They took it out of his arm like an adult. They had a lot of trouble finding his artery. The nurse said he did so well, even though he started crying.”

“Yeah, I bet other babies…”

“…loose their shit,” she finished laughing! “Yeah, it started when the nurse pulled his arm and started tapping on it. Benjamin was like, ‘I don’t know what this is, but I know I’m not going to like it.'”

Like father, like son.

It’s easy to walk into an Apple Store and find a great Apple Watch band. But we don’t think Apple’s bands — which start at $50 a pop — are worth the price when you can find more affordable options that are just as stylish and comfortable.

We’ve combed through hundreds of third-party Apple Watch bands and tried 63 for ourselves, with help from watch experts and our discerning staff. A watch band is a personal style choice, and there’s no single option that’s best for everybody. But if you want something different from what Apple offers, we’ve found picks in a range of styles and prices that should suit most tastes.

Just watched the Apple Event, like many of you. I am not going to list every detail announced. Other people do a much better job at this. Instead I am going to comment on what stuck in my head.

If you followed all the leaks, you were mostly mis-lead. I don’t pretend I know what will happen and that’s good. I would have been very wrong. No, Apple Watch 7 will not have a boxier design.

For the easy part: new iPhone in the same fashion as last year. iPhone 13 and 13 mini, iPhone Pro 13 and 13 Max. They look very similar, with new camera modules. On the iPhone the lenses are now in opposing corners, not on one side. The cameras are also noticeably larger. Beautiful new colors to show you have a new one.

It’s the cameras that blow me away each year. The Pro models now have a 3x tele. Last year it was 2x for the Pro and 2.5x for the Pro Max. Very welcome change. The ultrawide gets a much-needed macro mode. Video recording now provides a new mode, where it automatically sets the focus bracket “intelligently” and you can change it in post. You have to see it to believe it. Insanely great.

Apple Watch 7 gets a larger display that is rounded into the body. Apple claims it is also much more durable. The display remains brighter in the off state. The body keeps compatibility with all bands, Watch gets a faster charger that connects to USB-C. I hope they maintain compatibility to old chargers. Apparently there are no new sensors.

iPad gets a small spec bump to A13 Bionic which was introduced with iPhone 11 and new cameras with a 12MP ultrawide front camera supporting automatic framing in video calls. The enclosure keeps the old almost iconic form with wide bezels at the top and bottom. It now looks seriously dated.

The new iPad mini was the secret star of the show. It’s actually more of an iPad Air mini than an iPad mini. TouchID in power button, same body design, now with Truetone flash. Best of all, it now has a USB-C port, unlike the new/old iPad. This is now the real iPad lineup: 12.9″ Pro, iPad Air, iPad mini. The smaller Pro and the iPad somehow don’t fit in this range anymore.

Fitness+ is coming to more countries and Apple just subtitles instead of dubbing the audio. I am not sure how well that will work with people who have to stare at the screen during workouts. 1 month free trial, and 3 months if you buy a new Watch 7.

Speaking of which: Apple isn’t ready to ship. While you can preorder the iPhone from this Friday to be shipped a week later, Watch does not yet have a release date. Later this fall. I guess there will be a run on Xmas gifts.

It’s always interesting what Apple does NOT say. No performance comparisons between A14 and A15 for instance. I also missed updates on availability for iOS and watchOS, assuming they are coming next week. (Update. next Monday)

And to end on a high note: the iPhone Pro models get displays with variable refresh rates all the way from 10 to 120 Hz. That is going to save a ton of battery. It made the always-on Watch 5 display possible.

What were your highlights?

Today, Apple debuted its iPhone 13 lineup consisting of four new models:

• iPhone 13 mini
• iPhone 13
• iPhone 13 Pro
• iPhone 13 Pro Max

Similar to the iPhone 12 family of devices, the iPhone 13 and 13 mini are the more affordable options, while the two Pro models add premium features at a premium price. All four devices share the A15 Bionic processor, 5G networking, camera updates, MagSafe, and more.

Also, all four devices will be available for pre-order this Friday, September 17th, with availability beginning one week later on Friday, September 24th.

Here’s a complete overview of everything new coming to the iPhone 13 lineup.

iPhone 13 and 13 mini

The new iPhone 13 and 13 mini continue Apple’s latest flat-sided, aluminum iPhone aesthetic and come in five colors: (PRODUCT)RED, starlight, midnight, blue, and pink. The iPhone 13 features a 6.1” display, while the mini’s is 5.4”. Both displays have the protective Ceramic Shield layer on the front display as the iPhone 12 and 12 mini did. Apple says it has improved the iPhones’ Super Retina XDR OLED displays to reproduce truer blacks and has increased the maximum outdoor brightness to 800 nits (a 28% increase) with a peak brightness of 1200 nits for HDR content. Both iPhones also feature an IP68 water resistance rating.

The new pink iPhone 13.

Apple says that the A15 Bionic, which is built with a 5-nanometer process, is faster and includes 15 billion transistors. The company says the 6-core architecture – two high-performance and four efficiency cores – is up to 50% faster than its competition, though there is no mention of who that competition is. Likewise, Apple claims the 4-core GPU is up to 30% faster than its competition. Finally, the 16-core Neural Engine, which handles machine learning and similar tasks, can handle 15.8 trillion operations per second, driving features like Live Text in iOS 15. The iPhone 13 and 13 mini also feature more bands of 5G wireless, with over 200 carriers in 60 countries supporting the technology by the end of this year.

Apple claims it has improved battery life thanks to a larger battery and a more efficient A15 Bionic processor, display, and other components. The company says the changes enable the iPhone 13 mini to get 1.5 hours more battery life than the iPhone 12 mini and the iPhone 13 to get 2.5 hours more battery life than the iPhone 12.

One of the first things you’ll probably notice about the design of the iPhone 13 and 13 mini is that the rear cameras are arranged diagonally instead of stacked, as was the case with the iPhone 12 and 12 mini. Apple has also redesigned the TrueDepth camera system so that it takes up 20% less space. In other words, the notch is smaller.

As has become the norm with new iPhones, those 12MP rear-facing cameras are a big part of this year’s updates. The new camera system has been improved to allow 47% more light to reach the company’s biggest image sensor yet.

The Wide camera is available in both models and features Sensor-shift OIS, which was introduced with the iPhone 12 Pro Max to provide for more stable images by shifting the sensor to compensate for hand shake instead of moving the lens. Apple says the new Ultra Wide camera captures more detail in dark photos too.

On the software side, Apple announced Photographic Styles that take advantage of the A15 Bionic’s power along with a faster Neural Engine and new image signal processor. The feature, which you’ll find in the Camera app, comes with built-in presets that can be adjusted and will work across different scenes and photography sessions, allowing users to dial in a look all their own. Apple describes the feature as more than just a filter. Instead, Photographic Styles take advantage of computational photography and Apple’s unique multi-frame processing to treat different parts of a photo, like skin tones, differently.

Both cameras also add Smart HDR 4, which Apple says improves color, contrast, and lighting and improved Night mode. The front-facing TrueDepth camera supports all of the features of the back camera too.

Cinematic mode.

Video performance has been enhanced, with a new Cinematic mode that creates a depth effect by automatically shifting the focus from one subject of the video to another. Cinematic mode can also be adjusted manually while filming, or even after you have finished filming, using Photos or iMovie for iOS and soon, iMovie and Final Cut Pro on the Mac. Both iPhone models can also shoot in 4K at 60 fps in HDR with Dolby Vision.

Apple is also doubling the base and top storage for the iPhone 13 and 13 mini with available storage of 128, 256, and 512 GB.

iPhone 13 Pro and Pro Max

Like the iPhone 13 and 13 mini, the iPhone 13 Pro and Pro Max feature the A15 Bionic processor, a 20% smaller True Depth camera system, IP68 water resistance, and a bigger battery. The big differences are the colors, screens, and cameras.

The iPhone 13 Pro and Pro Max come in four colors: graphite, gold, silver, and sierra blue. The Super Retina XDR screen features ProMotion, the adaptive refresh rate technology that lets iOS change the refresh rate of the screen from 10 to 120 Hz, depending on how the iPhone is being used. According to the demos during the keynote, the iPhone 13 Pro and Pro Max can even match the speed of a user scrolling on their phone in real-time. Also, outdoor screen brightness has been increased 25% to 1000 nits.

ProMotion promises to improve the gaming experience.

The cameras combine new hardware and software innovations too. The Wide camera has a larger sensor with 1.9 µm pixels, which Apple says will mean less image noise and faster shutter speeds. The Wide camera features a ƒ/1.5 aperture lens for better low light photos too. The Ultra Wide camera has a wider ƒ/1.8 aperture as well as a new auto-focus system that Apple says provides a 92% improvement in low-light situations.

The Ultra Wide lens also supports macro photography for the first time, which I’m excited about. With add-on lenses, I’ve done some iPhone macro photography in the past, but now, you’ll be able to focus on subjects as little as 2 cm from the iPhone 13 Pro and Pro Max’s lens. Macro video is possible, too, including in Slow-mo and Time-lapse modes.

Macro photography is coming to the iPhone 13 Pro and Pro Max.

The Telephoto camera has been updated too, with a 77 mm lens that can shoot up to 3x zoom over the Wide camera.

The iPhone 13 Pro and Pro Max include the same software updates coming to the iPhone 13 and 13 mini, including:

• Photographic Styles
• Smart HDR 4
• Cinematic mode

All cameras on the iPhone 13 Pro and Pro Max support Night mode for the first time. Both models support shooting video in the ProRes codec shot in 4K at 30 fps as long as you have the models with 256 GB, 512 GB, or 1 TB of storage. There’s also a 128 GB iPhone 13 Pro and Pro Max, but it only supports ProRes at 1080p and 30 fps.

Like the iPhone 13 and 13 mini, Apple says that the new iPhone 13 Pro and Pro Max will get better battery life with the Pro adding 1.5 hours over the 12 Pro and the 13 Pro Max adding 2.5 hours compared to the 12 Pro Max.

iPhone Leather Wallet

Apple’s Leather Wallet for the iPhone, which attaches to the device using MagSafe, is adding Find My support. If your wallet becomes separated from your iPhone, the new feature will allow you to determine the last known location of your wallet if it becomes separated from your iPhone.

We won’t know for certain how much Apple has moved the ball forward with these updates until we’ve had a chance to try them, but I’m excited about this new iPhone lineup, particularly the Pro line. I’ve had ProMotion on an iPad for quite some time, and I can’t wait to see what it’s like on my iPhone, where smooth scrolling is more important due to its smaller screen size.

Of course, I’m also very interested in the new cameras. I take more photos with my iPhone these days than my semi-pro Sony camera, and based on what Apple claims, these should be meaningful updates, even coming from an iPhone 12 Pro Max.

You can follow all of our September Apple event coverage through our September 2021 event hub, or subscribe to the dedicated RSS feed.

Support MacStories Directly

Club MacStories offers exclusive access to extra MacStories content, delivered every week; it’s also a way to support us directly.

Club MacStories will help you discover the best apps for your devices and get the most out of your iPhone, iPad, and Mac. Plus, it’s made in Italy.

2014 was a wild time for iOS, and the creation of Becky! was truly the zenith

With the latest update to the CDC recommendations for international travel (trdr: don't), I wanted to visualize the stats of different countries. I found this neat little webapp that lets you interactively assign colors, export the data as JSON and then build a static PNG:

The programming related courses I work on are probably best described as introductory programming courses. Students are taught using an approach based on “a line of code at a time” within a Jupyter notebook environment which provides a REPL execution model. Students are encouraged to write a line of code in a cell, run it, and then inspect state changes arising from the code execution as displayed in the code cell output. Markdown cells before an after the code cell are use to explain the motivation for the next bit of code, or prompting students to predict what they think it will do. Markdown cells following a code cell can be use to review or explain what just happened, or prompt students to reflect on what they think happened.

In passing, I note that there are other models for providing text+code style annotations. For example the pycco-docs/pycco package will render side-by-side comments and code:

The view is generated from Python files containing inline comments and docstrings:

Something I haven’t yet tried is a workflow that renders the side-by-side view from a Python file generated from a Jupyter notebook using the jupytext file converter (I’m not sure if Jupytext can generate the python files using comment markup conventions that pycco expects?)

For simple code blocks, tools such as nbtutor provide a simple code stepper and tracer that can be used to explore the behaviour of a few lines of code.

I use nbtutor in some first year undergraduate notebooks and it’s okay, -ish (unfortunately it can break in combination with some other widgets running in the same notebook).

Another approach I am keen to explore in terms of helping students help themselves when it comes to understanding code they have written is the automated generation of simple flowchart visualisations from code fragements (see for example Helping Learners Look at Their Code).

Poking around looking for various Python packages that can help animate or visualise common algorithms (Bjarten/alvito is one; anyone got suggestions for others?) I came across a couple of other code stepping tools produced by Alex Hall / @alexmojaki.

The first one is alexmojaki/birdseye which can provide a step trace for code executed in a magicked notebook cell block:

You can also separately step though nested loops:

Another tool, alexmojaki/snoop, will give a linear trace from an executed code cell:

Alex also has a handy package for helping identify out of data Python packages based on the latest version availble on PyPi: alexmojaki/outdated.

When it comes to Python errors, for years we have used the Jupyter skip-traceback extension to minimise the traceback message displayed when an error is raised in a Jupyter notebook. However, there are various packages out there that attempt to provide more helpful error messages, such as SylvainDe/DidYouMean-Python (which is currently broken from the install – I think the package needs its internal paths fettling a bit!) and friendly-traceback. The latter package tidies up the display of error messages:

Note that the pink gutter to indicate failed cell execution comes from the innovationOUtside/nb_cell_execution_status extension.

You can then explore in more detail what the issue is and in some cases, how you might be able to fix it:

You can also start to tunnel down for more detail about the error:

This extension looks like it could be really handy in an introductory, first year undergraduate intro to programming module, but the aesthetic may be a bit simplistic for higher level courses.

From the repo, friendly-traceback/friendly-traceback, it looks like it shouldn’t be too hard to create your own messages.

This does make me wonder whether a language pack approach might be useful? That would allow for internationalisation but could also be used to easily support the maintenance of custom message packs for particular teaching and learning use cases?

With a couple of new modules presenting for the first time this year, I would argue we missed an opportunity to explore some of these ideas where we can start to use the technology as an illustrator of what’s going on with code we give to students, and more importantly that students might write for themselves.

There are several reasons why I think this probably hasn’t happened:

• no time to explore this sort of thing (with two years+ to produce a course, you might want to debate that…);
• no capacity in a module team to explore and test new approaches (I’d argue that’s our job as much as producing teaching material if the org is a beacon of best practice in the development and delivery of interactive online distance education materials);
• no capacity in support units to either research their effectiveness or explore such approaches and make recommendations into module teams about how they might be adopted and used, along with an examples gallery and sample worked examples based on current draft materials (I honestly wonder about where all the value add we used to get from support units years ago has gone and why folk don’t think we are the worse for not having units that explore emerging tech for teaching and learning. Folk too busy doing crapalytics and bollockschain, I guess);
• and I guess: “what value does it add anyway?” (which is to say: “why should we explore new ways of teaching and learning?”) and “you’re just chasing ooh, shiny” (which really doesn’t fit with 2+ year production cycles and material updates every five years where locking into an emerging technology is high risk, becuase rather than regularly updating around it, you are stuck with it for potentially up to a decade (2 years production, five years primary course life, 3 years course life extension)).

Bored, bored, bored, bored, bored…

Herewith a lost-pet story with (spoiler) a happy ending, starring a real bloodhound. Soon to be a major motion picture, I bet.

Here’s the hound, who’s called Hunter.

What happened was, we’ve been introducing our 9-month old cat þ (pronounced “Thorn”, because reasons) to the great outdoors. This is hard to avoid because we have a back porch we eat most meals on when it’s nice, and it’s a pain in the butt to not have the door to it open.

It’s been going great because þ is a cautious kind of cat. He skedaddles back inside or up a tree if a car goes down the road outside or someone drops a spoon on the floor or even sneezes. We kept him on a leash/harness but then after a while stopped, because he seemed to have no interest in going outside our yard. He’d flit briefly under the fence to the neighbor’s yard to investigate a chittering squirrel, but stays away from the street in front and the alley in back. And he never wanted to stay outside long.

Then, last Saturday, we needed to make an overnight trip to our cabin in connection with the renovations; our 22-year old son who lives in our basement was happy to mind the fort. When we left, þ was on a chair on the back porch, fascinated with all the to-ing and fro-ing with totes and duffel bags.

When we came home Sunday, we discovered our son hadn’t seen him since we left. Suddenly we were The People With The Lost Pet. We put up posters. We walked the neighborhood, whistling and calling. We advertised on the SPCA lost-animals page and one on Facebook. Since þ is microchipped, we updated that registry too. Lauren, who is more assiduous than I, talked to more neighbors in 48 hours than we had in the last year or two. Everyone was sympathetic.

Now, we live in a dense urban neighborhood with lots of cars and the occasional coyote, not to mention, from time to time, damaged and possibly predatory humans. So, there are risks for cats. But still, it didn’t make sense, we just couldn’t figure out the scenario in which he’d managed to get far enough away to be really lost. We’d had other cats that roamed far and wide and casually walked into neighbors’ houses, about whom we worried terribly, but they all had long peaceful lives.

Eventually we ran across Pet Searchers Canada, whose service is comprehensive and bloodhound-assisted. Also it’s not cheap, but we were feeling pretty emotionally beat-up. After we’d signed the contract and paid, Savannah the handler showed up with Hunter the bloodhound.

We provided þ’s harness as a scent sample, then Savannah and Hunter vanished for the best part of two hours. Savannah explained that she’d picked up his scent on the next street south and the intervening alley, and the western continuation of our alley. She told us she’d send a marked-up map showing where Hunter got the strongest signal, and advised us to go out after dark to check those areas out. Also to take along well-worn clothes, heavy with our scent, and drag them along behind to lay a scent trail us as we came home from our expedition, on the theory that the poor little unadventurous guy had got a little too far away and just didn’t know the way home.

She emailed us the marked-up map and Wednesday evening that’s what we did. And as we walked along, I occasionally offered the special “come here for a treat” whistle, and Lauren made “come for dinner” sounds. After dark is definitely the time to do this, because it’s quieter and also people aren’t going to be giving you the side-eye for walking along trailing a pair of jeans in the grass behind you.

While we were walking along the alley between our street and the next one south, territory we’d tried before albeit not in the quiet and the dark, suddenly there were plaintive cat cries answering our whistles and calls. þ has a high-pitched and penetrating voice when he has something he thinks it’s important that we hear.

We converged on someone’s tall back fence that faced the alley with an apparently-locked gate, but there was a gap under it and almost immediately, a pointy furry little black face looking through it. The gap was pretty low and þ had to put in some real squirming, but he made it out.

Now he’s at home. He won’t be going out for a while. He had a minor injury on one front leg, but already healing, no call for a vet visit.

What’s shocking is that he was in the back yard of a house across the street and maybe two houses east of us. I guess if you stay in your own yard you’re not going to learn how to find your way home. There’s a lesson in that.

Now, our neighborhood has plenty of cats and they do not co-exist very peaceably, so I suspect that poor þ was cornered and chased by one of the local feline bullies and that’s how he misplaced himself. Our daughter has vowed to walk him around the ’hood on his harness whether he wants to or not, so that he’ll at least know the nearby territory.

Judging by the number of posters on utility poles I see, pets often go missing. But you can do more than put up posters. I recommend bloodhounds.

I’ve been brooding my last post and trying to re-adjust to returning to work, so today’s Apple event was an interesting distraction. Some quick thoughts in no particular order:

• Not updating the base iPad to have USB-C seems backwards. “Parts bin” iPad indeed.
• I very much like the A15-toting 6th Gen iPad Mini (what it having USB-C and all), but can’t really justify getting one to replace my 5th Gen (I found it weird that they only mentioned the A15 when talking about the iPhone, but hey, that’s product placement for you).
• The loss of the headphone jack on the Mini is reason enough not to upgrade, to be honest. Cue an n-gate-style “Apple continues the fight against its own users” rant.
• With the addition of the wide-angle front camera and Center Stage to the standard iPad and the Mini, last year’s Air is pretty much dead for me. If I ever decide to get a bigger screen, it’ll have to be the Pro.
• Some of the videos were (at least partially) shot in Lisbon, which is pretty much insane as we still don’t have an official Apple store (or Continental Portuguese as a speech option for Siri).
• The phones are… OK, I guess. I’m not really interested in upgrading until (maybe) next year.
• Nobody got the new Watch design right. Which is a good thing, because I think it’s great. Again, not interested in upgrading, but nice to see.

Seeing the Sherlocking of FlickType streamed live to the world, however, was a major downer (I was one of the Watch beta testers, and loved it). Apple really ought to be better than this, and I expect a fair amount of fallout over the next few days.

So… when are we getting new Mac chipsets? That’s what I want to see, I need a new Mac desktop (preferably a new mini) that will last me for the next half decade.

The breakdown figures for macOS update overhead are… staggering, and I’m used to Windows updates by now.

There is simply no sane reason for things to be like this.

Datasette Desktop is a new macOS desktop application version of Datasette, an "open source multi-tool for exploring and publishing data" built on top of SQLite. I released the first version last week - I've just released version 0.2.0 (and a 0.2.1 bug fix) with a whole bunch of critical improvements.

You can see the release notes for 0.2.0 here, but as I've done with Datasette in the past I've decided to present an annotated version of those release notes providing further background on each of the new features.

The plugin directory

A new plugin directory for installing new plugins and upgrading or uninstalling existing ones. Open it using the "Plugins -> Install and Manage Plugins..." menu item. #74

This was the main focus for the release. Plugins are a key component of both Datasette and Datasette Desktop: my goal is for Datasette to provide a robust core for exploring databases, with a wide array of plugins that support any additional kind of visualization, exploration or data manipulation capability that a user might want.

Datasette Desktop goes as far as bundling an entire standalone Python installation just to ensure that plugins will work correctly, and invisibly sets up a dedicated Python virtual environment for plugins to install into when you first run the application.

The first version of the app allowed users to install plugins by pasting their name into a text input field. Version 0.2.0 is a whole lot more sophisticated: the single input field has been replaced by a full plugin directory interface that shows installed v.s. available plugins and provides "Install", "Upgrade" and "Uninstall" buttons depending on the state of the plugin.

When I set out to build this I knew I wanted to hit this JSON API on datasette.io to fetch the list of plugins, and I knew I wanted a simple searchable index page. The I realized I also wanted faceted search, so I could filter for installed vs not-yet-installed plugins.

Datasette's built-in table interface already implements faceted search! So I decided to use that, with some custom templates to add the install buttons and display the plugins in a more suitable format.

The first challenge was getting the latest list of plugins into my Datasette instance. I built this into the datasette-app-support plugin using the startup() plugin hook - every time the server starts up it hits that API and populates an in-memory table with the returned data.

The data from the API is then extended with four extra columns:

• "installed" is set to "installed" or "not installed" depending on whether the plugin has already been installed by the user
• "Installed_version" is the currently installed version of the plugin
• "upgrade" is the string "upgrade available" or None - allowing the user to filter for just plugins that can be upgraded
• "default" is set to 1 if the plugin is a default plugin that came with Datasette

The data needed to build the plugin table is gathered by these three lines of code:

plugins = httpx.get(
     "https://datasette.io/content/plugins.json?_shape=array"
).json()
# Annotate with list of installed plugins
installed_plugins = {
    plugin["name"]: plugin["version"]
    for plugin in (await datasette.client.get("/-/plugins.json")).json()
}
default_plugins = (os.environ.get("DATASETTE_DEFAULT_PLUGINS") or "").split()

The first line fetches the full list of known plugins from the Datasette plugin directory

The second makes an internal API call to the Datasette /-/plugins.json endpoint using the datasette.client mechanism to discover what plugins are currently installed and their versions.

The third line loads a space-separated list of default plugins from the DATASETTE_DEFAULT_PLUGINS environment variable.

That last one deserves further explanation. Datasette Desktop now ships with some default plugins, and the point of truth for what those are lives in the Electron app codebase - because that's where the code responsible for installing them is.

Five plugins are now installed by default: datasette-vega, datasette-cluster-map, datasette-pretty-json, datasette-edit-schema and datasette-configure-fts. #81

The plugin directory needs to know what these defaults are so it can avoid showing the "uninstall" button for those plugins. Uninstalling them currently makes no sense because Datasette Desktop installs any missing dependencies when the app starts, which would instantly undo the user's uninstall action decision.

An environment variable felt like the most straight-forward way to expose that list of default plugins to the underlying Datasette server!

I plan to make default plugins uninstallable in the future but doing so require a mechanism for persisting user preference state which I haven't built yet (see issue #101).

A log on the loading screen

The application loading screen now shows a log of what is going on. #70

The first time you launch the Datasette Desktop application it creates a virtual environment and installs datasette, datasette-app-support and the five default plugins (plus their dependencies) into that environment.

This can take quite a few seconds, during which the original app would show an indeterminate loading indicator.

Personally I hate loading indicators which don't show the difference between something that's working and something that's eternally hung. Since I can't estimate how long it will take, I decided to pipe the log of what the pip install command is doing to the loading screen itself.

For most users this will be meaningless, but hopefully will help communicate "I'm installing extra stuff that I need". Advanced users may find this useful though, especially for bug reporting if something goes wrong.

Under the hood I implemented this using a Node.js EventEmitter. I use the same trick to forward server log output to the "Debug -> Show Sever Log" interface.

Example CSV files

The welcome screen now invites you to try out the application by opening interesting example CSV files, taking advantage of the new "File -> Open CSV from URL..." feature. #91

Previously Datasette Desktop wouldn't do anything at all until you opened up a CSV or SQLite database, and I have a hunch that unlike me most people don't have good examples of those to hand at all times!

The new welcome screen offers example CSV files that can be opened directly from the internet. I implemented this using a new API at datasette.io/content/example_csvs (add .json for the JSON version) which is loaded by code running on that welcome page.

I have two examples at the moment, for the Squirrel Census and the London Fire Brigade's animal rescue data. I'll be adding more in the future.

The API itself is a great example of the Baked Data architectural pattern in action: the data itself is stored in this hand-edited YAML file, which is compiled to SQLite every time the site is deployed.

To get this feature working I added a new "Open CSV from URL" capability to the app, which is also available in the File menu. Under the hood this works by passing the provided URL to the new /-/open-csv-from-url API endpoint. The implementation of this was surprisingly fiddly as I wanted to consume the CSV file using an asynchronous HTTP client - I ended up using an adaption of some example code from the aiofile README.

Recently opened files and "Open with Datasette"

Recently opened .db and .csv files can now be accessed from the new "File -> Open Recent" menu. Thanks, Kapilan M! #54

This was the project's first external contribution! Kapilan M figured out a way to hook into the macOS "recent files" mechanism from Electron, and I expanded that to cover SQLite database in addition to CSV files.

When a recent file is selected, Electron fires the "open-file" event. This same event is fired when a file is opened using "Open With -> Datasette" or dragged onto the application's dock.

This meant I needed to tell the difference between a CSV or a SQLite database file, which I do by checking if the first 16 bytes of the file match the SQLite header of SQLite format 3\0.

.db and .csv files can now be opened in Datasette starting from the Finder using "Right Click -> Open With -> Datasette". #40

Registering Datasette as a file handler for .csv and .db was not at all obvious. It turned out to involve adding the following to the Electron app's package.json file:

  "build": {
    "appId": "io.datasette.app",
    "mac": {
      "category": "public.app-category.developer-tools",
      "extendInfo": {
        "CFBundleDocumentTypes": [
          {
            "CFBundleTypeExtensions": [
              "csv",
              "tsv",
              "db"
            ],
            "LSHandlerRank": "Alternate"
          }
        ]
      }

The Debug Menu

A new Debug menu can be enabled using Datasette -> About Datasette -> Enable Debug Menu".

The debug menu existed previously in development mode, but with 0.2.0 I decided to expose it to end users. I didn't want to show it to people who weren't ready to see it, so you have to first enable it using a button on the about menu.

The most interesting option there is "Run Server Manually".

Most of the time when you are using the app there's a datasette Python server running under the hood, but it's entirely managed by the Node.js child_process module.

When developing the application (or associated plugins) it can be useful to manually run that server rather than having it managed by the app, so you can see more detailed error messages or even add the --pdb option to drop into a debugger should something go wrong.

To run that server, you need the Electron app to kill its own version... and you then need to know things like what port it was running on and which environment variables it was using.

Here's what you see when you click the "Run Server Manually" debug option:

Here's that command in full:

DATASETTE_API_TOKEN="0ebb45444ba4cbcbacdbcbb989bb0cd3aa10773c0dfce73c0115868d0cee2afa" DATASETTE_SECRET="4a8ac89d0d269c31d99059933040b4511869c12dfa699a1429ea29ee3310a850" DATASETTE_DEFAULT_PLUGINS="datasette datasette-app-support datasette-vega datasette-cluster-map datasette-pretty-json datasette-edit-schema datasette-configure-fts datasette-leaflet" /Users/simon/.datasette-app/venv/bin/datasette --port 8002 --version-note xyz-for-datasette-app --setting sql_time_limit_ms 10000 --setting max_returned_rows 2000 --setting facet_time_limit_ms 3000 --setting max_csv_mb 0

This is a simulation of the command that the app itself used to launch the server. Pasting that into a terminal will produce an exact copy of the original process - and you can add --pdb or other options to further customize it.

Bonus: Restoring the in-memory database on restart

This didn't make it into the formal release notes, but it's a fun bug that I fixed in this release.

Datasette Desktop defaults to opening CSV files in an in-memory database. You can import them into an on-disk database too, but if you just want to start exploring CSV data in Datasette I decided an in-memory database would be a better starting point.

There's one problem with this: installing a plugin requires a Datasette server restart, and restarting the server clears the content of that in-memory database, causing any tables created from imported CSVs to disappear. This is confusing!

You can follow my progress on this in issue #42: If you open a CSV and then install a plugin the CSV table vanishes. I ended up solving it by adding code that dumps the "temporary" in-memory database to a file on disk before a server restart, restarts the server, then copies that disk backup into memory again.

This works using two custom API endpoints added to the datasette-app-support plugin:

• POST /-/dump-temporary-to-file with {"path": "/path/to/backup.db"} dumps the contents of that in-memory temporary database to the specified file.
• POST /-/restore-temporary-from-file with {"path": "/path/to/backup.db"} restors the content back again.

These APIs are called from the startOrRestart() method any time the server restarts, using a file path generated by Electron using the following:

backupPath = path.join(
  app.getPath("temp"),
  `backup-${crypto.randomBytes(8).toString("hex")}.db`
);

The file is deleted once it has been restored.

After much experimentation, I ended up using the db.backup(other_connection) method that was added to Python's sqlite3 module in Python 3.7. Since Datasette Desktop bundles its own copy of Python 3.9 I don't have to worry about compatibility with older versions at all.

The rest is in the milestone

If you want even more detailed notes on what into the release, each new feature is included in the 0.2.0 milestone, accompanied by a detailed issue with screenshots (and even a few videos) plus links to the underlying commits.

Quelques liens en vrac au retour de l’été, parfois très ancien, souvent plus récent. Bonne lecture !

Numérique

• Apple veut protéger les enfants mais met en danger le chiffrement. Ca me rappelle un vieux cartoon, où l’Oncle Sam a un paquet marqué « contrôle du discours sur Internet » et où on lui demande : « je vous l’emballe comment, anti-terrorisme ou protection des enfants ? » . J’en parle ici : Lutte contre la pédopornographie: Apple est-il trop curieux ?. Apple est surtout trop innocent et donc dangereux. Voir ma vision du sujet dans cet article. On lira avec intérêt les commentaires de l’EFF sur ce sujet épineux :
  • Apple’s Plan to ‘Think Different’ About Encryption Opens a Backdoor to Your Private Life. Le logiciel qui tourne dans votre téléphone, prévu pour repérer les images pédophiles peut être remplacé par un autre à la demande d’un gouvernement (c’est déjà le cas dans certains pays). “governments that outlaw homosexuality might require the classifier to be trained to restrict apparent LGBTQ+ content, or an authoritarian regime might demand the classifier be able to spot popular satirical images or protest flyers. ” / « Les gouvernements qui interdisent l’homosexualité peuvent exiger que le classifieur reconnaisse les contenus LGBTQ+ ou un régime autoritaire pourrait exiger que le classifieur puisse identifier des caricatures politiques ou des dépliants distribués dans les manifestations » ;
  • If You Build It, They Will Come: Apple Has Opened the Backdoor to Increased Surveillance and Censorship Around the World ;
  • Mise à jour : Apple retarde la mise en place de ses nouveaux outils contre la pédopornographie ;
• Il y a 30 ans, Linus Torvalds lançait (le noyau) Linux qui, complété par les utilitaires GNU, a formé le système d’exploitation GNU/Linux ;
• Pendant qu’on est dans la nostalgie : Windows XP a 20 ans, le Web a 30 ans, le PC a 40 ans ;
• Cryptomonnaies : What’s Chia, And Why Is It Eating All The Hard Drives? ;
• Quand Amazon utilise des algorithmes pour licencier ;
• Sympa d avoir l’auto-hébergement mis en avant par Vice : Meet the Self-Hosters, Taking Back the Internet One Server at a Time, avec des mentions du Fédiverse !
• WhatsApp sanctionné d’une amende record pour ne pas avoir respecté les normes de protection des données de l’UE. 225 millions d’Euros tout de même !  ;
• On pensait ProtonMail au dessus de tout soupçon. Comme toujours, c’est plus compliqué que cela. Protonmail a communiqué à la police l’adresse IP de militant·es anti-gentrification. Des militants de Youth for Climate ont été visés par la police française, qui a demandé à Europol de voir avec la Suisse si on pouvait obtenir leurs adresses IP. La justice Suisse a fait la demande à ProtonMail qui a du s’exécuter. Bref, si vous êtes un(e) militant(e) écolo qui fait de la désobéissance citoyenne, passez par TOR pour consulter votre compte ProtonMail…
• Cette étude sur l’impact économique des logiciels libres (OSS) et matériels libres (OSH) sur l’économie européenne est sans appel : « Les entreprises dans l’UE ont investi environ 1 milliard d’euros dans les logiciels libres en 2018, avec un impact sur l’économie européenne entre 65 et 95 milliards d’euros. L’analyse estime un rapport coûts-bénéfices supérieur à 1:4 et prédit qu’une augmentation de 10 % des contributions aux logiciels libres créerait annuellement 0,4 % à 0,6 % de PIB en plus ainsi que plus de 600 start-ups technologiques supplémentaires dans l’UE. » (voir page 21 du rapport (format PDF)) ;
• L’Allemagne réclame sept ans de mises à jour pour nos smartphones. « Le lobby Digitaleurope, qui représente notamment les intérêts d’Apple, Samsung, Xiaomi, Oppo ou Huawei, lutte contre ce durcissement ». Voir aussi chez Clubic ;
• Ah, si même Les Échos s’y mettent ! Le numérique face au défi environnemental ;
• Guillaume Pitron : « Internet a l’odeur du beurre rance, le goût du sel et le son strident des données dans les data centers ».

Vélo

• Avec le Brompton, j’ai l’impression de court-circuiter le système ;
• Attention coup de coeur : Velo Libre, dont le slogan est « Un vélo aussi libre qu’un logiciel libre » ;

• Bon Pote a publié une série de 5 articles sur le vélo par Guillaume Martin. Tous sont intéressants, les trois premiers sont vraiment tout-public, les deux derniers plus pointus :
  • Le vélo est le futur de nos mobilités (1/5) ;
  • Pourquoi développer le vélo ? Plaidoyer pour la bicyclette (2/5) ;
  • “Ma grand-mère ou mon plombier ne feront jamais de vélo” : stop aux idées reçues (3/5) ;
  • Le vélo est le meilleur moyen de décarboner nos mobilités (4/5) ;
  • Construire une France cyclable (5/5) ;

Climat

• Nicolas Hulot : « J’en appelle au sursaut des consciences ». Ouest France demande « L’écologie doit devenir le fil rouge de toutes les actions politiques ? ». Réponse de l’ancien ministre :

Bien sûr, et je ne comprends pas qu’elle soit encore un motif de divisions, de doutes, de railleries alors qu’elle doit être un impératif d’unité. Chaque matin, l’histoire donne raison à l’écologie. Les enfants qui naissent maintenant seront confrontés, dans trente ans, à un chaos monstrueux. C’est ça la réalité, elle ressort de toutes les pages du rapport du Giec.

• Climat : le nombre de catastrophes multiplié par cinq en cinquante ans ;
• « La France n’est pas assez préparée au réchauffement » explique Corinne le Quéré, présidente du Haut Conseil pour le climat ;
• Solarpunk Manifesto, trouvé via l’excellent . Et, oh miracle, la version française existe : Un Manifeste Solarpunk ;
• Inondations en Europe : comment le réchauffement climatique est passé de « menace pour les générations futures » à danger imminent ;
• Alors qu’il y a potentiellement 17,5 milliards de barils de pétrole et des milliards de m3 de gaz, le Groenland s’interdit toute recherche d’hydrocarbures sur son territoire ;
• Changement climatique : 73 % des habitants du G20 estiment que nous nous approchons d’un « point de non-retour » Source : Global Commons Survey: Attitudes to Transformation and Planetary Stewardship
• Non, le passage à 30 km/h en ville n’augmente pas la pollution ! ;
• Le système économique actuel, basé sur l’indice du Produit intérieur brut (PIB) n’est plus compatible avec l’urgence climatique, assure l’Easac, le conseil européen des sciences ;
• La société Harris Interactive a publié un sondage intitulé ‘Le cœur des Français’ (télécharger le rapport Harris Le Coeur des français, format PDF, 2 Mo, 77 pages). Il y a un focus, page 34, sur les français et l’environnement, et il donne de l’espoir :
  • le dérèglement climatique est la première préoccupation listée. C’est un sujet inquiétant pour 84 % des français, à égalité avec « l’avenir de vos enfants »
  • 63 % des français considèrent que les pouvoirs publics n’agissent pas assez en faveur de l’environnement (9 % pensent que c’est trop).
  • 84 % considèrent que « Même si de nombreuses personnes dans le reste du monde ne font pas vraiment d’efforts en faveur de l’environnement, il faut quand même en faire »
  • Sur les efforts à faire, 74 % sont prêts à moins prendre l’avion pour partir en vacances, et 69 % à moins utiliser la voiture pour se déplacer.
• DataClimat.fr, un projet open source et open data pour voir ce que le changement climatique pourrait nous réserver en France en terme de feux de forêt, de précipitations, de température et de sécheresse (porté par la députée indépendante Paula Forteza et son équipe. Rappelons, si le sujet vous intéresse qu’on peut la retrouver dans l’épisode #4 de mon podcast l’OCtet Vert ;
• Rapport du Giec : un changement climatique mondial «sans précédent», aux conséquences «irréversibles» ;
• Météo France fait un excellent travail de pédagogie sur son site. Par exemple, Changement climatique : 10 points clés pour comprendre le 6e rapport du GIEC ;
• La décarbonation est-elle vraiment possible ?, où JM Jancovici et chef économiste d’EDF Thomas-Olivier Léautier débattent de ce sujet ;
• Alexia Soyeux de Carbone4 a fait un bon résumé du rapport du GIEC : Rapport du GIEC : rappel des points clés . Extrait :
  • Les changements climatiques récents sont généralisés, rapides et s’intensifient. Ils sont sans précédent depuis des milliers d’années.
  • Il est incontestable que les activités humaines sont à l’origine du changement climatique, qui rend les phénomènes climatiques extrêmes, notamment les vagues de chaleur, les fortes précipitations et les sécheresses, plus fréquents et plus sévères, augmente la fréquente des conditions météo propices aux incendies, et cause le réchauffement, la perte d’oxygène et l’acidification des océans.
  • Le changement climatique affecte déjà toutes les régions habitées de la planète. Les preuves de l’attribution des phénomènes météorologiques et climatiques extrêmes à l’influence humaine se sont renforcées depuis le dernier rapport.
  • À moins d’une réduction immédiate, rapide et à grande échelle des émissions de gaz à effet de serre, limiter le réchauffement à 1,5°C sera hors de portée.
• Traduction par des bénévoles du résumé pour décideurs du rapport du GIEC. Beau boulot !
• Anaïs Rocci, sociologue à l’ADEME : « l’objectif de la sobriété est de trouver un modèle de société qui permette à la fois de respecter les limites des ressources planétaires et à chaque personne de vivre décemment ». Une excellente lecture !
• Petite fierté du moment : voir mon podcast l’Octet Vert listé comme ressource pour la conférence Reboot 2021, qui vise à « hacker les métiers de la com » ;

The post Apple releases OS updates to fix spyware hack uncovered by University of Toronto researchers appeared first on MobileSyrup.

Canada has a public health system – but it has some strange gaps. This may be because we are so close to the US, which has a completely broken “free market” system. We can see a doctor for free, and if we go to a hospital and need treatment there that is free too. But if we need medication and don’t need hospitalization then we have to pay for the treatment. Many people get additional coverage through health insurance for things like drugs, dentistry and spectacles (eyeglasses) that are not covered by the public system. We also have to pay for anything that the eye doctor might prescribe – and my particular plan does not cover things like eye drops.

The last time I went to see the opthamologist she suggested that I get eye drops for Dry Eye. In fact she gave me a sheet of recommended products – not just eye drops but a hot compress and even an Omega 3 supplement. Now that did surprise me. Because one of the things that we keep track of in this household is the effectiveness of nutritional supplements. Most of them have not been tested in the same way that medicines are, to show that they are both safe and effective. And frankly a lot of the nutritional supplements are not necessary, if you have a balanced diet. But I do have to report that since taking Vitamin B 100 once daily the irritating rash on my legs has cleared up completely.

But is it also the case that you can find out for yourself if the nutritional supplement actually has any benefits at all. It is claimed, on the document from my eye doctor, that Omega 3 “helps decrease inflammation, promotes good health and helps dry eyes”. I find that surprising since the National Institutes of Health said “Omega-3s from fish oil supplements are no better than placebo for dry eye. NIH-funded study finds omega-3 fails to yield beneficial results in the clinic.”

“The trial provides the most reliable and generalizable evidence thus far on omega-3 supplementation for dry eye disease,” said Maryann Redford, D.D.S., M.P.H., program officer for clinical research at NEI. Despite insufficient evidence establishing the effectiveness of omega-3s, clinicians and their patients have been inclined to try the supplements for a variety of conditions with inflammatory components, including dry eye. “This well-controlled investigation conducted by the independently-led Dry Eye Assessment and Management (DREAM) Research Group shows that omega-3 supplements are no better than placebo for typical patients who suffer from dry eye.”

This was published on line in the New England Journal of Medicine, April 13, 2018.

In the current Canadian Election campaign the New Democrats are suggesting that if they become the government they will include eyes and teeth in “Medicare” (which is actually the name of the US system, not ours) – but only for households with incomes under $90,000 a year, which indicates for me that they do not understand the meaning of free universal healthcare for all which is what we were supposed to be getting … in due course, I suppose.

I think is says a lot about North America when a country like Cuba has a better healthcare – and education come to that – than either Canada or the US.

Charging people $55 for Omega 3 which is no better than placebo is a pretty sharp practice, I think. But it seems to be common. After all, the optometrist who did my most recent eye test also said that I should take Omega 3 – but at least he didn’t try to sell me anything.