tsv-utils

Powerful collection of CLI tools for processing TSV files, written in D for performance and released by eBay. Includes a csv2tsv conversion tool. You can download an archive of pre-built binaries for Linux and OS X from their releases page: worked fine on my Mac.

Via @jeffsonstein

sillygwailo shared this story from Toronto – Streetsblog USA.

In crowded urban areas, cars aren’t the most efficient way to move people. That’s the lesson from Toronto’s one year King Street “pilot,” which prohibited through car traffic on the high-ridership streetcar corridor.

New data from the one-year pilot experiment shows the King Street corridor is moving more people than it did before the city cleared cars out to the way. That’s because removing car traffic from King Street improved streetcar service so much, ridership has grown 16 percent.

A total of about 84,000 people are using the King Street Streetcar along the east-west corridor daily, the city reports. But only 72,000 people were using the roadway before the street was redesigned to prioritize the streetcar.

The data comes from new report from the city’s transportation staff recommending the corridor design be made permanent. The city estimates it will cost just $1.5 million to make the improvements permanent, providing a dedicated lane for existing high-capacity surface transit one of the best transit investments available to cities.

Not only is the streetcar moving more people, thanks to its dedicated lane, riders are happier with service as well. The King Street Pilot resulted in about 30,000 minutes saved by riders daily, the city reports. It was especially effective reducing major delays. The slowest average journeys — at the evening rush hour — decreased by five minutes.

“Prior to the pilot, overall customer satisfaction with King streetcar service was low on key measures such as travel time, comfort, and wait time,” city staff writes. “Through the pilot period, customer satisfaction on all these measures have significantly improved.”

Cycling rates have also increased. There are about 380 more daily cyclists on the corridor at the afternoon peak, according to the city, “likely because reduced motor vehicle volumes made it more comfortable to cycle.”

Drivers haven’t been duly inconvenienced, the data show. Car travel times on parallel corridors is roughly the same as before the pilot began in January, 2017.

The pilot did show that retail sales were down very slightly, about 0.8 percent, most notably from restaurants. But the city says that reflects a trend that was underway before the pilot. There were a number of very outspoken business owners that opposed the pilot. But, overall, at least according to the city’s data, the experiment does not appear to have been the retail apocalypse some predicted and was very positive for tens of thousands of daily streetcar riders.

Function composition is taking the return value of one function and passing it as an argument to another function. It’s common enough that functional programmers have turned it into its own operation. In this episode, we go deep into why it’s important and how you can use it and write it yourself.

Transcript

Eric Normand: What is function composition? By the end of this episode, you will know what function composition is and why it’s useful. My name is Eric Normand, and I help people thrive with functional programming.

This is an important topic because it’s something we do all the time. We do function composition in both functional and non-functional styles of programming.

Functional programmers and mathematicians have named it, they’ve named it functional composition, and they’ve turned it into a higher order function. It’s something that we tend to talk about probably more than you would in a non-functional language.

What is it? It’s pretty simple. It’s when you take a function and you call it, and then you take the return value of that function and you pass it to a second function as an argument. You’re chaining the function calls.

Usually, in math or as examples, we use function f and g. You call function g, you get the result. You take that result and you pass it to function f. Of course, you get the result of function f.

There’s a special case because sometimes you’ll need some other arguments, like f will take it as the second argument. It will take the result of g as the second argument.

There’s a special case where you take that return value of g and you pass it as the first and only argument to f.

That’s a special case because it’s very simple and easy to reproduce in your code that you can just do f, open paren…I’m using JavaScript syntax, f, open paren, g, open paren. You put the arguments to g, and then you close the paren and you close the paren of f.

Now, the g is the…You don’t even need to name that return value, you don’t need to save it into a variable. You pass it directly to f.

If you take this special case — because it’s so regular — you can turn it into a higher-order function. That is, a function that takes f and g and returns a new function that does that same thing, that does the f open paren g.

Mathematicians usually use the dot as the operator for a function composition when they want to talk about it in a paper or something. They’ll write, “f.g,” to mean f composed with g. This gets confusing, and it’s one of the reasons why function composition is thought of as difficult.

The confusing part is that the g is called before f, even though it comes to the right of f. If I do f.g, the g is called first, and then the f is called on the result of g. It looks backwards. It looks like, “Call f,” and then, “Call g,” but the g is called first. That makes it confusing.

The reason it is that way is, like I said, it’s like doing f open paren g, open paren, put the args of g, and then close the parens. They’re in the same order as if you were to call it directly, but for clarity, you might want to move the g out of there, because it’s inside out.

The arguments get evaluated first before the function gets called on those arguments. If you move the g above there, and you called it g result equals g, and then you called it with its arguments, and then after that, you did f open paren g result…

I’m trying to talk through code, but the idea is that it’s in the same order as something you should be familiar with. It’s just when you look at it on it’s own and that’s not explained to you, it looks backwards.

Why do we talk about this? Why do we want to turn this into a higher-order function?

First of all, it’s very common to do this. It’s common to call a function and pass that argument to the next one. Even though it’s less common than the general case, the special case of the single argument is still really common.

Let’s say you wanted to double a number and then square it. You could double it, so you call the double function, then you take the result of that and you pass it to the square function, which then returns the value.

You could say, “This is a function composition,” and you can call compose with the two functions. That gives you a new function, which is the double square.

What it does is it reduces boilerplate. By reifying it…I have a whole episode on reification if you want to know what that is. I also have a whole episode planned for higher-order functions, if you want to know more about that.

You’re taking this idea that normally you’re just doing code by embedding the call to g as an argument to f. You’re now turning it into an object, a function, called compose, that can do that. Now, this thing can be manipulated like any other function, like any other value.

This compose function is not just a bit of syntax that your language understands. It’s a function that your code can understand, that your program can use. Then you can pass it to other higher-order functions like map, and other stuff. You’re getting leverage by turning it into a first-class thing.

I do want to say that it’s not always clearer to do this, to use compose. It’s something that people do use, though, and so, if you’re doing functional programming, you should probably understand it. It’s something that you’ll see a lot.

One place where it really is used a lot is in a thing called point-free style. I have a whole episode planned on point-free style. Basically, point-free style is very simple, is a way of defining functions without naming the arguments.

I could write a function. I call it fg, or f.g, that’s the name of the function. It takes arguments, those arguments get passed to g, and then g will return a value. That return value will get passed to f, and that return value f will get returned from f.g.

I can write that function, and I can write it over and over again for different fs and gs. I could write it for square and double, and I can write it for trim and capitalize on strings.

There’s all sorts of examples that I could write by hand each time, or I could call compose, and not have to name the arguments, not have to repeat that boilerplate.

This point-free style is a way of building functions without naming arguments. To be very frank, arguments take up a lot of brain space, if you have to name the argument, you have to think of a good name, otherwise, it’s going to be worse than not having one, and they take up code space.

You don’t want to be naming them if you don’t have to if it’s not going to add to the clarity.

There’s this thing called point-free style. Function composition is used a lot in it because you don’t have to name the arguments.

Another cool thing about function composition is that it is associative and it has an identity. There’s a whole episode on what an identity is. Notice if I do f composed with the identity function…

The identity function is just a function that returns its argument. You pass in the number five, it returns the number five. You pass in the string “Hello,” it returns the string “Hello.” It doesn’t do anything to it.

You think, “What’s the point of this function that doesn’t do anything to it?” It’s just like, “What’s the point of zero?” It represents nothing. Representing nothing is actually very useful, and so is representing a function that does nothing. [laughs]

It’s now able to be the identity value of function composition, so f composed with identity is the same as f. Identity composed with f is the same as f. The identity function is the identity value of function composition. You might want to write that down because it’s a mouthful.

It’s associative, which I have a whole episode on that. Real quick, it means the grouping doesn’t matter.

Imagine I have f composed with g, composed with h. I can put the parentheses around the f and the g, or I can put the parentheses around the g and the h. I get the same thing. I get the same result. The function at the end of the day that I get from that whole big expression is going to do the same thing.

When you have something that’s associative, and it has an identity, that means it is a monoid. I have an episode planned on monoids and what makes them cool.

That’s just a cool thing I wanted to mention at the end. If you don’t understand it, just wait for that episode and it will come.

This is the kind of thing that happens when you start dealing with functions as first-class objects, especially higher-order functions. You can start to get these kinds of cool algebraic properties that come out.

I just know when I’ve talked about this in the past, the really common question is, “Why do I need this? Why do you need it?” The truth is you don’t need it. You can do manual function composition each time.

You can, but then the other question is, “Why do you need anything?” You can always just do for-loops and gotos if you want if your language has gotos these days.

Why do you use map instead of for-loop? Basically, if you can use a map, map is way less error-prone than a for-loop. You don’t have to initialize any variables. You don’t have to get the end condition right.

If you can use a map, it’s probably less code than a for-loop. That’s not really what matters. What matters is that it’s less error-prone. The cool thing is that once you have map, map is a function. Map is something that can now participate in all these other higher-order functions.

Same with compose. You could do it manually, but then you don’t have a function called compose that you can use in other ways.

This is part of, I would call it, the leverage of functional programming. Just by turning this thing that you do a lot into a function that can now participate in the rest of the ecosystem, you now have this leverage.

For now, you’re acting in this higher level. Now, you’re composing compose. You’re mapping compose over things, which you couldn’t do before. You could do it, but you’d have to write out compose each time you wanted to.

Let me recap. Function composition is simply when you take the return value of one function and pass it as the argument to the next function. You can imagine making a function that just does that. You’ll realize that you do that over and over, that there’s a common pattern.

You could just make a function that takes the two functions and returns that common pattern function. That one is called function composition. It’s the special case.

Mathematicians use a dot. Even in Haskell, probably other languages, they use a full stop or the period character to indicate function composition, just because it looks like a dot. Because it’s so easy to do, it’s just a dot, you do it a lot.

It reduces boilerplate and then it reifies this concept into something that now you can leverage in with the rest of your functions. It’s used a lot in point-free style and it’s a monoid. It has an identity and it is associative.

At the end, I’d like to give you, I think of it like homework or something that would help you use this knowledge that I’ve just talked about. I think that it would be a good learning experience to write your own implementation of function composition. It’s not hard. Like I said, it’s just f, open paren g in JavaScript.

You need to write a function called compose, let’s say, that takes two functions as arguments, does the composition and returns the return value of them. It returns a new function that does the composition of the two.

You can look it up if you have to. If your language has first-class functions, it is really like a three-line function. It is nothing big. You don’t have to do the full-blown, industrial-strength one that can handle multiple arguments and things like that.

Just a function of one argument, composed with another function of one argument. Just try that out. That will really help you understand what it’s doing and why they’re in the order they are.

That was a favor for you. Go ahead, do that for yourself. Learn a little bit deeper than what you can in a podcast.

I’m going to ask a favor for myself. If you liked this episode, please share it with your friends and please subscribe. If you share it with your friends, then it’s just another thing to talk about with them.

You probably have coworkers who would like to know a little bit more about this functional programming stuff. If you subscribe, then you will get the next episode, which also has good stuff that you’ll probably like if you liked this one, because that’s how it is. I will have more stuff coming.

Thank you very much, and see you next time.

The post What is function composition? appeared first on LispCast.

Too tired to write much but I’ll note that I saw Handel’s Saul last night (Philharmonic Baroque) and then today, the all Bach concert at the Legion of Honor. Saul was great. I loved the music, (weird box of squirrels carillion! solemn and stately trombones! bassoons!) the choir was just brilliant, countertenor’s amazing voice gave me shivers, and I got an instant crush on the soprano singing Merab. Then as she was walking off stage and past my seat, all I can think is I must have been gazing at her adoringly or with a giant goofy grin, because quick as lightning she pulled a hilarious face and stuck her tongue out at me and then was all normal again. Sometimes it is the little things in life that are the most delightful!

The Wikipedia article on Saul quotes a letter describing a fabulous new instrument (the sackbut aka trombone):

He has also introduced the sackbut, a kind of trumpet,with more variety of notes,& it is 7 or 8-foot long,& draws in like a perspective glass, so may be shortened to 3-foot as the player chuses, or thrown out to its full length; despise not this description for I write from his own words.

Bits that I marked with a little star in the program: the opening hallelujah stuff, the duet between David and Michal, Merab in act 2 scene 9 singing “Author of peace, who canst control” and then the chorus “Oh fatal consequence” at the end of act 2. After that I stopped marking althought I certainly waggled my eyebrows thoughtfully when David sings “Great was the pleasure I enjoy’d in thee, and more than women’s love thy wondrous love to me!”

We must also note the excellence of the Witch of Endor’s vampirish cape.

I don’t have the right vocabulary to describe it, but there were glorious moments where the men of the chorus would start a sort of descending … cascade or something… in a minor key. Like in the end of act 1 where the chorus sings 2 lines for a few minutes. “Preserved him for the glory of thy name, thy people’s safety and the heathen’s shame” – I wished it would go on forever – I went into a complete trance. So great & complicated all at once.

It is sort of a libretto about toxic masculinity, like, at first i wanted it to be a bit more of a mirror for princes – don’t let paranoia drive you as a ruler, don’t attack the strongest allies you have out of fear or jealousy – But then pondering how even our paragons of rationality and control are mostly admired for their ability to deal out death, including gory details of how their swords reek from the entrails of the slain and their arrows always drink the blood of their victims, etc. Gross!

The Bach program today was pleasant and I liked the Coffee Cantata – very silly and adorable – the soprano was good, Shawnette Sulker. Afterwards wandered around the museum a little bit with Lisa but she had to leave. (But we did get to just stand around in front of The Russian Bride’s Attire. It’s really something! It would be amazing to cosplay with a big group. I call being the chick in the corner with the trash can hat!) I stayed a little longer and went to the Rubens exhibit by myself.

Why is it so damn hard to get over to that part of town on the bus? It took nearly 2 hours. The 38 Geary is hella slow.

Got some groceries & then crossed the street to get flowers from the woman who sets up in front of the abandoned liquor store. As always, I tried to choose flowers myself, but failed.

Flower lady: When you come to me you can get whatever you like, just tell me which ones…
Me: Um, some of these roses and these purple things and the purple thistles.
Flower lady: Don’t touch them, I will pick. It will ruin them.
Me: OK.
Flower lady: The thistles don’t go. You need this one. And some greens. Some leafs.
Me: Just the flowers is ok.
Guy with a chihuahua and groceries: You want to buy it?
Flower lady: But I don’t know if it works. No batteries.
(Lengthy conversation in Spanish, the guy is trying to sell her a floodlight)
Me: you need to sell stuff but people want to sell you stuff!
Flower lady: Tell me about it! Every day! Well, It’s good to buy from the little people. Like me. Because you can get what you like. Whatever you like! You choose!
Me: So…. Just the flowers then. Great thanks. I love them. (My back is hurting… i want to leave….)
Flower lady: I will choose for you. The greens will invigorate. Then something to make it pop. Here (stripping leaves brutally off the stems) I will arrange for you.
Me (resigned): Ok… you’re the expert! Lovely! Yes! Great! Thanks! Looks great!

*** A MILLION YEARS PASS WHILE SHE ARRANGES THE FLOWERS ***

Me: The thing is I have a pretty tall vase.
Flower lady: (does incomprehensible things)
Me: Very symmetrical! The yellow and orange roses look like flames in the middle!
Flower lady (giving me bay leaves): Put these leaves in the bath. Good for you.
Me: Thanks, I love bay leaves
Flower lady: And like this. (it starts to look super fancy!!!)
Me: Wow. OK!
Random white guy: I want one like that.
Me: Right? It’s nice.
Guy: I have bay leaves on my ranch and I like to just crush one and have it in my pocket.
Flower lady: I will make one for you.
Guy: Something something, guatemala.
Flower lady: Oh so! You are from Guatemala!
Guy (in spanish) No I just have a 2nd house there
Flower lady (in spanish): And a second woman and some extra children who look like you.
Me: *cracks up*
Guy (super uncomfortable): No! oh, no! My girlfriend is in the grocery store across the street!
Me: You better get her flowers, man!
Flower lady: Don’t let her know about the other wife in Guatemala. Where is your girlfriend? She can pick!
Me: She’s in the grocery store, he’s gonna surprise her.
Flower lady: I will give you this extra rose, for your bedside. The rose is INVIGORATING. It gives energy. You will keep it by the BED.
Me: Oh thanks. I actually have a little vase by my bed so , perfect
Flower lady: *starts to arrange more things with the extra rose*

*** NINETEEN MILLION HOURS LATER ***

I have a giant armful of flowers and am trying to balance it as I zoom down Cortland at like 15 miles an hour on my travelscoot driving it one-handed with a bag of groceries between my feet.

Dave Cormier calls on all of use to help build a better internet. "Please participate. Do it well. Put your values on the internet. Our society is literally being shaped by the internet right now, and will be for the foreseeable future." I'm all for that. But as I commented: " No amount of human posting will be sufficient to counter bot-driven (or mechanical-turk-driven) counter-content. No amount of careful contributions will turn off the surveillance and targeted advertising." And that's the conundrum we face.

I've had a podcast for many years - here it is. It's not a nice neat podcast with weekly episodes and such - that's far too organized for me. But any time I give a talk I record the audio and throw the result into my podcast, and you can use the RSS feed to subscribe to it in your favourite podcatcher. All this by way of saying that I'm invested in the podcast genre, and it's something that resonates with me. This article points to and documents therise of the medium. "The popularity of special interest audio/video podcasts has grown steadily among both sexes and all age groups.  With more than half of the world’s population now online, the breadth of audio and video shows capturing audience attention continues to grow." Podcasting will in the future be increasingly challenged by content silos with paywalls. I hope it survives.

A scientist who doesn’t publish dies in obscurity. Scientists know publishing results is simply part of the job and give it the time it deserves.

Community organizers (those in your local neighborhood) can spend as much time proving they had an impact as doing the work.

They know success is only one half of their job. The other half is proving their success.

It doesn’t matter if you do a good job this month if your funding is cut next month.

Failure to gain internal support kills a community as effectively as managing one badly.

The best community practitioners know this and spend 40% to 50% of their time (and budget) gaining and sustaining this support. They build sustaining support into their strategies and weekly plan of action.

Imagine the difference if you:

• Proactively schedule meetings with those who both support and don’t support the community each week to better understand their needs and align the community towards their goals.
• Systematically collect emotive community stories each week which you can drop into every conversation about the community.
• Take responsibility for measuring the community’s impact to world-class standards (not dubious call deflection or correlation metrics, but actual studies which demonstrate the statistically significant impacts of the community).

Much of the reason communities struggle to gain support is community managers don’t treat the process seriously enough. They don’t give it the time and budget it deserves.

No job description is going to read ‘you must persuade us not to kill the community each month’, but squint and read between the lines a little and it’s there.

And today another quick post on Eduroam, the federated Internet access authentication solution for students and researchers.

When I recently noticed that one of the certificates in the certificate chain I put together for an Austrian Eduroam user I support would expire early next year, I set out to do some preventive maintenance and put together a new chain. Quite to my surprise it seems to be no longer necessary!

According to their updated descriptions, their latest certificates are signed directly by the Digicert public root certificate and hence, no certificate chain is necessary anymore. What’s even better is that this public key is part of Linux’ standard SSL package so there is no need to get a key/certificate from anywhere and install it. The ‘Windows’ support pages of the Vienna Eduroam setup makes this quite clear but unfortunately the Linux version has not been updated. A bit of a shame.

But anyway, if you have a Linux device that you have configured yourself for Eduroam access with a certificate chain, check its validity when you have a minute. In case of the University of Vienna, this is the root certificate to use:

/etc/ssl/certs/DigiCert_Assured_ID_Root_CA.pem

I quickly had a look at which certificates other institutions use for their Eduroam setup and the two I looked at also get their setup signed by public root certificates. In the case of the University of Cologne, they even asked their users to update their configuration as soon as possible, as their previous certificate setup would stop working soon.

Deutsche Welle: Auschwitz-Überlebender Mano Höllenreiner: “Wir waren ja Deutsche!” “Sinti und Roma sind die größte Minderheit Europas. Im Nationalsozialismus wurden Hunderttausende ermordet. Mano Höllenreiner überlebte als Kind. Er warnt vor dem Vergessen. Andrea Grunau hat ihn besucht.”

“Einerseits würde er gerne alles vergessen, “aber es kommt immer wieder”, sagt Höllenreiner. Seine Frau berichtet von Alptraum-Nächten, in denen er fast unmenschliche Schreie ausgestoßen habe. Ihr Mann schaut sie dankbar an: “Sie hat viel mitgemacht mit mir.” Mano Höllenreiner setzt sich aktiv gegen das Vergessen ein. Viele Vorträge hat er gehalten, oft vor Schulklassen, damit “die jungen Deutschen wissen, was wir im KZ mitgemacht haben und dass es ein Verbrechersystem war”. Manche Schüler hätten geweint, manche geklatscht, ein Mädchen habe ihn schluchzend umarmt. “Es gibt gute Deutsche”, hält er fest. Für seine Aufklärungsarbeit erhielt er das Bundesverdienstkreuz.”

Der 8. April ist Internationaler Tag der Roma.

Als vor nicht einmal drei Wochen der neue Einsteiger-Kindle zur Vorbestellung für 80 Euro angeboten wurde, habe ich geraten, die Füße still zu halten und erst mal abzuwarten. Und nun gibt es genau zu diesem Preis den Kindle Paperwhite, der ein viel höher auflösendes Display hat. Jetzt ist der richtige Zeitpunkt zu kaufen.

Auch die Echos, FireTVs etc. sind diese Woche im Angebot.

More >

It appears Apple will break up iTunes into separate apps when it releases macOS 10.15 later this year.

Noted Apple developer Steve Troughton-Smith tweeted on Friday afternoon that he found evidence that suggests Apple is developing new UIKit-based Music, Podcasts and Books apps for macOS. This is in addition to the TV app Apple is also reportedly developing.

And yes, this means the much-discussed and long-awaited break up of iTunes. Finally!

— Steve Troughton-Smith (@stroughtonsmith) April 5, 2019

As Troughton-Smith notes, the move is likely part of Apple’s ongoing Marzipan initiative. With macOS Mojave, the company introduced Mac Stocks, Home, Apple News and Voice Memos apps that were based on iOS counterparts. With macOS 10.15, Apple is expected to introduce a software development kit that allows third-party developers to create cross-platform macOS and iOS apps.

Apple will preview macOS 10.15, as well as iOS 13, at its annual developer conference, WWDC. This year’s Worldwide Developer Conference will start on June 3rd.

Source: Twitter Via: MacRumors

The post Apple to break up iTunes with separate Music, Podcasts macOS apps: report appeared first on MobileSyrup.

Researching “the internet” (i.e. $DAYJOB) means having to deal with a ton of “unique” (I’m being kind) data formats. This is ultimately a tale of how I performed full-text searches across one of them.

It all started off innocently enough. This past week I need to be able to do full-text searches across metadata about who is using which parts of the internet. Normally I don’t need to do that at scale and can just go to RIPE’s excellent resource and manage to find what I need on the first page. However, this time I needed all the resultant info and noticed an interesting foible on that full text search interface. To reproduce it. Enter something like “domino's” (for the record, I’m not researching Domino’s Pizza — nor would I ever consume it — but a Twitter ad happened to fly by for Domino’s and I just typed it for kicks) into the field and page around, keeping an eye on the results. I think they still use Solr for indexing/searching and aren’t passing in all they need to keep session context or something. Anyway, suffice it to say it was fairly useless (I filed a bug report, so I’m not just complaining, and I wish more sites had the same easy error reporting filing capability the RIPE folks do).

If it were just searching for precise data in one field, that’s not really an issue since we have ALL THE WHOIS IP THINGS in Parquet. But:

• I really hate giving Amazon money (even if it’s $WORK money) for Athena queries
• Full text search across all columns is not one of Parquet’s strengths
• This is a third bullet b/c I feel compelled to have a minimum of three points in bullet lists likely thanks to an overbearing middle-school English teacher

Since I have a modest analytics server setup at home, I figured I’d take the opportunity to re-brush-up on either Elasticsearch or Couchbase since both are pretty great at free text searching JSON data. Except…this isn’t JSON data, It’s records formatted like this:

#
# The contents of this file are subject to 
# RIPE Database Terms and Conditions
#
# http://www.ripe.net/db/support/db-terms-conditions.pdf
#

as-block:       AS7 - AS7
descr:          RIPE NCC ASN block
remarks:        These AS Numbers are assigned to network operators in the RIPE NCC service region.
mnt-by:         RIPE-NCC-HM-MNT
created:        2018-11-22T15:27:05Z
last-modified:  2018-11-22T15:27:05Z
source:         RIPE
remarks:        ****************************
remarks:        * THIS OBJECT IS MODIFIED
remarks:        * Please note that all data that is generally regarded as personal
remarks:        * data has been removed from this object.
remarks:        * To view the original object, please query the RIPE Database at:
remarks:        * http://www.ripe.net/whois
remarks:        ****************************

as-block:       AS28 - AS28
descr:          RIPE NCC ASN block
remarks:        These AS Numbers are assigned to network operators in the RIPE NCC service region.
mnt-by:         RIPE-NCC-HM-MNT
created:        2018-11-22T15:27:05Z
last-modified:  2018-11-22T15:27:05Z
source:         RIPE
remarks:        ****************************
remarks:        * THIS OBJECT IS MODIFIED
remarks:        * Please note that all data that is generally regarded as personal
remarks:        * data has been removed from this object.
remarks:        * To view the original object, please query the RIPE Database at:
remarks:        * http://www.ripe.net/whois
remarks:        ****************************

They “keys” (the colon-ified line prefixes) vary and there are other record types (which I don’t need) that have other prefixes in them plus those #-prefixed comments are not necessarily only at the top. But, after judicious use of stringi::stri::stri_enc_toutf8(), stringi::stri_split_regex() and some vectorized record targeting they’re pretty easily converted to lovely ndjson data like this (random selection further in the conversion):

{"descr":"Reseau Teleinformatique de l'Education Nationale Educational and research network for Luxembourg","admin_c":"DUMY-RIPE","as_set":"AS-RESTENA","members":"AS2602, AS42909, AS51966, AS49624","mnt_by":"AS2602-MNT","notify":"noc@restena.lu","tech_c":"DUMY-RIPE"}
{"descr":"CWIX ASes announced to EBONE","admin_c":"DUMY-RIPE","as_set":"AS-TMPEBONECWIX","members":"AS3727, AS4445, AS4610, AS4624, AS4637, AS4654, AS4655, AS4656, AS4659 AS4681, AS4696, AS4714, AS4849, AS5089, AS5090, AS5532, AS5551, AS5559 AS5655, AS6081, AS6255, AS6292, AS6618, AS6639","mnt_by":"EBONE-MNT","notify":"staff@ebone.net","tech_c":"DUMY-RIPE"}
{"descr":"ASs accepted by DFN from the University of Cologne","admin_c":"DUMY-RIPE","as_set":"AS-DFNFROMCOLOGNE","members":"AS5520 AS6733","mnt_by":"DFN-MNT","tech_c":"DUMY-RIPE"}
{"descr":"NetMatters UK","admin_c":"DUMY-RIPE","as_set":"AS-NETMATTERS","members":"AS6765 AS3344","mnt_by":"AS8407-MNT","tech_c":"DUMY-RIPE"}

I went with Couchbase since it handles ndjson import by default and — as you know since you read the comparison in the aforelinked article — it can easily index all fields by default without you having to do virtually anything. Plus, Couchbase has been around long enough that it generally installs without pain and has a fairly decent web admin panel. Here’s a snapshot of the final import:

and here’s the config for the “all” full text index:

{
  "type": "fulltext-index",
  "name": "all",
  "uuid": "481bc7ed642dddfb",
  "sourceType": "couchbase",
  "sourceName": "ripe",
  "sourceUUID": "3ffbbe0c0923f233ffe0fc96c652262d",
  "planParams": {
    "maxPartitionsPerPIndex": 171
  },
  "params": {
    "doc_config": {
      "docid_prefix_delim": "",
      "docid_regexp": "",
      "mode": "type_field",
      "type_field": "type"
    },
    "mapping": {
      "analysis": {},
      "default_analyzer": "standard",
      "default_datetime_parser": "dateTimeOptional",
      "default_field": "_all",
      "default_mapping": {
        "dynamic": true,
        "enabled": true
      },
      "default_type": "_default",
      "docvalues_dynamic": true,
      "index_dynamic": true,
      "store_dynamic": false,
      "type_field": "_type"
    },
    "store": {
      "indexType": "scorch",
      "kvStoreName": ""
    }
  },
  "sourceParams": {}
}

You Said This Is A Post With R Code

Very true! We’ll get to that in a minute.

Going with Couchbase introduced a different problem: there’s almost no R support for Couchbase. Sure, Couchbase has a gnarly, two-year old, raw httr::-prefixed bit of a tutorial post but that’s not really as cool as if there were a library(couchbase). I mean, you can check GitUgh or CRAN or a more general search yourself if you’d like but it’s going to come up bupkis.

If you were expecting a big reveal, right now, that I’ve got a feature-packed, full R Couchbase package ready to roll…you didn’t actually read the title of the post. What I do have is a set of functions that — given server/connection metadata, a bucket, a full text index, and a query — will return all matching documents (I still do not like that term for “record”) for said set of parameters:

# function code is in: https://paste.sr.ht/~hrbrmstr/051f5d5400644952a3ad2cf8664b84e2cbb9ac6b

cb_fts("domino's", "all", "ripe")
## # A tibble: 120 x 9
##    admin_c   country descr                      inetnum                  mnt_by      netname  status    tech_c  notify         
##    <chr>     <chr>   <chr>                      <chr>                    <chr>       <chr>    <chr>     <chr>   <chr>          
##  1 DUMY-RIPE FR      OPEN IP DOMINO'S PIZZA     79.141.8.44 - 79.141.8.… ALPHALINK-… OPEN-IP  ASSIGNED… DUMY-R… NA             
##  2 DUMY-RIPE NL      Domino's Pizza TILBURG     62.21.176.160 - 62.21.1… AS286-MNT   OTS2634… ASSIGNED… DUMY-R… ip-reg@kpn.net 
##  3 DUMY-RIPE NL      Domino's Pizza EINDHOVEN   62.132.252.168 - 62.132… AS286-MNT   OTS2270… ASSIGNED… DUMY-R… ip-reg@kpn.net 
##  4 DUMY-RIPE NL      Domino's Pizza SPYKENISSE  194.123.233.232 - 194.1… AS286-MNT   OTS69259 ASSIGNED… DUMY-R… ip-reg@kpn.net 
##  5 DUMY-RIPE NL      Domino's AMSTERDAM         37.74.38.188 - 37.74.38… AS286-MNT   OTS6103… ASSIGNED… DUMY-R… kpn-ip-office@…
##  6 DUMY-RIPE NL      Domino's Pizza VOORSCHOTEN 92.66.116.136 - 92.66.1… AS286-MNT   OTS1914… ASSIGNED… DUMY-R… ip-reg@kpn.net 
##  7 DUMY-RIPE NL      Domino's Pizza Doetinchem… 212.241.42.136 - 212.24… AS286-MNT   OTS2301… ASSIGNED… DUMY-R… ip-reg@kpn.net 
##  8 DUMY-RIPE NL      Domino's Pizza AMSTERDAM   194.120.45.224 - 194.12… AS286-MNT   OTS82906 ASSIGNED… DUMY-R… ip-reg@kpn.net 
##  9 DUMY-RIPE NL      Domino's Pizza [Woerden] … 62.41.228.80 - 62.41.22… AS286-MNT   OTS2024… ASSIGNED… DUMY-R… ip-reg@kpn.net 
## 10 DUMY-RIPE NL      Domino's Pizza GRONINGEN   188.203.128.0 - 188.203… AS286-MNT   OTS3767… ASSIGNED… DUMY-R… kpn-ip-office@…
## # … with 110 more rows

It’s not fancy.

It’s meets the needs of a narrow use-case.

It’s not in a standalone package (which is triggering my R code OCD something fierce).

But, it’s seriously fast, got me back to “work mode” with a minimum of hassle, and now there’s some google-able Couchbase R code that isn’t just bare httr calls that may help someone else who’s on a quest for how to work with Couchbase in R.

The first primary function – cb_fts() — uses the /api/index/{index-name}/query API endpoint to paginate through results of the full text search and retrieves all matching record doc id keys, then calls the last primary function — cb_get_records_from_keys() — which uses the /query/service API endpoint, issues a SELECT * FROM {bucket} USE KEYS {keys} query with all the found document (record) key ids and returns the result set. Nothing more fancy than that.

FIN

While I do not have these functions in a standalone, Couchbase-focused package I do have them in the package associated with this particular project. If you do know of a Couchbase R package (please don’t link to JDBC/ODBC drivers as I’m not going to buy) please link to them in the comments.

If you have other strategies for how to deal with these “un-packages”, please blog about it and post a link as well! I’m curious how others balance the package/not-a-package/un-package tension, especially when you may need to depend on a series of functions across projects.

I finally purchased the Apple Pencil! Drawing zines is so much more fun now; I can easily erase, copy or move around the objects. All the drawings on this post are drawn on an Ipad Pro using the Apple Pencil.

Recently, I was invited to be on an online panel discussion on ‘Open Source Maintainer –  Women WILL Engineer the Future‘. Thank you Emma, for having me. Emma had compiled interesting questions; and a lot of thoughts, ideas and insights were exchanged during the course of 1 hour.

I will try to summarise the key take-aways for each question via a zine.

Q1: What is it like to be a Maintainer?

Q3: How can Open Source projects get better at Meritocracy?

Q3: How can projects encourage women in Open Source Leadership roles?

Q4: How does the Code of Conduct influence Open Source participation?

Q5: What are the challenges faced by women in Open Source?

Q6: What are the rewards of participating in Open Source?

Faruk macht einfach wunderbare Videos. Vor allem scheut er sich nicht, schlechte Produkte zu versenken.

mkalus shared this story from Toronto – Streetsblog USA.

In crowded urban areas, cars aren’t the most efficient way to move people. That’s the lesson from Toronto’s one year King Street “pilot,” which prohibited through car traffic on the high-ridership streetcar corridor.

New data from the one-year pilot experiment shows the King Street corridor is moving more people than it did before the city cleared cars out to the way. That’s because removing car traffic from King Street improved streetcar service so much, ridership has grown 16 percent.

A total of about 84,000 people are using the King Street Streetcar along the east-west corridor daily, the city reports. But only 72,000 people were using the roadway before the street was redesigned to prioritize the streetcar.

The data comes from new report from the city’s transportation staff recommending the corridor design be made permanent. The city estimates it will cost just $1.5 million to make the improvements permanent, providing a dedicated lane for existing high-capacity surface transit one of the best transit investments available to cities.

Not only is the streetcar moving more people, thanks to its dedicated lane, riders are happier with service as well. The King Street Pilot resulted in about 30,000 minutes saved by riders daily, the city reports. It was especially effective reducing major delays. The slowest average journeys — at the evening rush hour — decreased by five minutes.

“Prior to the pilot, overall customer satisfaction with King streetcar service was low on key measures such as travel time, comfort, and wait time,” city staff writes. “Through the pilot period, customer satisfaction on all these measures have significantly improved.”

Cycling rates have also increased. There are about 380 more daily cyclists on the corridor at the afternoon peak, according to the city, “likely because reduced motor vehicle volumes made it more comfortable to cycle.”

Drivers haven’t been duly inconvenienced, the data show. Car travel times on parallel corridors is roughly the same as before the pilot began in January, 2017.

The pilot did show that retail sales were down very slightly, about 0.8 percent, most notably from restaurants. But the city says that reflects a trend that was underway before the pilot. There were a number of very outspoken business owners that opposed the pilot. But, overall, at least according to the city’s data, the experiment does not appear to have been the retail apocalypse some predicted and was very positive for tens of thousands of daily streetcar riders.

The war against failed package deliveries continues, and although it’s been almost two months since it began, it’s still a work in progress. I’ve been doing a lot more stuff (including rebuilding my home infrastructure and running some cloud benchmarks), but a recent delivery miss bumped this back to the top of the order of business, so today I spent a couple of hours tinkering with electronics.

Current Ammunition

Last time I glued a contact sensor to the bell’s inductive coil, tacked a buzzer on an ESP-01, loaded it with Sonoff-Tasmota and called it a day. Later on, I took the time to solder the various bits onto a tiny piece of protoboard with a neater arrangement:

Since I treated the buzzer just like another device, it was trivial to plug in to my home automation setup¹ as another bit of MQTT/Node-RED logic:

As it turns out, the resident humans found the buzzer sound annoying (which was kind of the entire point, since I wanted to get them to actually act upon it). But it was also not loud/distinctive enough to carry to the back of the house, which eventually forced me to start looking for a “nicer” solution.

MP3 Playback On a Budget

I soon found out about the JQ6500, a tiny MP3 player that can be made to playback specific samples either via a UART or 5 pull-down pins, as well as having the ability to drive a sub-1W loudspeaker.

It comes with a micro-USB port (so no need to use a standalone micro-USB breakout), but I still need to add a voltage regulator to power the ESP-01 off it. To pull down the relevant input, I just tossed in an N-channel MOSFET, and the resulting circuit looks pretty much the same as before:

Uploading your own audio is as simple as cloning the rescue tool repository, building the jqtool binary on my lab Raspberry Pi and telling it to upload a folder’s worth of MP3 files, so I am trying out a few chimes as well as “regular” doorbell noises. Physically, the entire thing is only a slight bit larger due to the ESP-01 breadboard adapter:

Since I’m still using Sonoff-Tasmota, this was wired up on Node-RED in exactly the same way and there is zero code involved–which was rather disappointing, really, as I’ve been feeling the need to do something besides Spark and Python.

I’ve already established that the tinny little speaker I’m using isn’t going to cut it (but, as usual, planned ahead for that), so I’m just waiting on a few PAM8403 3W amplifier modules and beefier speakers to go with them. Once they arrive, I intend to build at least two much louder doorbell extenders, and 3D print cases to house them.

And if the resident humans don’t appreciate the volume, well… At least now I can take requests for which chimes to use.

For those who are interested in the interview of Carlos Ghosn,former CEO of Nissan, with French TV LCI, here is the transcript in English.
As you may know, Mr. Ghosn has been arrested four times, after Nissan executives went to the Tokyo Public Prosecutors Office and decided to make them the tool of getting rid of Carlos Ghosn, rather than settling the matter internally.
This is shared here for educational purposes.
The actual interview is available in French here: https://lnkd.in/giWYb9D
This comes courtesy of Jacques Deguest
Angel Investor, Co Founder & CEO in Japan, MBA, LLM

For the most part, pessimism about the likely effectiveness of regulation or counteroffensives—not to mention their feasibility—leads to one consistent, repeated recommendation: Let’s make voters more resistant to disinformation by strengthening digital literacy…But building strong critical thinking skills and smarter citizens is a long-term project—one that is hardly likely to enjoy continued support from governments who are elected on the basis of public ignorance. Democracy is on fire now. Fake news is a problem we can’t wait a generation to fix.

How can we address fake news vulnerability now? I delve into the latest research on misinformation and social media contagion in my latest Digital Voyage column for JSTOR Daily.

[T]he demands of the digital world need not consign us to a catatonic state. It may still be possible to anticipate and manage the steady acceleration of incoming stimuli, if we adopt lessons from community of people who are already experts in handling overload: people with autism.

That’s the premise of my first story for Medium’s monthly magazine. I spoke with autistic people and autism researchers to find out how neurotypical (i.e. non-autistic) people can borrow from autistic experiences in order to manage the growing problem of digital overload. Read the whole story on Medium.

What happened was, the girls are finishing Grade Seven so we walked the Great Wall of China. This actually makes perfect sense. By “the girls” I mean my daughter and a schoolfriend; they’ve been in Mandarin Bilingual elementary and have learned quite a bit of Chinese. They may be at their maximum proficiency for a while, since their high schools’ Mandarin offerings aren’t that great. So we (I mean the girls’ parents) thought we should expose them to some Real Chinese. Except for none of the adults speak any, so we went shopping for tours and picked Walk the Great Wall of China.

The Surface of China series includes this fragment and also:

• Visiting Buddha, which happens in Hong Kong.

• 2000km, about the train ride from Hong Kong to Beijing.

• Fujifilm X-T30, about the camera I bought on the first day of this vacation.

• On Disliking Beijing; what the title says, with strong language about China’s current rulers.

• On Liking Beijing, because it wasn’t all bad. With pretty pictures and dance videos.

• Walking the Great Wall, being what we went to China to do. Justly on many bucket lists.

• Tianjin, about a lovely museum, impressive tombs, and the Dowager Empress.

It was a three-legged trip; we flew to Hong Kong and hung out for a couple of days, then took the bullet train to Beijing — 300 or so km/h for nine hours. Then a day in Beijing, six days out of town, five them at various Great Wall locations, and a final stretch back in Beijing, then a direct flight home.

Just the surface?

I’ve mostly recovered from the jetlag hangover and see that I have 500-ish photos worth keeping and several screens-full of raw notes. So, as photog and blogger, I ought to be eager to share. There’s a problem: It’s all surface stuff. Did I get lots of interesting visuals? Did I eat lots of interesting food? Did I get off the beaten track? Yes to all of those.

But, how many Chinese people did I get into serious conversations with? Three. Did those conversations go near any of the difficult subjects of history or government or truth? Nope. Do I understand what life feels like for any of the people I saw and occasionally photographed? No.

So I’ll share pictures because they’re pretty and stories that I think interesting or entertaining or maybe useful to other Westerners planning a visit. But, don’t kid yourself that you’re going to learn anything deep or important about China beyond what it looks like. This wasn’t research, it was tourism.

Notes on getting around

• Vancouver to HK is 12½ hours; that’s rough.

• Hong Kong is easy to get around in; get an Octopus card at the first opportunity and public transit will take you more or less anywhere at a reasonable price. Taxis are OK but (like everywhere in the world) traffic is terrible.

• The long fast train from HK to Beijing was a treat. The train food is trash but the seats are comfy and you’ll see a whole lot of China really fast. The train stations at either end are reasonably efficient and manageable.

• The G Adventures tour was excellent. I’m not normally a guided-tour kind of person, and have enough travel experience and language smatterings to get by most places, not including mainland China. Our guide and driver were excellent, the route well-chosen, and the price very reasonable.

• Beijing is tough to get around in. The subway system is easy to figure out but the ticket-selling machines are klunky and failure-prone, and the place is so freaking huge that you’re often a long walk from the nearest station. Taxi fares are reasonable, but near any major tourist attraction the drivers will refuse to go on-meter and demand exorbitant prices.

• The traffic in Beijing is terrible too. Imagine that.

Sharing plans

I have way too many pictures to blog or Tweet or whatever; come over for dinner if you’re in town and I’ll do a slide show. I’ll write a few blog fragments and share some of the prettier pix, and after all a close look at the surface of China is better than no look at all.

One of a few articles I’m writing about accessibility. The previous one was about color contrast.

---

Last time I wrote about accessibility, I complained about a common accessibility test that I think is inaccurate and counterproductive: color contrast. This time the problem was just my mistake.

The takeaway: if you want an element on a website to have an action when you click it, it should probably be a button.

Buttons have useful default behavior

With JavaScript, you can make anything clickable by adding an onclick handler. So your <div> element can turn from static UI to an interactive element just by adding that behavior. Unfortunately, this only adds one particular type of interaction to the element: the ability to respond to a click.

Adding an onclick handler does little to signal that the element is interactive, because lots of elements (sometimes all elements) have onclick handlers, and only a subset of those elements are really click targets.

<button> elements, on the other hand, are always obviously clickable (and non-clickable, when they have a disabled property). They also have useful default behaviors:

1. You can Tab to focus a button on a webpage. Buttons are, by default, part of the page’s tab order.
2. When a button is focused, it has a default style that shows its focus - usually a blue ring if you’re using macOS.
3. When a button is focused, you can hit Enter or Space to activate it via the keyboard.

Why I strayed

ObservableHQ ended up with a lot of <div> and <a> elements where it should have had buttons. In part, this was refactoring debt. But it was also from another cause: the user agent stylesheet.

The user agent stylesheet is a built-in set of styles that are the defaults for elements on a webpage. They vary between browsers.

Now, to dive in really quickly, this goes back to the founding principles of the web and how they don’t line up to web expectations today. The idea was that individual users would be able to set their favorite fonts and read webpages in those fonts. Same with background color and font color, and other particulars. Part of the ‘cascade’ in Cascading Style Sheets was cascading from user choices.

This is an admirable idea! But it reflects two big differences between the early web and today’s web:

1. Standards of polish and presentation are vastly different today than they were in the early web. With the exception of the personal websites of Computer Science professors, most webpages have very opinionated style choices.
2. Web applications have an entirely different set of design constraints than blogs or papers. When an application needs to squeeze five buttons into a tiny UI space, or align different UIs perfectly, arbitrary variance in font size, face, and other parameters can easily break the layout.

And in this lens, buttons are a severe annoyance. The default user stylesheet for buttons in Chrome, for example, resets the font in the button to -system-ui. The user style for a button looks something like:

letter-spacing: normal;
text-rendering: auto;
text-align: center;
text-transform: none;
text-indent: 0px;
text-shadow: none;
word-spacing: normal;
font: 400 11px system-ui;
display: inline-block;
margin: 0em;
align-items: flex-start;
cursor: default;
color: buttontext;
background-color: buttonface;
box-sizing: border-box;
padding: 2px 6px 3px;
border-width: 2px;
border-style: outset;
border-color: buttonface;
border-image: initial;

That’s a lot of defaults. And in different browsers, different defaults. Compared to the default style of a <div>:

display: block;

This is part of my lame excuse for why ObservableHQ dodged buttons: their styling on the web is a weird attempt to make them seem like ‘native UI’ even in an age when our expectation is that web pages look uniform across platforms. Resetting all the defaults is a hassle and it’s easy to get wrong - just look at the many sites that use a nice system font stack but still have bits of Helvetica in their buttons.

How to go without buttons

When you switch to buttons, you pay with CSS styles. But you can go the other route, of using <div> elements (or any other element), and making them accessible and adding the same affordances. But it’s arguably a bit trickier. You’ll need

• role=button, to make it noticeable as a button
• focusindex=0, to make it keyboard-focusable
• Event handlers for Enter & Space keys to provide the keyboard affordances.

MDN has a very helpful page on this process, with role=button. From my perspective, this is the hard way. Checking out some of the sites I rely on, I found a few that use role=button, like are.na - but are.na doesn’t have a visible focus state, and doesn’t have keyboard handlers. Instapaper, unfortunately, has a bunch of interactive but non-focusable, non-detectable elements.

Unknown unknowns

My last post in this area was about color contrast, a property which automated tests can detect and do eagerly, constantly dole out disapprovals. In stark contrast, interactivity - and, I fear, a lot of the properties of web applications, as I will contrast with websites, brochureware, and blogs - are not easily testable. Google’s Lighthouse and the axe extension found plenty of problems with the site - most of which we’ve fixed - but our clickable divs weren’t one of them.

Fin

This is the latest piece in my day-to-day effort to improve my work. It’s not advanced technique or particularly novel, but might be useful to other folks building applications.

Observable is a lot better than it was a few months ago, but there are still big challenges ahead, some of which are going to require novel approaches. For example, two things we build the product on are iframes - which are the environment for running code - and CodeMirror, which is the editor for writing code. Navigating in and out of these abstractions is tricky, when, for example, Tab indents text in CodeMirror, instead of moving on to the next element.

But that’s for a future blog post.

The other thing is that technically this is more universal design at this point than capital-a accessibility: we’re making sure that the interactivity of the application is broad and obvious, in part by making sure pages are keyboard-accessible. And it’s elementary at that - there’s a lot more to do.

On the Joe Moakley Courthouse in Boston, the words of Felix Frankfurter.