Rolandt

It was probably not difficult to discern from my previous Drill-themed post that I’m fairly excited about the Apache Drill 1.15.0 release. I’ve rounded out most of the existing corners for it in preparation for a long-overdue CRAN update and have been concentrating on two helper features: configuring & launching Drill embedded Docker containers and auto-generation of Drill CTAS queries.

Drill Docker Goodness

Starting with version 1.14.0, Apache provides Drill Docker images for use in experimenting/testing/building-off-of. They run Drill in single node standalone mode so you’re not going to be running this in “production” (unless you have light or just personal workloads). Docker is a great way to get to know Drill if you haven’t already played with it since you don’t have do do much except run the Docker image.

I’ve simplified this even more thanks to @rgfitzjohn’s most excellent stevedore package which adds a robust R wrapper to the Docker client without relying on any heavy external dependencies such as reticulate. The new drill_up() function will auto-fetch the latest Drill image and launch a container so you can have a running Drill instance with virtually no effort on your part.

Just running the vanilla image isn’t enough since your goal is likely to do more than work with the built-in cp data source. The default container launch scenario also doesn’t hook up any local filesystem paths to the container so you really can’t do much other than cp-oriented queries. Rather than make you do all the work of figuring out how to machinate Docker command line arguments and manually configure a workspace that points to a local filesystem area in the Drill web admin GUI the drill_up() function provides a data_dir argument (that defaults to the getwd() where you are in your R session) which will then auto-wire up that path into the container and create a dfs.d workspace which auto-points to it for you. Here’s a sample execution:

library(sergeant)
library(tidyverse)

dr <- drill_up(data_dir = "~/Data")
## Drill container started. Waiting for the service to become active (this may take up to 30s).
## Drill container ID: f02a11b50e1647e44c4e233799180da3e907c8aa27900f192b5fd72acfa67ec0

You can use dc$stop() to stop the container or use the printed container id to do it from the command line.

We’ll use this containerized Drill instance with the next feature but I need to thank @cboettig for the suggestion to make an auto-downloader-runner-thingy before doing that. (Thank you @cboettig!)

Taking the Tedium out of CTAS

@dseverski, an intrepid R, Drill & sergeant user noticed some new package behavior with Drill 1.15.0 that ended up spawning a new feature: automatic generation of Drill CTAS statements.

Prior to 1.14.0 sergeant had no way to accurately, precisely tell data types of the columns coming back since the REST API didn’t provide them (as noted in the previous Drill post). Now, it did rely on the JSON types to create the initial data frames but id also did something **kinda horribad*: it ran readr::type_convert() on the result sets . Said operation had the singular benefit of auto-converting CSV/CSVH/TSV/PSV/etc data to something sane without having to worry about writing lengthy CTAS queries (at the expense of potentially confusing everyone, though that didn’t seem to happen).

With 1.15.0, the readr::type_convert() crutch is gone, which results in less-than-helpful things like this when you have delimiter-separated values data:

# using the Drill container we just started above

write_csv(nycflights13::flights, "~/Data/flights.csvh")

con <- src_drill("localhost")

tbl(con, "dfs.d.`flights.csvh`") %>% 
  glimpse()
## Observations: ??
## Variables: 19
## Database: DrillConnection
## $ year           <chr> "2013", "2013", "2013", "2013", "2013", "2013", "2013", "2013…
## $ month          <chr> "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "…
## $ day            <chr> "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "…
## $ dep_time       <chr> "517", "533", "542", "544", "554", "554", "555", "557", "557"…
## $ sched_dep_time <chr> "515", "529", "540", "545", "600", "558", "600", "600", "600"…
## $ dep_delay      <chr> "2", "4", "2", "-1", "-6", "-4", "-5", "-3", "-3", "-2", "-2"…
## $ arr_time       <chr> "830", "850", "923", "1004", "812", "740", "913", "709", "838…
## $ sched_arr_time <chr> "819", "830", "850", "1022", "837", "728", "854", "723", "846…
## $ arr_delay      <chr> "11", "20", "33", "-18", "-25", "12", "19", "-14", "-8", "8",…
## $ carrier        <chr> "UA", "UA", "AA", "B6", "DL", "UA", "B6", "EV", "B6", "AA", "…
## $ flight         <chr> "1545", "1714", "1141", "725", "461", "1696", "507", "5708", …
## $ tailnum        <chr> "N14228", "N24211", "N619AA", "N804JB", "N668DN", "N39463", "…
## $ origin         <chr> "EWR", "LGA", "JFK", "JFK", "LGA", "EWR", "EWR", "LGA", "JFK"…
## $ dest           <chr> "IAH", "IAH", "MIA", "BQN", "ATL", "ORD", "FLL", "IAD", "MCO"…
## $ air_time       <chr> "227", "227", "160", "183", "116", "150", "158", "53", "140",…
## $ distance       <chr> "1400", "1416", "1089", "1576", "762", "719", "1065", "229", …
## $ hour           <chr> "5", "5", "5", "5", "6", "5", "6", "6", "6", "6", "6", "6", "…
## $ minute         <chr> "15", "29", "40", "45", "0", "58", "0", "0", "0", "0", "0", "…
## $ time_hour      <chr> "2013-01-01T10:00:00Z", "2013-01-01T10:00:00Z", "2013-01-01T1…

So the package does what it finally should have been doing all along. But, as noted, that’s not great if you just wanted to quickly work with a directory of CSV files. In theory, you’re supposed to use Drill’s CREATE TABLE AS then do a bunch of CASTS and TO_s to get proper data types. But who has time for that?

David had a stellar idea, might sergeant be able to automagically create CTAS statements from a query?. Yes. Yes it just might be able to do that with the new ctas_profile() function.

Let’s pipe the previous tbl() into ctas_profile() and see what we get:

tbl(con, "dfs.d.`flights.csvh`") %>% 
  ctas_profile() %>% 
  cat()

-- ** Created by ctas_profile() in the R sergeant package, version 0.8.0 **

CREATE TABLE CHANGE____ME AS
SELECT
  CAST(`year` AS DOUBLE) AS `year`,
  CAST(`month` AS DOUBLE) AS `month`,
  CAST(`day` AS DOUBLE) AS `day`,
  CAST(`dep_time` AS DOUBLE) AS `dep_time`,
  CAST(`sched_dep_time` AS DOUBLE) AS `sched_dep_time`,
  CAST(`dep_delay` AS DOUBLE) AS `dep_delay`,
  CAST(`arr_time` AS DOUBLE) AS `arr_time`,
  CAST(`sched_arr_time` AS DOUBLE) AS `sched_arr_time`,
  CAST(`arr_delay` AS DOUBLE) AS `arr_delay`,
  CAST(`carrier` AS VARCHAR) AS `carrier`,
  CAST(`flight` AS DOUBLE) AS `flight`,
  CAST(`tailnum` AS VARCHAR) AS `tailnum`,
  CAST(`origin` AS VARCHAR) AS `origin`,
  CAST(`dest` AS VARCHAR) AS `dest`,
  CAST(`air_time` AS DOUBLE) AS `air_time`,
  CAST(`distance` AS DOUBLE) AS `distance`,
  CAST(`hour` AS DOUBLE) AS `hour`,
  CAST(`minute` AS DOUBLE) AS `minute`,
  TO_TIMESTAMP(`time_hour`, 'FORMATSTRING') AS `time_hour` -- *NOTE* You need to specify the format string. Sample character data is: [2013-01-01T10:00:00Z]. 
FROM (SELECT * FROM dfs.d.`flights.csvh`)


-- TIMESTAMP and/or DATE columns were detected.
Drill's date/time format string reference can be found at:
--
-- <http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html>

There’s a parameter for the new table name which will cause the CHANGE____ME to go away and when the function finds TIMESTAMP or DATE fields it knows to switch to their TO_ cousins and gives sample data with a reminder that you need to make a format string (I’ll eventually auto-generate them unless someone PRs it first). And, since nodoby but Java programmers remember Joda format strings (they’re different than what you’re used to) it provides a handy link to them if it detects the presence of those column types.

Now, we don’t need to actually create a new table (though converting a bunch of CSVs to Parquet is likely a good idea for performance reasons) to use that output. We can pass most of that new query right to tbl():

tbl(con, sql("
SELECT
  CAST(`year` AS DOUBLE) AS `year`,
  CAST(`month` AS DOUBLE) AS `month`,
  CAST(`day` AS DOUBLE) AS `day`,
  CAST(`dep_time` AS DOUBLE) AS `dep_time`,
  CAST(`sched_dep_time` AS DOUBLE) AS `sched_dep_time`,
  CAST(`dep_delay` AS DOUBLE) AS `dep_delay`,
  CAST(`arr_time` AS DOUBLE) AS `arr_time`,
  CAST(`sched_arr_time` AS DOUBLE) AS `sched_arr_time`,
  CAST(`arr_delay` AS DOUBLE) AS `arr_delay`,
  CAST(`carrier` AS VARCHAR) AS `carrier`,
  CAST(`flight` AS DOUBLE) AS `flight`,
  CAST(`tailnum` AS VARCHAR) AS `tailnum`,
  CAST(`origin` AS VARCHAR) AS `origin`,
  CAST(`dest` AS VARCHAR) AS `dest`,
  CAST(`air_time` AS DOUBLE) AS `air_time`,
  CAST(`distance` AS DOUBLE) AS `distance`,
  CAST(`hour` AS DOUBLE) AS `hour`,
  CAST(`minute` AS DOUBLE) AS `minute`,
  TO_TIMESTAMP(`time_hour`, 'yyyy-MM-dd''T''HH:mm:ssZ') AS `time_hour` -- [2013-01-01T10:00:00Z].
FROM (SELECT * FROM dfs.d.`flights.csvh`)
")) %>% 
  glimpse()
## Observations: ??
## Variables: 19
## Database: DrillConnection
## $ year           <dbl> 2013, 2013, 2013, 2013, 2013, 2013, 2013, 2013, 2013, 2013, 2…
## $ month          <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
## $ day            <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
## $ dep_time       <dbl> 517, 533, 542, 544, 554, 554, 555, 557, 557, 558, 558, 558, 5…
## $ sched_dep_time <dbl> 515, 529, 540, 545, 600, 558, 600, 600, 600, 600, 600, 600, 6…
## $ dep_delay      <dbl> 2, 4, 2, -1, -6, -4, -5, -3, -3, -2, -2, -2, -2, -2, -1, 0, -…
## $ arr_time       <dbl> 830, 850, 923, 1004, 812, 740, 913, 709, 838, 753, 849, 853, …
## $ sched_arr_time <dbl> 819, 830, 850, 1022, 837, 728, 854, 723, 846, 745, 851, 856, …
## $ arr_delay      <dbl> 11, 20, 33, -18, -25, 12, 19, -14, -8, 8, -2, -3, 7, -14, 31,…
## $ carrier        <chr> "UA", "UA", "AA", "B6", "DL", "UA", "B6", "EV", "B6", "AA", "…
## $ flight         <dbl> 1545, 1714, 1141, 725, 461, 1696, 507, 5708, 79, 301, 49, 71,…
## $ tailnum        <chr> "N14228", "N24211", "N619AA", "N804JB", "N668DN", "N39463", "…
## $ origin         <chr> "EWR", "LGA", "JFK", "JFK", "LGA", "EWR", "EWR", "LGA", "JFK"…
## $ dest           <chr> "IAH", "IAH", "MIA", "BQN", "ATL", "ORD", "FLL", "IAD", "MCO"…
## $ air_time       <dbl> 227, 227, 160, 183, 116, 150, 158, 53, 140, 138, 149, 158, 34…
## $ distance       <dbl> 1400, 1416, 1089, 1576, 762, 719, 1065, 229, 944, 733, 1028, …
## $ hour           <dbl> 5, 5, 5, 5, 6, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 5, 6, 6, 6, 6, 6…
## $ minute         <dbl> 15, 29, 40, 45, 0, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 59, 0, 0, 0…
## $ time_hour      <dttm> 2013-01-01 10:00:00, 2013-01-01 10:00:00, 2013-01-01 10:00:0…

Ahhhh… Useful data types. (And, see what I mean about that daft format string? Also, WP is mangling the format string so add a comment if you need the actual string.)

FIN

As you can see questions, suggestions (and PRs!) are welcome and heeded on your social-coding platform of choice (though y’all still seem to be stuck on GH ).

NOTE: I’ll be subbing out most install_github() links in READMEs and future blog posts for install_git() counterparts pointing to my sr.ht repos (as I co-locate/migrate them there).

You can play with the new 0.8.0 features via devtools::install_git("https://git.sr.ht/~hrbrmstr/sergeant", ref="0.8.0").

The Levchin Prize awards two entrepreneurs every year for significant contributions to solving global, real-world cryptography issues that make the internet safer at scale. This year, we’re proud to announce that our very own Firefox CTO, Eric Rescorla, was awarded one of these prizes for his involvement in spearheading the latest version of Transport Layer Security (TLS). TLS 1.3 incorporates significant improvements in both security and speed, and was completed in August and already secures 10% of sites.

Eric has contributed extensively to many of the core security protocols used in the internet, including TLS, DTLS, WebRTC, ACME, and the in development IETF QUIC protocol.  Most recently, he was editor of TLS 1.3, which already secures 10% of websites despite having been finished for less than six months. He also co-founded Let’s Encrypt, a free and automated certificate authority that now issues more than a million certificates a day, in order to remove barriers to online encryption and helped HTTPS grow from around 30% of the web to around 75%. Previously, he served on the California Secretary of State’s Top To Bottom Review where he was part of a team that found severe vulnerabilities in multiple electronic voting devices.

The 2019 winners were selected by the Real-World Cryptography conference steering committee, which includes professors from Stanford University, University of Edinburgh, Microsoft Research, Royal Holloway University of London, Cornell Tech, University of Florida, University of Bristol, and NEC Research.

This prize was announced on January 9th at the 2019 Real-World Crypto Conference in San Jose, California. The conference brings together cryptography researchers and developers who are implementing cryptography on the internet, the cloud and embedded devices from around the world. The conference is organized by the International Association of Cryptologic Research (IACR) to strengthen and advance the conversation between these two communities.

For more information about the Levchin Prize visit www.levchinprize.com.

The post Eric Rescorla Wins the Levchin Prize at the 2019 Real-World Crypto Conference appeared first on The Mozilla Blog.

The Globe and Mail: Forget the wall. Trump is the national security crisis. By Sarah Kendzior.

“The speech was akin to a hostage video, and American viewers were his captive audience. We watched because the stakes felt too high to turn away. We watched because Mr. Trump has taunted us with talk of declaring a “national emergency” – an act which gives him the power to do things like kill the internet, freeze bank accounts, and turn military troops into a domestic police force. We watched because Mr. Trump has long applauded death through his praise of dictators and criminals. We watched because the path to American autocracy was laid out upon his election, and we wanted to know which victims were next.”

Link via MetaFilter.

undefined

Only in Germany would they need to put a sticker on your dashboard reminding you of your new tires max speed when you switch them out for snow tires in the winter. Because, you know, the autobahn.

For my American friends, 210kph equals 130mph.

Elon Musk says Tesla is twice as safe with Autopilot on, but his stats don’t add up. Here’s Musk’s tweet. Here’s the quote from Tesla’s report, as published in Teslarati: “In the 4th quarter, we registered one accident for every 2.91 million miles driven in which drivers had Autopilot engaged. For those driving without Autopilot, … Continued

The post Is a Tesla on Autopilot safer? Maybe, but Elon Musk’s stats are crooked appeared first on without bullshit.

More than just sergeant has been hacked on recently, so here’s a run-down of various updates:

waffle

The square pie chart generating waffle package now contains a nascent geom_waffle() so you can do things like this:

library(hrbrthemes)
library(waffle)
library(tidyverse)

tibble(
  parts = factor(rep(month.abb[1:3], 3), levels=month.abb[1:3]),
  values = c(10, 20, 30, 6, 14, 40, 30, 20, 10),
  fct = c(rep("Thing 1", 3), rep("Thing 2", 3), rep("Thing 3", 3))
) -> xdf

ggplot(xdf, aes(fill=parts, values=values)) +
  geom_waffle(color = "white", size=1.125, n_rows = 6) +
  facet_wrap(~fct, ncol=1) +
  scale_x_discrete(expand=c(0,0)) +
  scale_y_discrete(expand=c(0,0)) +
  ggthemes::scale_fill_tableau(name=NULL) +
  coord_equal() +
  labs(
    title = "Faceted Waffle Geoms"
  ) +
  theme_ipsum_rc(grid="") +
  theme_enhance_waffle()

and get:

It’s super brand new so pls file issues (wherev you like besides blog comments as they’re not conducive to package triaging) if anything breaks or you need more aesthetic configuration options. NOTE: You need to use the 1.0.0 branch as noted in the master branch README.

markdowntemplates

I had to take a quick peek at markdowntemplates due to a question from a blog reader about the Jupyter notebook generation functionality. While I was in the code I added two new bits to the knit: markdowntemplates::to_jupyter code. First is the option to specify a run: parameter in the YAML header so you can just knit the document to a Jupyter notebook without executing the chunks:

---
title: "ggplot2 example"
knit: markdowntemplates::to_jupyter
run: false
--- 

If run is not present it defaults to true.

The other add is a bit of intelligence to whether it should include %load_ext rpy2.ipython (the Jupyter “magic” that lets it execute R chunks). If no R code chunks are present, rpy2.ipython will not be loaded.

securitytrails

SecurityTrails is a service for cybersecurity researchers & defenders that provides tools and an API to aid in querying for all sorts of current and historical information on domains and IP addresses. It now (finally) has a mostly-complete R package securitytrails. They’re research partners of $DAYJOB and their API is so give it a spin if you are looking to broaden your threat-y API collection.

astools

Keeping the cyber theme going for a bit, next up is astools) which are “Tools to Work With Autonomous System (‘AS’) Network and Organization Data”. Autonomous Systems (AS) are at the core of the internet (we all live in one) and this package provides tools to fetch AS data/metadata from various sources and work with it in R. For instance, we can grab the latest RouteViews data:

(rv_df <- routeviews_latest())
## # A tibble: 786,035 x 6
##    cidr         asn   minimum_ip maximum_ip  min_numeric max_numeric
##    <chr>        <chr> <chr>      <chr>             <dbl>       <dbl>
##  1 1.0.0.0/24   13335 1.0.0.0    1.0.0.255      16777216    16777471
##  2 1.0.4.0/22   56203 1.0.4.0    1.0.7.255      16778240    16779263
##  3 1.0.4.0/24   56203 1.0.4.0    1.0.4.255      16778240    16778495
##  4 1.0.5.0/24   56203 1.0.5.0    1.0.5.255      16778496    16778751
##  5 1.0.6.0/24   56203 1.0.6.0    1.0.6.255      16778752    16779007
##  6 1.0.7.0/24   56203 1.0.7.0    1.0.7.255      16779008    16779263
##  7 1.0.16.0/24  2519  1.0.16.0   1.0.16.255     16781312    16781567
##  8 1.0.64.0/18  18144 1.0.64.0   1.0.127.255    16793600    16809983
##  9 1.0.128.0/17 23969 1.0.128.0  1.0.255.255    16809984    16842751
## 10 1.0.128.0/18 23969 1.0.128.0  1.0.191.255    16809984    16826367
## # ... with 786,025 more rows

That, in turn, can work with iptools::ip_to_asn() so we can figure out which AS an IP address lives in:

rv_trie <- as_asntrie(rv_df)

iptools::ip_to_asn(rv_trie, "174.62.167.97")
## [1] "7922"

It can also fetch AS name info:

asnames_current()
## # A tibble: 63,453 x 4
##    asn   handle       asinfo                                                iso2c
##    <chr> <chr>        <chr>                                                 <chr>
##  1 1     LVLT-1       Level 3 Parent, LLC                                   US   
##  2 2     UDEL-DCN     University of Delaware                                US   
##  3 3     MIT-GATEWAYS Massachusetts Institute of Technology                 US   
##  4 4     ISI-AS       University of Southern California                     US   
##  5 5     SYMBOLICS    Symbolics, Inc.                                       US   
##  6 6     BULL-HN      Bull HN Information Systems Inc.                      US   
##  7 7     DSTL         DSTL                                                  GB   
##  8 8     RICE-AS      Rice University                                       US   
##  9 9     CMU-ROUTER   Carnegie Mellon University                            US   
## 10 10    CSNET-EXT-AS CSNET Coordination and Information Center (CSNET-CIC) US   
## # ... with 63,443 more rows

which we can use for further enrichment:

routeviews_latest() %>% 
  left_join(asnames_current())
## Joining, by = "asn"

## # A tibble: 786,035 x 9
##    cidr         asn   minimum_ip maximum_ip  min_numeric max_numeric handle            asinfo                     iso2c
##    <chr>        <chr> <chr>      <chr>             <dbl>       <dbl> <chr>             <chr>                      <chr>
##  1 1.0.0.0/24   13335 1.0.0.0    1.0.0.255      16777216    16777471 CLOUDFLARENET     Cloudflare, Inc.           US   
##  2 1.0.4.0/22   56203 1.0.4.0    1.0.7.255      16778240    16779263 GTELECOM-AUSTRAL… Gtelecom-AUSTRALIA         AU   
##  3 1.0.4.0/24   56203 1.0.4.0    1.0.4.255      16778240    16778495 GTELECOM-AUSTRAL… Gtelecom-AUSTRALIA         AU   
##  4 1.0.5.0/24   56203 1.0.5.0    1.0.5.255      16778496    16778751 GTELECOM-AUSTRAL… Gtelecom-AUSTRALIA         AU   
##  5 1.0.6.0/24   56203 1.0.6.0    1.0.6.255      16778752    16779007 GTELECOM-AUSTRAL… Gtelecom-AUSTRALIA         AU   
##  6 1.0.7.0/24   56203 1.0.7.0    1.0.7.255      16779008    16779263 GTELECOM-AUSTRAL… Gtelecom-AUSTRALIA         AU   
##  7 1.0.16.0/24  2519  1.0.16.0   1.0.16.255     16781312    16781567 VECTANT           ARTERIA Networks Corporat… JP   
##  8 1.0.64.0/18  18144 1.0.64.0   1.0.127.255    16793600    16809983 AS-ENECOM         Energia Communications,In… JP   
##  9 1.0.128.0/17 23969 1.0.128.0  1.0.255.255    16809984    16842751 TOT-NET           TOT Public Company Limited TH   
## 10 1.0.128.0/18 23969 1.0.128.0  1.0.191.255    16809984    16826367 TOT-NET           TOT Public Company Limited TH   
## # ... with 786,025 more rows

Note that routeviews_latest() and asnames_current() cache the data so there is no re-downloading unless you clear the local cache.

docxtractr

The docxtractr package recently got a CRAN push due to some changes in the tibble but it also include a new feature that lets you accept or reject “tracked changes” before trying to extract tables/comments from a document without harming/changing the original document.

ednstest

DNS Flag Day is fast approaching. What is “DNS Flag Day”? It’s a day when yet-another cabal of large-scale DNS providers and tech heavy hitters decided that they know what’s best for the internet and are mandating compliance with RFC 6891 (EDNS). Honestly, there’s no good reason to run crappy DNS servers and no good reason not to support EDNS.

You could just go to the flag day site and test your provider (by entering your domain name, if you have one). But, you can also load the package, and run it locally (it still calls their API since it’s open and provides a very detailed results page if your DNS server isn’t compliant). You can just run it to get compact output and an auto-load of the report page in your browser or save off the returned object and inspect it to see what tests failed.

I ran it on a few domains that are likely familiar to readers and this is what it showed:

edns_test("rud.is")
## EDNS compliance test for [rud.is] has ✔ PASSED!
## Report URL: https://ednscomp.isc.org/ednscomp/60049cb032

edns_test("rstudio.com")
## EDNS compliance test for [rstudio.com] has ✖ FAILED
## Report URL: https://ednscomp.isc.org/ednscomp/54e2057229

edns_test("r-project.org")
## EDNS compliance test for [r-project.org] has ✔ PASSED!
## Report URL: https://ednscomp.isc.org/ednscomp/839ee9c9af

The print() function in the package also has some minimal cli and crayon usage in it if you’re looking to jazz up your R console output.

ulid

Finally, there’s ulid which is a package to make “Universally Unique Lexicographically Sortable Identifiers in R”. These ULIDs have the following features:

• 128-bit compatibility with UUID
• 1.21e+24 unique ULIDs per millisecond
• Lexicographically sortable!
• Canonically encoded as a 26 character string, as opposed to the 36 character UUID
• Uses Crockford’s base32 for better efficiency and readability (5 bits per character)
• Case insensitive
• No special characters (URL safe)
• Monotonic sort order (correctly detects and handles the same millisecond)

They’re made up of

 01AN4Z07BY      79KA1307SR9X4MV3

|----------|    |----------------|
 Timestamp          Randomness
   48bits             80bits

The timestamp is a 48 bit integer representing UNIX-time in milliseconds and the randomness is an 80 bit cryptographically secure source of randomness (where possible). Read more in the full specification.

You can get one ULID easily:

ulid::ULIDgenerate()
## [1] "0001E2ERKHVPKZJ6FA6ZWHH1KS"

Generate a whole bunch of ’em:

(u <- ulid::ULIDgenerate(20))
##  [1] "0001E2ERKHVX5QF5D59SX2E65T" "0001E2ERKHKD6MHKYB1G8JHN5X" "0001E2ERKHTK0XEHVV2G5877K9" "0001E2ERKHKFGG5NPN24PC1N0W"
##  [5] "0001E2ERKH3F48CAKJCVMSCBKS" "0001E2ERKHF3N0B94VK05GTXCW" "0001E2ERKH24GCJ2CT3Z5WM1FD" "0001E2ERKH381RJ232KK7SMWQW"
##  [9] "0001E2ERKH7NAZ1T4HR4ZRQRND" "0001E2ERKHSATC17G2QAPYXE0C" "0001E2ERKH76R83NFST3MZNW84" "0001E2ERKHFKS52SD8WJ8FHXMV"
## [13] "0001E2ERKHQM6VBM5JB235JJ1W" "0001E2ERKHXG2KNYWHHFS8X69Z" "0001E2ERKHQW821KPRM4GQFANJ" "0001E2ERKHD5KWTM5S345A3RP4"
## [17] "0001E2ERKH0D901W6KX66B1BHE" "0001E2ERKHKPHZBFSC16FC7FFC" "0001E2ERKHQQH7315GMY8HRYXV" "0001E2ERKH016YBAJAB7K9777T"

and “unmarshal” them (which gets you the timestamp back):

unmarshal(u)
##                     ts              rnd
## 1  2018-12-29 07:02:57 VX5QF5D59SX2E65T
## 2  2018-12-29 07:02:57 KD6MHKYB1G8JHN5X
## 3  2018-12-29 07:02:57 TK0XEHVV2G5877K9
## 4  2018-12-29 07:02:57 KFGG5NPN24PC1N0W
## 5  2018-12-29 07:02:57 3F48CAKJCVMSCBKS
## 6  2018-12-29 07:02:57 F3N0B94VK05GTXCW
## 7  2018-12-29 07:02:57 24GCJ2CT3Z5WM1FD
## 8  2018-12-29 07:02:57 381RJ232KK7SMWQW
## 9  2018-12-29 07:02:57 7NAZ1T4HR4ZRQRND
## 10 2018-12-29 07:02:57 SATC17G2QAPYXE0C
## 11 2018-12-29 07:02:57 76R83NFST3MZNW84
## 12 2018-12-29 07:02:57 FKS52SD8WJ8FHXMV
## 13 2018-12-29 07:02:57 QM6VBM5JB235JJ1W
## 14 2018-12-29 07:02:57 XG2KNYWHHFS8X69Z
## 15 2018-12-29 07:02:57 QW821KPRM4GQFANJ
## 16 2018-12-29 07:02:57 D5KWTM5S345A3RP4
## 17 2018-12-29 07:02:57 0D901W6KX66B1BHE
## 18 2018-12-29 07:02:57 KPHZBFSC16FC7FFC
## 19 2018-12-29 07:02:57 QQH7315GMY8HRYXV
## 20 2018-12-29 07:02:57 016YBAJAB7K9777T

and can even supply your own timestamp:

(ut <- ts_generate(as.POSIXct("2017-11-01 15:00:00", origin="1970-01-01")))
## [1] "0001CZM6DGE66RJEY4N05F5R95"

unmarshal(ut)
##                    ts              rnd
## 1 2017-11-01 15:00:00 E66RJEY4N05F5R95

FIN

Kick the tyres & file issues/PRs as needed and definitely give sr.ht a spin for your code-hosting needs. It’s 100% free and open source software made up of mini-services that let you use only what you need. Zero javacript on site and no tracking/adverts. Plus, no evil giant megacorps doing heaven knows what with your browser, repos, habits and intellectual property.

I’m not a big fan of the currently fashionable “iPhones are bad, m’kay!” thinking. Yes, there are apps and social networks that are bad, but the technology itself is neutral. After spending a day without my phone while it was in getting a new camera module, however, I will admit that it was really nice spending some time away from it.

Sitting on the train, for example, with no magic device in hand was a treat. I just watched the scenery going by. The people getting on and off. The signs and graffiti. I enjoyed the space to think without somebody else’s thoughts jumping in at any moment.

I might have to do that again. On purpose.

Compelling visualization doesn't just conjure itself out of nowhere. The ideas come from somewhere, and oftentimes they build off previous ones. Read More

For the fourth year in a row, I present to you my state of the union address.

I had to return my old Surface Pro and the Yoga C930 will also eventually have to go back. But I needed a good permanent Windows 10 machine. So here is a new Surface Pro, this time with a grey Signature Type Cover. It's my favorite Windows machine by quite a margin. And I am getting better at this. Wiped the machine, reinstalled Windows 1809, all patches and firmware updates, connected to my Microsoft account, installed all my applications and filled it up with about 200 GB of data. All within one afternoon while doing other stuff.

Not immediately visible while I am working, but the Keyboard Cover has style.

A reader of my newsletter wrote in, talking about the problem of finding a job with work/life balance in Silicon Valley:

It seems like us software engineers are in a tough spot: companies demand a lot of hard work and long hours, and due to the competitiveness here in Silicon Valley, you have to go along with it (or else there’s some bright young Stanford grad just dying to work all nighters on Red Bull to take your place).

But they throw you aside once the company has become established and they no longer need the “creative” types.

In short, how do you get a job with work/life balance when you’re competing against people willing to work long hours?

All nighters make for bad software

The starting point is realizing that working long hours makes you a much less productive employee, to the point that your total output will actually decrease (see Evan Robinson on crunch time). If you want to become an effective and productive worker, you’re actually much better off working normal hours and having a personal life than working longer hours.

Since working shorter hours makes you more productive, that gives you a starting point for why you should be hired.

Instead of focusing on demonstrative effort by working long hours, you can focus on identifying and solving valuable problems, especially the bigger and longer term problems that require thought and planning to solve.

Picking the right company

Just because you’re a valuable, productive programmer doesn’t mean you’re going to get hired, of course. So next you need to find the right company.

You can imagine three levels of managerial skill:

• Level 1: These managers have no idea how to recognize effective workers, so they only judge people by hours worked.
• Level 2: These managers can recognize effective workers, but don’t quite know how to create a productive culture. That means if you choose work long hours they won’t stop you, however pointless these long hours may be. But, they won’t force you work long hours so long as you’re doing a decent job.
• Level 3: These managers can recognize effective workers and will encourage a productive culture. Which is to say, they will explicitly discourage working long hours except in emergencies, they will take steps to prevent future emergencies, etc..

When you look for a job you will want to avoid Level 1 managers. However good your work, they will be happy to replace you with someone incompetent so long as they can get more hours out of them. So you’ll be forced to work long hours and work on broken code.

Level 3 managers are ideal, and they do exist. So if you can find a job working for them then you’re all set.

Level 2 managers are probably more common though, and you can get work/life balance working for them—if you set strong boundaries. Since they can recognize actual competence and skills, they won’t judge you during your interview based on how many hours you’re willing to work. You just need to convey your skill and value, and a reasonable amount of dedication to your job.

And once you’ve started work, if you can actually be productive (and if you work 40 hours/week you will be more productive!) they won’t care if you come in at 9 and leave at 5, because they’ll be happy with your work.

Unlike Level 3 managers, however, you need to be explicit about boundaries during the job interview, and even more so after you start. Elsewhere I wrote up some suggestions about how to convey your value, and how to say “no” to your boss.

Employment is a negotiated relationship

To put all this another way: employment is a negotiated relationship. Like it or not, you are negotiating from the moment you start interviewing, while you’re on the job, and until the day you leave.

You are trying to trade valuable work for money, learning opportunities, and whatever else your goals are (you can, for example, negotiate for a shorter workweek). In this case, we’re talking about negotiating for work/life balance:

1. Level 1 managers you can’t negotiate with, because what they want (long hours) directly conflicts with what you want.
2. Level 2 managers you can negotiate with, by giving them one of the things they want: valuable work.
3. Level 3 managers will give you what you want without your having to do anything, because they know it’s in the best interest of everyone.

Of course, even for Level 3 managers you will still need to negotiate other things, like a higher salary.

So how do you get a job with work/life balance? Focus on providing and demonstrating valuable long-term work, avoid bad companies, and make sure you set boundaries from the very start.

---

Struggling with a 40-hour workweek? Too tired by the end of the day to do anything but collapse on the sofa and watch TV?

Learn how you can get a 3-day weekend, every single week.

Deutsche Welle: Homeschooling und Freilernen: Eltern gegen Schulpflicht. “”Wunderlich gegen Deutschland” – Eine Familie aus Hessen klagte vor dem Europäischen Gerichtshof für Menschenrechte, weil die Polizei ihre Kinder abholte. Auch andere Eltern sind gegen die Schulpflicht für alle.”

“Die Verletzung der Schulpflicht gilt in den meisten Bundesländern als Ordnungswidrigkeit, die mit einem Bußgeld geahndet wird. In Hessen können Haftstrafen bis zu sechs Monaten verhängt werden. Deutsche Behörden und Gerichte betonen, dass die Schulpflicht dazu diene, allen die demokratischen Werte des Grundgesetzes zu vermitteln. Sie sorge dafür, dass sich niemand in Parallelgesellschaften absondere oder dem Dialog mit Andersdenkenden verschließe. Dirk Wunderlich weist den Verdacht der Abschottung zurück: Kinder von Heimschuleltern seien oft in Vereinen aktiv, seine Kinder seien im ganzen Ort und darüber hinaus mit unterschiedlichsten Menschen vernetzt, dafür habe er dem Familiengericht viele Zeugen präsentiert.”

Was ich als Lehrer von Homeschooling halte, dürfte klar sein. Es gibt sicher Dinge, die Kinder und Jugendliche sich selbst beibringen können, sei es mit Hilfe von Büchern, Baukästen oder dem allgegenwärtigen Internet. Es gibt aber auch Dinge, die können sie nur in der Schule lernen, z. B. in naturwissenschaftlichen Fächern, wo die Schule eine ganz andere Ausstattung zum Experimentieren bieten kann als das traute Heim. Nicht umsonst studieren Lehrer jahrelang.
Auch die Tatsachen, dass der Besuch einer Schule demokratische Werte vermittelt, zum kritischen Denken und zum Hinterfragen ermutigt und unter Umtänden einen wichtigen Kontrast zur Einstellung des Elternhauses bietet, sind nicht zu unterschätzen.

This seems almost too good to be true, but Snopes and others have proven that this TV show actually existed

On May 8, 1958, art imitated life in 2018. In an episode of a TV show called Trackdown, there was a conman named Trump, who tried to scare the bejeezus out of a town by preaching, “at midnight tonight, without my help and knowledge, every one of you will be dead.” The only way he could save them is by building a wall.

Source: A 1950s TV show had a fear-mongering conman named Trump who wanted to build a wall / Boing Boing

By now, even remote villages on uncharted islands in the Pacific know that the U.S. is in the midst of a protracted partial government shutdown. It’s having real impacts on the lives of Federal government workers but they aren’t the only ones. Much of the interaction Federal agencies have with the populace takes place online and the gateway to most of these services/information is a web site.

There are Federal standards that require U.S. government web sites to use SSL/TLS certificates and those certificates have something in common with, say, a loaf of bread you buy at the store: they expire. In all but the best of orgs — or we zany folks who use L e t ‘ s E n c r y p t and further propel internet denizens into a false sense of safety & privacy — renewing certificates involves manual labor/human intervention. For a good chunk of U.S. Federal agencies, those particular humans aren’t around. If a site’s SSL certificate expires and isn’t re-issued, it causes browsers to do funny things, like this:

Now, some of these sites are configured improperly in many ways, including them serving pages on both http and https (vs redirecting to https immediately upon receiving an http connection). But, browsers like Chrome will generally try https first and scare you into not viewing the site.

But, how big a problem could this really be? We can find out with a fairly diminutive R script that:

• grabs a list of Federal agency domains (thanks to the GSA)
• tries to make a SSL/TLS connection (via the openssl package) to the apex domain or www. prefixed apex domain
• find the expiration date for the cert
• do some simple date math

I’ve commented the script below pretty well so I’ll refrain from further blathering:

library(furrr)
library(openssl)
library(janitor)
library(memoise)
library(hrbrthemes)
library(tidyverse)

# fetch the GSA CSV:

read_csv(
  file = "https://raw.githubusercontent.com/GSA/data/master/dotgov-domains/current-federal.csv",
  col_types = "ccccccc"
) %>% 
  janitor::clean_names() -> xdf

# make openssl::download_ssl_cert calls safer in the even there
# are network/connection issues
.dl_cert <- possibly(openssl::download_ssl_cert, otherwise = NULL)

# memoise the downloader just in case we need to break the iterator
# below or another coding error causes it to break (the cached values
# will go away in a new R session or if you manually purge them)
dl_cert <- memoise::memoise(.dl_cert)

# we'll do this in parallel to save time (~1,200 domains)
plan(multiprocess)

# now follow the process described in the bullet points
future_map_dfr(xdf$domain_name, ~{

  who <- .x

  crt <- dl_cert(who)  

  if (!is.null(crt)) {
    # shld be the first cert and expires is second validity field
    expires <- crt[[1]]$validity[2] 
  } else {
    crt <- dl_cert(sprintf("www.%s", who)) # may be on www b/c "gov"
    if (!is.null(crt)) {
      expires <- crt[[1]]$validity[2]
    } else {
      expires <- NA_character_  
    }
  }

  # keep a copy of the apex domain, the expiration field and the cert
  # (in the event you want to see just how un-optimized the U.S. IT 
  # infrastructure is by how many stupid vendors they use for certs)
  tibble(
    who = who,
    expires = expires,
    cert = list(crt)
  )

}) -> cdf

Now, lets make strings into proper dates, count only the dates starting with the date of the shutdown to the end of 2019 (b/c the reckless human at the helm is borderline insane enough to do that) and plot the timeline:

filter(cdf, !is.na(expires)) %>% 
  mutate(
    expires = as.Date(
      as.POSIXct(expires, format="%b %d %H:%M:%S %Y")
    )
  ) %>% 
  arrange(expires) 
  count(expires) %>% 
  filter(
    expires >= as.Date("2018-12-22"), 
    expires <= as.Date("2019-12-31")
  ) %>% 
  ggplot(aes(expires, n)) +
  geom_vline(
    xintercept = Sys.Date(), linetype="dotted", size=0.25, color = "white"
  ) +
  geom_label(
    data = data.frame(), 
    aes(x = Sys.Date(), y = Inf, label = "Today"),
    color = "black", vjust = 1
  ) +
  geom_segment(aes(xend=expires, yend=0), color = ft_cols$peach) + 
  scale_x_date(name=NULL, date_breaks="1 month", date_labels="%b") +
  scale_y_comma("# Federal Agency Certs") +
  labs(title = "2019 Federal Agency ShutdownCertpoalypse") +
  theme_ft_rc(grid="Y")

Now, I’m unwarrantedly optimistic that this debacle could be over by the end of January. How many certs (by agency) could go bad by then?

left_join(cdf, xdf, by=c("who"="domain_name")) %>% 
  mutate(
    expires = as.Date(
      as.POSIXct(expires, format="%b %d %H:%M:%S %Y")
    )
  ) %>% 
  filter(
    expires >= as.Date("2018-12-22"),
    expires <= as.Date("2019-01-31")
  ) %>% 
  count(agency, sort = TRUE)
## # A tibble: 10 x 2
##    agency                                          n
##    <chr>                                       <int>
##  1 Government Publishing Office                    8
##  2 Department of Commerce                          4
##  3 Department of Defense                           3
##  4 Department of Housing and Urban Development     3
##  5 Department of Justice                           3
##  6 Department of Energy                            1
##  7 Department of Health and Human Services         1
##  8 Department of State                             1
##  9 Department of the Interior                      1
## 10 Department of the Treasury                      1

Ugh.

FIN

Not every agency is fully shutdown and not all workers in charge of cert renewals are furloughed (or being forced to work without pay). But, this one other area shows the possible unintended consequences of making rash, partisan decisions (something both Democrats & Republicans excel at).

You can find the contiguous R code at 2018-01-10-shutdown-certpocalypse.R and definitely try to explore the contents of those certificates.

Yes, Facebook Groups are free but you’re still paying for it.

You’re paying for it in missed opportunities, lost knowledge, and the members who drift away from the noise.

You’re paying when they make changes and don’t tell you until they’re live.
You’re paying when you can’t design the community to support what your audience needs.
You’re paying when you can’t move to a better platform when you outgrow the group.
You’re paying for it when you don’t get any SEO traffic which sustains most long-term communities.
You’re paying for it when you can’t design an onboarding journey for newcomers which helps them take their first steps and become top members.
You’re paying for it when you can’t treat members differently based upon their previous contributions.
You’re paying for it when Facebook’s reputation takes a hit and people leave the platform.
You’re paying for it when you can’t identify and reward members for great contributions.

The list goes on and on.

Is saving a few hundred (or even a few thousand) dollars a month really worth the cost?

p.s. The fee for the Strategic Community Management and Psychology of Community courses rises after today.

How we use war games to train engineers to find and fix issues in production

Several months ago, The New York Times completed the replatform of our core news website. Initially this was supported in production by a tight on-call rotation of a few engineers who were responsible for handling any complications that might arise. But, as we began to port over features from our previous platform and expand our on-call rotation, it became clear that we needed to teach more engineers how to triage production issues.

A key indicator that we needed to spread this knowledge was the discovery that not all engineers were completely comfortable or confident in being assigned to an on-call rotation, even though they had been consistently delivering features and product improvements to the code base. We did our best to produce robust and detailed playbooks, troubleshooting documentation and demos. While these materials were as thorough as possible, they were not something one could easily navigate to quickly determine action steps while alarms are blaring at 2 a.m.

To help our engineers feel more comfortable, we developed a framework for simulating “war game” scenarios. War games is a process often used in software engineering to simulate defects in production — without breaking production. We use war games in our development process as a way to refine our own tooling, documentation and procedures for supporting production services. Because the process is simulated, it allows engineers to practice being on-call without the stress of a real production issue.

Setting Up Our Test Environment

We knew we wanted to simulate production issues and make them as realistic as possible, which meant that we had to create a new environment in which we could safely test our scenarios without affecting production. To do this, we forked our repository and renamed production items to instead be tied to a new “war games” label in Google Cloud Platform (GCP). We duplicated our CI/CD pipeline to mimic exactly how releases and rollback procedures worked.

Since these releases needed to actually do something, we created a new production-like war games cluster that would be tied to deployments in this pipeline. This allowed us to simulate triaging issues in GCP and Stackdriver as if they were actually in production.

Creating Scenarios

We had been in production for some time before we started the war games process, so we had the benefit of hindsight on a few production incidents and could use our experience to create easily reproducible and realistic scenarios. Some scenarios included introducing buggy code updates that sneaked through our testing layers, while others included short-circuiting calls to our backend services in order to simulate upstream issues. Because we were more interested in testing our production defect procedures, we focused on simulating easily reproducible problems rather than stress testing the system architecture and resiliency.

The war games framework

In order to put the framework into practice we spent some time designing the overall process, including thoughtful handouts that outlined how each session should run and a feedback form so we could iterate on the framework as we went. At a high level, the war games process is composed of four parts: an invite to the engineer going through the scenario, a handout to cover prerequisites and provide any necessary context, the session itself and the feedback process.

Invite

We usually allot one hour for each session. When we ran shorter sessions, we ended up rushing through and were not able take time to find gaps in documentation or tooling. We found that it was best to spend about a half hour on each scenario; this way, one engineer can run through two scenarios or two engineers can run through one scenario each.

Handout

As part of the invite we link to a handout that covers everything the engineer will need ahead of the session. This includes any prerequisites (such as having kubectl running locally), or any /etc/hosts entries if DNS isn’t available.

Session

For the session itself, we found it helpful to clearly define three roles: the engineer, the conductor and the instigator. The engineer is simply responsible for running through the scenarios. The conductor leads the session and walks the engineer through the process, reminding them that they are not the one being tested, rather our own procedures and documentation are.

The conductor is also primarily responsible for jotting down notes during the session to find areas of improvement around where the engineer had the most trouble, such as gaps in our documentation or dashboards, or any ideas for improving future sessions. Lastly, the instigator’s role is to setup and execute the various scenarios for the engineer to run through. It’s also good to have this person take notes for a second perspective.

Feedback

After the session, we send out a survey to the engineer to gather some feedback. As part of the survey we try to quantify how the engineer feels about going on-call before and after the session. We also try to gather feedback on our troubleshooting playbooks, documentation, dashboards and also the war game process itself.

We have since employed this framework across the organization, with a few tweaks relevant to each stack. This has not only helped new on-call engineers become more comfortable with a particular stack, but has also helped us improve our tooling, documentation and procedures for production defects in a way that is quite eye opening.

---

Congrats, You’re On Call! Now What? was originally published in Times Open on Medium, where people are continuing the conversation by highlighting and responding to this story.

Technologische Innovationen haben die Macht, unsere Welt zu beeinflussen. Was dabei jedoch oftmals untergeht, wie schnell die Erträge solcher Innovationen sinken. Ein Beispiel ist der Verbrennungsmotor. Er revolutionierte den Verkehr und führte zu einem Gesellschaftswandel. [...]

The testimonials from famous athletes and celebrities are pouring in: apparently getting up at 4am is life-changing, revolutionary and will make you a new person.Apple chief executive officer Tim Cook, Dwayne “The Rock” Johnson, actor Mark Wahlberg and former US Navy Seal turned fitness instructor and motivational speaker Jocko Willink all rise before the sun, primarily for one reason: to get a workout in with little distraction.Putting the benefits of being an early riser aside (more time to…

“The table is too small!” the little one says. “Not everyone can fit round it! Can you help me?”. Thus I helped to put together a bigger table, so she could also seat her Playmobil figures around the Lego table. I like her thinking.

Emojis can articulate anything.

This is the cycle that got me hooked on running, thanks to my son and of course Parkrun (the weekly timed, volunteer-driven 5km run/jog/walk.)

Taken from this edition of the Visual Thinkery newsletter.

The post My First Ten Parkruns appeared first on Visual Thinkery.

I love my job, but the coffee provided in our offices is terrible. Luckily, I’ve found a great solution: the AeroPress Coffee and Espresso Maker, a portable hybrid brewer that combines some of the best qualities of other brewing systems. It is fast like a Nespresso machine and easy to use like a French press, and makes coffee that tastes almost as great as a lovingly prepared pour-over.

Not everyone wants to talk about money, so it can be hard to get solid advice from people you know. That’s where Wirecutter Money can help. The Money team takes the deep-dive Wirecutter approach and applies it to credit cards, software, and services to make recommendations that can set you on the path to a secure financial future. We asked Wirecutter Money senior editor Korrena Bailie to tell us what her team plans to review this year, how the Money team keeps up with changing rates, and what her own terrifying new year’s money resolution is.

“Wirecutter-ing your stuff” means buying things that make life easier and more enjoyable. What does “Wirecutter-ing your finances” mean?

We believe that Wirecutter-ing your finances is primarily about reducing friction or stress in your life—mostly in small ways, but sometimes in big ones.

There’s no reason money topics have to cause fear or tension, and our job is to recommend the credit cards, software, and behavioral tricks that will ease those feelings.

Wirecutter-ing your finances can mean paying a little up front for greater benefits later on. For example, we love the Chase Sapphire Reserve, our pick for the best travel credit card. You pay a $450 annual fee up front, but the rewards and travel credits will most likely put more money in your pocket than with any other card. In addition, you get excellent travel insurance, which means your vacation won’t crumble beneath you if your flight is late or someone rear-ends your rental car.

It also means picking the right software to do your taxes or the best budgeting app, or even thinking about saving for retirement. How many fights have you had with a significant other over money? For nearly all of these problems, there’s a better solution. And we’re committed to finding them for our readers.

Wirecutter-ing your finances can mean paying a little up front for greater benefits later on.

What are some of the most surprising things you and your staff have learned since we started Wirecutter Money?

This one may not be surprising to everyone, but we’ve found again and again that money begets money. So if you already have a substantial amount of money (and, ideally, good credit), opportunities to make that money work in your favor come at you again and again, from lucrative sign-up bonuses to 0% interest offers to waived fees on bank accounts—I could go on forever.

The flip side is how limited things are for you if you’re struggling financially. Financial products for people who are starting out or recovering from financial troubles are pretty poor as a rule. In this case, banks have no incentive to help you because you’re part of a group of people who will earn them the least money. We’re starting to see some innovation in this regard, but it remains an uphill battle. Whenever we review cards or services for people who have poor credit or little cash flow, we do a lot of research to ensure we’re recommending something that’s actually good, not just the least bad of a terrible bunch.

How does Wirecutter Money make its recommendations? How do you keep up with changing rates?

Wirecutter Money is a team of three personal finance experts who have several years of collective experience—we’re very familiar with product gotchas, and we have a lot of experience weeding out the good from the bad.

We make our recommendations based on a lot of old-fashioned legwork. There’s no shortcut to finding the best credit card, financial software, or bank; for every category, we create an inventory of everything that’s out there, including offerings from local banks or credit unions. We speak to bank spokespeople, Wirecutter readers, real people who use these products, and financial experts to build a picture of what the products offer and how easy or difficult they are to use in real life. We create comprehensive comparison tables to evaluate offers side by side and overlay our calculations on existing public data from sources such as the Federal Reserve or the Bureau of Labor Statistics.

Keeping up with changing rates is an endless challenge, but we want our guides to be totally up to date and accurate at all times for our readers. We work closely with our awesome compliance manager, Jeremy Zafiros, who has the thankless task of tracking and then sharing all the latest credit card rate changes and ensuring they’re implemented sitewide. We’ve also created a central database for rates and fees, which when updated will roll those updates out across all our guides and articles.

What advice do you have for people who are trying to build their credit, either from scratch or back up from hard times?

Building credit is tough, but it can also be rewarding—both to see that three-digit number tick up and to see the doors that better credit can open for you. If you’re building credit from scratch, you might be surprised at how quickly you can get to a good place; many secured-card issuers, for example, favor applicants with no credit as opposed to poor credit.

It can be more challenging to rebuild credit, and in this case you need to be more patient, but it is achievable. Generally, even if you have a severe negative mark on your credit (say, a collections account), the impact on your credit will lessen over time. And during that time, you should focus on what you can do immediately to improve your situation. For example, pay off any outstanding accounts that you can, and start to build good credit habits such as making consistent on-time payments and keeping your credit use below 30% of your total credit limit across your cards.

What do you plan to cover for 2019? Any money resolutions of your own?

We have more credit card topics to cover, including airline, business, hotel, 0% APR, and best credit card combos. We’re also going to update our tax software, travel insurance, and budgeting guides. We’ll dive into banking accounts, credit unions, how to accomplish your savings goals, money and relationships, and money for families—and we’ll partner with The New York Times, Wirecutter’s parent company, on some very exciting projects.

My money resolution is a petrifying one: Buy my first house. I’m 35, my husband is 40, and we’re finally in a place where we can buy a home. I’m terrified—but also excited to see years of hard work pay off. I’m also lucky enough to work with several very seasoned homeowners at Wirecutter. I know they’ll give me great advice on buying and owning a home!

Jeff Atwood @codinghorror
Pay attention to the way @natfriedman is running GitHub. Over the past two months, he's announced multiple great ne… twitter.com/i/web/status/1…

Going home tonight down Guerrero from the NERT training, I paused on the sidewalk as a couple of cars were coming out of the driveway by Mitchell’s Ice Cream. One pulled out onto Guerrero and the other started to inch out. I was waiting for them both to leave before crossing the driveway in the dark. The second car’s driver noticed me and motioned for me to cross in front of him. But, he was pulled up so far that to get by I would have had to go on the uncomfortably angled part of the driveway as it went down into the street.

So I waved him on, smiling and nodding as if to say thanks for the thought but no… and waited. There was no traffic at all at 9:30pm. So he could just go and be done with it.

He shook his head and started to back up his car a little bit, maybe a foot or so, but it still wasn’t really enough and I decided what I usually do, which is: do I want to wait 10 seconds for this car to move along, or do I want to put myself in front of a 2 ton death machine in the dark when someone could come along and rear end it, etc? I mean, why bother? Am I in a hurry? No and I’m even comfortably sitting down in my nice powerchair.

So I shook my head and waved at the guy again and said “It’s ok I’ll just go after you” and he shook HIS head and started to roll down the window and told me to go on and I said “No I really don’t have any interest in being in front of your vehicle I’ll just go after you go.” He looked angry and started to roll forward while still telling me I should go! As he pulled out into the street he yelled “BITCH!”

Yeah that guy’s heart was definitely in the right place! He was so kind! So charitable! So thoughtful! Too bad his head was up his own ass!

Our friends at small places have produced another multi-sensory feast of city cycling splendour, this time featuring Delft, Netherlands — just one stop on their summer 2018 tour of northern Europe.

“Old enough to have a historic centre, large enough for it to be vibrant, yet small enough to make that centre mostly car-free. The suburbs of these cities grew up in the decades where protected bike lanes were standard on all streets, avoiding the awkward middle ring of cities like Amsterdam and The Hague.”

You can almost smell fragrant, summer air while all manner of bikes criss-cross intersections, public squares and underpasses. Bells brrrringing, hair flying in the wind, people smiling — where are the cars?

Spoor en centrum translates roughly as “rail and city centre”, a pragmatic reference to the new heart of mobility within some of the world’s most active and accessible urban communities, at least those of a certain size:

It’s been said that the Netherlands is both a dense country and a sprawling city, but neither is quite right.

Large cities quickly give way to countryside, and a constellation of dozens of cities sit in a sweet spot around 100,000 people – where every trip is a comfortable cycling distance and frequent rail service to every other city is close at hand.

Before comparing your city to Delft, first bear in mind that, to adopt the spoor en centrum approach, a city must feature, by definition, rail at the centre of its transportation strategy. Beyond simply connecting to cycling infrastructure, such rail facilities would integrate with and support (and thus be supported by) bespoke infrastructure and facilities for all modes and accessibility requirements in society. All in all, a multi-modal system that anticipates the needs of high volumes of commuters — of all ages, all abilities.

Consider: trains from Delft to The Hague (Den Haag) run every six minutes, and can accomplish the 10km in about seven minutes. World-class. Similarly, the main bike parking garage at Delft station is underground, with over 5,000 spots, and at least as much again in a second underground facility, plus outdoors around the station.

End-of-trip facilities are thus in keeping with the city’s transportation strategy, and the travel modes encouraged by civic leadership for citizens within spoor en centrum. Multiple levels of government get what they want and need. Commuters get what they want and need. Quid pro quo.

Bike mode share in the city: 40%. As such, a seemingly appealing place to walk and wheel, less so for driving.

While this is the direction many North American cities, including those in our growing region, are heading towards, we obviously aren’t there yet. But we could be — let’s simply consider the basic ingredients, population, area and, thus, density.

At 100,000 people in less than 25km², Delft clocks in at 4,443 people/km². Vancouver is over 5,500 people/km²…but not a great comparison from the perspective of population and land area. (We’re more like Amsterdam — about five to seven times the scale of Delft, with a variety of high, medium and low-density neighbourhoods.)

Port Coquitlam and City of Langley aren’t terribly far off on both factors, and with densities between 2,000-2,500people/km². But change is harder — one is not much of a transit, business or post-secondary education hub (PoCo), and the other — possibly — less ready culturally (CoL).

The best fits? The City of North Vancouver and New Westminster — both have worked ardently over the decades, relative to the region, to incorporate cycling infrastructure on roads and recreational paths when it wasn’t necessarily politically expedient, or reflective of present-day use.

Both are also smaller in size and population, but at Delft-esque population densities. Both also have the benefit of past and current mayors, members of council and staff who follow European-style approaches to integrating community and transportation planning with future visions of the cities they want to be, in addition to the needs of the community today.

Lastly, both have looked to Vancouver for inspiration, while also plowing ahead with different strategies that meet their specific conditions; mostly, this involves being perched on the sides of slopes, and  squeezed between bridges and highways controlled by other levels of government.

Does British Columbia, population ~5 million, have the potential to build a few Delfts? Perhaps. According to small places, at a population of 17 million, “there are a couple dozen Delfts in the Netherlands.” So yes, a few more Delfts is the least we can do.

Delft will no doubt continue to inspire comparison and challenge for the Lower Mainland, as the city will soon be home for Chris & Melissa Bruntlett, Vancouver authors of Building the Cycling City. (New employer? The Dutch Cycling Embassy, natch!)

Read more about their move here, and tune into their future adventures between spoor en centrum.

Take a walk on the Arbutus Greenway in this unusually warm January weather. Stop at 11th Avenue and look north towards the mountains.

Then turn around and look south. There are cars along the greenway. How is that happening?

Somehow the section of the greenway between 11th and 12th Avenue has become a car storage facility on the west side of the greenway. No indication on how those vehicles got there, But since there is no road…

 

 

Matt Higginson, Marie-Claude Nadeau, Kausik Rajgopal, McKinsey, Jan 10, 2019

---

The Occam problem is this: unless and until blockchain becomes the simplest and most effective technology to do a job (any job) it will not be widely adopted. Yet despite huge investments, blockchain has yet to meet this challenge. We shouldn't be surprised. "It is an infant technology that is relatively unstable, expensive, and complex. It is also unregulated and selectively distrusted." As I commented to a colleague today, the applications of blockchain will not be the obvious ones (like, for example, registering credentials) but rather the rhizomatic ones (where, for example, some underlying technology (like, say, merkle trees) spreads from industry to industry in a generally underground manner.

Web: [Direct Link] [This Post]

Scott McLean, Jaya Dixit, Canadian Journal of Education, Jan 10, 2019

---

This paper (27 page PDF) uses the frame of 'identity work' described by Snow and Anderson to interpret identity statements made by interviewees describing their experience with self-help books. Snow and Anderson identified four activities common to the practice of identity work including "verbal construction and assertion of personal identities." This activity has two modes, 'distancing' and 'embracement', and many of the responses to the survey were found to fit neatly into one of the other of these modes. The authors write that these results should suggest to us that "learning is more than the straightforward reception of content; the reception of content is influenced by one’s positioning of oneself as a subject vis-à-vis the content, the person or people communicating that content, the media through which that content is being communicated, and other recipients of that content."  Image: SWSCmedia.

Web: [Direct Link] [This Post]

Google is rolling out an important update to its first-party Podcasts app. Continue reading →

Google will no longer manufacturer its Chromecast Audio digital media player.

A Google spokesperson confirmed the product’s discontinuation in an email to MobileSyrup.

“Our product portfolio continues to evolve, and now we have a variety of products for users to enjoy audio. We have therefore stopped manufacturing our Chromecast Audio products,” said Google.

“We will continue to offer assistance for Chromecast Audio devices, so users can continue to enjoy their music, podcasts and more.”

Users can still buy them if they’re still in stock at stores, but the product isn’t expected to get restocked.

First announced in September 2015, Google’s smart audio jack allowed users to turn any speaker or audio system into a connected speaker. In the simplest terms, the device was a wireless audio receiver.

The product’s discontinuation was first spotted by a U.K.-based Redditor who posted that Google cancelled their Chromecast Audio order, with customer support confirming that it was because the product is no longer in production.

Since Google is getting rid of the Chromecast Audio, it’s anticipated that the company is going to replace it with some other device or add an audio jack to the Google Home Mini to release a product similar to the Amazon Echo Dot.

Via: Android Central

The post Google confirms Chromecast Audio has been discontinued appeared first on MobileSyrup.