Rolandt

I haven’t written any magics for IPython before (and it probably shows!) but I started sketching out some magic for the Contentmine command-line container I described in Using Docker as a Personal Productivity Tool – Running Command Line Apps Bundled in Docker Containers,

What I’d like to explore is a more general way of calling command line functions accessed from arbitrary containers via a piece of generic magic, but I need to learn a few things along the way, such as handling arguments for a start!

The current approach provides crude magic for calling the contentmine functions included in a public contentmine container from a Jupyter notebook running inside a container. The commandline contentmine container is started from within the notebook contained and uses a volume-from the notebook container to pass files between the containers. The path to the directory mounted from the notebook is identified by a bit of jiggery pokery , as is the method for spotting what container the notebook is actually running in (I’m all ears if you know of a better way of doing either of these things?:-)

The magic has the form:

%getpapers /notebooks rhinocerous

to run the getpapers query (with fixed switch settings for now) and the search term rhinocerous; files are shared back from the contentmine container into the .notebooks folder of the Jupyter container.

Other functions include:

%norma /notebooks rhinocerous
%cmine /notebooks rhinocerous

These functions are applied to files in the same folder as was created by the search term (rhinocerous).

The magic needs updating so that it will also work in a Jupyter notebook that is not running within a container – this should simply be just of case of switching in a different directory path. The magics also need tweaking so we can pass parameters in. I’m not sure if more flexibility should also be allowed on specifying the path (we need to make sure that the paths for the mounted directories are the correct ones!)

What I’d like to work towards is some sort of line magic along the lines of:

%docker psychemedia/contentmine -mountdir /CALLING_CONTAINER_PATH -v ${MOUNTDIR}:/PATH COMMAND -ARGS etc

or cell magic:

%%docker psychemedia/contentmine -mountdir /CALLING_CONTAINER_PATH -v ${MOUNTDIR}:/PATH
COMMAND -ARGS etc
...
COMMAND -ARGS etc

Note that these go against the docker command line syntax – should they be closer to it?

The code, and a walked through demo, are included in the notebook available via this gist, which should also be embedded below.

---

You get in to work after a long weekend and that little number flashes away. 159 unread emails....

On Monday May 16th, you’re invited to join a webinar with Ron Friedman and I.

Ron is the author of the The Best Place To Work, my favourite book of the past year.

Ron is going to talk about what the world’s best workplaces do differently, the science of getting things done, how to motivate employees towards excellence, and the psychology behind productive work teams.

You can sign up here: https://attendee.gotowebinar.com/register/4439499572157766914

My colleague Sarah Hawk, recently teamed up with HigherLogic to publish this free eBook on automation and motivation. I strongly recommend you download a free copy.

User security is paramount. Vulnerabilities can weaken security and ultimately harm users. We want people who identify security vulnerabilities in our products to disclose them to us so we can fix them as soon as possible. That’s why we were one of the first companies to create a bug bounty program and that’s why we are taking action again – to get information that would allow us to fix a potential vulnerability before it is more widely disclosed.

Today, we filed a brief in an ongoing criminal case asking the court to ensure that, if our code is implicated in a security vulnerability, that the government must disclose the vulnerability to us before it is disclosed to any other party. We aren’t taking sides in the case, but we are on the side of the hundreds of millions of users who could benefit from timely disclosure.

The relevant issue in this case relates to a vulnerability allegedly exploited by the government in the Tor Browser. The Tor Browser is partially based on our Firefox browser code. Some have speculated, including members of the defense team, that the vulnerability might exist in the portion of the Firefox browser code relied on by the Tor Browser. At this point, no one (including us) outside the government knows what vulnerability was exploited and whether it resides in any of our code base. The judge in this case ordered the government to disclose the vulnerability to the defense team but not to any of the entities that could actually fix the vulnerability. We don’t believe that this makes sense because it doesn’t allow the vulnerability to be fixed before it is more widely disclosed.

Court ordered disclosure of vulnerabilities should follow the best practice of advance disclosure that is standard in the security research community. In this instance, the judge should require the government to disclose the vulnerability to the affected technology companies first, so it can be patched quickly.

Governments and technology companies both have a role to play in ensuring people’s security online. Disclosing vulnerabilities to technology companies first, allows us to do our job to prevent users from being harmed and to make the Web more secure.

The deterioration in the trend of Swiss watch exports observed since July 2015 gathered pace in March. With a decline of 16.1% compared to last year their value totalled only 1.5 billion francs, making these the lowest March figures since 2011. The scale of the downturn is also unusual, since we must go back to the crisis of 2009 to find rates of variation of this order.

My wonder.

More >

Pondering how to put together some Docker IPython magic for running arbitrary command line functions in arbitrary docker containers (this is as far as I’ve got so far), I think the commands must include a couple of things:

1. the name of the container (perhaps rooted in a particular repository): psychemedia/contentmine or dockerhub::psychemedia/contentmine, for example;
2. the actual command to be called: for example, one of the contentine commands: getpapers -q {QUERY} -o {OUTPUTDIR} -x

We might also optionally specify mount directories with the calling and called containers, using a conventional default otherwise.

This got me thinking that the called functions might be viewed as operating in a namespace (psychemedia/contentmine or dockerhub::psychemedia/contentmine, for example). And this in turn got me thinking about “big-L, big-A” Linked Applications.

According to Tim Berners Lee’s four rules of Linked Data, the web of data should:

1. Use URIs as names for things
2. Use HTTP URIs so that people can look up those names.
3. When someone looks up a URI, provide useful information, using the standards (RDF*, SPARQL)
4. Include links to other URIs. so that they can discover more things.

So how about a web of containerised applications, that would:

1. Use URIs as names for container images
2. Use HTTP URIs so that people can look up those names.
3. When someone looks up a URI, provide useful information (in the minimal case, this corresponds to a Dockerhub page for example; in a user-centric world, this could just return a help file identifying the commands available in the container, along with help for individual commands; )
4. Include a Dockerfile. so that they can discover what the application is built from (also may link to other Dockerfiles).

Compared with Linked Data, where the idea is about relating data items one to another, the identifying HTTP URI actually represents the ability to make a call into a functional, execution space. Linkage into the world of linked web resources might be provided through Linked Data relations that specify that a particular resource was generated from an instance of a Linked Application or that the resource can be manipulated by an instance of a particular application.

So for example, files linked to on the web might have a relation that identifies the filetype, and the filetype is linked by another relation that says it can be opened in a particular linked application. Another file might link to a description of the workflow that created it, and the individual steps in the workflow might link to function/command identifiers that are linked to linked applications through relations that associate particular functions with a particular linked application.

Workflows may be defined generically, and then instantiated within a particular experiment. So for example: load file with particular properties, run FFT on particular columns, save output file becomes instantiated within a particular run of an experiment as load file with this URI, run the FFT command from this linked application on particular columns, save output file with this URI.

Hmm… thinks.. there is a huge amount of work already done in the area of automated workflows and workflow execution frameworks/environments for scientific computing. So this is presumably already largely solved? For example, Integrating Containers into Workflows: A Case Study Using Makeflow, Work Queue, and Docker, C. Zheng & D. Thain, 2015 [PDF]?

A handful of other quick points:

• the model I’m exploring in the Docker magic context is essentially stateless/serverless computing approach, where a commandline container is created on demand and treated in a disposable way to just run a particular function before being destroyed; (see also the OpenAPI approach).
• The Linked Application notion extends to other containerised applications, such as ones that expose an HTML user interface over http that can be accessed via a browser. In such cases, things like WSDL (or WADL; remember WADL?) provided a machine readable formalised way of describing functional resource availability.
• In the sense that commandline containerised Linked Applications are actually services, we can also think about web services publishing an http API in a similar way?
• services such as Sandstorm, which have the notion of self-running containerised documents, have the potentially to actually bind a specific document within an interactive execution environment for that document.

Hmmm… so how much nonsense is all of the above, then?

Global temperature is on the rise, as most of us know. Ed Hawkins charted it in this spiral edition of temperature over time.

Spiralling global temperatures from 1850-2016 (full animation) https://t.co/YETC5HkmTr pic.twitter.com/Ypci717AHq

— Ed Hawkins (@ed_hawkins) May 9, 2016

See also the Quartz chart that uses a standard coordinate system but stacks lines on top of one another.

Tags: temperature, time series, weather

Last week at RailsConf in Kansas City, Terence Lee and Richard Schneeman of Heroku’s Ruby Task Force sat down with the legendary Aaron Patterson (AKA tenderlove).

Aaron has been working hard to make Ruby three times faster — a goal that Matz called Ruby 3x3. Along the way, Aaron has discovered that Ruby may face a hard decision. On one side, Ruby can continue to be the productive, general-purpose scripting language that it looks like today. But the other side of Ruby is that it’s used to run long-running processes in Rails applications, pushing it to be more performant, strongly-typed, and memory-heavy. Ruby can't prioritize both.

To find out where Aaron thinks Ruby’s going, you can read the abridged transcript below the fold — but to hear all about his new job at Github, Ruby performance, mechanical keyboards, grumpy cats, and more, you should listen to the whole recording right here.

---

Richard Schneeman: The stuff you've been working on — we've seen you tweeting a lot about Ruby performance — how's that coming along?

Aaron Patterson: Good, although, we just stepped out of Koichi's talk, and he basically ruined my talk [laughs]. But I've been working on different stuff. Like loading precompiled byte code, and looking at faster method caches or different types of method caches. Other things I've been thinking about is just — I'll cover this in my talk tomorrow — but improving memory efficiency, that type of stuff. Mostly what i've been poking at.

Though in Koichi's talk, [he said] loading precompiled code isn't helping that much, but I got different numbers than he did, so I'm hopeful, I guess? The thing is though, when Koichi's putting out numbers, I watch and i'm like - he's probably right! So yeah, I'll be talking about that stuff tomorrow.

Richard: so you just threw out a bunch of performance optimization stuff. I'm kinda curious, where do you get inspiration from? Do you just pick up and say, 'i'm gonna try this technique?' or do you look at what other languages are doing, or are there resources or some VM implementer's handbook for performance that you're skimming off of?

Aaron: No, typically, I just read about stuff like what JRuby — to be honest, it's mostly me just talking to the JRuby folks. 'So, hey, what do you folks do to improve performance on your VM?' And they tell me that stuff, and I'm like, 'hm, I wonder if MRI does that? Oh no they don't? Let's try that!' So it's mostly just stealing. That's basically where I get my inspiration from — all the hard work that everyone else has done.

Terence Lee: Well I feel like all the great software, a lot of the great software eventually does that, right? I remember Yehuda was talking about how Ember will eventually steal all the great ideas from everyone else, and then it will be great again.

Aaron: I think it's interesting though — one thing, you know, I don't know, Koichi keeps talking about these incremental speedups and stuff, and he's like, 'well, it's not that much.' And I understand it's not that much, but cumulatively, we combine all these not-that-much optimizations together and eventually we'll be, you know, 3x faster. So I don't think he should discount that type of work.

Richard: Absolutely, Koichi has done some fantastic work.

Terence: So with regards to Ruby 3x3 and performance, I know last year you were working on a JIT [just-in-time compiler]. Is it still legit?

Aaron: No it's not. It's not [laughs]. It's too hard. It's way too hard. It's super hard. And there's concerns about, like, how much memory consumption, stuff like that. I haven't been working on that lately. Basically, I got a little bit going, and then I was like, you know what? This is hard. So I quit. [laughs]

Richard: Would any of the ahead-of-time compilation work help the JIT at all? Or is it really tangential?

Aaron: It's tangential. I mean you can do some stuff, like if you do ahead-of-time compilation and you, let's say, run your application and take statistics on the stuff that you execute, maybe you can take that bytecode and perform optimizations on it afterwards.

But that's not just in time. We did some analysis and we can take this bytecode and I can actually improve the performance of this bytecode based on the stats that we took from running the website or whatever. But that's not just-in-time, it means you have to run your slow code for a while to figure out what to do and modify the bytecode later. That's one technique you can do, but I don't think that technique is nearly as popular as just doing a JIT. Because the JIT, you're essentially doing that, but you're doing it then, while you're running.

Richard: Doing it live.

Aaron: Yes, doing it live. I mean I think that we'll have, I hope that we'll get a JIT, it would be nice.

Terence: Do you think a JIT is required to hit the 3x3 performance goal?

Aaron: Yeah, definitely. I absolutely believe we'll need a JIT for the 3x3.

Every time I've talked to Koichi about doing a JIT, he's like, 'no, dude, it'll take too much memory,' is basically the argument. So I know that memory requirements are important for people who run on platforms like Heroku, so having a low-memory solution is important. On the other hand, I keep thinking like, well, it may take more memory but we could have a flag for it, or something like that. You could opt-in, like ruby --fast, and have it go–

Terence: The JVM has development and server mode too.

Aaron: Yeah, exactly. You can flip that with flags and stuff. So I don't really see how memory usage would be that much of an argument. Though, when you look at the stuff that Chris Seton has done on Truffle Ruby, so Truffle + JRuby — that thing is super fast, but you'll notice that in presentations that they never talk about memory consumption. They never talk about it.

Terence: He also doesn't talk about the warm-up time.

Aaron: Nope, they don't talk about the warm-up time. Nope. Not at all. It was funny, I read his PhD thesis, which was really awesome, it was a good paper, I recommend people to read this thesis. But it's very specific: 'we do not care about' — there is a line — 'we do not care about memory.' [laughs]

Richard: I mean memory's cheap, right? Just go to that website, downloadmoreram.com! [laughs]

Terence: Presumably, if it initially took a lot of memory, there's work and optimizations that can be done to ratchet that down over time too, right?

Aaron: Sure. I guess… you can't please everybody. If you make it use more memory, people are gonna complain that it uses more memory. If you make it use less memory but it's slower, people complain that it's slow. Everybody's gonna complain.

Richard: I guess Ruby was a originally a general-purpose scripting language, and I think that that's some of the emphasis behind, 'we don't want to emphasize too much on too much memory usage.' Do you ever think that — [Aaron pulls out a keyboard] — do you ever think that attitude will change from Matz and the Ruby core to say, yes! We are more of a long-lived process type of a language? Or more like living with some of these tradeoffs, where maybe we run with this flag, but I guess then we kind of end up in Java land where there's like a billion flags?

Aaron: Well, I think we have to move in that direction. I mean I know that one thing I really like about Ruby, or why I love Ruby so much is part of that fast bootup. I can write a quick little script, run it and it's like, I did my job, now go off to do something else.

But on the other hand, when you consider how people are making a living off of Ruby, how Ruby is feeding your family, it's typically with long-lived server processes. You're writing websites and stuff with processes that live for a long time. If we wanna continue to use Ruby as a way to make money, we're gonna have to support that stuff long-term.

I'm sure that the attitude will change, but I don't think it's gonna be an extreme direction. It'll be somewhere in the middle. We'll find some sort of compromises we can make.

Poster sessions are an important part of any academic conference – providing a way for researchers (including both faculty and students) to share their research in a format that supports describing methods, discussion, and results as well as fostering discussion about the project. Normally, these posters are printed on large format printers, carefully rolled into tubes for travel, and hung from poster boards or walls in a conference venue. It works, but requires the posters to be completed days (or weeks) ahead of time to allow for layout and printing (and any revisions to fix typos or omissions). It also requires a the content to be static – it’s a printed poster – and the format usually involves a 4’x6′ sheet of paper packed with dense micro-print and footnotes.

When we were planning the 2016 University of Calgary Conference on Postsecondary Learning and Teaching, we knew it would be the first conference to be held in the new Taylor Institute for Teaching and Learning. It’s a facility designed from the ground up to be pervasively digital, and it felt wrong to be doing printed posters when we have 37 high quality screens for use in the learning studios.

So. We committed to doing the poster session in digital format. Something we’ve never done before. I’d never designed a digital poster – there are different affordances and constraints. Although the screens are full 1080p HD resolution, that’s actually much lower resolution and lower data density than people are used to with traditional large-format printed posters. We had to do some experimentation, but it turns out that it’s possible to take a poster file that has been prepared for print and display the PowerPoint or PDF version of the poster just fine on the 50″ displays on our collaboration carts1.

Another thing we had to plan for – the rooms where the posters would be displayed were being actively used during the conference. So, the rooms would need to be combined by raising 2 Skyfold walls and moving all of the tables and chairs out of the way. It took a grand total of 40 minutes from start to finish, and the entire combined Learning Studios ABC was ready for the poster session. Amazing. Many hands make light work, and the technology worked flawlessly2.

After the poster session (which worked – I was totally not anxious about that. At all.), I did some quick napkin math. We were using 18 of the displays, which works out to 75 feet of digital posters. And we only used half of the Collaboration Carts in the building3. Wow.

I drafted a one-page “How to design a digital poster” document for participants to get an idea of what might be involved. It grew to 2 pages. It’s a lame and incomplete guide, but I needed to give something to the people who were being asked to prepare digital posters… I also ran a drop-in session for people to try their poster designs, and my team consulted with several people about how they might prepare their material for the poster session. Best team ever.

We realized that most people will be familiar with creating PowerPoint and PDF files, and our Collaboration Carts4 don’t have the full Microsoft Office suite, so we needed to find a way to reliably display these standard files through a web interface. I initially thought of using Google Slides (as indicated in the howto document), but quickly discovered that Slides renders many things… non-optimally. But – there is a company that has some experience with Office files, and who provides a cloud-based web renderer for such things.

OneDrive to the rescue. The PowerPoint rendering is flawless, and the more-conventional posters with microtext rendered perfectly – and were completely legible on the 50″ HD displays! I set up a shared OneDrive folder so my team had access to upload copies of peoples’ poster files (most showed up with a PowerPoint file on a USB thumbdrive), so a quick drag-and-drop to the shared OneDrive folder took care of webifying the poster for display during the session. Easy peasy.

But not all posters were conventional print-posters-displayed-on-digital-display things. A handful tried something completely new – one team created an amazing graphic poster inspired by Nick Sousanis’ work on Unflattening. Amazing. And it won the “Best Poster” prize!

I created my poster (in collaboration / based on work by Brian Lamb) as a self-playing looped video file (authored in Keynote and Procreate ) – and really struggled with the format. The pacing is difficult to get right, because the poster runs in the background to foster conversation. For a video poster, people may not be able to stay for long, and wind up missing large chunks of the concept. I’ll work on it some more… Perhaps with a layer of navigation and interactivity, so participants can trigger brief video (or other) segments as they explore, rather than sitting and watching something like a museum exhibit…

Quick observations from the session… It was by far the highest energy academic poster session I’ve ever seen. There wasn’t the usual “I’m here for the wine and food, and will talk to enough people that I won’t feel guilty about leaving early” kind of thing – people were genuinely having great discussions. Fantastic. We actually had to kick people out of the room after the end of the session because we had to reset the room for conference mode before day 2 of the conference (and we only had our volunteers for so long, so needed to finish that conversion before they headed out). We had to give 4 announcements, flash the lights, and eventually just throw the switch to turn off the poster displays before people started leaving. Amazing. I’ve never seen such a thing. I don’t know if it was a result of the novelty of the session – nobody had seen digital posters done in that way, at that scale, before – or if it’s yet another indicator that we have an absolutely incredible community of instructors and students at the University of Calgary…

One thing we’ll need to do is facilitate a series of sessions throughout the year to provide opportunities for people from across campus to come together and think creatively about what a digital academic poster could be – how can interactivity be a part of the experience? How can live data or external media be pulled in? What happens when the design constraints of a physical large-format printed poster are let go, and a poster is designed to take advantage of the affordances offered by the medium?

If anyone knows of any really good resources to help support mindful design of digital/interactive academic posters, I’d love to see them!

• Nancy Chick [wrote a brief essay about creating posters for the scholarship of teaching and learning, and visual representation of concepts](http://sotl.ucalgaryblogs.ca/posters/).

1. we REALLY need to come up with a better name for these things – 50″ touch-enabled display, embedded Windows PC for Internet connectivity, and a Mersive Solstice Pod, on wheels, times 37. Collaboration Cart doesn’t really work, but it’s all we’ve come up with that isn’t a corny forced acronym… The cart things are able to be moved throughout the learning studios on the main floor of the building, and the actual hardware that runs them is safely tucked into a server room on another floor – all we have on the carts themselves is the display, and a converter to send the video signal down from the server room over HD-BaseT and then into the display via HDMI, and another wire that sends the USB signal for the touch interface over Ethernet up to the server room
2. which was a huge concern for me – this is the first full-scale event we’ve run in the building, aside from grand-opening types of things – and was also the first time all of the Carts were fired up and running independently (usually they’d been run in presentation mode, mirroring the instructors’ display that’s on the Big Screen, but for this session, each Cart was a separate thing which put a HUGE load on the network because of all of the video being chucked around to drive the units…
3. again. Better name?
4. again. Horrible name.

There's a debate running around a few posts recently on the question of whether to reuse old presentations or offer new ones.

Martin Weller writes about it in The New or Reused keynote Dilemma. There are advantages and disadvantages to both, he says.

This is in response to a James Clay post on The Half Life of a Keynote. He writes,

I know of one individual who delivered forty eight virtually identical keynotes over a four year period across different events, I never even came close to that!

I think I might know who that is; I think I've seen some of them.

How do you decide whether or not to reuse a previous presentation? Clay writes,

Sometimes I feel that I have a back catalogue of keynotes and that though I may want to deliver the new album, people would rather hear the classic hits from the past!

This carries flavours of Kathy Sierra's attitude, which Alan Levine cites in a comment on Weller's post:

this presentation is a user experience. And if it's a user experience, then what am I? Ah... now we’re at the place where stage fright starts to dissolve. Because if the presentation is a user experience, than I am just a UI. That’s it. I am a UI. Nothing more... I am not important. What is important is the experience they have. My job is to provide a context in which something happens for them.

OK, there's a point to that, but if I am indeed nothing more than a UI, then they could bring in just anyone to deliver the same content, or they could just play one of my videos. But I don't thunk that's what people want.

I've done more than 350 presentations. As Jerry Seinfled would say, "that's a lot". And as you can see from my Presentation Page, I rarely if ever use the same presentation, and while I reuse slides, you can actually see them evolve over time.

In other words, I treat each presentation as a new creation, and strive as much as possible to give people a unique experience. It's harder, I think - a lot harder.

As Martin Weller says, it takes time. Now it takes a lot less time for me than for him, but for me a presentation, even if it's only an hour, can be a full day event. But I consider it time well spent.

And alsso, I think, it's what people have come to expect and want from me. Because I'm not just delivering content. Lt me explain.

I sometimes think in terms of the music metaphor the way James Clay does. But the presentation isn't the song or the album.

For me, my 'greatest hits' are my ideas. I've had a bunch of them over the years, some more popular than others. Like all greatest hits, they're derived from previous work done by others, but with a solid dose of my own creativity.

You're probably familiar with them (at least, I hope you are). Here's some of them:

- relevant similarity
- distributed online community
- hacking memes
- rules for good technology
- learning objects as tools for conversation
- knowing the future by reading the signs
- learning management as content syndication (gRSShopper)
- syndicated learning audio streams (Ed Radio)
- topic-based learning content feeds (Edu-RSS)
- resource profiles
- community-based sustainability for OERs
- distributed digital rights management
- syndication-based adaptive workflow (Synergic)
- learning networks qua networks that learn
- personal learning environments
- the semantic condition (autonomy, diversity, openness, interactivity)
- the Downes theory of learning
- personal prrofessional development (relevance, usability, interactivity)
- the massive open online course (MOOC)
- content as a McGuffin
- the critical literacies
- learning as recognition
- personal learning

And there's a lot more; I'm not exactly Prince but I have a pretty good catalogue. And together these ideas (along with a liberal borrowing of iddeas from other people) form a comprehensive theory of learning which has come to be known under the heading of 'connectivism'.

But the think with ideas, as opposed to content, is that you're not just an interface. You don't just deliver them. You have to use them (and hopefully in a way that prompts the audience to use them as well).

Like Prince, what I feel is that I am delivering an experience. The content per se is irrelevant (I could literally talk for an hour on any topic, without notes; I've learned how). My ideas are the tools I use to frame the talk, to give it a perspective.

Having said that, there are two major principles I use when giving a presentation; each of these constitute key reasons why each presentation must be unique, at least for me.

The first is the Mike Bullard approach to comedy:

- Find something you have in common with the audience
- Turn them around to make them see it from your perspective
- Make them laugh at themselves

Now I do like to make people laugh :) but my final point is:

- Make them learn something about themselves

And that's the basis of the experience. I come into a presentation not thinking that the audience is lacking something which I can provide, I come in thinking that the audience already has the essentual skills or abilities, which I can help them realize.

This means every presentation is different, because every audience is different. Even the same group from the same place (as I discovered delivering this talk three times in two days to the same conference). Every place is different. Every context is different.

To use the jargon of the discipline, my talks are localized. But for me, localization is a bit different; it's a process where I fuse my own understanding of the locale (which you see though my photos I take, the references I make to other talks in the same conference, and the stories I tell about my local experience, plus a je ne sais quoi). You can't repeat that one talk to the text.

And, of course, my talks are interactive. They're not interactive in a typcial sense (they're not, for example, 'question and answer' sessions). But like any experienced presenter, I read the audience. I adapt what I'm doing on the fly. I sometimes make use of backchannels (I built my own software to do this before Twitter became a thing).

The second principle, which is essential for removing the pre-presentation jitters (as Kathy Sierra says, they never disappear), is this:

- Love your audience

This piece of wisdom is attributed to Luciano Pavarotti. As an opera singer he has a unique challenge: he is rendering interpretations of songs that aren't even his. Anyone could sing them. Yet the audience is in place to hear Pavarotti sing them. It isn't the content, it's the experience.

Loving your audience isn't just a one-way relationship. It's not simply about serving your audience or meeting their needs, as Kathy Sierra suggests. It's more. It's a sharing of the experience, where I bring what I have, and the audience brings what it has, and we see what we can create.

People want to be heard, they want to contribute (before, after and during the talk) and I want to let them, because each piece of audience engagement leads to the next idea (and my unique skill, if I have one, is to be able to recognize them, pull them out of the conversation like a diamond in the rough, polish them, and share them for all of us to see, as something we created).

Finally, there's one other reason why keep each presentation unique: since 2004, when James Morrison suggested it to me, I have recorded nearly every talk, and so now the slides, audio and video for each of my talks is presented as its own presentation page. As I said, there are more than 350 of them. And just as I could not post 350 identical essays, so also I can't post 350 identical presentations.

Not everyone needs to do it this way. But this is what I do.

Apple appears to be shortening review times for new app and update submissions to the App Store. According to data collected by independent app review tracking website AppReviewTimes and as reported by Bloomberg today, review times have approached 2 days as opposed to the 7-10 days it took Apple to review apps in the past.

Alex Webb, writing for Bloomberg:

Apple Inc. has cut the approval time for new submissions to its App Store from more than a week to less than two days, part of a broader push to increase revenue from services including mobile applications.

The accelerated pace allows app developers to fix bugs faster, try out new features more regularly and better react to market changes, while building developer loyalty to Apple’s iOS mobile operating system. The mean approval time has fallen from 8.8 days a year ago to 1.95 days in the past two weeks, according to AppReviewTimes.com, which analyzes user-submitted data. In December, the average was more than five days.

The Mac App Store, which is smaller in volume than the iOS App Store, is currently seeing app review times of 24 hours.

In addition to AppReviewTimes' crowdsourced data, anecdotal experience from my own conversations with developers indicate that, particularly over the past two weeks, approval times of 2-3 days have become quite common for iPhone and iPad apps.

Apple's App Review has been a contentious topic in the Mac and iOS developer community for years. In March, we surveyed over 170 developers and, among various issues such as inconsistent enforcement of guidelines and lack of communication, the majority of developers noted that App Review needed to be faster.

App Review needs to get faster; one week is too long. This was a nearly universal sentiment amongst the developers surveyed. In addition to the obvious “hire more people”, there were a range of other suggestions for how Apple might accomplish faster reviews.

In March, we were also able to put together a chart plotting iOS App Store review times since January 2015 using 7,000 data points provided by AppReviewTimes. As we noted then, the average "rarely dropped below 6 days and was often at 8 or more days". Having a processing time of 2 days is, therefore, quite an improvement on Apple's part.

The question is whether such change is a temporary anomaly or if it will be the new standard going forward. Bloomberg's story associates shorter review times with Apple's goal to retain developers on their platforms and increase revenue from the App Store. It's important to note, though, that Apple hasn't promoted shorter review times as a recent addition to their developer program. On their App Review Status webpage, the company simply states that 99% of new app submissions and updates for iOS and Mac have been reviewed in the last 5 business days as of May 6, 2016.

However, data from AppReviewTimes has proven to be accurate in the past, and I'd tend to believe that review times are indeed at the lowest they've ever been in an effort to improve App Review and build up developer goodwill ahead of WWDC.

Apple's slow app review times have become a staple of the app economy over the past several years, with developers having to plan app releases and bug fix updates well in advance, often to their detriment, to cope with Apple's slow process. A turnaround time of 2 days – and even if Apple manages to stay between 2-4 days on average – could mean happier developers, faster releases and updates, and, as a consequence, happier customers.

It'll be interesting to see if these shorter review times will last until WWDC and if they're part of a bigger plan to meaningfully improve App Review for developers.

---

Like MacStories? Become a Member.

Club MacStories offers exclusive access to extra MacStories content, delivered every week; it's also a way to support us directly.

Club MacStories will help you discover the best apps for your devices and get the most out of your iPhone, iPad, and Mac. Plus, it's made in Italy.

Join Now

Stephen Downes, Half an Hour, May 15, 2016

---

I come into a presentation not thinking that the audience is lacking  something which I can provide, I come in thinking that the audience already  has the essential skills or abilities, which I can help them realize. This means every presentation is different, because every audience is different.

[Link] [Comment]

There is an excellent Tim Minchin song called If You Open Your Mind Too Much, Your Brain Will Fall Out. I'm sad to report that the same is also true of your data and your science.

At this point in the story I'd like to introduce you to Emil Kirkegaard, a self-described "polymath" at the University of Aarhus who has neatly managed to tie every single way to be irresponsible and unethical in academic publishing into a single research project. This is going to be a bit long, so here's a TL;DR: linguistics grad student with no identifiable background in sociology or social computing doxes 70,000 people so he can switch from publishing pseudoscientific racism to publishing pseudoscientific homophobia in the vanity journal that he runs.

Yeah, it's just as bad as it sounds.

The Data

Yesterday morning I woke up to a Twitter friend pointing me to a release of OKCupid data, by Kirkegaard. Having now spent some time exploring the data, and reading both public statements on the work and the associated paper: this is without a doubt one of the most grossly unprofessional, unethical and reprehensible data releases I have ever seen.

There are two reasons for that. The first is very simple; Kirkegaard never asked anyone. He didn't ask OKCupid, he didn't ask the users covered by the dataset - he simply said 'this is public so people should expect it's going to be released'.

This is bunkum. A fundamental underpinning of ethical and principled research - which is not just an ideal but a requirement in many nations and in many fields - is informed consent. The people you are studying or using as a source should know that you are doing so and why you are doing so.

And the crucial element there is "informed". They need to know precisely what is going on. It's not enough to simply say 'hey, I handed them a release buried in a pile of other paperwork and they signed it': they need to be explicitly and clearly informed.

Studying OKCupid data doesn't allow me to go through that process. Sure: the users "put it on the internet" where everything tends to end up public (even when it shouldn't). Sure: the users did so on a site where the terms of service explicitly note they can't protect your information from browsing. But the fact of the matter is that I work in this field and I don't read the ToS, and most people have a deeply naive view of how 'safe' online data is and how easy it is to backtrace seemingly-meaningless information to a real life identity.

In fact, gathering of the data began in 2014, meaning that a body of the population covered had no doubt withdrawn their information from the site - and thus had a pretty legitimate reason to believe that information was gone - when Kirkegaard published. Not only is there not informed consent, there's good reason to believe there's an implicit refusal of consent.

The actual data gathered is extensive. It covers gender identity, sexuality, race, geographic location; it covers BDSM interests, it covers drug usage and similar criminal activity, it covers religious beliefs and their intensity, social and political views. And it does this for seventy thousand different people. Hell, the only reason it doesn't include profile photos, according to the paper, is that it'd take up too much hard-drive space.

Which nicely segues into the second reason this is a horrifying data dump: it is not anonymised in any way. There's no aggregation, there's no replacement-of-usernames-with-hashes, nothing: this is detailed demographic information in a context that we know can have dramatic repercussions for subjects.

This isn't academic: it's willful obtuseness from a place of privilege. Every day, marginalised groups are ostracised, excluded and persecuted. People made into the Other by their gender identity, sexuality, race, sexual interests, religion or politics. By individuals or by communities or even by nation states, vulnerable groups are just that: vulnerable.

This kind of data release pulls back the veil from those vulnerable people - it makes their outsider interests or traits clear and renders them easily identifiable to their friends and communities. It's [happened before] (https://www.reddit.com/r/legaladvice/comments/3edf1s/imagaysinglemanfromacountrywhere_gaydeath/). This sort of release is nothing more than a playbook and checklist for stalkers, harassers, rapists.

It's the doxing of 70,000 people for a fucking paper.

The paper

And speaking of the paper, let's talk about what fascinating and vital research that overrides the pillar of informed consent this data was used for.

Oh: it wasn't.

This data release wasn't as part of a wider work - more the other way around. It was a self-contained data release of the type beloved by the open data movement (and me!): making information available so other researchers could pick it up and answer interesting questions. The paper attached to it is just to explore the data briefly and explain what it is.

On its own, this isn't a problem. It's a pretty nice thing to do. But when the data is this toxic, you should be able to point to an actual research justification for why this is a valuable way to spend your time, and why the questions it can answer justifies your grossly improper behaviour.

And that's when things go from "terribly poor judgment" to "actively creepy". Some research questions were used, as a way of demonstrating what the data can be used for, and Kirkegaard's choices definitely say...something, about him.

His first research question was: what if gay men are basically just women? We have data on gender and we have data on sexuality; let's see if LGB people provide different answers from straight people for their given gender! Let's see if they're basically the opposite gender!

You'll be amazed to know he didn't end up incorporating this into the paper, presumably because the answer was "of course not, you bigot". But he did find time to evaluate whether religious people are just plain ol' idiots - the methodology for which is checking the user's response to various questions akin to IQ test entries. You know, the racist classist sexist thing.

As an aside, this kind of creepy superpositional homophobia is actually an improvement on much of the work I've found from Kirkegaard while digging into this, which is not superpositional at all: previous credits include such pseudoscience as arguing that letting low-IQ immigrants in will damage Danish society, and they should be subjected to brain implants and genetic engineering for entry, and (I wish this was a joke) checking whether immigrants commit more crime if they have tiny penises.

Neither of these questions are things that would pass actual institutional review as justifications for gathering data of this depth without informed consent. Neither of these questions are particularly original - there's literature on both, using information gathered by qualified people in responsible situations. And no vaguely competent peer reviewer would look at this methodology, this dataset, and this set of conclusions, and determine that it constituted a publication.

The process

At this point any of you who are still reading and haven't given up due to length, horror or disagreement might be going: how the heck did this ever get published? Even if the institution didn't catch it - because it was recreational, because it was outside the department's scope, whatever - the journal should've, right? There's no way any reputable psychology journal would have let this slide without asking some hard questions.

Well, that's where we descend yet further into academic ineptitude: you see, Kirkegaard submitted it to an open access journal called Open Differential Psychology, which is very very highly reputed. Mostly that's thanks to their work of their editor, a Mr...oh.

Emil Kirkegaard.

Yep, Kirkegaard is publishing in a journal he's the editor of, which isn't actually new for him - the thing looks pretty much like a vanity press. In fact, of the last 26 papers it "published", he authored or co-authored 13. When 50% of your papers are by the editor, you're not an actual journal, you're a blog.

Moreover, there wasn't actually any review. There's review for the paper, sure, but the underlying dataset? Pfft. Just throw that out, whatever. Which is pretty infuriating given that ODP reviews are handled by forum thread (open means open!) and the very first comment on the paper itself is that it is highly deficient in its approach to ethical questions around the collection of this data.

So no wonder this got out so easily, beyond the author's own callous irresponsibility; the task of checking it fell to nobody. Those checks that were done happened after the release, not before.

The response

Unsurprisingly, when Kirkegaard publicised that he'd done this, a lot of people took issue, including myself, Kelly Hills and Ethan Jewett. Kirkegaard's replies to questions opened with smug, self-assured 'pfft, I did nothing wrong':

That's just on specific, answerable questions; I didn't do any work to anonymise stuff, it's already public, who cares! Check with OKC? Eh, if I don't ask they can't say no.

When things get more abstract and philosophical, responses get worse. Kirkegaard first says that he'll not discuss ethics at all - to be fair, given the irresponsibility of his actions thus far the motivation may just have been "not digging any further" if he has a lick of sense, but I doubt it - and then that he will, but only when "the [Social Justice Warrior] storm" goes away.

This is not how academia works, at least not in the field Kirkegaard is publishing this data in and about: responsiveness to questions about the merits or implications of your work is as essential to good research as consent, as picking appropriate methods, as peer review. Refusal to discuss is not professional - and neither is throwing derogatory collective nouns at the people taking issue.

But there's no indication here that Kirkegaard cares about professionalism; a very, very generous read is that he's out of his depth in a space and field he doesn't understand, was never trained in how to do this right, and panicking. A less generous read is that he's privileged, willfully obtuse and deeply, deeply arrogant about where he has expertise and the work he does. Either way, this work is an embarrassing and unprofessional offering from an institute-affiliated researcher that puts vulnerable populations at risk.

The wider questions

While Kirkegaard himself engenders precisely zero sympathy - see the pseudoscientific racism above - there are wider questions here, for both academia and the open data community.

Academia needs to confront the fact that what happens outside the house follows you home. Kirkegaard is clearly out of his depth on everything he's doing here, and just as clearly, he hasn't been called on it. A lot of institutions and researchers I know insist that researchers restrict their work to areas where they can do a good job if they want support or university affiliation; they also insist those works go through the same ethical vetting that an in-school paper would, whether they're a hobby work or not.

It's safe to say that when you're in the linguistics department and one of your advisees spends his spare time arguing for racially-biased gene therapy while doxing marginalised people, you've fucked this bit up quite severely.

At the same time, though, this is not a situation where we can point to institutional standards and individual impropriety, fix them and call it done. Unprofessional behaviour of this scale and with this number of layers is rare. Messing up the ethical implications of your work is common; it happens with grad students all the time. Ethical training in grad school is quite clearly not fit for purpose.

This isn't to say it doesn't cover valuable bits; this is just to that what it covers is nowhere near the breadth of modern data or modern scenarios. Ethical training tends to be oriented around two core sets of examples and illustrations - biomedical shitshows like Tuskegee or Milgram, and traditional survey design and data. These are good things to cover, but they don't touch on the impact higher computing power has had on the ease of deanonymisation. They don't include the ambiguity of data's status and publicity on the internet, and the increased nuance it adds to the question of whether using something is Right or Wrong.

I'm not arguing lecturers shouldn't be teaching Tuskegee or Milgram. I'm just saying they should teach AOL as well.

The open data problem is a lot simpler; we need to stop treating openness as an inherently good thing. Don't get me wrong, I consider closed publishers to be festering sores on the body academic, but open data carries with it a risk to study participants. In the same way that papers go through a peer review process, datasets should also be evaluated and hammered and considered before they're released, and sent back for more anonymisation if they're not kosher. The rise of convenient code and data platforms has been amazing, and helped us be more transparent about our methods - but it's also pushed the responsibility of ensuring that releasing the data does not pose harm back on to the researcher.

As this incident shows a dozen times over, that's not something we can rely on.

(Update: there is now an open letter aimed at the parent institutions to get this properly dealt with. Please do sign if you agree with it, whether you're in academia or industry or simply a concerned citizen)

Before I got into working on add-ons I didn't realise the burden that the old style add-ons have created for everyone in the add-on. I do think add-ons are great and what they've done to help Firefox users and Firefox has been excellent. But the more I've understood the ecosystem, the more I'm concerned by it.

This is a negative post, focusing on the problems, but I don't think these are a surprise for people involved in add-ons. No matter how fondly some people might look at add-ons of the past, they have some real problems that haven't been addressed.

For users

• There is no information what an add-on does beyond the description. It might slow down the browser in various ways. It might mess with the user interface or settings.

• Because an add-on can do pretty well anything, the security can be problematic.

• For most users there is no difference between Firefox and the add-ons. There's no way to tell that an add-on caused problems or wierd behaviour.

For Firefox developers

• Because there is no layer between many of the XUL and XPCOM internals of Firefox, its really hard for Firefox developers to change things without being hampered by the effect on add-on developers. It's really hard to move Firefox forward when every change might break the experience for an unknown number of users.

• Problems reported in Firefox are harder to diagnose and add-ons are the first line of inquiry.

For Add-on developers

• Every release of Firefox can break your add-on because Firefox developers are changing the internals and progressing. You might get a notice about it if we can find it (see below).

• The API for what an add-on can do is as about as large the Firefox internals itself. Thats a huge, sporadically documented and continually shifting API. Developers shouldn't have to be Mozilla experts to produce an add-on.

For the add-ons team

• Every release the add-ons team trawls through bugs and finds things that could cause problems for add-ons. We then alter the addons.mozilla.org validator, run add-ons through them. Squint at the result, see if they sound right and then repeat. Then tell all the add-on developers about them.

• Reviewing add-ons places a security burden upon Mozilla to prevent malicious add-ons going through. Add-ons could be obfuscated or just plain complicated and add-on developers want their add-ons approved. This creates a huge amount of tension between everyone.

People have asked me questions about where WebExtensions will go and I have to refer back to this list and make sure that when we implement that feature we don't fall into the same trap. So to repeat: what's really important is to look back at the decisions that led to this and stop ourselves repeating these problems.

WebExtensions and the path forward is looking really good and I know that it will get so much better, I'm genuinely excited by that.

There are many traits that I value in developers, one I keep reiterating to anyone who will listen is developer responsibility. What does that mean?

So you write a peice of code based on a bug report or a feature request. You get it past code review, merge it and move to the next bug. Does that mean your responsibility ends there? No.

If you are working on something that is being deployed to a server, did you make sure it actually landed in on the production server? Did you do a quick check that it actually worked in production? Did it need in any metrics? Did you look at any logs to see if it threw some errors? Did something come up a few days later, a bug report maybe? Did you follow up on that?

Most of those questions and checks shouldn't take long to do, if they do then that's a problem.

But just throwing out code and then ignoring it afterward isn't ok. I've seen that happen and then been shocked to find that the developer never actually checked even very quickly that the feature went out. It created bugs or in some cases didn't even work because settings hadn't been flipped. That's not cool. It's your code, it's your responsibility.

...and by the way everyone on my team at the time of writing this is awesome at this.

In part so I don’t lose the recipe, here are some notes for getting up and running with IPython on a Lego EV3 brick.

The command lines are prefixed to show whether we’re running them on the Mac or the brick…

To start with, we need to flash a microSD card with an image of the ev3dev operating system. The instructions are on the ev3dev site – Writing an SD Card Image Using Command Line Tools on OS X. – , but I repeat the ones for a Mac here for convenience.

1. Download an image from the repository – I used the ev3dev-jessie-2015-12-30 image because the current development version is in a state of flux and the python bindings don’t necessarily work with it just at the moment…
2. Assuming you have downloaded the file to your home Downloads directory (that is, ~/Downloads), launch a new terminal and run:cd Downloads
   [Mac] unzip ev3-ev3dev-jessie-2015-12-30.img.zip
   [Mac] diskutil list
3. Put the microSD card (at least 2GB, but no more than 16GB, I think? I used a 4GB microSD (HC) card) into an SD card adapter and pop it into the Mac SD card reader. Check that it’s there and what it’s called; (you’re looking for the new disk…):
   [Mac] diskutil list
4. Now unmount the new disk corresponding the the SD card:diskutil unmountDisk /dev/disk1s1
   If you don’t see the /dev/desk1 listing, check that the write protect slider on your SD card holder isn’t in the write protect position.
5. We’re going to write some raw bits to the card (/dev/disk1 in my example), so we need to write to /dev/rdisk1 (note the r before the disk name). The write will take some time – 5 minutes or so – but if you’re twitchy, ctrl-T will show you a progress message). You’ll also need to enter your Mac password. (NOTE: if you use the wrong disk name, you can completely trash your computer. So be careful;-)
   [Mac] sudo dd if=~/Downloads/ev3dev.1900MB.img of=/dev/rdisk1 bs=4mGOTCHA: when I tried, I got a Permission Denied message at first. It seems that for some reason my Mac thought the SD card was write protected. On the SD card adapter is a small slider that sets the card to “locked” or “unlocked”. The SD card reader in the Mac is a mechanical switch that detects which state the slider is in and whether the card is locked. I don’t know if it was a problem with the card adapter or the Mac reader, but I took the card reader out of the Mac, changed the slider setting, put card reader back in, and did the unmount and then sudo dd steps again. It still didn’t work, so I took the card out again, put the slider back the other way, and tired again. This time it did work.
6. Once the copy is complete, take the SD card adapter out, remove the microSD car and put in in the EV3 brick. Start the EV3 brick in the normal way.

Hopefully, you should the brick boot into the Brickman UI (it may take two or three minutes, include a period when the screen is blank and the orange lights are ticking for a bit…)

Navigate the brick settings to the Networks menu, select Wifi and scan for your home network. Connect to the network (the password settings will be saved, so you shouldn’t have to enter them again).

By default, the IP address of the brick should be set to 192.168.1.106. To make life easier, I set up passwordless ssh access to the brick. Using some guidance in part originally from here, I accessed the brick from my Mac terminal and set up an ssh folder. First, log in to the brick from the Mac terminal:

[Mac] ssh robot@192.168.1.106

When prompted, the password for the robot user is maker.

This should log you in to the brick. Run the following command to create an ssh directory into which the shh key will be placed, and then exit the brick commandline to go back to the Mac command line.

[Brick] install -d -m 700 ~/.ssh
[Brick] exit

Create a new key on your Mac, and then copy it over to the brick:

[Mac] ssh-keygen -R 192.168.1.106
[Mac] cat ~/.ssh/id_rsa.pub | ssh robot@192.168.1.106 'cat > .ssh/authorized_keys'
You will be prompted again for the password – maker – but next time you try to log in to the brick, you should be able to do so without having to enter the password. (Instead, the ssh key will be used to get you in.)

If you login to the brick again – from the Mac commandline, run:

[Mac] ssh robot@192.168.1.106

you should be able to run a simple Python test program. Attach a large motor to input A on the brick. From the brick commandline, run:

[Brick] python

to open up a Python command prompt, and then enter the following commands to use the preinstalled ev3dev-lang-python Python bindings to run the motor for a few seconds:

[Python] import ev3dev.ev3 as ev3
[Python] m = ev3.LargeMotor('outA')
[Python] m.run_timed(time_sp=3000, duty_cycle_sp=75)

Enter:

[Python] exit

to exit from the Python interpreter.

Now we’re going to install IPython. Run the following commands on the brick command line (update, but DO NOT upgrade the apt packages). If prompted for a password, it’s still maker:

[Brick] sudo apt-get update
[Brick] sudo apt-get install -y ipython

You should now be able to start an IPython interpreter on the brick:

[Brick] ipython

The Python code to test the motor should still work (hit return it you find you are stuck in a code block). Typing:

[Brick] exit

will take you out of the interpreter and back to the brick command line.

One of the nice things about IPython is that we can connect to it remotely. What this means is that I can start an IPython editor on my Mac, but connect it to an IPython process running on the brick. To do this, we need to install another package:

[Brick] sudo apt-get install -y python-zmq

Now you should be able to start an IPython process on the brick that we can connect to from the Mac:

[Brick] ipython kernel

The process will start running and you should see a message of the form:

To connect another client to this kernel, use:
--existing kernel-2716.json

This file contains connections information about the kernel.

Now open another terminal on the Mac, (cmd-T in the terminal window should open a new Terminal tab) and let’s see if we can find where that file is. Actually, here’s a crib – in the second terminal window, go into the brick:

[Mac] ssh robot@192.168.1.106

And then on the brick command line in this second terminal window, show a listing of a the directory that should contain the connection file:

[Brick] sudo ls /home/robot/.ipython/profile_default/security/

You should see the –existing kernel-2716.json file (or whatever it’s exactly called) there. Exit the brick command line:

[Brick] exit

Still in the second terminal window, and now back on the Mac command, copy the connection file from the brick to your current directory on the Mac:

[Mac] scp robot@192.168.1.106:/home/robot/.ipython/profile_default/security/kernel-2716.json ./

If you have IPython installed on you Mac, you should now be able to open an IPython interactive terminal on the Mac that is connected to the IPython kernel running on the brick:

[Mac] ipython console --existing ${PWD}/kernel-2716.json --ssh robot@192.168.1.106

You should be able to execute the Python motor test command as before (remember to import the necessary ev3dev.ev3 package first).

Actually, when I ran the ipython console command, the recent version of jupyter on my Mac gave me a depreaction warning which means I would have been better running:

[Mac] jupyter console --existing ${PWD}/kernel-2716.json --ssh robot@192.168.1.106

So far so good – can we do any more with this?

Well, yes, a couple of things. When starting the IPython process on the brick, we could force the name and location of the connection file:

[Mac] ipython kernel -f /home/robot/connection-file.json

Then on the Mac we could directly copy over the connection file:

[Mac] scp robot@192.168.1.106:/home/robot/connection-file.json ./

Secondly, we can configure a Jupyter notebook server running on the Mac to so that it will create a new IPython process on the brick for each new notebook.

Whilst you can configure this yourself, it’s possibly easy to make use of the remote_ikernel helper:

[Mac] pip3 install remote_ikernel
[Mac] remote_ikernel manage --add --kernel_cmd="ipython kernel -f {connection_file}" --name="Ev3dev" --interface=ssh --host=robot@192.168.1.106

Now when you should be able to connect to a notebook run against an IPython kernel on the brick.

Note that it may take a few seconds for the kernel to connect and the first cell to run – but from then on it should be quite responsive.

To show a list of available kernels for a particular jupyter server, run the following in a Jupyter code cell:

import jupyter_client
jupyter_client.kernelspec.find_kernel_specs()

PS for ad hoc use, I thought it might be useful to try setting up a quick wifi router that I could use to connect the brick to my laptop in the form of an old Samsung Galaxy S3 android phone (without a SIM card). Installing the Hotspot Control app provided a quick way of doing this… and it worked:-)

PPS for a more recent version of IPython, install it from pip.

If you installed IPython using the apt-get route, uninstall it with:

[Brick] sudo apt-get uninstall ipython

Install pip and some handy supporting tools that pip may well require at some point:

[Brick] sudo apt-get install build-essential python-dev

Running:

[Brick] sudo apt-get install python-pip

would run an old version of pip – pip --version shows 1.5.6 – which could be removed using sudo apt-get remove python-setuptools.

To grab a more recent version, use:

[Brick] wget https://bootstrap.pypa.io/get-pip.py
[Brick] sudo -H python get-pip.py

which seems to take a long time to run with no obvious sign of progress, and then tidy up:

[Brick] rm get-pip.py

Just to be sure, then update it:

[Brick] sudo pip install --upgrade setuptools pip

which also seems to take forever. Then install IPython:

[Brick] sudo pip install -y ipython

I’m also going to see if I can give IPythonwidgets a go, although the install requirements looks as if it’ll bring down a huge chunk of the Jupyter framework too, and I’m not sure that’s all necessary?

[Brick] sudo pip install ipywidgets

For a lighter install, try sudo pip install --no-deps ipywidgets to install ipywidgets without dependencies. The only required dependencies are ipython>=4.0.0, ipykernel>=4.2.2 and traitlets>=4.2.0;.

The widgets didn’t seem to work at first, but it seems that following a recent update to the Jupyter server on host, it needed a config kick before running jupyter notebook:

jupyter nbextension enable --py --sys-prefix widgetsnbextension

PPS It seems to take a bit of time for the connection to the IPython server to be set up:

The Timeout occurs quite quickly, but then I have to wait a few dozen seconds for the tunnels on ports to be set up. Once this is done, you should be able to run things in a code cell. I usually start with print("Hello World!") ;-)

PPPS For plotting charts:

sudo apt-get install -y python-matplotlib

Could maybe even go the whole hog…

sudo apt-get install -y python-pandas
sudo pip install seaborn

PPPPS Here’s my current build file (under testing atm – it takes about an hour or so…) – ev3_ipy_build.sh, so:
[Mac] scp ev3_ipy_build.sh robot@192.168.1.106
[Brick] chmod +x ev3_ipy_build.sh
[Brick] sudo ./ev3_ipy_build.sh

[sourceode language=’bash’]sudo apt-get update
sudo apt-get install -y build-essential python-dev
sudo apt-get install -y python-zmq python-matplotlib python-pandas

wget https://bootstrap.pypa.io/get-pip.py
sudo -H python get-pip.py
rm get-pip.py
sudo pip install –upgrade setuptools pip

sudo pip install ipython ipykernel traitlets seaborn
sudo pip install –no-deps ipywidgets
[/sourcecode]

From Ian: One more in the series on new forms of living. $1800 doesn’t seem so bad in Vancouver, though I’m sure it would be pretty spendy elsewhere.

From co.exist:

.

Andrew Jackson, in the Globe and Mail Report On Business, writes about carbon, taxes, troubled economies and his view of a prosperous future (or at least less dangerous one).  To me, his thoughts are starting to seem like orthodoxy, and the truly radical views are held by those who advocate business as usual. What, he asks, would be the path to a lower-carbon world? And what would happen if we took it, and didn’t keep motoring along fossil-fuel freeway?

It is now often said that the environment and the economy are not in conflict. But it is even more true to say that addressing the crisis of global climate change in a serious way could revive a moribund global economy. . . .

. . . . But the world does, as the IMF underlines in a special feature on the energy transition, have a massive challenge to deal with in terms of the needed reduction of greenhouse gas emissions. . . .

The sheer scale of the needed energy transition is such that it could drive a global economic recovery. But only if our political leaders are up to the challenge.

Andrew Jackson is an adjunct research professor in the Institute of Political Economy at Carleton University in Ottawa and senior policy adviser to the Broadbent Institute.

 

From The New Yorker:

The car-sharing industry has its roots in mid-century Europe, whose coöperative vehicle pooling was mimicked, in the nineteen-eighties, by American nonprofits and city governments. But only with the spread of G.P.S. and smartphone technologies have today’s private companies been able to produce real efficiencies for the user and measurable benefits for the environment.

In 2010, Susan Shaheen, a transportation expert at the University of California, Berkeley, published a ten-year review of car-sharing data, concluding that the model basically works: those with new access to cars will drive more, but their impact is more than offset by that of car owners who begin using the programs. In other words, were car sharing to replace private car ownership in the city, the effect would likely be more drivers doing less driving. Car2Go bridges the gap between American hearts, which are still piston powered, and the typical American salary, which is just equal to the average cost of a new car.

.

For a local view, here is a summary from the Urban Land Institute BC of a seminar on car-sharing as part of the ‘Next Million’ series.

WeChat public accounts, China’s most influential social media channel for enterprises, are overwhelmingly dominated by tech, a study by social marketing startup Robin8 has found.

Robin 8 released the results of the study at CES on Thursday, which crawled through over half a million WeChat public accounts across one month.

Each account was scored according to a number of factors including their average page views per post, their total number of post page views, and their total number of ‘likes’ on posts.

The results unanimously towards technology as  the hottest topic on WeChat’s public accounts. According to Robin8’s ‘influence’ calculation, the top five WeChat public accounts were:

1.  科技美学 (Technology Aesthetics, our translation)
2.  纵观互联网 (Overview of the Internet, our translation)
3. 小米公司 (Xiaomi)
4. 互联网观察 (Internet Watch, our translation)
5. 科技每日推送 (Daily Tech Recommendations, our translation)

All five had more than half a million page views in April, with Xiaomi’s public account gaining 228,601 page views for a single post. Not all of the public accounts on Robin8’s top 100 list were companies – quite a few are operated by individuals, such as Yinghuan Li, a chief correspondent and tech journalist for The Time Weekly (时代周报) who is considered a wang hong or internet celebrity. Some accounts were incredibly prolific in April, such as 电脑报 (Computer Report, our translation), which published 163 posts. Others, such as Xiaomi’s public account, had less than ten.

In addition to the top 100 list, Robin8 also shared other insights from its one-month study, such as popular keywords and brands. Again, technology had a strong presence as “mobile phones,” “design,” “products”, and “technology” were identified as popular keywords, and tech brands such as Apple, Huawei, Tencent, and Alibaba were especially of interest to large numbers of WeChat users.

“In the future, we’ll publish more data like this,” Duan Yong, the CTO of Robin8, told TechNode. “We plan to launch a [wang hong] search engine, so everyone can search for this kind of information themselves.”

According to Robin8’s CEO, Miranda Tan, the company’s “wang hong search engine” will launch next Friday. The search engine will leverage the company’s natural language processing (NLP) algorithms and big data analytics to pull and analyze data from social platforms, such as Weibo and WeChat public accounts.

The Power of the Masses: China’s Wang Hong Phenomenon

Robin8’s results were part of a larger discussion on KOLs (Key Opinion Leaders) and wang hong in China. Though Robin8’s study revealed the popularity of technology-related public accounts, Mr. Yong said that public accounts on any topic have the potential to be influential.

“In every vertical, you can find a wang hong. In their small circle, they have influence. You can find wang hong in VR, robotics, electric cars – anything,” said Mr. Yong during a panel discussion following his presentation. Robin8’s product helps companies find wang hong, using content from social platforms like WeChat, Weibo, and Douban, and

“Anyone can be a wang hong, as long as they have a lot of reach, relevance, and influence,” said Arlene Ang, the CEO of OMD (Omnicom Media Group), a global communications agency that hosted Robin8’s presentation and the KOL panel discussion at CES Asia 2016. “It might be a cook, it might be a designer, it might be a teacher, as long as they have this relevance and influence.”

An increasing number of brands are seeing wang hong, from WeChat public accounts to live-streaming hosts, as a way to market their products, increase brand awareness, and produce interesting content with high conversion rates. Last month, one of China’s most well-known internet celebrities, online video comedian Papi Jiang, raised 22 million RMB (about $3.4 USD) in a bid from cosmetics startup Lily & Beauty during an ad auction.

Though Papi Jiang is exceptional, the diehard nature of wang hong fans is a unique characteristic of China’s internet celebrities. It’s that obsessive fandom that makes wang hong, even those with smaller or more niche fan bases, so valuable to brands. In addition, unlike companies, wang hong can come across as more genuine and sincere.

“I think [the] wang hong’s biggest selling point is they’re down-to-earth. That’s why a lot of consumers trust them,” said Ms. Ang. “They trust the content that is being put out by the wang hong.”

However, as one of the panelists noted, companies will need to be careful when dealing with wang hong. Unlike more traditional KOLs, such as actors and sports stars, wang hong are more independent and, in some ways, unpredictable.

“Part of the wang hong phenomenon…is about how they can help brands,” said David Li, the founder of Maker Collider, a platform that helps hardware makers bring their products to market. “But something you need to be aware of is, one day, if the wang hong and the company have a falling out, the wang hong can recreate one of the company’s products from scratch…perhaps making it even cheaper…and resell it on their own.”

Correction (5/13/2016 17:19): This post was updated to correct the spelling of Robin8’s CTO’s name.

With the latest update to Google Photos, Google is making shared albums smarter and more social by adding a couple of new features to them. Firstly, people whom you share an album to can now comment on the photos right from within the app. Continue reading →

Dark Sky, the popular weather app for iOS, is finally making its way to the Google Play Store. The app is known for its “scary precise” weather information and timely notifications, which for some reason plethora of other weather apps have been unable to match so far. Continue reading →

Parveen Kaler @kaler
Instagram changes. People hate it. Corollary: People are unable to make a change in their life because they are predisposed to hate change.

The recently released Redis 3.2 now has an official Geo API in the mainstream branch of the in-memory database. If you are curious on how you could use this, read on...

The Geo API has been around for a while, appearing in the Redis unstable branch about ten months ago and that was, in turn, based on work from 2014. There's a bit of history in that development process, which being practical folk we'll skip past and go straight to the stuff that makes your development day better.

At its simplest, the GEO API for Redis reduces longitude/latitude down into a geohash. Geohash is a technique developed in 2008 to represent locations with short string codes. The Geohash of a particular location, say Big Ben in London, would come out as "gcpuvpmm3f0" which is easier to pass around than "latitude 51.500 longitude -0.12455". The longer the string, the more precise the geohash code.

That encoding into a string is good for humans and URLs but it isn't particularly space efficient. The good news is geohashes can be encoded as binary and using 52 bits, a geohash gets down to 0.6 meter accuracy which is good enough for most uses. A 52-bit value which just happens to be able to be a small-enough integer to live in a Redis floating-point double safely and that's what the Geo API works with behind the scenes.

From the user's point of view though, it's all about the longitude and latitude – yes, we switch them around when working with the API – and working with locations. So let's make some locations:

> GEOADD locations -0.12455 51.5007 "Big Ben" -0.12520 51.50115 "Westminster Station" -0.11358 51.50482 "BFI IMAX" 
(integer) 3

locations is a Redis key and what follows is a list of longitudes and latitudes followed by labels for those locations. I just picked some places in Central London to work with. Those longitudes and latitudes are being hashed together to make a numeric geohash and the label is saved in a sorted set with the geohash as a score in a sorted set (ZSET). We're able to do this multiple times in one command line. We can now ask the geo set what it contains using sorted set commands:

> ZRANGE locations 0 -1
1) "Big Ben"  
2) "Westminster Station"  
3) "BFI IMAX"  

We can start querying this. Let's say we're at longitude -0.11759 latitude 51.50574 – or as I call it "just by the Royal Festival Hall" – and we want to know if any of those locations are within half a kilometre:

> GEORADIUS locations -0.11759 51.50574 500 m
1) "BFI IMAX"  

We use the GEORADIUS command on our geo set and then give it our coordinates. That's followed by a number and by a unit type for that number; in this case 500 metres (m for meters, mi for miles, km for kilometers and ft for feet). Well, we've found out it's within range. Let's find out how far by adding WITHDIST:

> georadius locations -0.11759 51.50574 500 m WITHDIST
1) 1) "BFI IMAX"  
   2) "295.9825"

So it's 295M – in a straight line – to the BFI IMAX. Let's open up the range to a kilometre and ask for the results in ascending order of distance:

> GEORADIUS locations -0.11759 51.50574 1 km WITHDIST ASC
1) 1) "BFI IMAX"  
   2) "0.2960"
2) 1) "Westminster Station"  
   2) "0.7335"
3) 1) "Big Ben"  
   2) "0.7392"

Note the distances are in the same unit as our query. If something is already in the set, we can query how close a member of the list is to other members. For that we have GEORADIUSBYMEMBER, so if we want to know what's within 100 meters of Westminster Station, we can ask using:

> GEORADIUSBYMEMBER locations "Westminster Station" 100 m withdist
1) 1) "Westminster Station"  
   2) "0.0000"
2) 1) "Big Ben"  
   2) "67.3659"

We asked for the distances so you can see that at 0.0M away is Westminster Station. "Big Ben" is a whole 67M away and if we widened our radius we'd find the IMAX is 900M away. There's a more direct way to find that distance though:

> GEODIST locations "Westminster Station" "BFI IMAX"
"902.1221"

The GEODIST command takes two members from a geo set and returns the distance between them (in meters by default but you can add mi, ft or km to a GEODIST query to get it in those units).

Well, now we've got distances, we also want to think about how we convert these Geo set members back into usable coordinates. The GEOPOS command comes into play here:

> GEOPOS locations "Big Ben"
1) 1) "-0.12454837560653687"  
   2) "51.50069897715604839"

That converts a Geo set member back into the longitude and latitude. You can also get the string-based geohash back with GEOHASH in the same way:

> GEOHASH locations "Big Ben"
1) "gcpuvpmm3f0"  

And that's it for the Geo API. Wait, you say, there's no command to remove a member of geo sets. Remember this is still a sorted set underneath, so to remove a member use Redis's ZREM like this:

> ZREM locations "Big Ben"
(integer) 1

And Big Ben is gone... from your set.

So that's the Geo API for Redis. It's a simple implementation ideal for quickly working out the proximity of locations to other locations, like restaurants as a user walks through a city, vehicles and garages or any other safe to approximate geo-data. If you want the whole GIS environment, you'll probably want to look to something like PostgreSQL and PostGIS, but if you just need the more common "What's close to me?" query done quickly, Redis's Geo API should be a first port of call.

The gas-tank indicator you didn’t know was there…

When you pull up to a gas station, it’s helpful to know which side of your car the fuel tank is on. That piece of information determines which side of the gas-pump islands you’ll drive up.

You probably know by now which side your car’s gas cap is on, the driver’s side or the passenger’s side. But what if you’re driving a rental or someone else’s car, or what if you’ve just bought a new car? 

In those situations, it’s worth knowing about the handy triangle on the fuel gauge of every single car:

See the arrow? It points to the side of the car your fuel tank is on

Armin Vit has a smart take on the Instagram icon redesign (via John Gruber):

Unlike Uber, that replaced it’s “U” for a metaphysical atom, the change here is only aesthetic. It’s still a camera. Yes, at first it will be hard to recognize it, but when you have 200 million people tapping on it everyday, multiple times a day, that’s the kind of brand engagement that Coca-Cola or Nike would kill for. When it comes to “brand impressions” and “brand touchpoints”, Instagram (and Facebook and Twitter and, yes, even Uber) have no shortage of opportunities so it will only be a matter of time — three months, probably — before this is known, recognized, and considered as the Instagram app icon. Simply by repetition and usage. Hell, I was starting to get used to the Uber icon until they pulled out of Austin this Monday.

If people ever stop checking Instagram obsessively, I don't think it'll be because of an icon change.

→ Source: underconsideration.com

My favorite feature of my home office is the paint on the walls. It’s blood red. I’m talking deep scarlet burgundy. The ceiling is a single solid red, but the walls are a macabre mix of every horror film red on top of a slightly textured wall. When people walk in the office, it’s this color and texture they see and they often comment, “It feels so, well, oppressive.”

To which I answer, “There’s the door – right there.”

With respect to friends and family, my office is not for you; it’s for me. My office is not designed to welcome nor entertain anyone except me. It’s intended to be a place where I feel productively safe. Those blood red walls? There is not an ounce of horror in them for me. It’s my cave and a cave is a dark place hidden away from all to see. I wear those blood red walls like a warm blanket.

There are three other essentials that, for me, represent a proper cave. They are:

Your forever desk. My primary location in my office is at my desk. As you can see from the photo, I have two 27” monitors – one is an iMac, and the other is a display. Family photos on the left. Secondary desk with the PS4 and laser printer on the right. You might not have even seen the primary desk at first glance which is surprising because it’s the most important part of the picture.

Everything on that desk will be replaced in the next five years. New iMac, new PS4. The family photos will be upgraded with the latest scholastic accomplishments. There will be more pens. That desk isn’t going anywhere.

A desk’s job is to build productivity, and for me, it achieves this by first providing an immense amount of clear working space. When I put my hands on the keyboard, I want nothing around them except a cup of black coffee. There’s space for memorabilia, but it is well outside my line of sight. Second, a desk must be built like a tank. The surface of my desk is two inches of solid wood. The legs and support beams are similarly sturdy. When I put my feet up on those beams, the desk doesn’t budge.

While I, too, take the desk for granted, there are moments when I stop and admire the slightly discolored oddly shaped patch to the right of my keyboard. It’s where I’ve worn through the finishing clicking and dragging various mice over the years. I run my hands over the surface of the deck. It’s smooth, but there are dents and divots. Some of those imperfections are stories, some are simply mistakes, but like a great bag, a desk’s character is one that improves with age.

A deep leather couch. True story. I owned the Pandora.com domain name many years ago. When I sold the domain for less money than you think to the company that became Pandora, I explained to my wife I wanted three things: whatever the fancy SLR camera was at the time, carte blanche to buy a shit ton of books, and a leather couch so deep that when leaned back, you crossed a time zone.

The couch is from Restoration Hardware and it’s nearly four feet deep and almost seven feet wide. If I put my back squarely against the back cushion, my legs stick straight out like a toddler and I’m tall. When visitors sit down and discover this depth, they tilt their head, look at me, and are about say something snarky about feeling like a toddler, so I quickly explain…

“The door… it’s right there.”

This couch speaks to me. This couch says, “You. You there. You looked stressed and I have just the thing. Fire up Netflix, turn on a random Star Trek Voyager, lay down, and how about a quick snooze? Not interested in watching something? How about we re-read the Planetary Omnibus because we’re still not clear what the hell was going on there, right?”

If my desk is where I am productive, my couch in my cave is where I relax. Perhaps I am serendipitously productive or maybe I just find essential quiet between the thoughts on 28 square feet of leather.

Lovingly curated bookshelves. I’ve already waxed poetic about book shelves here. In preparation for this piece, I embarked on the weekend-long task of – once again – curating my bookshelves. If my desk is where I work, and my couch is where I contemplate, my bookshelves are my life resume.

The multi-day process of reviewing and sorting these books is not just organizationally cathartic; it’s a mental adventure where I perform a deep assessment of my current mental state. For example, multiple books on the craft of poker were removed from the shelf. Poker had a good long run – 5+ years – but during my last Vegas stint, I didn’t even think to visit the poker room. Those poker books – gone to goodwill. The writing shelf, the leadership shelf – all well stocked and full of decades of wisdom. The comic book shelf is now shelves as I’ve been on a very satisfying comic book kick for the past six months.

Unlike the desk or the couch, I don’t spend much time at my bookshelf, but like the desk and the couch, my bookshelf defines my office. These are the ideas and the words that I care about. I’ve spent thousands of hours of my time quietly contemplating each one of those books, often multiple times. For each one of these books, there are ten more that didn’t make the shelf. I could buy another bookshelf, but I enjoy the constraint of these 14 shelves. A book must distinguish itself in some way to make the shelf and when it does I want to see it every day.

A desk, a couch, a bookshelf all surrounded by blood red walls. This is the office I’ve designed for myself. I’m sitting here right now listening to Arcade Fire and appreciating that I prefer my coffee mug on the left – far away from the chaotic and spill-inducing movements of my mouse. The white stone polar bear is still sitting there, staring at me, reminding me that everything good that has happened to me is because I chose to write.

This is cross-posted on the CUNY Academic Commons News blog as part of the Citation Needed series I am writing there.

I was trawling through the posts on the CUNY Academic Commons last week and I came across this post celebrating the recently crowned PH.D.’s in the CUNY Grad Center’s English program. This was of particular interest to me because I was once a part of the venerable Ph.D. program in English at the Grad Center. Back in my salad days when I was but green in judgement I was hoping to accomplish the Herculean task of completing a dissertation just as those 31 relieved folks listed in that post have done. Alas, ed-tech got in the way and I became a beauty school drop-out.

I was intrigued by a few of the dissertation titles and did some searching. In particular, I did a search for Amanda Licastro‘s composition and rhetoric research on e-portfolios, a topic I have some vague familiarity with—if only to refute how we blanketly define them in higher ed. And bam, I found that a vast majority of the recent Ph.D.’s in the English program have their dissertations freely available for download through the CUNY Academic Works site. How cool is that? A local case of open access research for the win, now it’s time to explore open formats  It also made me wonder why the English department post didn’t link to the dissertations given they listed them all and provided the text I needed to discover them via search. That should be a basic requirement, links still make the web the web.

Fact is, in just the last ten years with the explosion of fairly simple web publishing frameworks we have entered a new age for open access research, and even as the publishing companies try ever harder to control the means of academic research production, a quick search on the web suggests the forces are everywhere against them. But it doesn’t end at research, a quick search for professor Licastro’s dissertation brings up her personal site on the Commons, various presentations, and even a short video about the work she is doing in the classroom at Stevenson University

Fact is access and openness can start with one’s research, dissertation, etc., but it quickly becomes a multi-faceted frame of people working through ideas in time and space. I never cease to find this element of the web compelling, and it always seemed to me to fit so beautifully with the pursuits of scholarship.

People are focusing on the wrong thing when analyzing Apple's path forward in the face of slowing iPhone sales. Instead of debating how much Apple will try to monetize the iPhone user base with services (not as much as consensus thinks), the company is instead planning its largest pivot yet. There are only a handful of logical explanations for Apple's current R&D expense trajectory, and all of them result in a radically different Apple. In a few years, we are no longer going to refer to Apple as the iPhone company. 

Apple R&D: By the Numbers

As I pointed out last May, Apple's R&D expense saw a significant bump up beginning in mid-2014. It was clear Apple was up to something big. However, after looking at Apple's 2Q16 results, it appears I underestimated the situation. As depicted in Exhibit 1, Apple is now on track to spend more than $10 billion on R&D in 2016, up nearly 30% from 2015 and ahead of even my aggressive estimate. This is a remarkable feat considering that Apple was spending a little over $3 billion per year on R&D just four years ago.

Exhibit 1: Apple R&D Expense (Annual)

One of the more interesting aspects of Apple's R&D expense trajectory in recent years is that the increase has been outpacing revenue growth. As seen in Exhibit 2, given my current iPhone sales expectations for FY16 and FY17, Apple is on track to approach a multi-decade record in terms of amount spent on R&D as a percent of revenue. 

Exhibit 2: Annual Apple R&D Expense (Percent of Revenue)

Unusual R&D Perceptions

The most shocking aspect about the amount of money Apple is spending on R&D is how little attention it has garnered in Silicon Valley and on Wall Street. Other than my R&D post last year, there is rarely any mention of Apple's R&D, and this doesn't seem to make much sense.

I suspect most of this has been due to the fact that Apple does not draw attention to its product pipeline and long-term strategy, choosing instead to embrace secrecy and mystery. Now compare this to Mark Zuckerberg laying out his 10-year plan for Facebook. It is easy and natural for people to then label Facebook as innovative and focused on the future. The same principle applies to Larry Page reorganizing Google to make it easier for investors to see how much is being spent on various moonshot projects. Jeff Bezos is famous for his attitude towards failing often and in public view, giving Amazon an aura of being a place of curiosity and boldness when it comes to future projects and risk taking. 

Meanwhile, Tim Cook has remained very tight-lipped about Apple's future, which gives the impression that Apple isn't working on ground-breaking ideas or products that can move the company beyond the iPhone. Instead of labeling this as a mistake or misstep, Apple's product secrecy is a key ingredient of its success. People like to be surprised. Another reason Apple takes a much different approach to product secrecy and R&D is its business model. Being open about future product plans will likely have a negative impact on near-term Apple hardware sales. Companies like Facebook and Google don't suffer from a similar risk. The end result is that there is a legitimate disconnect between Apple's R&D trends and the consensus view of the company's product pipeline. Apple is telling us that they are working on something very big, and yet no one seems to notice or care. I find that intriguing.

Logical Explanations for Apple R&D

Even though Apple remains tight-lipped about its dramatic increase in R&D expense, there are three logical explanations for what may be happening.

1) Apple's expanded product line requires additional R&D. This theory represents the most straightforward explanation. Essentially, because Apple has grown significantly over the years, the company needs to spend more on R&D just to keep up with its more expansive product line and greater competition. The company is now invested in four hardware categories (iPhone, iPad, Mac and Apple Watch), not to mention various software and services initiatives. 

2) Apple plans on doing more. Keeping with another simple explanation, Apple's increased R&D spend could signal that the company is willing to try its hand at more things. The expectation would be that Apple will begin releasing a greater number of products in terms of hardware, software and services. 

3) Apple is looking to pivot. Apple is ramping up R&D because they have a few big bets that require a massive increase in investment. The two most logical areas for these bets are wearables and personal transport initiatives. In both cases, Apple is moving well beyond its comfort zone of selling pieces of glass that can be held in one's hand. Instead, Apple is literally building a new company with additional capabilities and strengths.

The Most Likely Explanation

After analyzing the three preceding possible explanations for Apple's R&D increase, we can conclude the only one that actually makes sense is the third choice: Apple is looking to pivot. The first two theories fail to hold much water since they do not mesh with Apple's functional organizational structure. Since each senior Apple executive is in charge of his or her domain across Apple's product line, it is not possible for Apple to simply keep expanding the product line without negative consequences. At a certain point, Apple's resources are just stretched too thin to be effective. Some have argued that Apple had experienced some of this resource strain towards the end of Apple Watch development. In reality, Apple is constantly suffering from this resource strain despite having $233 billion of cash and cash equivalents on the balance sheet. 

It is this functional organizational structure that explains why Apple management talks about the need to remain focused and saying no to certain products and industries even though Apple could conceivably see much success. This rules out the explanation that Apple is spending more on R&D with the intent of doing a greater number of things. Apple's R&D follows a similarly focused mantra. While there are always scattered teams of people focused on far-fetched ideas and products, these activities do not amount to much of Apple's $10 billion budgeted for R&D in 2016. Instead, sudden and dramatic increases in Apple R&D are a result of new product initiatives.

One way of validating the claim that R&D is very much product focused is to graph the year-over-year change in R&D in absolute terms. As shown in Exhibit 3, a step pattern becomes apparent over the past 10 years. There have been two discernible increases in R&D expenses followed by periods of flat growth. When taking this step pattern and then overlaying it with Apple product launches, three product development stages become apparent: iPhone and iPad in the mid-to-late 2000s, Apple Watch beginning in 2012, and something new beginning around the Spring 2014. I suspect this latest item is primarily related to Apple working on its own electric car (Project Titan).

Exhibit 3: Apple R&D Expense Growth (Quarterly)

Apple Will Pivot

Apple is not spending $10 billion on R&D just to come up with new Watch bands, larger iPads, or a video streaming service. Instead, Apple is planning on something much bigger: a pivot into the automobile industry. 

The word "pivot" has become a buzzword lately, often misused to simply mean change. In reality, pivoting is actually a sign of strength as a company takes what it learns from one business model in one market and applies it a new one with a different business model. Apple would be taking lessons learned from its long-standing view on the world based on the Mac, iPod, and broader iOS lineup to begin selling an electric car.

This sounds incredibly ambitious and bold, and that is the point. Apple wants to move beyond the iPhone. In this regard, pivot seems like the wrong word to use since the iPhone is a very successful product generating more cash flows than the rest of Apple's product line put together times two. However, it is this success that ultimately serves as the greatest motivation for Apple management to figure out the next big thing.

In terms of how Apple can physically pivot, Apple's functional organizational structure needs to once again enter the conversation. An ability to pivot is the primary reason Apple has a rare functional organizational structure in the first place. By allowing management to put all of its attention on the product, and not internal politics, Apple's organizational structure is a major strength, not weakness. Apple is designed to move from product to product, industry to industry. We see the company do just that by entering the smartphone market, followed by the wearables market and soon, the auto market. 

Project Titan: A Finished Product is Likely

It seemed like many in Silicon Valley and on Wall Street spent most of 2015 debating if Apple even wanted to design its own car. (The answer was obvious: yes.) The discussion has now turned to whether Apple will actually end up selling a car or if management will conclude there isn't enough there to actually lead to a finished product. While Apple does indeed say no to most products and projects, Project Titan is not just any R&D project. 

In reality, people are grossly underestimating the odds that Project Titan will lead to Apple actually shipping an electric car. At this point, I peg odds of Apple selling its own electric car to be at least 80 percent. There is one very simple reason for my high degree of confidence: Project Titan is a long-term pivot. I don't consider Titan to be just another project that Apple has been tinkering around with in the lab for years like an Apple television set or Apple Pencil. Instead, Project Titan is much more about building a foundation for Apple that will literally represent the company's future.

It was recently revealed that Apple has set up a web of Project Titan buildings and infrastructure spread across Santa Clara, Sunnyvale and San Jose. This means that it is incorrect to think of Project Titan as just being about one product or one feature. Instead, Apple is building an entire start-up focused on the electric car industry, giving me a high level of confidence that Apple's efforts will lead to products. When diving deeper into Project Titan, this is where there is greater unknown as to whether a certain technology will ever ship, such as various autonomous driving features, different features for new internal passenger compartments, unique car materials, and the list goes on. Each one of those items should be thought of as an individual project that may not see the light of day. 

Apple has likely spent upwards of a few billion dollars on Project Titan so far when including real estate and stock-based compensation. When considering that Apple will likely be spending upwards of $14 billion per year on R&D by 2017 or 2018, Project Titan could easily end up being a $10-$15 billion project before Apple even ships a product. This is uncharted territory not just for Apple, but for the entire auto industry.

There is much to be discovered from tracking just one line item on Apple's income statement. R&D expense tells me that Apple is planning its most significant pivot yet. 

Receive my analysis and perspective on Apple throughout the week via exclusive daily updates (2-3 stories a day, 10-12 stories a week). To sign up, visit the membership page. 

When people ask Jean Yang this question, she reminds them that most of the features in mainstream languages follow decades of research:

Yes, Guido Van Rossum was a programmer and not a researcher before he became the Benevolent Dictator of Python. But Python's contribution is not in innovating in terms of any particular paradigm, but in combining well features like object orientation (Smalltalk, 1972, and Clu, 1975), anonymous lambda functions (the lambda calculus, 1937), and garbage collection (1959) with an interactive feel (1960s).

You find a similar story with Matz and Ruby. Many other languages were designed by people working in industry but drawing explicitly on things learned by academic research.

I don't know what percentage of mainstream languages were designed by people in industry rather than academia, but the particular number is beside the point. The same is true in other areas of research, such as databases and networks. We need some research to look years and decades into the future in order to figure what is and isn't possible. That research maps the terrain that makes more applied work, whether in industry or academia, possible.

Without academics betting years of their career on crazy ideas, we are doomed to incremental improvements of old ideas.