slogger, Metafilter, May 27, 2016

This is a video of a machine that picks up randomly organized rocks and places them into very neat rows according to type and geological age. For all kinds of reasons (chief among them being sorting things into very need and organized rows) this machine really appeals to me. "Maus manually trained a machine learning algorithm to recognize features in 30 different types of stones." From the video site Jller – Prokop Bartoní ček & Benjamin Maus:  "the machine works with a computer vision system that processes the images of the stones and maps each of its location on the platform throughout the ordering process. The information extracted from each stone are dominant color, color composition, and histograms of structural features such as lines, layers, patterns, grain, and surface texture."

Keith Lyons, Clyde Street, May 27, 2016

Herodotus  is a terrific read, so if you haven't yet, you should. It's also an interesting backdrop against which to frame this discussion of George Siemens's recent talk (and mammoth slide deck) on the fragmentation and reassembly of knowledge. "Think about the parallels between ‘ historia’ (critical thinking) and...  flourishing in a world that welcomes diversity of views woven into new sense-making," writes Keith Lyons. This view resonates with me. My 'histories' consist of some 26,000 individual posts like this one. They can be combined and recombined to create any sort of narrative. Here's the secret: the narrative, and the way of making the narrative, is not sacrosanct. Any of a hundred ways of doing ti will work equally well. And the same applies ro science and enquiry (and, for that matter, literature and art).

From CityLab: Mapping the Incredible Spread of Million-Dollar Homes Across San Francisco

.

One of the seminal books on culture is Edgar Schein’s Organizational Culture and Leadership. Schein writes in the introduction:

Perhaps the most intriguing aspect of culture as a concept is that it points us to phenomena that are below the surface, that are powerful in their impact but invisible and to a considerable degree unconscious. In that sense, culture is to a group what personality or character is to an individual. We can see the behavior that results, but often we cannot see the forces underneath that cause certain kinds of behavior. Yet, just as our personality and character guide and constrain our behavior, so does culture guide and constrain the behavior of members of a group through the shared norms that are held in that group.

In Schein’s telling, things like ping pong tables and kegerators are two (small) examples of artifacts — the visible qualities of an organization. They are easy to observe but their meaning is usually indecipherable and unique to a particular group (to put it another way, copying Google’s perks is missing the point).

The next level down are espoused beliefs and values, what everyone in an organization understands consciously: “openness,” for example, or “the customer is always right”; as you might expect espoused beliefs and values devolve rather easily into cliché.

It’s the third level that truly matters: underlying assumptions. Schein writes:

Basic assumptions, in the sense in which I want to define that concept, have become so taken for granted that one finds little variation within a social unit. This degree of consensus results from repeated success in implementing certain beliefs and values, as previously described. In fact, if a basic assumption comes to be strongly held in a group, members will find behavior based on any other premise inconceivable.

The implications of this definition are profound: culture is not something that begets success, rather, it is a product of it. All companies start with the espoused beliefs and values of their founder(s), but until those beliefs and values are proven correct and successful they are open to debate and change. If, though, they lead to real sustained success, then those values and beliefs slip from the conscious to the unconscious, and it is this transformation that allows companies to maintain the “secret sauce” that drove their initial success even as they scale. The founder no longer needs to espouse his or her beliefs and values to the 10,000th employee; every single person already in the company will do just that, in every decision they make, big or small.

Microsoft’s Blindness

As with most such things, culture is one of a company’s most powerful assets right until it isn’t: the same underlying assumptions that permit an organization to scale massively constrain the ability of that same organization to change direction. More distressingly, culture prevents organizations from even knowing they need to do so. Schein continues:

Basic assumptions, like theories-in-use, tend to be nonconfrontable and nondebatable, and hence are extremely difficult to change. To learn something new in this realm requires us to resurrect, reexamine, and possibly change some of the more stable portions of our cognitive structure…Such learning is intrinsically difficult because the reexamination of basic assumptions temporarily destabilizes our cognitive and interpersonal world, releasing large quantities of basic anxiety. Rather than tolerating such anxiety levels, we tend to want to perceive the events around us as congruent with our assumptions, even if that means distorting, denying, projecting, or in other ways falsifying to ourselves what may be going on around us. It is in this psychological process that culture has its ultimate power.

Probably the canonical example of this mindset was Microsoft after the launch of the iPhone. It’s hard to remember now, but no company today comes close to matching the stranglehold Microsoft had on the computing industry from 1985 to 2005 or so.¹ The company had audacious goals — “A computer on every desk and in every home, running Microsoft software” — which it accomplished and then surpassed: the company owned enterprise back offices as well. This unprecedented success changed that goal — originally an espoused belief — into an unquestioned assumption that of course all computers should be Microsoft-powered. Given this, the real shock would have been then-CEO Steve Ballmer not laughing at the iPhone.

A year-and-a-half later, Microsoft realized that Windows Mobile, their current phone OS, was not competitive with the iPhone and work began on what became Windows Phone. Still, unacknowledged cultural assumptions remained: one, that Microsoft had the time to bring to bear its unmatched resources to make something that might be worse at the beginning but inevitably superior over time, and two, that the company could leverage Windows’ dominance and their Office business. Both assumptions had become cemented in Microsoft’s victory in the browser wars and their slow-motion takeover of corporate data centers; in truth, though, Microsofts’ mobile efforts were already doomed, and nearly everyone realized it before Windows Phone even launched with a funeral for the iPhone.

Steve Ballmer never figured it out; his last acts were to reorganize the company around a “One Microsoft” strategy centered on Windows, and to buy Nokia to prop up Windows Phone. It fell to Satya Nadella, his successor, to change the culture, and it’s why the fact his first public event was to announce Office for iPad was so critical. I wrote at the time:

This is the power CEOs have. They cannot do all the work, and they cannot impact industry trends beyond their control. But they can choose whether or not to accept reality, and in so doing, impact the worldview of all those they lead.

Microsoft under Nadella’s leadership has, over the last three years, undergone a tremendous transformation, embracing its destiny as a device-agnostic service provider; still, it is fighting the headwinds of Amazon’s cloud, open source tooling, and the fact that mobile users had six years to get used to a world without Microsoft software. How much stronger might the company have been had it faced reality in 2007, but the culture made that impossible.

Steve Jobs’ Leadership

Shein defines leadership in the context of culture:

When we examine culture and leadership closely, we see that they are two sides of the same coin; neither can really be understood by itself. On the one hand, cultural norms define how a given nation or organizations will define leadership—who will get promoted, who will get the attention of followers. On the other hand, it can be argued that the only thing of real importance that leaders do is to create and manage culture; that the unique talent of leaders is their ability to understand and work with culture; and that it is an ultimate act of leadership to destroy culture when it is viewed as dysfunctional.

A great example of this sort of destruction was Steve Jobs’ first keynote as interim CEO at the 1997 Boston Macworld, specifically the announcement of Apple’s shocking partnership with Microsoft:

When Jobs said the word Microsoft, the audience audibly groaned. A few minutes later, when Jobs clicked to a slide that said Internet Explorer would be the default browser on Macintosh, the audience booed so loudly that Jobs had to stop speaking. When Jobs finally said the actual words “default browser” the audience booed even louder, with several individuals shouting “No!” It is, given the context of today’s Apple keynotes, shocking to watch.

Then, after Bill Gates spoke to the crowd via satellite (in what Jobs would call his “worst and stupidest staging event ever”), Jobs launched into what his biographer Walter Isaacson called an “impromptu sermon”:

If we want to move forward and see Apple healthy and prospering again, we have to let go of a few things here. We have to let go of this notion that for Apple to win Microsoft has to lose. OK? We have to embrace a notion that for Apple to win Apple has to do a really good job, and if others are going to help us, that’s great, cause we need all the help we can get. And if we screw up and we don’t do a good job, it’s not somebody else’s fault. It’s our fault. So, I think that’s a very important perspective.

I think, if we want Microsoft Office on the Mac, we better treat the company that puts it out with a little bit of gratitude. We like their software. So, the era of setting this up as a competition between Apple and Microsoft is over as far as I’m concerned. This is about getting healthy, and this is about Apple being able to make incredibly great contributions to the industry, to get healthy and prosper again.

Here’s Shein:

But as the group runs into adaptive difficulties, as its environment changes to the point where some of its assumptions are no longer valid, leadership comes into play once more. Leadership is now the ability to step outside the culture that created the leader and to start evolutionary change processes that are more adaptive. This ability to perceive the limitations of one’s own culture and to evolve the culture adaptively is the essence and ultimate challenge of leadership.

Make no mistake: even though he had been gone for over a decade, Steve Jobs was responsible for that booing.

Jobs had set up Apple generally and the Macintosh specifically as completely unique and superior to the alternatives, particularly the hated IBM PC and its Windows (originally DOS) operating system. By 1997, though, Microsoft had won, and Apple was fighting for its life. And yet the audience booed its lifeline! That is how powerful culture can be — and that is why Jobs’ “impromptu sermon” was so necessary and so powerful. It was Apple’s version of Office on the iPad, and a brilliant display of leadership.

Warning Signs for Apple and Google

Over the weekend Marco Arment wrote a widely-read piece (now) called If Google’s Right About AI, That’s a Problem for Apple:

The BlackBerry’s success came to an end not because RIM started releasing worse smartphones, but because the new job of the smartphone shifted almost entirely outside of their capabilities, and it was too late to catch up. RIM hadn’t spent years building a world-class operating system, or a staff full of great designers, or expertise in mass production of luxury-quality consumer electronics, or amazing APIs and developer tools, or an app store with millions of users with credit cards already on file, or all of the other major assets that Apple had developed over a decade (or longer) that enabled the iPhone. No new initiative, management change, or acquisition in 2007 could’ve saved the BlackBerry. It was too late, and the gulf was too wide.

Today, Amazon, Facebook, and Google are placing large bets on advanced AI, ubiquitous assistants, and voice interfaces, hoping that these will become the next thing that our devices are for. If they’re right — and that’s a big “if” — I’m worried for Apple…If the landscape shifts to prioritize those big-data AI services, Apple will find itself in a similar position as BlackBerry did almost a decade ago: what they’re able to do, despite being very good at it, won’t be enough anymore, and they won’t be able to catch up.

Arment is exactly right. What is fascinating, though, is that, as I wrote last week, Google has their own set of problems: users actually spend their time in social apps, mostly owned by Facebook, and while Google has a critical asset in Android, its most valuable users (from a monetization standpoint) are on iOS. How will users actually access Google’s AI capabilities (if they turn out to matter), and how will Google monetize them?

To be sure, neither company is struggling today. Apple may have failed to achieve record results for the first time in 13 years, but their 2Q 2016 revenue of $50.6 billion was more than the revenue of Microsoft, Google, and Facebook combined; Google, meanwhile, is still setting year-over-year records, with $17.3 billion in revenue.

That, though, is the challenge: BlackBerry wasn’t struggling in 2006, nor was Microsoft in 2007, or even Apple as late as 1993. There was no obvious reason to think that anything was amiss, and it was culture that ensured that whatever hints there were would be ignored. Shein again:

Culture as a set of basic assumptions defines for us what to pay attention to, what things mean, how to react emotionally to what is going on, and what actions to take in various kinds of situations. Once we have developed an integrated set of such assumptions—a “thought world” or “mental map”—we will be maximally comfortable with others who share the same set of assumptions and very uncomfortable and vulnerable in situations where different assumptions operate, because either we will not understand what is going on, or, worse, we will misperceive and misinterpret the actions of others.

And so BlackBerry thought Apple was lying about the iPhone; Steve Ballmer declared “He liked Microsoft’s chances”; and Apple, well, Apple had already decided to, in Jobs’ view, sacrifice product for profits. The time to act was at the moment of denial, not the moment of crisis.

Paths Forward

That said, both Apple and Google are still operating from positions of considerable strength going forward: iPhone growth may or may not have peaked, but it’s not going anywhere for a good long while, and the company is almost certainly working on a car. Google, meanwhile, is arguably in even better shape: the company has a massive lead in machine learning, which could manifest itself in all kinds of interesting applications, and here Android looms large.

Still, there are very obvious steps both companies could do to entrench their advantages:

• Apple could partner with a company like Microsoft (again) to build out its services layer, both on the backend (Azure) and, if they want to get really radical, the front-end (combining Siri and Cortana). The most radical solution, though, would be fully opening up iOS in such a way that users could set Google (or any other company’s) services as defaults. This would foreclose any medium-term threat to the iPhone from an Android experience that is fully-infused with Google’s AI capabilities (more on the long-term problems in a moment)
• Google could — should! — build a bot for Facebook Messenger. More than that, they should build an entire backend for Facebook Messenger developers. Do people want to live in Facebook? Very well, meet them there, just as Google found its user base on Windows through the browser.

Both ideas (and there are certainly others) have their issues: Apple would be foreclosing their future as a services provider, but frankly, I am extremely skeptical about this regardless. Not only does the company have the wrong organizational structure but, similar to Microsoft, the company’s overwhelming success has had far-reaching effects on the culture; in this case, the company is so focused on making physical products that it’s doubtful an effective services mentality could ever emerge, not to mention the company’s (at times disingenuous) absolutism about privacy.²

Google, meanwhile, would be supporting its most dangerous competitor. At the end of the day Google and Facebook share the exact same customers — advertisers — and even though it’s not clear how Google can steal attention back it’s also not obvious that they should aid their rival.³

The Curse of Culture

The biggest problem for both, though, is culture. Apple, beyond everything else — and in part because of the humiliation of that 1997 keynote — desires complete control; Google, for its part, desires information, and can’t tolerate the idea of Facebook having more.

The rigidity of both is the manifestation of the disease that affects every great company: the assurance that what worked before will work eternally into the future, even if circumstances have changed. What makes companies great is inevitably what makes companies fail, whenever that day comes.⁴

1. Yes, Apple ultimately came to earn much more revenue that Microsoft ever did, and Google has come close, but both did so in the context of a much larger industry
2. This too is why I don’t buy the “Wait for WWDC” response to Marco’s article; the reasons to be skeptical about Apple’s prospects here are structural
3. That, in some respects, gets to the tragedy of this piece: Apple and Google are the most natural of partners. Neither has to lose for the other to win, and both have wasted far too much valuable time fighting a war that was never necessary.
4. One final quote from Shein:

   If one wishes to distinguish leadership from management or administration, one can argue that leadership creates and changes cultures, while management and administration act within a culture. By defining leadership in this manner, I am not implying that culture is easy to create or change, or that formal leaders are the only determiners of culture. On the contrary, as we will see, culture refers to those elements of a group or organization that are most stable and least malleable. Culture is the result of a complex group learning process that is only partially influenced by leader behavior. But if the group’s survival is threatened because elements of its culture have become maladapted, it is ultimately the function of leadership at all levels of the organization to recognize and do something about this situation. It is in this sense that leadership and culture are conceptually intertwined.

   Are Tim Cook and Sundar Pichai managers, or leaders? And which do they need to be?

From Metro Vancouver & TransLink Update

.

Concord Pacific Developments has purchased the Vancouver Molson Coors brewery and disclosed plans to transform one of the city’s last remaining industrial sites into a “mixed-use residential neighbourhood.” The developer bought the site for $185-million and the deal closed March 31, according to real estate information service RealNet.

The three-hectare property is assessed in documents at $49,019,400. The City of Vancouver, which has repeatedly said it has no plans to rezone the site for anything other than industrial use, sent an e-mail to the Globe & Mail stating it had not received a rezoning application for the site.

“Any change to that would require a regional amendment by Metro. The city’s policies for these lands are set and staff are not contemplating any changes to current policies,” it said.

Tom Davidoff, economics professor at the University of British Columbia’s Sauder School of Business, is in favour of the Concord Pacific plan, but not necessarily the approach. Davidoff told the Globe and Mail that “the city has to be careful. My understanding is that land prices are going nuts and developers buy assuming [there will be] upzoning. Arguably, the city shouldn’t reward such behaviour – it’s critical that paying a lot for land should not entitle the owner to demand city action.”

After a couple of years working on Mozilla’s mobile operating system project, I’m coming back to Firefox!

I’ll be doing some familiar things and some new things. My official title is Product Manager, Firefox Roadmap and Community. What that means, first and foremost, is that I’ll be returning as our storyteller, making sure that we’re communicating regularly about where Firefox is heading, and that we’re fully engaged with Firefox users, fans, and contributors.

My first few weeks will be spent getting up to speed with the Firefox teams, from Product  Management and User Experience to Engineering and Program/Project Management. We’re doing a lot with Firefox in 2016 and 2017. I can’t wait to start sharing that story.

If you’ve got ideas about what needs improving first with Firefox communications, perhaps the Monday all-hands meeting content, or the roadmap documents on the wiki, or something completely different, please let me know in comments or email.

I’m over the moon excited about this role. Stay tuned. It’s gonna be great.

How Complete Streets Improve Commutes For City Cyclists—and Motorists

By Claire McFarlane

This article was originally posted on Torontoist

After heated debate in City Council and the eventual passage of the Bloor bike-lane pilot project, the feasibility of cycling on Toronto’s roadways has been brought to the fore. How can cyclists, pedestrians, and drivers share the city’s streets safely and fairly? Can Toronto’s roads accommodate all forms of transportation?

That’s the question the Toronto Centre for Active Transportation’s Nancy Smith Lea and researchers set out to answer in the new book, Complete Street Transformations in the Greater Golden Horseshoe Region.

The book, penned alongside researchers from Ryerson University and the University of Toronto, examines nine Complete Street transformations—that is, roadways that are upgraded to better accommodate all users, including pedestrians, cyclists, drivers, and public transit commuters—in southern Ontario, including two in downtown Toronto at Queens Quay and Richmond and Adelaide.

The study of the improvements made to Richmond and Adelaide streets, which included the addition of a cycle track separated from vehicle traffic by flexi-posts and planter boxes, concluded that the upgrades resulted in an increased number of cyclists using the roadways and reduced travel times for drivers. During off-peak hours, a motorist’s trip was 30 per cent faster after the cycle track was installed and 12 per cent faster during peak hours.

KEEP READING: How Complete Streets Improve Commutes For City Cyclists—and Motorists

Related on the dandyBLOG:

From the Horse's Mouth: Nancy Smith Lea

Building community through bike-friendly planning

New Complete Streets book launched at Ryerson University by TCAT

Bike Spotting: Have you heard the city is planning more protected bike lanes?

Paul Kim, On Dynamism:

My problems aren’t performance. They aren’t type-safety (maybe it’s just me, but I rarely have issues with it and when I do, the time it takes to fix it is far less than the time specifying everything to a tee everywhere else). They aren’t being able to write clever VDLs. For me, it’s writing apps that solve my users’ problems and getting them out in a timely fashion. As it stands now, Swift (at least pure-Swift, or even current Swift as a non-ABI-stable moving target) does not do that for me.

Now maybe these same problems can be solved in a static way but what I’m not seeing from the static-camp are (decent) solutions. What I’m seeing are either hand-waving or the same crufty code-generation, write tons of repetitive boilerplate type of solutions that I had hoped we had left behind in the 90s.

What I do see makes me worry that it’s not the experienced app-writers that are being heard.

* * *

Daniel Jalkut, Not Perfected Here:

Swift is a fascinating, beautiful language. How will it evolve to prove superior, in the long-run to Objective-C? By providing a suite of impedance-matched frameworks that fulfill all the needs of current iOS and Mac developers.

* * *

Dave DeLong posted 12 tweets on the subject, starting here, and including:

5/ but one of the things I love about @SwiftLang is that, since the dynamism isn’t there, I have to think about problems in new ways

…and…

8/ I know that the people working on @SwiftLang are some of the most brilliant people in the world, and they all also love Objective-C

…and…

10/ so maybe we’ll end up dynamism. Maybe we won’t. What we WILL end up with is one of the most kick-ass languages in the world

Though I could ask about the frameworks (and I did), I can’t argue with Dave’s conclusion. I’ve said before, and will say again, that I’m writing all my new code in Swift and I’m enjoying it a ton, and I get bugged when I have to write Objective-C.

In other words — if you think that the various people writing about Swift and dynamism are anti-Swift, you’ve got it wrong.

Let’s imagine your data and survey results tell you that members don’t like to create guest columns about the topic.

Should you do more of it or less of it?

Most people see the data and decide to do less of what members don’t like.

But this is a slippery slope. What members like best is precisely what they already enjoy doing. Members can’t like what they haven’t tried. In fact, most of the things they enjoy now they didn’t like before they tried them.

If you build your community around data alone, you are putting up a fence around existing behaviors…only now you have to pay to maintain the land.

If you want to be the leader chasing after his people, follow the data.

The alternative is to see that dislike of guest columns as a persuasion challenge and do more of it. The problem isn’t that members don’t like doing it. The problem is you haven’t persuaded them why they should do it yet.

Your message isn’t getting through.

That means finding better stories, shrinking the behavior change, working with a group of others to highlight positive examples, equipping people with new information.

If guest columns is one of the overlapping valuable behaviors, you’ve got to persuade people to create them. There is very little value in putting up a fence around existing behaviors. There is a lot of value in successfully changing behavior in a way that benefits you and your members.

If we’re not trying to tip behavior in our favour, what are we trying to do?

Today I’m presenting a webinar with Vanilla to explain how you can change the behavior of your audience. I hope you will join us. 1pm Eastern. Click here to signup.

Three months ago Sonos published a blog post announcing a change of direction:

As we look to the future there are two big areas that we’re leaning into: paid streaming services, and voice control.

Now Amir Afrati writes:

After years of internal debate and discussion about how to do so, the company is preparing to open up Siri to apps made by others. And it is working on an Amazon Echo-like device with a speaker and microphone that people can use to turn on music, get news headlines or set a timer.

Sonos PLAY:5 is a fantastic speaker with two microphones.

The Washington Post interviewed Chen in a quiet Bay Area conference room on a rainy afternoon for about an hour. He discussed everything from the influence of his Catholic high school to why almost no job is beneath a person to how he plans to rescue BlackBerry.

This is the kind of manager you want.

More >

I’m not swayed by the Core and not really sold on the looks of the Pebble 2, but the Time 2 looks very nice — finally, a reasonably-sized bezel that doesn’t get in the way of enjoying the screen, and that should do away with most of the criticisms regarding their square design.

On the other hand, my expectations towards a possible Round 2 are now pretty high — especially considering that the Pebble Time Round was launched right after the end of the last kickstarter, annoying a bunch of people.

Overall, I’m sold on smartwatches and happy with the Apple Sports Watch I brought back from Seattle, but I’ll be keeping an eye on what Pebble does (even though they’ll never have full iOS integration, they certainly build great devices).

I want to change the way the next generation writes. If you teach them, I’d like to help you. See if this description fits you: You teach writing. That could be English composition, marketing communications, public relations, technical writing, journalism, or any other non-fiction writing class. You are teaching a class in Fall of 2016. … Continue reading Seeking professors who teach writing without bullshit →

The post Seeking professors who teach writing without bullshit appeared first on without bullshit.

After a short delay, Telus Samsung Galaxy Note 4 owners are reporting that Marshmallow is now available to download. The update, which is Android 6.0.1, is approximately 1GB in size.

If you’re not seeing a download notification, manually check for it via Menu > Settings > About Device > Software Update > Update.

Update: Koodo Mobile customers are now reporting the Note 4 has been updated to Marshmallow.

Related reading: Note 4 review

(Thanks, IceR2)

From ULI’s UrbanLand magazine:

During the discussion that followed the ggplot2 posts from David and I last week we started talking about tidy data and the man himself noted that matrices are often useful instead of “tidy data” and I mentioned there might be other data that are usefully “non tidy”. Here I will be using tidy/non-tidy according to Hadley’s definition. So tidy data have:

• One variable per column
• One observation per row
• Each type of observational unit forms a table

I push this approach in my guide to data sharing and in a lot of my personal work. But note that non-tidy data can definitely be already processed, cleaned, organized and ready to use.

@hadleywickham @drob @mark_scheuerell I'm saying that not all data are usefully tidy (and not just matrices) so I care more abt flexibility

— Jeff Leek (@jtleek) February 12, 2016

This led to a very specific blog request:

@jtleek @drob I want a blog post on non-tidy data!

— Hadley Wickham (@hadleywickham) February 12, 2016

So I thought I’d talk about a couple of reasons why data are usefully non-tidy. The basic reason is that I usually take a problem first, not solution backward approach to my scientific research. In other words, the goal is to solve a particular problem and the format I chose is the one that makes it most direct/easy to solve that problem, rather than one that is theoretically optimal.   To illustrate these points I’ll use an example from my area.

Example data

Often you want data in a matrix format. One good example is gene expression data or data from another high-dimensional experiment. David talks about one such example in his post here. He makes the (valid) point that for students who aren’t going to do genomics professionally, it may be more useful to learn an abstract tool such as tidy data/dplyr. But for those working in genomics, this can make you do unnecessary work in the name of theory/abstraction.

He analyzes the data in that post by first tidying the data.

library(dplyr)
library(tidyr)
library(stringr)
library(readr)
library(broom)
 
original_data %
  separate(NAME, c("name", "BP", "MF", "systematic_name", "number"), sep = "\\|\\|") %>%
  mutate_each(funs(trimws), name:systematic_name) %>%
  select(-number, -GID, -YORF, -GWEIGHT) %>%
  gather(sample, expression, G0.05:U0.3) %>%
  separate(sample, c("nutrient", "rate"), sep = 1, convert = TRUE)

It isn’t 100% tidy as data of different types are in the same data frame (gene expression and metadata/phenotype data belong in different tables). But its close enough for our purposes. Now suppose that you wanted to fit a model and test for association between the “rate” variable and gene expression for each gene. You can do this with David’s tidy data set, dplyr, and the broom package like so:

rate_coeffs = cleaned_data %>% group_by(name) %>%
     do(fit = lm(expression ~ rate + nutrient, data = .)) %>%
     tidy(fit) %>% 
     dplyr::filter(term=="rate")

On my computer we get something like:

system.time( cleaned_data %>% group_by(name) %>%
+               do(fit = lm(expression ~ rate + nutrient, data = .)) %>%
+                tidy(fit) %>% 
+                dplyr::filter(term=="rate"))
|==========================================================|100% ~0 s remaining 
user  system elapsed 
 12.431   0.258  12.364

Let’s now try that analysis a little bit differently. As a first step, lets store the data in two separate tables. A table of “phenotype information” and a matrix of “expression levels”. This is the more common format used for these type of data. Here is the code to do that:

expr = original_data %>% 
  select(grep("[0:9]",names(original_data)))
 
rownames(expr) = original_data %>%
  separate(NAME, c("name", "BP", "MF", "systematic_name", "number"), sep = "\\|\\|") %>%
  select(systematic_name) %>% mutate_each(funs(trimws),systematic_name) %>% as.matrix()
 
vals = data.frame(vals=names(expr))
pdata = separate(vals,vals,c("nutrient", "rate"), sep = 1, convert = TRUE)
 
expr = as.matrix(expr)

If we leave the data in this format we can get the model fits and the p-values using some simple linear algebra

expr = as.matrix(expr)
 
mod = model.matrix(~ rate +  as.factor(nutrient),data=pdata)
rate_betas = expr %*% mod %*% solve(t(mod) %*% mod)

This gives the same answer after re-ordering

all(abs(rate_betas[,2]- rate_coeffs$estimate[ind]) < 1e-5,na.rm=T)
[1] TRUE

But this approach is much faster.

 system.time(expr %*% mod %*% solve(t(mod) %*% mod))
   user  system elapsed 
  0.015   0.000   0.015

This requires some knowledge of linear algebra and isn’t pretty. But it brings us to the first general point: you might not use tidy data because some computations are more efficient if the data is in a different format. 

Many examples from graphical models, to genomics, to neuroimaging, to social sciences rely on some kind of linear algebra based computations (matrix multiplication, singular value decompositions, eigen decompositions, etc.) which are all optimized to work on matrices, not tidy data frames. There are ways to improve performance with tidy data for sure, but they would require an equal amount of custom code to take advantage of say C, or vectorization properties in R.

Ok now the linear regressions here are all treated independently, but it is very well known that you get much better performance in terms of the false positive/true positive tradeoff if you use an empirical Bayes approach for this calculation where you pool variances.

If the data are in this matrix format you can do it with R like so:

library(limma)
fit_limma = lmFit(expr,mod)
ebayes_limma = eBayes(fit_limma)
topTable(ebayes_limma)

This approach is again very fast, optimized for the calculations being performed and performs much better than the one-by-one regression approach. But it requires the data in matrix or expression set format. Which brings us to the second general point: **you might not use tidy data because many functions require a different, also very clean and useful data format, and you don’t want to have to constantly be switching back and forth. **Again, this requires you to be more specific to your application, but the potential payoffs can be really big as in the case of limma.

I’m showing an example here with expression sets and matrices, but in NLP the data are often input in the form of lists, in graphical analyses as matrices, in genomic analyses as GRanges lists, etc. etc. etc. One option would be to rewrite all infrastructure in your area of interest to accept tidy data formats but that would be going against conventions of a community and would ultimately cost you a lot of work when most of that work has already been done for you.

The final point, which I won’t discuss here is that data are often usefully represented in a non-tidy way. Examples include the aforementioned GRanges list which consists of (potentially) ragged arrays of intervals and quantitative measurements about them. You could force these data to be tidy by the definition above, but again most of the infrastructure is built around a different format that is much more intuitive for that type of data. Similarly data from other applications may be more suited to application specific formats.

In summary, tidy data is a useful conceptual idea and is often the right way to go for general, small data sets, but may not be appropriate for all problems. Here are some examples of data formats (biased toward my area, but there are others) that have been widely adopted, have a ton of useful software, but don’t meet the tidy data definition above. I will define these as “processed data” as opposed to “tidy data”.

• Expression sets for expression data
• Summarized experiments for a variety of genomic experiments
• Granges Lists for genomic intervals
• Corpus objects for corpora of texts.
• igraph objects for graphs

I’m sure there are a ton more I’m missing and would be happy to get some suggestions on Twitter too.

Soon after my discussion with Hilary Parker and Jenny Bryan about spreadsheets on Not So Standard Deviations, Brooke Anderson forwarded me this article written by Steven Levy about the original granddaddy of spreadsheets, VisiCalc. Actually, the real article was written back in 1984 as so-called microcomputers were just getting their start. VisiCalc was originally written for the Apple II computer and notable competitors at the time included Lotus 1-2-3 and Microsoft Multiplan, all since defunct.

It’s interesting to see Levy’s perspective on spreadsheets back then and to compare it to the current thinking about data, data science, and reproducibility in science. The problem back then was “ledger sheets” (what we might now call a spreadsheet), which contained numbers and calculations related to businesses, were tedious to make and keep up to date.

Making spreadsheets, however necessary, was a dull chore best left to accountants, junior analysts, or secretaries. As for sophisticated “modeling” tasks – which, among other things, enable executives to project costs for their companies – these tasks could be done only on big mainframe computers by the data-processing people who worked for the companies Harvard MBAs managed.

You can see one issue here: Spreadsheets/Ledgers were a “dull chore”, and best left to junior people. However, the “real” computation was done by the people the “data processing” center on big mainframes. So what exactly does that leave for the business executive to do?

Note that the way of doing things back then was effectively reproducible, because the presentation (ledger sheets printed on paper) and the computation (data processing on mainframes) was separated.

The impact of the microcomputer-based spreadsheet program appears profound.

Already, the spreadsheet has redefined the nature of some jobs; to be an accountant in the age of spreadsheet program is — well, almost sexy. And the spreadsheet has begun to be a forceful agent of decentralization, breaking down hierarchies in large companies and diminishing the power of data processing.

There has been much talk in recent years about an “entrepreneurial renaissance” and a new breed of risk-taker who creates businesses where none previously existed. Entrepreneurs and their venture-capitalist backers are emerging as new culture heroes, settlers of another American frontier. Less well known is that most of these new entrepreneurs depend on their economic spreadsheets as much as movie cowboys depend on their horses.

 If you replace "accountant" with "statistician" and "spreadsheet" with "big data" and you are magically teleported into 2016.

The way I see it, in the early 80's, spreadsheets satisfied the never-ending desire that people have to interact with data. Now, with things like tablets and touch-screen phones, you can literally "touch" your data. But it took microcomputers to get to a certain point before interactive data analysis could really be done in a way that we recognize today. Spreadsheets tightened the loop between question and answer by cutting out the Data Processing department and replacing it with an Apple II (or an IBM PC, if you must) right on your desk.

Of course, the combining of presentation with computation comes at a cost of reproducibility and perhaps quality control. Seeing the description of how spreadsheets were originally used, it seems totally natural to me. It is not unlike today's analytic dashboards that give you a window into your business and allow you to "model" various scenarios by tweaking a few numbers of formulas. Over time, people took spreadsheets to all sorts of extremes, using them for purposes for which they were not originally designed, and problems naturally arose.

So now, we are trying to separate out the computation and presentation bits a little. Tools like knitr and R and shiny allow us to do this and to bring them together with a proper toolchain. The loss in interactivity is only slight because of the power of the toolchain and the speed of computers nowadays. Essentially, we've brought back the Data Processing department, but have staffed it with robots and high speed multi-core computers.

Thinking about my post on Theranos yesterday it occurred to me that one thing that’s great about all of the innovation and technology coming out of places like Silicon Valley is the tremendous reduction of friction in our lives. With Uber it’s much easier to get a ride because of improvement in communication and an increase in the supply of cars. With Amazon, I can get that jug of vegetable oil that I always wanted without having to leave the house, because Amazon.

So why is there all this friction? Sometimes it’s because of regulation, which may have made sense at an earlier time, but perhaps doesn’t make as much sense now. Sometimes, you need a company like Amazon to really master the logistics operation to be able to deliver anything anywhere. Otherwise, you’re just stuck driving to the grocery store to get that vegetable oil.

But sometimes there’s friction for a reason. For example, Ben Thompson talks about how previously there was quite a bit more friction involved before law enforcement could listen in on our communications. Although wiretapping had long been around (as noted by David Simon of…The Wire) the removal of all friction by the NSA made the situation quite different. Related to this idea is the massive data release from OkCupid a few weeks ago, as I discussed on the latest Not So Standard Deviations podcast episode. Sure, your OkCupid profile is visible to everyone with an account, but having someone vacuum up 70,000 profiles and dumping them on the web for anyone to view is not what people signed up for—there is a qualitative difference there.

When it comes to Theranos and diagnostic testing in general, there is similarly a need for some friction in order to protect public health. John Ioannides notes in his commentary for JAMA:

Even if the tests were accurate, when they are performed in massive scale and multiple times, the possibility of causing substantial harm from widespread testing is very real, as errors accumulate with multiple testing. Repeated testing of an individual is potentially a dangerous self-harm practice, and these individuals are destined to have some incorrect laboratory results and eventually experience harm, such as, for example, the anxiety of being labeled with a serious condition or adverse effects from increased testing and procedures to evaluate false-positive test results. Moreover, if the diagnostic testing process becomes dissociated from physicians, self-testing and self-interpretation could cause even more problems than they aim to solve.

Unlike with the NSA, where the differences in scale may be difficult to quantify because the exact extent of the program is unknown to most people, with diagnostic testing, we can precisely quantify how a diagnostic test’s characteristics will change if we apply it to 1,000 people vs. 1,000,000 people. This is why organizations like the US Preventative Services Task Force so carefully considers recommendations for testing or screening (and why they have a really tough job).

I’ll admit that a lot of the friction in our daily lives is pointless and it would be great to reduce it if possible. But in many cases, it was us that put the friction there for a reason, and it’s sometimes good to think about why before we move to eliminate it.

Designers often restrict themselves to a medium/pursuit (e.g., print, digital, environments). Increasingly, though, I think such constraints are unnecessary. In fact, I suspect that these sorts of bounds can arrest your growth. This can result in myopic behavior/solutions.

As such, I encourage you to start with a different question. I want you to reflect on what sort of work you are most suited to. Put another way: Are you the sort of person who plants a garden, or the kind who tends to it?

I’d bet that most young designers would pick the first option. New projects are exciting. They bring the possibility of discovery. Additionally, such work helps you build your portfolio. (In spite of arguments to the contrary, this remains a worthwhile pursuit.) For such designers, work at a studio/agency is quite often rewarding.

Completing new exercises—especially those with time-limits—is mentally rigorous. As such, studio projects allow you to work your conceptual muscles. They afford you opportunities to hone your skills/craft. This sort of work can also stave off boredom. At its best, studio work makes the exploration of new ideas, styles, industries, and beliefs compulsory.

However, product-based work (e.g. an internal app, or a startup project) can also be gratifying. These sorts of projects allow for more holistic decision-making. One can collect data, and refine a solution accordingly. Additionally, there’s more room in such projects to shape a design vocabulary that provides lasting value.

I’ve worked in design services for the bulk of my career. I like parts of this work, but feel that certain personal quirks leave me better suited to designing products. I’m a bit of a generalist. I enjoy defining a UX convention one day, scripting a video the next, and toying with an illustration the one following. I’m also a bit of a control freak. I want to build things well. This means I enjoy refining details others might not. But, that’s just me.

The options I present above might seem muddy or clear, depending on your current situation. If you work at an agency where you feel that your work is compromised, producing design for a startup might seem more attractive. Conversely, if you’ve been at a startup for years, having something fresh to work on is likely appealing.

I can’t tell you which one to choose. Odds are you’d likely benefit from trying both settings and seeing which feels best. What works for one designer might not work for you. In any event, the question of products or services is worth asking. In contemplating this question, you might uncover new insight into the kind of work you’ll find most fulfilling.

Like clockwork, perennial leaker Evan Blass has released fresh information on the upcoming OnePlus 3.

According to Blass, the phone will feature a 5.5-inch 1080p screen, Qualcomm’s new Snapdragon 820 processor, 64GB of internal storage and a 16 megapixel rear-facing camera.

It will be interesting to see how OnePlus decides to justify a 1080p display once again. After all, if Blass’ report is accurate — and there’s little reason to believe it isn’t since it looks like the company is leaking this information directly to him — then this will be the second year in a row that OnePlus has decided against adding a QHD display to its flagship device.

On the other hand, it’s laudable when an OEM loads their device with ample internal storage. when Blass says “One Plus 3 basics” before listing some of the OnePlus 3’s specs, he seems to suggest all OnePlus 3 models with start with 64GB of internal storage. Given the fact OnePlus currently only sells a 64GB version of the OnePlus 2, and has done so for a number of months, a 64GB base model OnePlus 3 seems like a good bet.

OnePlus 3 basics: 5.5-inch 1080p, Snapdragon 820, 64GB storage, 16MP rear camera, NFC. SS from an N preview build. pic.twitter.com/u1a0hQoEIP

— Evan Blass (@evleaks) May 24, 2016

Blass doesn’t mention other important specs like battery capacity, but past rumours have pointed to the OnePlus 3 including 3GB of RAM and 3,000mAh battery. The leaked render Blass shared last week also suggests the phone will feature a USB-C port and a front-facing fingerprint sensor.

Lastly, the screenshot Blass included with his tweet is interesting because it shows his OnePlus 3 running Android N. Android Authority speculates this might mean the OnePlus 3 will ship with Android N. Dare to dream, certainly, but this seems unlikely. At the very least, it suggests OnePlus is already working on a Android N version of its Oxygen OS skin.

OnePlus is expected to announce the OnePlus next month. The company plans to announce during a special virtual reality event. In an interview with CNET, OnePlus founder Carl Pei said the OnePlus 3 will launch sometime at the end of Q2 2016.

This story is a version of my talk “Managing Difficult Stakeholders,” given at ProductCamp Portland in 2016. When you’re managing a valuable product, working with difficult stakeholders becomes crucial and can make or break your product. As a product manager, I interact with over 20 people a day; each person is unique, with different backgrounds, roles, industries and stories. My job revolves...

Source

Great story by Christina Warren at Mashable on how the iPad is being used to teach interactive problem solving in a fun, new way:

I’m sitting on the floor at The Academy of Talented Scholars (PS 682) in Bensonhurst, watching kindergarteners create robots on an iPad.

It’s one of the cutest things I’ve ever seen, and I don’t even like children.

The exercise is part of the curriculum led by co-teachers Stacy Butsikares and Allison Bookbinder, focused on helping the 5- and 6-year-old students come up with ways to solve problems.

I often wish I had an iPad when I was in elementary school 20 years ago.

In the same story, Christina focuses on the Hopscotch programming app and kids who grow up using it:

So what happens after kids master Hopscotch? Do they continue coding? Conrad says that the team receives fan mail all the time (something she calls “really gratifying”) from kids who have parlayed their experience with Hopscotch into learning other languages too.

I wonder what Apple thinks of teaching Swift to a new generation of programmers on an iPad.

→ Source: mashable.com