Shared posts

30 Sep 05:41

When The Community Is Indispensable, You Overcome Other Problems

by Richard Millington

Last week StarlingBank announced they were closing their brand community.

“We had hoped the forum would be a place where customers could openly discuss with each other questions they may have about our app. In reality, however, we’ve found that a small but vocal minority of forum members have had unrealistic expectations of what it is for. In addition, the conversations that take place there often expose potentially sensitive customer information.”

A lot of communities have closed down in the past few years because of similar complaints.

i.e.

  • “A lot of our members now talk on social media”
  • “Too much spam”
  • “Vocal minority causing problems”
  • “Legal liability”

These aren’t the real reasons of course. They’re simply the straw that broke the camel’s back.

Every one of these problems has been overcome by others in the past. Facebook doesn’t shut down when moderating becomes a pain, they hire 3000 more moderators.

The real reason is the organization isn’t willing to invest the resources to overcome these problems. They might have plenty of engagement, but it’s not delivering results that matter.

Do you think the community would shut down if it had delivered 500+ great sales leads, nurtured 250 raving advocates, converted 2000 newcomers into premium customers, been the primary source for great PR stories, was the main tool to collect service feedback etc…etc…?

The community is closing down because it’s not delivering the results that make investing extra resources a no-brainer.

Indispensable communities, those which are absolutely necessary, don’t get shut down.

30 Sep 05:41

Are micro-credentials the mortal enemy of culture?

by Serge

The title of this post is inspired by a text written in 1935 by the French poet Paul Valéry which I received from my friend and colleague Philippe Petiqueux (@misterppqx).

The opening words of Paul Valéry’s text are:

“I never hesitate to declare that the diploma is the mortal enemy of culture. The more important diplomas have become in life (and this importance has only increased because of economic circumstances), the lower the performance of education has been. The more control was exercised and extended, the worse the results became.”

What could that signify in the realm of Open Badges, in particular when they are used as a means to deliver micro- or nano- diplomas and micro-credentials? Does the size of the credential matter in its deleterious effects on culture? Are micro-credentials less harmful than macro-credentials? Or more?

Problems start to arise when the means take precedence over the ends, the digital token (the badge) over the human process (the recognition), the institutional recognition (the test, the examination leading to the diploma) over the human process (the learning of individuals and communities), what Paul Valéry identifies as the prevalence of control over action:

“I find that control, in all matters, results in vitiating the action, in perverting it… I have already told you: as soon as an action is subjected to control, the deep purpose of the one who acts is no longer the action itself, but designing the forecast of the control, the defeat of the means of control. The control of studies is just a special instance and a striking demonstration of this very general observation.”

Alfie Kohn develops a similar view: “Heidegger said that life is lived toward—informed by and in anticipation of—death (Sein zum Tode). By analogy, a classroom where learning is always pointed to a test (Lernen zum Examen?) is one where ideas, and the act of reading, are experienced as just so many means to an end.” (Alfie Kohn. Feel-Bad Education: And Other Contrarian Essays on Children and Schooling (p. 90). Beacon Press).

Are micro-credentials less harmful than macro-credentials?

There is no reason to believe that micro-credentials could be any less harmful than macro-credentials. I would even contend that they are potentially more lethal as they are more pervasive and easier to implement. Any person with a bit of authority within the formal education system could create their own micro-credentials supported by their own paraphernalia of micro-assessments, micro-tests or micro-exams, paving the path to the next step: creating macro-credentials out of the compilation of micro-credentials.

It does not mean that all micro-credentials are harmful. Micro-things can be useful and even indispensable, like the non-human micro-organisms living in our body (they are estimated to represent ¾ of all our body’s cells). That does not mean that a human body can be described as the result of the aggregation of micro-organisms! Nor does it mean that a well formed human mind can be described as the result of the aggregation of micro-credentials.

Of course, if we need to hire a plumber or put our brain into the hands of a brain surgeon, we would like to know what their credentials are, an information that goes beyond certificates and diplomas delivered at the end of their study and includes professional development, how they are recognised by their peers, clients/patients, colleagues, how they engage with the world, etc.

If a well formed human mind, whether that of a plumber of a brain surgeon, could probably get many micro-credentials, the aggregation of micro-credentials, on the other hand, will unlikely lead to the creation of a well-formed mind. Probably as likely as the work of Shakespeare could be produced by the combination of a monkey and a typewriter. One does not become a plumber, a brain surgeon or an Elizabethan play author through the mere accumulation of discrete skills or micro-credentials. It is about how they engage with the world.

As Paul Valéry writes about the French Baccalaureate:

“The fundamental diploma in our country is the baccalaureate. It has led to the orientation of studies towards a strictly defined programme and in consideration of tests which, above all, represent, for examiners, professors and patients, a total, radical and uncompensated loss of time and work. […]
“The purpose of teaching is no longer to train the mind, but to acquire the diploma, it is the minimum required that becomes the subject of the studies. It is no longer a question of learning Latin, or Greek, or geometry. It is about borrowing, not acquiring, borrowing what it takes to complete the baccalaureate.”

Beyond micro-credentials (and credentials!)

If Open Badges do not have to be micro-credentials, what could they be? I like to define them as a means to capture recognition; they are snapshots of recognition. The recognition can be formal (in the name of an institution) or informal (in the name of an individual or a community). While micro-credentials provide a fragmented representation of the world and individuals, recognition could provide a continuous and holistic view of the past, present and future of one’s life path and engagement with the world.

Open Badges could pave the way to empower communities as loci of recognition. Institutions might be delegated the power to deliver credentials, but it is the community which is the actual environment where recognition takes place.

Thomas Quasthoff, is testimony to that. Denied admission to the music conservatory in Hanover, Germany, owing to his physical inability to play the piano (his mother’s exposure used thalidomide during her pregnancy) Thomas Quasthoff was later recognised as one of the great baritones of the 20th century.

There are legions of school leavers, whether truants, dropouts (or rather pushedouts!) like  Jean-Paul Gaultier, who became successful in life. The real world offered them opportunities that the micro-world of schooling denied them.

Reference:

Paul Valéry, Le bilan de l’intelligence (1935), in Variété, Œuvres, t. 1, Gallimard, Pléiade, p. 1076. French and English translation.

30 Sep 05:24

Vancouver’s Chinese community has finally embraced hockey. With an Asian superstar surely not far away, how did it happen?

by Patrick Blennerhassett
In Richmond, south of Vancouver, there’s one of the most beautiful sporting venues you’ll ever see. Perched on the edge of the Fraser River is the Richmond Olympic Oval with its spectacularly arching rooftop, built for the 2010 Winter Olympics. During the Games it played host to speed skating, but has since been converted into a multi-purpose sports and event venue. The building, which features pine beetle damaged wood for its roofing, stretches 2.6 hectares, and now houses two ice hockey…
30 Sep 05:24

‘The Misgendering Machines’ at CSCW 2018

This is the associated FAQ for The Misgendering Machines: Trans/HCI Implications of Automatic Gender Recognition, the solo-author paper I got into CSCW 2018 (and am super excited to present!)

What’s this about then?

There’s a subfield of facial recognition called gender classification, or gender recognition, which purports to, well, recognise people’s gender based on photos of them. This is not only worth exploring directly (look at the red flags on that), but it’s also an opportunity to explore HCI’s approach to trans issues and whether that has got any better over time, because HCI has repeatedly used the technology in papers.

What we found was that the technology was, indeed, just as dodgy as it sounds - and so was HCI’s use of it, ranging from papers that did not define ‘gender’ but used it to mean whatever the heck gender classification APIs meant, to papers that implicitly or explicitly defined gender in a trans-exclusionary way. Overall, it suggests the field has made very little progress on undertaking work that is trans-inclusive, as exampled by the wide array of explicitly essentialist work, particularly around health

And what’s the solution?

More trans people. But in the meantime, people actually educating themselves, explicitly operationalising gender in a way they sure as shit haven’t been, and trying to deconstruct what precisely they’re looking for when one of their variables is called “gender”.

Also ending gender classification as a technology in its entirety.

Why did you write a solo-author paper as a first year?

The professionally beneficial answer: I’m a wunderkind whose name you should keep in mind for postdocs, low-work, high-impact publications and grant submissions with multiple commas in the dollar amount.

The actual answer: Everyone Knows that first-years do not write solo-author papers because it is a terrible, exhausting idea, and because Everyone Knows that nobody felt a need to tell me until I’d already done it. I and my sleep cycle have learned our lesson, probably, although CHI suggests that’s a very overestimated “probably”.

What soundtrack would you recommend for the reading of this paper?

30 Sep 05:23

Shaping Cities contribution, “Of Systems and Purposes: Emergent technology for the skeptical urbanist”

by AG

I am very pleased, and every bit as proud, to announce the publication of the latest SUPERTOME to emerge from the Urban Age process, Shaping Cities in an Urban Age, and with it my essay “Of Systems and Purposes.” It won’t contain anything to startle those of you who have been following my work for awhile — you’ll see, for example, that I once again return to the Beer well — but I do think it’s a pretty neat distillation of my thought about cities and technology as it’s developed over the past several years. I reprint it here for your enjoyment.

I’m particularly delighted that my work is featured alongside that of so many urbanists I respect enormously, in such a physically beautiful edition. My congratulations to Ricky Burdett, Philipp Rode, and especially the book’s indefatigable production team.

The legendary technologist Alan Kay once said that “the best way to predict the future is to invent it.”
 
Kay could perhaps be forgiven for the comment’s Promethean hubris, central as he was to the intellectual life of Xerox’s celebrated Palo Alto Research Center (PARC), where so many of the interface conventions we continue to rely upon today were invented. The plain fact of the matter is that an ensemble of techniques invented or extensively developed at PARC, over a period of a very few years in the early 1970s — among them the graphical user interface, the mouse, the windowing system and the kind of multitasking it enabled, laser printing — remain at the core of home and office computer use some forty years down the line. The tools and techniques that Kay and his colleagues at PARC experimented with for their own use really did change the way we all work, think and play, generating a multitrillion-dollar market in the process of doing so.
 
This unimpeachable set of facts certainly does seem to legitimate the premise at the heart of Kay’s claim: that collective futures are something that can be architected at will by the sufficiently visionary. Perhaps unsurprisingly, given the heroic role it casts them in, this notion has been embraced by successive generations of technologists, very much including those now busily at work “disrupting” the processes that have defined urban life since time immemorial. Judging from the frequency with which Kay is invoked in their PowerPoint decks and TED talks, at least, their various interventions in commerce and mobility, self-presentation and socialization, and production, distribution and consumption are consciously intended to realize coherent visions of the future.
 
But which visions? Where Kay’s work at PARC was at least liminally inspired by the liberatory ethos of the Bay Area 1960s — an intellectual current nurtured by the work of thinkers like Illich, Marcuse, Carson and Fuller, the upwelling of the Black Power, feminist and gay-rights movements, the anarchist Diggers and their experiments with Free Stores, Clinics and crashpads, the encounter with mystical-ethical systems of the East, and above all copious amounts of high-grade LSD — his latterday descendants appear to imagine futures of a rather different stripe. Those taking the boldest strides to transform urban life today range from explicit neo-Randians like Uber’s Travis Kalanick, to the avowedly technolibertarian developers of Bitcoin and the technology undergirding it, the blockchain, to those whose political projects — beyond a clear commitment to the standard tenets of entrepreneurial capitalism, as it expresses itself in the neoliberal period — are as yet unclear, like Elon Musk, Jeff Bezos and Mark Zuckerberg.
 
From the public comments, commitments and investments of these and other would-be disruptive innovators of their ilk, it is possible to assemble at least a rough picture of the world they wish to call into being, and therefore the urban forms and rituals that are likely to predominate in that world:
 
Where on-demand, local digital fabrication of goods (via 3D printing, numerically-controlled laser cutting and milling, etc.) is not possible, conventionally-manufactured products will be shipped, warehoused and distributed to the consumer via an almost fully automated supply and fulfillment chain. While it won’t be possible to do without human labor completely, entire job categories — warehouse worker, commercial truck driver, deliveryperson — will disappear from the economy, never to be replaced.
 
The means of production will be held (and such employment contracts as remain necessary issued) by distributed autonomous organizations, corporations manifested in and as self-directing software. With the greater part of the built environment networked at high resolution, and truly economic microtransactions enabled by digital currency, every market for mobility and commercial or residential space is “liquified,” or ruthlessly optimized for efficient, moment-to-moment value extraction. Access to space (microflats, single rooms, or even workstations) will be leased by the minute, while very, very few spatial resources will escape being harnessed for revenue generation.
 
For those who can afford it, on-demand, point-to-point mobility will be undergirded in most cities by a permanently orbiting fleet of autonomous vehicles. And all the while, thanks to the myriad sensors of the so-called internet of things, everything from physical location to social interaction to bodily and affective states becomes grist for the mill of powerful machine-learning algorithms set to anticipate a wide range of needs and desires, and fulfill them before they quite breach the surface of awareness.
 
In this world, the art of governmentality has been refined to a very high degree. Custodial organizations, State or otherwise, are furnished with a torrential flow of information about our choices, and the unparalleled insight into human motivation that can be gleaned from analysis of that flow. Prudent behavior on the part of the consumer-citizen is enforced by an array of personalized performance targets, incentives and disincentives presented in the form of brightly-gamified “social credit” schemes — networked carrots and sticks sufficient to keep all but the irredeemably anti-social acting within permissible bounds.
 
If this sounds like a grim, dispiriting and airless set of possibilities — and it certainly does to me — it is fortunately unlikely that this particular future will unfold in quite the way imagined by those now busily engaged in the attempt to realize it. Several decades’ accumulated experience with networked technologies suggests that whatever actual impact they do have in the fullness of time often bears little to no resemblance to the visions of the people who devised them, or indeed the concrete experiences of their earliest adopters. It would be profoundly foolish to suppose that technologies like 3D printing, the blockchain or machine learning will have no bearing on the form or function of large-scale urban environments. They undoubtedly will. But when would-be innovators promise that their inventions will directly drive radical change — whether undermining material scarcity and the commodity form (as the inventor of the RepRap 3D printer originally imagined his device would do), stripping bias from the operations of the criminal justice system (as the promoters of risk-assessment algorithms promise) or even allowing exchanges of value to abscond from the visibility of the State entirely (as ideologues of the blockchain hope) — we have reason to believe that circumstances will conspire to confound or even subvert their intentions.
 
Recall Steve Jobs’s astonished comment, upon being shown the algorithmically self-righting Segway scooter for the first time, that “they’ll architect cities around these things.” With this technology in hand, the prospect of undoing at least some of the damage done to cities of the twentieth century by the internal-combustion engine suddenly seemed a great deal more credible. The formless sprawl, the environments legible only at speed, the dependence for mobility on capsular vehicles that isolated occupants from their surroundings and one another, above all the air pollution: in the minds of its earliest advocates and enthusiasts, all of these circumstances stood to be transformed by the Segway. But compare this rather pleasant vision to the world we actually live in some two decades downstream from the Segway’s commercial appearance, where the vehicles remain limited to ferrying around annoying platoons of helmeted tourists, and perhaps the occasional airport security officer. Instead of compelling any gross transformation of the urban environment, let alone the way we collectively think about urban mobility, thus far the Segway’s primary contribution to everyday life has been inspiring the cheap, Chinese-made “hoverboards” whose lithium-ion batteries burst into flames with distressing frequency.
 
Or consider what eventually happened to Craigslist — when it first emerged in the San Francisco of the late 1990s, a virtually utopian space in which goods, skilled services and shelter circulated for free. A passionate community of users grew up around the early Bay Area Craigslist, and something very close to a true gift economy sprung into existence among them: a functioning ecosystem of exchange founded on goodwill and mutuality, in the very heart of the late-capitalist West. For these early users, much of what they’d previously resorted to accomplishing at retail was, for a time, furnished by a single humble, all-but-rudimentary website.

And yet, for all its promise and sustaining optimism, this apparition of an entirely different mode of citying somehow failed to take the rest of the world by storm. Putting the indifferent stewardship of its management team to one side, Craigslist was ultimately undone by nothing other than scale. As the userbase drawn by the enticing prospect of free or ultra-low-cost services spiked beyond the Dunbar number — the notional upper bound of a human community in which all the members know one another by name — the bonds of implicit trust necessary to any agalmic community became first harder to sustain, and then impossible to construct at all. And this was replicated in city after city, as the service was rolled out across the planet’s major metropolitan markets, in accordance with the build-once/deploy-many-times ethos that drives the software industry and the logic of unlimited scalability that governs the network. In many ways a victim of its own success, Craigslist just about everywhere soon became cluttered with nakedly commercial listings — listings whose propositions were virtually impossible to verify independently, which flowed onto the site at such implacable velocity that they crowded out the community-generated posts that had so strongly characterized its early days. (What’s more, the platform badly undercut the classified advertising-based business model most free local weeklies depended on, driving many of them to extinction.) None of this looked anything like the neighborly, human-scaled, practically utopian community of exchange its gentle founder Craig Newmark had intended to realize. The Craigslist at scale that we know today, harbor for slumlords, haven for scammers and human traffickers, isn’t so much a negation as an outright renunciation of its initial promise.  

Some technical innovations, of course, actually do result in profound alterations in the form, tenor and distribution of city life. For every internal combustion engine, safety elevator, tungsten-filament lightbulb or mobile phone, though, there are dozens of Segways or Craigslists. It is striking, furthermore, how often the technologies with truly transformational implications for the city were originally intended to address some other order of challenge or problem entirely. I very much doubt, for example, that Jeff Bezos had the cratering of high-street retail, the choking of big-city streets with parcel-delivery traffic or the staggering reduction in demand for warehousing labor in mind when he sat down to draft his first plans for an online book market.

This is a history we might wish to bear in mind when inventors, developers and other interested parties present us with claims that some new technology on offer will surely give rise to radically new (and invariably radically better) permutations of the city. We would be wise to consider that the things they propose will invariably be constrained by what the philosopher Jane Bennett thinks of as “the material recalcitrance of cultural products.” Deeply entrenched systems, structures that are psychic every bit as much as they are political or economic, lay in wait to capture and redirect the energies unleashed by emergent technology, and very often the result of this encounter is something starkly other than any innovators had intended. In this light, we should consider the possibility that Kay’s promise might have been little more than bravado all along, and the successful scaling-up to worldwide hegemony of the ensemble of tools he helped to develop at PARC a one-time, more or less irreproducible fluke, with no particularly salient implications for innovators in other times or places.

For all the sweep and verve of his framing of things, then, I personally prefer the perspective offered by another technologist: the great British cybernetician Stafford Beer, who argued that “the purpose of a system is what it does.” We should evaluate a technology, that is, by considering the outcomes it is actually seen to produce when deployed in the world at scale — and not the reputations of its authors, their intentions, institutional affiliations or prior successes, or the ostensible benefits that supposed to attend its adoption.
 
Applying Beer’s bracing realism, the most pernicious words in the technologist’s vocabulary are “might,” “could” and “can,” and the only meaningful test of a proposed technical intervention are the conditions it empirically gives rise to when deployed into a recalcitrant world. We oughtn’t properly even be speaking of “potential”; the only way to ascertain whether or not a given technical or techno-social proposition is indeed within the space of possibility is to build a prototype, deploy it, and await the results. And what we learn when we consider past innovations in the light of this unremitting standard is that technical development, for all its rigors, is the easy part of inventing the future. Seeing an innovation bedded in at the core of some longer-lasting transformation requires the much harder work of making space for it in all the interlocking systems that give shape to our lives: systems of law, governance and regulation, infrastructures both physical and financial (e.g. insurance), social conventions and practices, language, even entrenched habits of mind.
 
And this is perhaps truer still for those who intend to realize progressive urban futures. It is still possible to dream of cities in which the flows of matter and energy necessary to an equitable distribution of goods and lifechances are sustainable over the longer term, in which the rather abstract, Lefebvrian “right to the city” is made concrete in accessible, universal mobility and participatory political processes, and above all in which dignified, decent lives are possible. But translating these aspirations into conditions on the ground will require urbanists to develop fluency with a set of conditions that by and large remain opaque to them, even threatening.
 
We must in the first instance have the courage to think the city in the light of the more outré technical possibilities suddenly available to us. Just what does public space look or feel like, when each of the people occupying it is surrounded at all times by a cloud of semi-autonomous servitors and companions, virtual as well as materially embodied? What remains of high streets, Main Streets or malls once retail as we have known it, with all its ability to communicate, seduce and gather, is exploded into ten thousand separate acts of on-the-spot production or just-in-time fulfilment? What do prospects for entry-level or otherwise unskilled employment look like in that unbundled world, and how will that be felt in the tenor of street life? The ways in which these questions come to be answered will set the boundary conditions for everyday urban life, for the kinds of political struggle that are possible in the urban frame and for the subjectivities and selves that arise there.
 
As we reckon with the lines of flight that now open up to us, however, we must retain the clarity and integrity to ground these possibilities against everything we know about the fate of interventions past. We need to understand the captures, detours and reversals that perennially afflict emergent technologies at the point where they intersect with existing ways of doing, making, dwelling and being, taking note particularly of the fact that technologies that prosper and find traction in the world are very often those which reinforce existing inequities of power. What this implies for urbanists of a politically progressive stripe is that, for a given struggle, conventional community organizing may offer a far better return on investments in energy, effort or other resources than an attempt to drive change via technical means.
 
Working fluently with technology means stripping it of its unearned gloss of neutrality. All technologies are, without exception, expressions of one or another set of values, and therefore by any sane accounting ought to be contested terrain. When Uber becomes popular in a given city, for instance, and that popularity is explicitly cited as justification for not maintaining an adequate level of investment in public transit, we can be sure that what we are seeing is somebody’s values being enacted, if not necessarily our own.

Like any other professional or disciplinary community, the adepts of network technology hold tacit beliefs in common. They hold certain conceptions of the just, the true and the beautiful, think the world in certain distinct ways. If they cannot always realize their aims directly, it still behooves us to know what they believe, and understand what it is that they are trying to achieve.

Further, the particular set of values inscribed in a technology may have a great deal to do with its fortunes in the world, and how well it is able to function as a purposive invention of the future. Uber is a particularly resonant example; whatever else it may be, it enacts a kind of propaganda of the deed, or what the media scholar Alison Powell calls the argument-by-technology. The vision of hyperindividualism, invidious interpersonal competitiveness and unlimited-convenience-for-those-who-can-afford-it inscribed in the service dovetails perfectly with — one might even say “embraces and extends” — the neoliberal ethos that has prevailed in the developed world for the past four decades. And this perhaps explains why it has been realized, where the rather more humane visions undergirding Craigslist or the Segway plainly have not been. Wherever services like Uber go unchallenged, the imposition of these values is effectively a fait accompli — and with future resource commitments tending to be entrained by path dependence, that achievement sets the initial conditions for everything that follows in its wake.

In the end, perhaps the crucial insight is this: urbanists can no longer ignore the impact of developments like machine learning, large-scale data analysis and automation, or treat them as something external to our field of inquiry. Operating at every scale and level of urban life, from vehicle guidance to the mediation of sociality to the aesthetics of the built environment, they are clearly set to exert the most profound influence on the physical spaces of our cities, the things we do in them, the ways they generate meaning and value, and the very selves we understand ourselves to be. It’s no longer tenable for anyone who cares about the life of cities to hold this set of facts at bay, especially those of us who nurture some remaining hope that the master’s tools can be used to build other sorts of houses entirely. And while we needn’t and oughtn’t build our practices exclusively around this class of technologies, we might want to consider how to fold a nuanced, properly skeptical engagement with them into our approach to the design of urban space and experience.

30 Sep 05:22

Markets can support competition

by Paul Jarvis
“Why are you building X when Y exists?”
30 Sep 05:22

✚ Practicing with a Visualization Toolbox, Tools and Additional Resources Roundup – September 2018

by Nathan Yau

I hope that if you're trying to learn how to work with data that you make time to fiddle with the toys in your growing toolbox. Otherwise, you just have a bunch of bookmarks and no new skills. Here are some things you can do to hone your skills. Read More

30 Sep 05:22

Same Difference

by David A. Banks

As an aspirational ideal, “creativity” has a uniquely insidious siren song: It promises escape from the system that defines it. To be creative is to transcend or recombine the established order, but there’s always the danger of cooptation and appropriation. As Dr. Malcolm, Jeff Goldblum’s character in Jurassic Park puts it to the park’s CEO, “You stood on the shoulders of geniuses to accomplish something as fast as you could, and before you even knew what you had, you patented it, packaged it, and slapped it on a plastic lunch box and you’re selling it, and you’re selling it!” But of course they are — why bother with scientific breakthroughs if you are not going to commodify them? Figuring out how to sequence DNA is one thing, but turning that idea into a theme park? That is true creativity.

In Against Creativity, Oli Mould, a lecturer in Human Geography at the University of London, explores this phenomenon of how radical and revolutionary ideas become mere fodder for lunch boxes. Creativity, Mould claims, is often invoked to describe not how ideas break free of capitalism but are made compatible with it. It recasts kinds of labor that may have seemed outside capitalist exploitation — care, emotion, art, design — as the most exploitable form of production. The way creativity is used today, Mould writes, “feeds the notion that the world and everything in it can be monetized.” Accordingly, creativity has become a means to rent, sell, or offer subscriptions to something that was once free or otherwise disconnected from the profit motive. Graffiti artists are hired by real estate firms to bring a safe level of grittiness to a neighborhood. Ebay asks us to choose between passing on a valuable collectable to a relative or finding the highest bidder.

The “power to create something from nothing” can only derive from cooperation. No one does great things alone

In Mould’s account, capitalism is especially hungry for subcultures — hippies, punks, skaters, etc., not to mention black and queer culture in general — which can be transformed into “vehicles for its proliferation.” These subcultures are mined for new symbols, artifacts, and practices that can be scaled up from regional favorites to international sensations.

This vision of creativity’s close bond with economic growth is epitomized by urban-planning theorist Richard Florida, who has convinced cities and towns to cater to the “creative class” as the central engine of their economic development. One need only install the sorts of converted living spaces, artisanal bars, and totems of tolerance and “authenticity” that “creatives” enjoy and sit back and watch as game developers, interior designers, and small-batch chocolatiers jump-start the local economy.

“Contemporary capitalism,” Mould argues, “has commandeered creativity to ensure its own growth.” As that claim suggests, he believes that creativity pre-existed capitalism and has not always just been a euphemism for economic exploitation. In his view, the word originally signified the generation of something from nothing, or the uniting of two previously separate ideas. But since something truly new or different is unassimilable to capitalism’s techniques for value extraction — economies of scale, interchangeable workers, mass production — much of creativity under capitalism is, as Mould argues, “newness to maintain more of the same,” rather than the development of new ways of being.

As a result, creativity under neoliberalism can be found only “by looking to your own agency; any appeal to wider structures do not matter.” Regardless of whether you are a poet or a software engineer, once you are convinced that creativity begins and ends within yourself, your work stops destabilizing the power structures that divvy out the rewards.

Nevertheless, Mould holds out hope for “an alternative, perhaps revolutionary creativity” capable of “creating new phenomenon to which capitalism is unaware.” This would require redefining creativity in terms of evading capitalist co-optation. It also means surrendering the individualist ideology that is central to “creativity” as an aspirational ideal — the idea that you alone can make a difference. If you believe the change agent is yourself and not your relationships with other people, you are already working in a capitalist mode.

One doesn’t even need to be critical of capitalism to acknowledge this: The “power to create something from nothing” and turn impossibilities into lived experiences can only derive from cooperation. From the moon landing to the polio vaccine, no one does great things alone.

Contrary to capitalist myth, individual genius can produce nothing in itself. Neither people nor things are creative; ideas and processes and relations are. True creativity, according to Mould, is inherently social, a matter of reorganizing collective effort rather than iterating on products and services. He points to Britain’s National Health Service and other cooperative work arrangements as fertile ground for authentic creativity: “Co-operative models of labor organization, self-management and equality of pay; these are the antitheses of the current vernacular of creative work, but they should be its driving force.” To redeem creativity, in other words, we should stop seeing it as the basis for rewarding individuals with fame and fortune and instead focus on the creative potential of organizations.

But if Mould is right that true creative work is the opposite of selling out, then how do you do creative work to survive? It is not easy to delineate between capitalistic behavior and unalienated “authentic” activity, because capitalism constantly demands that individuals do things in exchange for money. If anything short of a complete overthrow of capitalism will fail to permit a “true” creativity (which may indeed be the case), we’re boxed in to an analytic corner: If creativity is best understood as anything that is not capitalism, then creativity actually depends on capitalism. In a post-capitalist world, by this view, no one would be creative.

To define “true” creativity against the workings of capitalism, Mould must be able to spell out exactly where creativity ends and marketing begins. But these are not so easy to clearly differentiate, and the process ends up requiring a kind of exceptionalism. At one point, he tries to frame true “creativity” through an analysis of the work of Jack Coulter, “a synesthete who paints the colors he hears.”

Coulter’s paintings are, of course, themselves consumable, they are for sale as prints, and his Instagram account has over 66,000 followers. The outputs therefore become commodified. But it is the experiences that Coulter has before putting brush to canvas that is “creative” in the non-capitalist sense. He is experiencing stimuli that are simply not achievable by “normal” sensibilities. The resultant works are vivid depictions of the world that he has access to; visual representations of a world that a non-synesthete would never be able to experience.

By this account, what makes Coulter’s work creative rather than capitalistic is rooted in his having a particular kind of body with particular capacities. Mould labels this “diffability” — his neologism for disability that emphasizes difference over lack. While this is clearly meant to betoken respect for the differently abled, it also complicates his earlier definitions of creativity. To say that an “experience” like Coulter’s synesthesia is creative until it leaves his mind seems to contradict his claim that creativity is a social enterprise. Here, one can simply experience something unique to become creative. More troubling is how this appears to pin creativity on something one has no control over, opening the door for a kind of biological determinism — though Mould does not step through it himself. He notes instead that the diffabled face a more acute version of the dilemma everyone faces under neoliberal capitalism: whether they should turn the qualities that make their experience unique into products that make them commensurate with everything else that can be bought with money.


In 2017, Pepsi ran a commercial in which Kendall Jenner attends a vaguely sketched protest march (protesters carry signs that read “join the conversation) and gives a Pepsi to a cop. The cop tentatively accepts it and, after drinking it, is rewarded with inexplicable applause. The commercial is nearly three minutes long — so long that it has a B plot where a cellist and a photographer (i.e. creatives) are unhappy with their work until they are inspired by seeing the protest. It is not hard to imagine the ad agency brief that led to this: a few breathless paragraphs about a new generation trying to balance their care for the issues with feelings of belonging and personal inspiration.

What’s missing from Mould’s argument about creativity is precisely this kind of scaffolding: What kinds of creative work are systematically prevented?

The rebuke from the ad’s targeted demographic was swift — clearly Pepsi had appropriated the image of protest to try to profit off of a sterilized version of what thousands of people found very meaningful — and the ad was pulled. But it had already done its work and garnered dozens of headlines and a few television news segments. In an economy where attention begets profit, confronting corporate malfeasance has a good chance of creating more value than headaches for shareholders. By conforming to a culture of popular outrage, the liberal commentariat participated in a Pepsi ad campaign. The ad even made it into the introduction of Mould’s book, published a year later, and now the review that you are currently reading. Should we have ignored the ad and risked tacitly endorsing it, or was critique more appropriate, even as it generated the sort of buzz that ad executives dream about?

Whitney Phillips, in a white paper written for journalists about the risks of amplifying far-right groups, addresses another version of the same question: She writes that while it may have been “unethical” to completely ignore a resurgent, organized white supremacist movement online, it is also true that “news coverage of those messages helped make the messages, and their messengers, much more visible than they would have been otherwise, even when the reporting took an explicitly critical stance.”

One of the main catalysts of this no-win scenario, Phillips writes, is the “tyranny of analytics”: “The emphasis on quantifiable metrics stacks the news cycle with stories most likely to generate the highest level of engagement possible, across as many platforms as possible. Things traveling too far, too fast, with too much emotional urgency, is exactly the point, but these are also the conditions that can create harm.” We don’t know how to confront the emotional and physical violence of multinational brands or white supremacists without giving them valuable attention.

Revolutionary creativity would need to circumvent this dilemma as well as the conundrum of “selling out.” It would have to invent a new way to contend with the logic of attention, the political economy of clicks and views. Exactly what that way is a known unknown. Knowing what we do not know — what Robert K. Merton called “specified ignorance” — allows researchers to build a scaffold for knowledge creation. The state of particle physicists in the 1960s suggested the existence of the Higgs boson, but it wasn’t actually discovered until half a century later. It was the specific lack of knowledge that drove scientific investigation.

What’s missing from Mould’s argument about creativity is precisely this kind of scaffolding: What kinds of creative work are systematically prevented due to our collective ignorance? Mould implores readers to practice “imagining the impossible” as a means of confronting capitalist appropriation. But how do we make these individual efforts sync up and scale?

This seems more urgent than rehashing how capitalism neutralizes and assimilates novelty produced at the margins. Even as we fight capitalism, we conform to its logic not by selling out but by desiring what it teaches us to desire. In his quest to defrock capitalist creativity and uncover a true, authentic creative power, Mould loses sight of this power of conformity. We need a method that is itself the product of creative thinking but has its sights on creating a new normal and not personal distinction. To conform is to give up individual freedoms in service of a greater good — if we are truly casting aside our individualist ideas of what counts as creativity, then we need to start getting specific about the kind of box we are willing to think in. Rather than get bogged down in the dubious binary of “Is it revolutionary or is it appropriated?” we should instead ask what kind of anti-capitalist culture would be worth conforming to.

30 Sep 05:22

Scientific Facts — Are they like Myths, Told through Fairytales and Spread by Gossip?

Anita de Waard, The Scholarly Kitchen, Oct 09, 2018


Icon

This paper caught my imagination despite myself, and then convinced me that the answer to the question in the headline is "yes". The key is in understanding the semantics in articles: "In stories and rhetoric, a statement exists not as a separate entity, but plays a role in the overarching narrative." When we look at scientific papers, this role is often signified by tense. In conclusions, the results are typically qualified ("It appears that..." or "One explanation may be..."). Over time, and in the wider community, this hypothesis is accepted. Then, in citing papers, the qualifiactions are dropped, and the conclusion is simply stated as a fact. "In short, scientific facts are based on a game of telephone between cited and citing authors. Besides being pulled from its experimental context, claims are often validated (and turned into ‘known facts’) by the simple act of being cited."

Web: [Direct Link] [This Post]
30 Sep 05:22

Surprising Ways AI Can Improve eLearning Accessibility

Pamela Hogle, Learning Solutions, Oct 09, 2018


Icon

I don't thing the solutions are particularly "surprising" but I think the linkage between AI and accessibility is a good one. Pamela Hogle describes how AI can help with authoring, with translation and transcribing, with vision assistance (including reading and recognition), with voice commands, and with eye tracking to improve the VR experience, for example, through foveated rendering.

Web: [Direct Link] [This Post]
30 Sep 05:21

Beautiful Bikes from Boston's Builders' Ball - Chapman Cycles 650b Tourer

by noreply@blogger.com (VeloOrange)
by Igor

Builders' Ball was an absolute whirl of excitement and energy. It was one jam-packed day of amazing bikes, people, and conversation. Some of New England's best builders were here along with lots of old and new friends to see. This will be a three-part series of three bikes over the next three days that really stood out to me regarding craftsmanship, utility, and beauty. Enjoy!

To kick off this trio, check out this Chapman Cycles 650b Flat-Bar Tourer. It was an absolutely stunning machine with details for days, flawless paintwork, and custom components that would make even the most known, classic constructeurs to pick their jaws off the floor. Here is Brian's transcribed blog post about this build along with my photos from the show.


Gorgeous paintwork and lining on our Wavy Fenders as always from Brian
"I’ve known Frank for years as he has been a staple at many handmade bike shows. He even puts on the Ann Arbor Handmade Bicycle Show in Michigan. He’s well known in the cycling community as the curator of Lugs Not Drugs on Instagram which features beautiful handmade bikes from around the world. So when Frank asked me to build him a touring bike, I was honored.


Many don't know that our draw-bolts are actually plated brass. By machining the ends, you can make some gorgeous contrast with silver and Noir fenders.


Frank has many bikes and knew what he was looking for in regards to fit and components. It was definitely a collaboration between the two of us but what you see here is mostly the brainchild of Frank. The drivetrain is almost exclusively SunXCD with their cranks, chainrings, front, and rear derailleurs. The downtube shifters are Huret friction with a wide range double up front and a 9-speed 11-34 on the rear. The wheels were built by Tommy at Cutlass Wheels in Baltimore and have the Pacenti Brevet rims laced to a White Industries MI5 rear hub and Schmidt SL wide body dynamo in the front.
Integrated spoke holder
There is a large part of the cycling community who are embracing the 45° back sweep on these Jones handlebars. Jeff Jones has been developing and perfecting these bars for a long time and has a lot to say on how they should be set up. In this case, with some insight directly from Jeff, I set the 710mm Bend H-Bar up for Frank by not cutting them down and using long grips. The grips are the extra long ODI longnecks of which I used two pairs in order to get the full bar covered. I also used downtube shifters and smooth clamp locking brake levers to maximize the amount of hand positions. I find the locking brake levers essential for these bars because if you’ve ever tried to lean one that is loaded against a wall, it really just wants to roll and fall.




Some of the traditional features I like to include on the bikes I build are the integrated pump, integrated lighting, handmade racks, and handmade stem. But on this bike I tried to take it a little farther with the handmade cantilever brakesstainless straddle cable hangers, and rear cable hanger. The geometry of my brakes hasn’t changed since I built my first set 3 years ago. I have changed some small details but I’m very happy with their performance as is. The custom straddle hangers are minimal and use pinch bolts to hold the cable as opposed to a set screw style. These were quickly cobbled together the morning after I was thinking of them and future iterations may vary a little.

Because of the non-traditional geometry, I was not able to build using lugs. To get a lug edge though, I chose to do a bilaminate construction combining fillet brazing and faux lugs. This can be seen on the head tube and in the seat cluster. The frame has clearance for 650b knobbies but it is shown here with 48mm wide Compass tires and the new Wavy fenders from Velo-Orange. The frame obviously has a sloping top tube and I wanted to make it look as integrated into the design as possible. Choosing the angle was not arbitrary and was based off of many criteria including frame size, head tube length, stem angle, and bar angle. I’m convinced that 11° is perfect on a bike this size. Frank is 6’1″ and I wanted the bike to be proportionally balanced.
Additional lining on the headtube and bi-lam lugs
Frank sent me a photo of the gray he liked and I matched him up with this warm gray. The head tube is a very light shade of it and I did the lining in burgundy. I did many mock-ups in Adobe Illustrator going back and forth with Frank in regards to that third color. Burgundy won out over pink, red, and navy blue. There are no straight lines on these Wavy fenders except for the edge but I wanted to play around and try and paint them. A couple yards of 3M blue and green tape later and I had my edge. It’s actually difficult to see the transition in certain lights but the burgundy stripe is clearly visible. This gray bike may be understated from a distance but once you get up close, there’s a lot to take in."
30 Sep 05:21

Adding Device Frames to iPhone XS and XS Max Screenshots with Shortcuts

by Federico Viticci

Update 10/10: A newer version of this shortcut, which can apply frames to screenshots taken on multiple Apple devices, is available here.

MacStories readers may be familiar with the way I like to present iPhone screenshots in app reviews and other stories – particularly for "hero" images, such as the one above, I want my screenshots to be contained in device frames that resemble official marketing images from Apple. They're prettier, and they do a better job at communicating what an app looks like on an actual device. I could create these images manually using apps like Affinity Photo and Pixelmator on iOS, but the process would be slow, boring, and time-consuming. Instead, for years now I've been using Workflow and its 'Overlay Image' action to get this done in an automated fashion.

With Shortcuts and the new iPhone XS and XS Max, it was time for an update to my old workflow. While I could have kept using the same iPhone X assets for the XS given their physical resemblance, I upgraded to a XS Max this year, which meant that my screenshots wouldn't have fit the old device frames natively anyway. Fortunately, Apple uploaded official marketing assets for the XS and XS Max a couple of days ago, so with the help of my girlfriend (who's better at Photoshop than I am) I was able to update my workflow for the new devices and add a few extra options in the process as well.

My new 'XS Frames' shortcut supports both the XS and XS Max in portrait and landscape orientations; it's based on official assets from Apple, but it doesn't require any configuration on your end because graphics are embedded within the shortcut as base64 text strings; and, in addition to saving framed screenshots into the Photos app, it can also combine screenshots in the same image if you're dealing with multiple images at the same time (again, just like the hero image above).

The key feature of this shortcut is that it doesn't require any management on your end. Screenshots are automatically filtered from your photo library, and the shortcut understands whether it's processing portrait or landscape orientations based on the width of each image. Because assets are encoded inside the shortcut, you don't have to manually download any templates. You can run the shortcut as an extension when sharing images in Photos or in the Shortcuts app, where it'll bring up a screenshot picker UI; in my experience, due to memory constraints neither Siri nor the widget were able to successfully execute the shortcut. I recommend running the shortcut in the app to avoid any memory-related issues with extensions.

When the shortcut first runs, it checks for images in the input; if no images were passed in the first step, the shortcut is not running within the share sheet in Photos, which means it needs to display an image picker containing XS and XS Max screenshots. This is done using a combination of two 'Find Photos Where' actions that filter screenshots and subsequently filter screenshots that also match specific width parameters. In the picker, screenshots are sorted by date (with the newest ones at the bottom, like in Photos) and you can select multiple ones before continuing.

Filtering screenshots based on their width values.

Filtering screenshots based on their width values.

At this point, the shortcut starts a repeat loop for each screenshot chosen at the beginning. You don't have to care about any of these technical details, but under the hood the shortcut creates a dictionary for each image based on the type of device and orientation using coordinates (x and y values for where to paste a screenshot into a device frame) and the aforementioned encoded assets. I think it's pretty neat. Screenshots and associated device frames are then decoded, combined, and added to a variable. At the end of the shortcut, multiple framed screenshots are combined in a single composite image, but otherwise the final output is always saved to Photos into the default 'All Photos' album.

An example of landscape XS Max screenshots, framed and combined in the same image.

An example of landscape XS Max screenshots, framed and combined in the same image.

I tested this shortcut with up to 13 XS and XS Max screenshots at a time, and it worked without issues; in just a few seconds I was able to pick screenshots, frame them, combine them, and end up with hero images I can use in my articles.

Framing the latest screenshot, which can be shared in any app just by pasting it.

Framing the latest screenshot, which can be shared in any app just by pasting it.

As a bonus, I also created a separate version of this shortcut that frames the latest screenshot you took, saves it into your photo library, and also copies the image to the system clipboard. This can be a nice way to quickly "prettify" a screenshot that you want to share on Twitter or iMessage. You can invoke this shortcut with a Siri phrase or from the widget, but it'll always open the Shortcuts app to avoid any memory constraints.

Updated shortcuts for the Apple Watch Series 4 and MacBook Pro line are on my list to work on next. In the meantime, I'm going to be using this shortcut to frame all my XS Max screenshots, which you're going to see here on MacStories as well as the Club. You can download the shortcuts below.

XS Frames

Add device frames to iPhone XS and XS Max screenshots. The shortcut supports portrait and landscape orientations. If multiple screenshots are passed as input, they will be combined in a single image.

Get the shortcut here.

Frame Latest

Add an iPhone XS or XS Max frame to the latest screenshot from your library. The composite image is saved into the Photos app and copied to the clipboard.

Get the shortcut here.


Support MacStories Directly

Club MacStories offers exclusive access to extra MacStories content, delivered every week; it's also a way to support us directly.

Club MacStories will help you discover the best apps for your devices and get the most out of your iPhone, iPad, and Mac. Plus, it's made in Italy.

Join Now
30 Sep 05:21

Architecture of Nautilus, the new Dropbox search engine

by Diwaker Gupta

Over the last few months, the Search Infrastructure engineering team at Dropbox has been busy releasing a new full-text search engine called Nautilus, as a replacement for our previous search engine.

Search presents a unique challenge when it comes to Dropbox due to our massive scale—with hundreds of billions of pieces of content—and also due to the need for providing a personalized search experience to each of our 500M+ registered users. It’s personalized in multiple ways: not only does each user have access to a different set of documents, but users also have different preferences and behaviors in how they search. This is in contrast to web search engines, where the focus on personalization is almost entirely on the latter aspect, but over a corpus of documents that are largely the same for each user (localities aside).

In addition, some of the content that we’re indexing for search changes quite often. For example, think about a user (or several users) working on a report or a presentation. They will save multiple versions over time, each of which might change the search terms that the document should be retrievable by.

More generally, we want to help users find the most relevant documents for a given query—at this particular moment in time—in the most efficient way possible. This requires being able to leverage machine intelligence at several stages in the search pipeline, from content-specific machine learning (such as image understanding systems) to learning systems that can better rank search results to suit each user’s preferences.

The Nautilus team worked with our machine intelligence platform to scale our search ranking and content understanding models. These kind of systems require a lot of iteration to get right, and so it is crucial to be able to experiment with different algorithms and subsystems, and gradually improve the system over time, piece-by-piece. Thus, the primary objectives we set for ourselves when starting the Nautilus project were to:

  • Deliver best-in-class performance, scalability, and reliability to deal with the scale of our data
  • Provide a foundation for implementing intelligent document ranking and retrieval features
  • Build a flexible system that would allow our engineers to easily customize the document-indexing and query-processing pipelines for running experiments
  • And, as with any system that manages our users’ content, our search system needed to deliver on these objectives quickly, reliably, and with strong safeguards to preserve the privacy of our users’ data

In this blog post we describe the architecture of the Nautilus system and its key characteristics, provide details about the choices we made in terms of technologies and approaches we chose for the design, and explain how we make use of machine learning (ML) at various stages of the system.

High level architecture

Nautilus consists of two mostly-independent sub-systems: indexing and serving.

The role of the indexing pipeline is to process file and user activity, extract content and metadata out of it, and create a search index. The serving system then uses this search index to return a set of results in response to user queries. Together, these systems span several geographically-distributed Dropbox data centers, running tens of thousands of processes on more than a thousand physical hosts.

The simplest way to build an index would be to periodically iterate through all files in Dropbox, add them to the index, and then allow the serving system to answer requests. However, such a system wouldn’t be able to keep up with changes to documents in anything close to real-time, as we need to be able to do. So we follow a hybrid approach which is fairly common for search systems at large scale:

  • We generate “offline” builds of the search index on a regular basis (every 3 days, on average)
  • As users interact with files and each other, such as editing files or sharing them with other users, we generate “index mutations” that are then applied to both the live index and to a persistent document store, in near real-time (on the order of a few seconds).

Some other key pieces of the system that we’ll talk about are how to index different kinds of content, including using ML for document understanding and how to rank retrieved search results (including from other search indexes) using an ML-based ranking service.

Data sharding

Before we talk about specific subsystems in Nautilus, let’s briefly discuss how we can achieve the level of scale we need. With hundreds of billions of pieces of content, we have an enormous amount of data that we need to index. We split, or “shard,” this data across multiple machines. To do this, we need to decide how to shard files such that search requests for each user complete quickly, while also balancing load relatively evenly across our machines.

At Dropbox, we already have such a schema for grouping files, called a “namespace,” which can be thought of as a folder that one or more users have access to. One of the benefits of this approach is it allows us to only see search results from files that they have access to, and it is how we allow for shared folders. For example: the folder becomes a new namespace that both sharer and share recipient have access to. The set of files a Dropbox user can access is fully defined by the set of underlying namespaces she was granted access to. Given the above properties of namespaces, when a user searches for a term, we need to search all of the namespaces that they have access to and combine the results from all matches. This also means that by passing the namespaces to the search system we only search content that the querying user can access at the time the search is executed.

We group a number of namespaces into a “partition,” which is the logical unit over which we store, index, and serve the data. We use a partitioning scheme that allows us to easily repartition namespaces in the future, as our needs change.

 

Indexing

Document extraction and understanding

What are the kinds of things users would like to search by? Of course there is the content of each document, i.e., the text in the file. But there are also numerous other types of data and metadata that are relevant.

We designed Nautilus to flexibly handle all of these and more, through the ability to define a set of “extractors” each of which extracts some sort of output from the input file and writes to a column in our “document store.” The underlying technology has extra custom built layers that provide access control and data encryption. It contains one row per file, with each column containing the output from a particular extractor. One significant advantage of this schema is that we can easily update multiple columns on a row in parallel without worrying about changes from one extractor interfering with those from others.

For most documents, we rely on Apache Tika to transform the original document into a canonical HTML representation, which then gets parsed in order to extract a list of “tokens” (i.e. words) and their “attributes” (i.e. formatting, position, etc…).

After we extract the tokens, we can augment the data in various ways using a “Doc Understanding” pipeline, which is well suited for experimenting with extraction of optional metadata and signals. As input it takes the data extracted from the document itself and outputs a set of additional data which we call “annotations.” Pluggable modules called “annotators” are in charge of generating the annotations. An example of a simple annotator is the stemming module which generates stemmed tokens based on raw tokens. Another example is converting tokens to embeddings for more flexible search.

Offline build

The document store contains the entire search corpus, but it is not well-suited for running searches. This is because it stores extracted content mapped by document id. For search, we need an inverted index: a mapping from search term to list of documents. The offline build system is in charge of periodically re-building this search index from the document store. It runs the equivalent of a MapReduce job on our document store in order to build up a search index that can be queried extremely fast. Each partition ends up with a set of index files that are stored in an “index store.”

By separating the document extraction process from the indexing process, we gain a lot of flexibility for experiments:

    • Modification of the internal format of the index itself, including the ability to experiment on a new index format that improves retrieval performance or reduces storage costs.
    • Applying a new document annotator to the entire corpus. Once an annotator has demonstrated benefits when applied to the stream of fresh documents flowing in the instant indexing pipeline, it can be applied to the entire corpus of documents within a couple days by simply adding it to the offline build pipeline. This increases experimentation velocity compared to having to run large backfill scripts for updating the data corpus in the document store.
    • Improving the heuristics used for filtering the data that gets indexed. Not surprisingly, when dealing with hundreds of billions of pieces of content, we have to protect the system from edge cases that could cause accuracy or performance degradations, e.g., some extremely large documents or documents that were incorrectly parsed and generate garbled tokens. We have several heuristics for filtering out such documents from the index, and we can easily update these heuristics over time, without having to reprocess the source documents every time.
    • Ability to mitigate an unforeseen issue caused by a new experiment. If there is some error in the indexing process, we can simply rollback to a previous version of the index. This safeguard translates to a higher tolerance for risk and speed of iteration when experimenting.

 

Serving

The serving system is comprised of a front-end, which accepts and forwards user search queries; a retrieval engine which retrieves a large list of matching documents for each query; and a ranking system named Octopus that ranks results from multiple back-ends using machine learning. We’ll focus here on the latter two, as the front-end is a fairly straightforward set of APIs that all our clients use (web, desktop, and mobile).

Retrieval engine

The retrieval engine is a distributed system which fetches documents that match a search query. The engine is optimized for performance and high recall—it aims to return the largest set of candidates possible in the given allocated time budget. These results will then be ranked by Octopus, our search orchestration layer, to achieve high precision, i.e., ensure that the most relevant results are highest in the list. The retrieval engine is divided into a set of “leaves” and a “root”:

  • The root is primarily in charge of fanning out incoming queries to the set of leaves holding the data, and then receiving and merging results from the leaves, before returning them to Octopus.
    • The root also includes a “query understanding” pipeline which is very similar to the doc understanding pipeline we discussed above. The purpose of this is to transform or annotate a query to improve retrieval results.
  • Each leaf handles the actual document lookup for a group of namespaces. It manages the inverted and forward document index. The index is periodically seeded by downloading a build from the offline build process, and is then continuously updated by applying mutations consumed from Kafka queues.

Search Orchestrator

Our Search Orchestration layer is called Octopus. Upon receiving a query from a user, the first task performed by Octopus is to call Dropbox’s access-control service to determine the exact set of namespaces the user has read access to. This set defines the “scope” of the query that will be performed by the downstream retrieval engine, ensuring that only content accessible to the user will be searched.

Besides fetching results from the Nautilus retrieval engine, we have to do a couple things before we can return a final set of results to the user:

  • Federation: In addition to our primary document store and retrieval engine (described above), we also have a few separate auxiliary backends that need to be queried for specific types of content. One example of this is Dropbox Paper documents, which currently run on a separate stack. Octopus provides the flexibility to send search queries to and merge results from multiple backend search engines.
  • Shadow engines: The ability to serve results from multiple backends is also extremely useful for testing updates to our primary retrieval engine backend. During the validation phase, we can send search queries to both the production system and the new system being tested. This is often referred to as “shadow” traffic. Only results from the production system are returned to the user, but data from both systems is logged for further analysis, such as comparing search results or measuring performance differences.
  • Ranking: After collecting the list of candidate documents from the search backends, Octopus fetches additional signals and metadata as needed, before sending that information to a separate ranking service, which in turn computes the scores to select the final list of results returned to the user.
  • Access Control (ACL) checks: In addition to the retrieval engine restricting the search to the set of namespaces defined in the query scope, an additional layer of protection is added at the Octopus layer by double checking that each result returned by the retrieval engine can be accessed by the querying user before returning them.

Note that all of these steps have to happen very fast—we target a budget of 500ms for the 95th percentile search (i.e., only 5% of searches should ever take longer than 500ms). In a future blog post, we will describe how we make that happen.

Machine learning powered ranking

As mentioned earlier, we tune our retrieval engine to return a large set of matching documents, without worrying too much about how relevant each document is to the user. The ranking step is where we focus on the opposite end of the spectrum: picking the documents that the user is most likely to want right now. (In technical terms, the retrieval engine is tuned for recall, while the ranker is tuned for precision.)

The ranking engine is powered by a ML model that outputs a score for each document based on a variety of signals. Some signals measure the relevance of the document to the query (e.g., BM25), while others measure the relevance of the document to the user at the current moment in time (e.g., who the user has been interacting with, or what types of files the user has been working on).

The model is trained using anonymized “click” data from our front-end, which excludes any personally identifiable data. Given searches in the past and which results were clicked on, we can learn general patterns of relevance. In addition, the model is retrained or updated frequently, adapting and learning from general users’ behaviors over time.

The main advantage of using an ML-based solution for ranking is that we can use a large number of signals, as well as deal with new signals automatically. For example, you could imagine manually defining an “importance” for each type of signal we have available to us, such as which documents the user interacted with recently, or how many times the document contains the search terms. This might be doable if you only have a handful of signals, but as you add tens or hundreds or even thousands of signals, this becomes impossible to do in an optimal way. This is exactly where ML shines: it can automatically learn the right set of “importance weights” to use for ranking documents, such that the most relevant ones are shown to the user. For example, by experimentation, we determined that freshness-related signals contribute significantly to more relevant results.

Conclusion

After a period of qualification where Nautilus was running in shadow mode, it is currently the primary search engine at Dropbox. We’ve already seen significant improvements to the time-to-index new and updated content, and there’s much more to come.

Now that we have solid foundations in place, our team is busy building on top of the Nautilus platform to add new features and improve search quality. We’re exploring new capabilities, such as augmenting the existing posting-list-retrieval-algorithm with distance-based retrieval in an embedding space; unlocking search for image, video, and audio files; improving personalization using additional user activity signals; and much more. Find out about how we validated the performance and reliability of Nautilus in the second part of this post.

Nautilus is a prime example of the type of large scale projects involving data retrieval and machine learning that engineers at Dropbox tackle. If you are interested in these kinds of problems, we would love to have you on our team.

Thanks to: Adam Faulkner, Adhiraj Somani, Alan Shieh, Annie Zhou, Braeden Kepner, Elliott Jin, Franck Chastagnol, Han Lee, Harald Schiöberg, Ivan Traus, Kelly Liu, Michael Mi, Peng Wang, Rajesh Venkataraman, Ross Semenov, and Sammy Steele.

30 Sep 05:21

Dollars for Booze

This week, we’ll transform some data on US alcohol sales – data that wants to make us believe that US-Americans spent 800 times more on alcohol in 2014 than in 1935. As we’ll find out, this incredible increase exists mainly thanks to two other phenomena, which we will get rid of. We’ll adjust for purchasing power and take population into account to arrive at the following useful pattern:

Be aware that the chart doesn’t show drinking behavior, but the price (including taxes) for that behavior. Other charts show that the peak of drinking in the 20th century was actually in 1980. And the chart above starts in 1935, two years after the Prohibition ended (this banned the production, importation & sale of alcohol for thirteen years in the US): The most interesting part of the history – the sales before the prohibition and the huge gap – is left out.

Data transformation

If you download alcohol expenditure data from the US Department of Agriculture for any year before 1997, the prices in the data are nominal. And every time you see the word “nominal” in a data description, your internal alarm system should start urgently blinking red. Nominal dollars are dollars as you’ve paid (or received) them back then. If you paid 1 dollar for chocolate in 2000, the data will list this as “$1”.

“Well, what’s the problem?” you might ask. The problem is that prices increase, and by far more than you’d probably expect. That same chocolate from 2000 costs 1.50 USD today. And for 1.50, you could have bought four of the same chocolate bars in 1970.

So if we plot our nominal alcohol sales, the increase in price is so huge that it hides subtle patterns. The following heat map shows that. The higher the sales, the darker the red:

But luckily, there’s a great little index called the “Consumer Price Index”. How does it work? Imagine some people go shopping every month. Every month they buy exactly the same things: The same shoes, the same house, the same chocolate. Then they check how much the price of the same goods changed compared to last month. And there it is, there’s your Consumer Price Index (CPI).

The US Bureau of Labor Statistics created a tool called CPI Inflation Calculator that gives you back the adjusted price. If we use this tool on our alcohol sales prices[1]here’s the spreadsheet – and plot a heat map again, we start to see patterns:

We can see a peak in the 1940s – but since around the 70s, Americans have spent more money on alcohol than in the 40s. That’s true. But did we just prove that Americans are drinking more and more?

Here’s another truth: There were only 140 million US-Americans in the 40s, but more than 200 million in 1970 (and almost 320 million in 2014). Like the nominal prices, the population increased. And like with the nominal prices, we need to get rid of the population increase in the data to see the actual pattern:

Finally we have data that we can work with! It’s the same one that you can find in the chart above. This week, we were not able to see patterns in the “raw” data since they were “polluted” by two other phenomena. Our data didn’t just tell us something about alcohol sales, but also about price changes and the increase of the US population. Only after calculating them out, our data became useful.


In case you missed it, I announced a new Data Vis Book Club edition this week: We will read Cole Nussbaumer Knaflic’s “Storytelling with Data” and discuss it on the 30th of October. You can find more information in this blog post. It would be great to have you on board. I’ll see you next week!


  1. To do so, we can use the IMPORTXML formula to get the results directly from the CPI Inflation Calculator into our Google Sheets. This is how it works: Open a Google Sheet. Put the year in cell A1, e.g. 1935; and the price from back then in A2. Then you can use the following formula in cell A3: =IMPORTXML("https://data.bls.gov/cgi-bin/cpicalc.pl?cost1="&A2&"&year1="&$A1&"01&year2=201808","//span[@id='answer']"). This formula opens the website of the CPI Inflation Calculator with our cost from A2 and the year from A1 (“193501”, for January 1935) already filled in. Then we ask the calculator to give us back the adjusted price in August 2018 (201808). The Google Sheets will get this adjusted price from a div with the id answer. To see the formula in action, head over to the Google Sheets I’ve used to calculate the adjusted prices for the chart above. And to learn how to target specific elements on a website, visit this explanation of XPath.

30 Sep 05:21

A New and Deliciously Dark Theme for macOS

Postbox Dark Mode

When running macOS Mojave, Postbox automatically adapts to your Light and Dark Mode preferences, located at macOS > System Preferences > General.

You easily switch between themes (independent of your macOS mode) by adding our new "Theme" button to the Postbox toolbar. Right-click the toolbar and select "Customize..." from the pop-up menu. Then drag and drop the "Theme" icon to the desired location in your toolbar.

pb6-theme-switch@2x.png?mtime=20180924104636#asset:2241:url

The toolbar button will allow you to switch modes with a single click. On macOS Mojave and later, a press-and-hold action will allow you to select from Light, Dark, or Default to follow the macOS setting.

We've also brought Dark Mode to previous versions of macOS, starting with El Capitan 10.11 and later.

Don't think that we've forgotten about our Windows users! A Dark Mode theme will be arriving for Windows in late 2018.

30 Sep 05:21

Avoiding burnout: lessons learned from a 19th century philosopher

You’re hard at work writing code: you need to ship a feature on time, or release a whole new product, and you’re pouring all your time and energy into it, your heart and your soul. And then, an uninvited and dangerous question insinuates itself into your consciousness.

If you succeed, if you ship your code, if you release your product, will you be happy? Will all your time and effort be worth it?

And you realize the answer is “no”. And suddenly your work is worthless, your goals are meaningless. You just can’t force yourself to work on something that doesn’t matter.

Why bother? Why work at all?

This is not a new experience. Almost 200 years ago, John Stuart Mill went through this crisis. And being a highly verbose 19th century philosopher, he also wrote a highly detailed explanation how he managed to overcome what we would call depression or burnout.

And this explanation is useful not just to his 19th century peers, but to us programmers as well.

“Intellectual enjoyments above all”

At the core of Mill’s argument is the idea that rational thought, “analysis” he calls it, is corrosive: “a perpetual worm at the root both of the passions and of the virtues”. He never rejected rational thought, but he concluded that on its own it was insufficient, and potentially dangerous.

Mill’s education had, from an early age, focused him solely on rational analysis. As a young child Mill was taught by his father to understand—not just memorize—Greek, arithmetic, history, mathematics, political economy, far more than even many well-educated adults learned at the time. And since he was taught at home without outside influences, he internalized his father’s ideas prizing intellect over emotions.

In particular, Mill’s father “never varied in rating intellectual enjoyments above all others… For passionate emotions of all sorts, and for everything which has been said or written in exaltation of them, he professed the greatest contempt.” Thus Mill learned to prize rational thought and analysis over other feelings, as many programmers do—until he discovered the cost of focusing on those alone.

“The dissolving influence of analysis”

One day, things went wrong:

I was in a dull state of nerves, such as everybody is occasionally liable to; unsusceptible to enjoyment or pleasurable excitement; one of those moods when what is pleasure at other times, becomes insipid or indifferent…

In this frame of mind it occurred to me to put the question directly to myself: “Suppose that all your objects in life were realized; that all the changes in institutions and opinions which you are looking forward to, could be completely effected at this very instant: would this be a great joy and happiness to you?” And an irrepressible self-consciousness distinctly answered, “No!”

From this point on Mill suffered from depression, for months on end. And being of an analytic frame of mind, he was able to intellectually diagnose his problem.

On the one hand, rational logical thought is immensely useful in understanding the world: “it enables us mentally to separate ideas which have only casually clung together”. But this ability to analyze also has its costs, since “the habit of analysis has a tendency to wear away the feelings”. In particular, analysis “fearfully undermine all desires, and all pleasures”.

Why should this make you happy? You try to analyze it logically, and eventually conclude there is no reason it should—and now you’re no longer happy.

“Find happiness by the way”

Eventually an emotional, touching scene in a book he was reading nudged Mill out of his misery, and when he fully recovered he changed his approach to life in order to prevent a recurrence.

Mill’s first conclusion was that happiness is a side-effect, not a goal you can achieve directly, nor verify directly by rational self-interrogation. Whenever you ask yourself “can I prove that I’m happy?” the self-consciousness involved will make the answer be “no”. Instead of choosing happiness as your goal, you need to focus on some other thing you care about:

Those only are happy (I thought) who have their minds fixed on some object other than their own happiness; on the happiness of others, on the improvement of mankind, even on some art or pursuit, followed not as a means, but as itself an ideal end. Aiming thus at something else, they find happiness by the way.

It’s worth noticing that Mill is suggesting focusing on something you actually care about. If you’re spending your time working on something that meaningless to you, you will probably have a harder time of it.

“The internal culture of the individual”

Mill’s second conclusion was that logical thought and analysis are not enough on their own. He still believed in the value of “intellectual culture”, but he also aimed to become a more balanced person by “the cultivation of the feelings”. And in particular, he learned the value of “poetry and art as instruments of human culture”.

For example, Mill discovered Wordsworth’s poetry:

These poems addressed themselves powerfully to one of the strongest of my pleasurable susceptibilities, the love of rural objects and natural scenery; to which I had been indebted not only for much of the pleasure of my life, but quite recently for relief from one of my longest relapses into depression….

What made Wordsworth’s poems a medicine for my state of mind, was that they expressed, not mere outward beauty, but states of feeling, and of thought coloured by feeling, under the excitement of beauty. They seemed to be the very culture of the feelings, which I was in quest of. In them I seemed to draw from a Source of inward joy, of sympathetic and imaginative pleasure, which could be shared in by all human beings…

Both nature and art cultivate the feelings, an additional and distinct way of being human beyond logical analysis:

The intensest feeling of the beauty of a cloud lighted by the setting sun, is no hindrance to my knowing that the cloud is vapour of water, subject to all the laws of vapours in a state of suspension…

The practice of happiness

Mill’s advice is not a universal panacea; among other flaws, it starts from a position of immense privilege. But I do think Mill hits on some important points about what it means to be human.

If you wish to put it into practice, here is Mill’s advice, insofar as I can summarize it (I encourage you to go and read his Autobiography on your own):

  1. Aim in your work not for happiness, but for a goal you care about: improving the world, or even just applying and honing a skill you value.
  2. Your work—and the rational thought it entails—will not suffice to make you happy; rational thought on its own will undermine your feelings.
  3. You should therefore also cultivate your feelings: through nature, and through art.


There’s always too much work to do: too many features to implement, too many bugs to fix—and working evenings and weekends won’t help.

The real solution is working fewer hours. Learn how you can get a 3-day weekend.

30 Sep 05:21

Designing E-Learning 3.0 in gRSShopper - 2

by Stephen Downes
[Part 1 - Part 2 - Part 3 - Part 4 - Part 5 - Part 6 - Part 7]

In the previous segment I decided to create a course with modules and sections, and set up the modules. I'll fill out the actual content of those modules over time. But now I need to stance face-to-face with something I've been dreading...

Events

I had a colleague tell me a few years ago that "calendars are solved" and that it was therefore of no interest to work on them. Then why won't Outlook and Google Calendar talk with each other? I asked. Suffice to say they're not completely solved.

But it's also true that nobody wants to use the course calendar. People have their own calendars, and the course calendar has to work with those. So there's a requirement to make it easy both to import and export event data, and while it has been solved by a lot of people in a lot of places, it's still a pain to work with.

In gRSShopper as I write now there is a predefined 'event' table, but the form isn't defined as well as I would like, and reflects my previous efforts to work in a useful way with events. So let's solve this thing. Here's the 'event' table as it looks now in the Form Editor:


Figure 10 - Events table.
I'm pretty sure I won't need all of those fields, but let me set up a basic input form and then we'll see. Here's the form in the Form Editor:

Figure 11 - Form Editor - Event



And my new form:

Figure 12 - Event Editor



I'm not sure I have the 'datetime' type done yet, while I know I have 'date' done - something to check. (Update - I had to update the code for this. This took me far longer than it should have because I kept running into errors. 24 hours later, the errors turn out to be due to failing (sloppily) to ensure my JQuery calls were prefaced with document.ready() statements.)

Also, 'location' is just a text field. If I were thinking about normal form I would actually use a 'place' or 'location' table and define the locations, but I'm not even sure there will be locations, since the events are online. Or, maybe some people will create Meet-Ups that I should add to the events list. I don't know yet. So I won't define it ahead of time.

OK, let's try making an event:


Figure 13 - Completed Event Form, Edit and Resources tabs


Thus far I've made a lot of progress creating and filling forms, but there isn't really anything for people to look at yet. The best way to give people something to look at is to use a page.

So I'll make a page I'll provisionally call 'Course Outline'. The Page Editor is already defined (and even has Help so I won't review how it all works here). So I'll open it up.

The main thing I'm interested in right now is the big box labeled 'Page code'. This is where I define the content of the page to be displayed to the reader. What I want to do is list the course modules and events (there's only one course so I don't need to list courses). I do this using a keyword command (you can see all the page commands here).

Here are the keyword command instructions:

keyword xxx    Keyword command; xxx = commands:
db=ddd select records from ddd database
expires=nnn; do not display records older than nnn seconds
number=nn display no more than nn records
sort=xx sort on field xx
sort=xx,yy sort on field xx first, then yy second
format=fff format using the view ddd_fff
lookup create lookup from another table

So if I want to list some modules, I'll do this:


This will display all the modules, sorting them according to their module number. The format will default to HTML, and I'll want to change that, but I'll test this first. I enter the code into the form (it autosaves, so I never need to click save). Then I go to the preview tab.

(Pause)

OK, that didn't work. A couple of things. First, here is the error I got:

Not Found

The requested URL /page/226 was not found on this server.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
Here's why. The actual preview URL is https://el30.mooc.ca/cgi-bin/page.cgi?page=226 but the preview window wants to show https://el30.mooc.ca/page/226 but it can't find it. What should happen is that a .htaccess file should redirect the URL. But the .htaccess file is probably missing. I'm always forgetting to put it in. :p

(I have to do a Google search on 'WinSCP show hidden files' to remember how to find hidden files like .htaccess -- ah, here it is.) So I edited it and loaded it in the main directory (on the to-do list is to make sure .htaccess is part of the out-of-the-box distribution). Here's what it looks like:

Figure 14 - .htaccess file
It's line 5 that actually rewrites the URL. Trying again and...

Figure 15 - where are my modules?
... nothing.

The problem is that gRSShopper doesn't know how to display modules. After all, I only created them yesterday.

What I need to do is to define a View that will display modules. So I'll select Create a New View in the left-hand window and open the View Editor. It's pretty boring; there's only two fields, 'View Title' and 'View Text'.

The title of the view is based on the table being displayed and the format it's being displayed in, like this: 'title_format'. (I used to have fancy dropdowns that let you select each and then would automatically create a title, but it ended up being a lot easier just to name the title directly). So for my module html view I'll use the title 'module_html'.

In the 'Text' field we define what we want to see. We can put in any command that works in a page, any HTML, any Javascript, indeed, pretty much anything at all. The View is an incredibly powerful tool (and can be used to break a lot of things, so be warned).

We can also insert any field from the record we're displaying using special commands. For example, if I want to insert the module title, I use the command:  [*module_title*]

Notice that I'm using the full name of the field; this is because sometimes content from different tables blends into a single view and I want to keep it all straight. So let me define a pretty simple module view:

Figure 16 - View Editor - module HTML



The will automatically insert the site URL (notice that this means views can be reused on different sites). The ID number of the module and the title will be inserted in the [* *] blocks. The result should be a clickable link to the module.

Figure 17 - Course Outline, first draft
But I would also like to include the events in the outline. I use a keylist command to do this.

Figure 18 - keylist command



I cheated a bit. I knew 'event_list' was already defined, so I just used that, just to test it out. I'll probably make a new view. Even so, we can see the result pretty clearly:

Figure 19 - Course Outline, second draft
None of these landing pages work yet - you'll get a very basic page if you click on any of the links. Also, all this could use some styling and more, and yes, I really should create a proper event view for the course outline. I should put a header and footer template on the page that will contain the styling I need, plus any Javascript libraries. Once I've done that, I'll set the page to 'autopublish' and never worry about it again - if I make changes to the modules or events, they'll be reflected automatically in the course outline.

(You might be thinking - but wait, none of this is rocket science! I could have done this! Well - yes. But my own experience has been that it's never this easy to do any of this stuff, which is why I created gRSShopper (and again, now that I've done it once, I never have to do it again - I can apply these forms and events and modules and views to any course, any website, that I want.)

OK, let's set up the new Event view and while we're at it set up a page that will list all of the events in the course. First the view, which will include the description, event time and place. I'll call it event_outline.

Figure 20 - Events View

Figure 21 - Page Editor - Events Page

Notice the events page defines the sort to use the 'start' field. We won't see this in action in this post but it will be useful when we have more than one event.


And here's what the output looks like.

Figure 22 - Events Page Output


Sure, it lacks styling, and could probably look better. But this is the basic idea. Since it's an online course I'm not concerned about the location (though maybe I should say 'Google Hangout' or 'Zoom' or whatever). The main thing is the link, which should take me directly to the event itself, launching the event viewer application, whatever it may be.

Done? Well, no. What about time zones? After all, people from around the world are taking this course. I could just put 'Eastern Standard Time' or whatever it is in October after the date, but that leaves the reader needing to figure it out for themselves. Not very convenient. What I'd like to be able to do is display it in the reader's local time.

Well, Javascript has a lot of date-time functions, which is what I'll use.

gRSShopper stores datetimes as 'epoch' dates. This is a single very large integer counting the number of seconds since January 1, 1970 UTC.  So if I put [*event_start*] into the page code, gRSShopper will print out that number. We can use that as the input to a Javascript command. Javascript, however, inputs time in the number of milliseconds since January 1, 1970 UTC. So we need to add some zeros to the end. So here's how we get the datetime into Javascript:

                var event = new Date([*event_start*]000);

Once the date is in Javascript, we can print it out as a local date:

               document.write(event.toString());

So here's our newly defined view:

Figure 23 - Event view with Javascript-defined date

And here's the output:

Figure 24 - Event listing with date-time in the local time zone

Ah, but we're not done. How do we get it from here into someone's calendar? Because, as I said, people won't typically use their gRSShopper calendar - they'll use Outlook or Google Calendar or whatever.

What we want is one of those little links that allows people to click on a button and enter the event into their calendar. Those are easy to create, but there are services that will create them. Once such is AddEvent, which is what I'll use here. Following the instructions here I add their code to my event_outline view:

Figure 25 - Event View with AddEvent Button Code
Notice that I use some of the page formatting commands in my view to configure the data mfor AddEvent. In particular:

    date format=iso;time=[*event_start*]

This takes the epoch date specified by [*event_start*] and formats it as an ISO date string. There are other formats available: RFC822 (for RSS feeds), nice (for nice dates), and more. Also notice the use of the timezone command to insert the site's local time zone.

Here's the output:

Figure 26 - Event listing with AddEvent button
OK, it's still pretty ugly and needs some nice formatting. But now that the functionality is there, all I need to do is add events in gRSShopper and the system will take care of the rest.

And even better, now that I've done all this, you don't have to, because all you need to do is clone the course from the gRSShopper database and this will all work in your course on your instance of gRSShopper.

Like I said, I was dreading doing events. This whole thing, beginning to end (and including fixing scripts where needed, and writing this text and taking all the images), took me a good two days. It should be a lot easier for you now that I've run through the steps.

30 Sep 05:21

Social Working Array [Flickr]

by vanderwal

vanderwal posted a photo:

Social Working Array

30 Sep 05:20

NetNewsWire Diary #2: Switching to OPML

Since the earliest days of NetNewsWire, before 1.0 even shipped, I wanted to make the subscriptions list on disk an OPML file.

It seemed like using the standard format for listing RSS subscriptions would be a good idea. But I was never able to make that happen — until now, with NetNewsWire 5.0d7.

Why It’s a Good Idea

Given that OPML import and export is a must-have feature — which NetNewsWire already has — then storing the subscriptions on disk as OPML means less code, because there’s just one format to support, instead of supporting both OPML and a custom format for just the app.

It means that the OPML reading and writing code will run for every user on every session, which means any bugs in OPML support won’t be able to hide.

It means a user can get their subscriptions list and move it to another system without first having to launch the app and do an export — the OPML file is already there on disk already.

It also means a user could replace their subscriptions list on disk by swapping in a different OPML file, without having to figure out how to do it through the app.

Why I Never Did This Before

The problem, though, is that the standard OPML format for listing RSS feeds doesn’t have slots for additional metadata. I want to store things like Conditional GET information, the hash of the contents from the last fetch, the edited name as well as the feed’s declared name, and so on.

So for all these years I thought, “Well, that’s too bad. I wish I could use OPML, but I can’t.”

Why I Could Do It Now

It occurred to me I could use a “side table” — that is, a separate place to store feed metadata. This would allow me to store whatever additional data I wanted to store in a separate location.

The go-to for this would normally be SQLite, with some kind of table for feed metadata.

But this has another problem: SQLite is structured storage. There’s a schema. But this doesn’t quite suit feed metadata — especially not when you add in syncing. I wanted the ability to store attributes for each feed without knowing, in advance, what all those attributes might be.

Enter Rainier, the inspired-by-UserLand-Frontier app I’m working on. It includes an object database — that is, it includes a hierarchical, schema-less, key-value database. It’s like a dictionary that can contain nested dictionaries.

Which is perfect for what I wanted to do: I could make each feed a table in an ODB database, and then each feed can store arbitrary key-value pairs. (Think of a “table” in an ODB database as really a dictionary.)

So I built this ODB storage — perhaps ironically, it’s built on top of SQLite, but why not? SQLite is amazing — and its first use is in NetNewsWire.

Rainier-the-app isn’t far enough along to actually do anything at all yet. But a key component of it is now part of NetNewsWire. I expect this pattern to get repeated for a few more things, too — including the (not-nearly-finished) scripting language.

30 Sep 05:19

SF City Guide

by brandon

Whether on an extended layover, sneaking in a weekend trip, or staying for the long haul, San Francisco has no shortage of places to see and things to do. Sometimes that makes it hard to decide on where you're headed.

Luckily for you, our staff has rounded up all their favorites and created a downloadable .pdf so you can take it with you and access it anywhere.

 

30 Sep 05:17

The Best Bluetooth Headphone Adapter

by R. Matthew Ward
The Best Bluetooth Headphone Adapter

After spending another 21 hours this year listening to the top new Bluetooth headphone adapters, we think the FiiO μBTR is the best way to add Bluetooth to your favorite set of wired headphones—it sounds good, and a convenient clip makes it easier to use on the move than competitors. But unless you really love your old headphones, the best option for most people who want to listen wirelessly to their phone, tablet, or computer is to buy a set of good Bluetooth headphones, which don’t have a cable to get in the way and generally offer easier-to-use controls and a better-placed microphone for improved call quality.

30 Sep 05:17

I got it slightly wrong - NFC mobile payments are not a complete dud

by Dean Bubley
I have long been a critic of NFC-based mobile payments from smartphones. It was originally touted as a mobile operator service, debiting value (or adding to your monthly bill) with a tap. I was deeply skeptical.

Almost 10 years ago, I wrote that for it to work, "I ought to be able to check my balance from the phone screen, look at recent transactions and so on. At present, the UI/app side of NFC appears woefully weak to me". (link)

Six years ago, I was more emphastic still: "the tap-to-pay thing is a nonsense, a solution looking for a problem. The involvement of a telco adds zero value and lots of friction". (link)

I have to admit to being partly wrong. Yes, I know, it's rare for me to issue a mea culpa, but here, I have to hold my hands up!

I do actually see (some) people paying for things with their smartphones, as well as using them for stored tickets / travelcards. I also know that in China, South Korea and some other places, QR codes are popular, but that's a different technology. But for the most part, I was right about MNOs being an obstacle rather than a catalyst.

A few things have changed:
  • Linking NFC chips to native-OS capabilities like Apple and Android Pay, allowing the creation of good apps & UIs.
  • Integration with on-device biometrics, notably fingerprint- and facial-recognition
  • People commonly carrying charger cables or USB power banks, so there's less fear that running out of power = running out of money.
  • Very wide adoption of contactless plastic-card payments. In many UK and other European countries, a huge bulk of low-value retail and services-sector payments have moved to this model. This has allowed the broader concept of tap-to-pay to be normalised with plastic, and then a (currently small) % switch further to phones. It has also catalysed merchants to move to tap-to-pay terminals.
  • Most telcos bowing out of the payments value-chain, replaced by banks (old & new fintech ones), plus Apple, Google & Samsung. This has taken out cost, friction - and put the user back in control with familiar, mostly-trusted payment brands.
But it's still definitely not a "revolution". It's one of those secondary things that some people have adopted, without becoming universal. It's a bit like wearing a FitBit - people don't look at you weirdly any more, but they mostly don't intend getting one themselves. (If you pay with a watch, though, you're definitely still weird). 

NOTE: I realise that "normal" is a very geo-specific culture judgement. Within a country or even city, you may find greater levels of acceptance or scorn depending where you shop or drink coffee / beer. 

NFC phone-payment is now somewhere on the geek spectrum in between using a voice assistant (fairly normal), and wearing an AR/VR headset in public (not). There are some fairly clear demographics about who does/doesn't use NFC, as well.

This morning, for an entirely unscientific experiment, I counted 9 out of 100 people exiting my local tube station in central London using a phone rather than a card. Among card users, I couldn't tell how many used a proprietary TFL Oyster card vs. a normal credit/debit card that is now accepted at Oyster terminals. 

On closer inspection however, some of those people had phone-cases which also held some plastic cards as a physical wallet (like these - link), so given not all had been thumb-ing the screens, the actual number of proper NFC users was probably 5-7%. And that is among commuters who mostly travel every day, at 9am, so we can perhaps expect routines to be optimised. I might try another time to see how daytime "casual" usage differs.

Without having done a similar count at shops and restuarants, but having been generally observant over recent months, I'd guess that about the same percentage applies for retail transactions. Common-ish, but certainly atypical.

(As a sidenote - I'm writing this in a very nice cafe in London, which is having to apologise to customers that its card/NFC reader isn't working at all. Cash is still a critical backup).

Personally, I don't use my phone for retail payments. I use my contactless bank or credit cards, and I have a proper pre-pay Oyster. I prefer to keep my various payment and online relationships completely separate - Apple doesn't even have my credit card details.

I'm curious what the situation is elsewhere, and I'll keep an eye out when I'm travelling. My expectation is that phones will become somewhat more common for payments over time, but there's not going to be a sudden flip, as there was with contactless cards. I think geographic differences will persist too - I'd be surprised if Germany was a fast adopter of phone-based NFC, while China could well move much mor decisively. 

There might even be one or two places with telcos in the NFC-payments loop, as they are in some developing markets for messaging-based payments - but I don't see that in markets with existing high % of banking and card adoption.

Bottom line: I was definitely a bit too negative 10 years ago about the long-term future of NFC. But the transition remains slow, patchy, and dependent on the UX skills of the smartphone manufacturers. Also, few saw the behavioural acceptance of consumers being driven by tap-to-pay plastic cards as a first (and for many, last) step.

30 Sep 05:17

Online Is Offlife

by Ton Zijlstra

I very much disagree, as deep and rich parts of our social lives only exist because our myriad of online interactions and manifest themselves most definitely in our offline lives, are indistinguishable from it.

But I get the sentiment, as endlessly scrolling in your FB or other timelines is indeed taking you out of your life. That addictive and numbing timeline is a dark pattern worth breaking.

Online = Offlife

Found on a wall in Novi Sad, Serbia.

30 Sep 05:14

A City-Wide Plan: Gil Kelley’s amazing announcement that almost went unnoticed

by Gordon Price

The Urbanarium event on September 20 – Planning West Coast Cities – may be remembered more for the complexion of its panel than the significance of the announcement made by Vancouver Planning Director Gil Kelley, subtly integrated into his presentation.  Too subtly, apparently.  While it didn’t get the attention it deserved, it may well mark the beginning of a new era in city planning.

Vancouver, in his opinion, should have a city-wide plan, and the work should start soon after the election of a new council.  It could take two to three years to achieve, but given it’s ambition, so is the timeframe.

It’s been argued for years whether there’s actually a need for such a planning document to be used as the basis for shaping the city, rather than relying on the Vancouver equivalent, the Zoning and Development Bylaw – a hodepodge of incremental change, unfathomable to most citizens.  Some say the last city plan of any clarity was devised by Harland Bartholomew, though never passed by council, in 1928.  Other would contend that it was the 1995 ‘CityPlan,’  – “a framework for decisions on City funding, programs, and actions” that could have been more effective if it had been translated into actual zoning at the neighbourhood level.

Kelley is proposing a version of both: a visioning exercise, bringing together already established policy combined with the values and aspirations of Vancouverites today, followed by a physical plan which would translate chosen directions into actions and zoning.

I doubt he would have used a major public event to announce that intention if it had not been vetted, minimally, by the city manager and mayor.  I also assume it reflects the belief of the planning staff generally and his reading of community sentiment.  (Gil is welcome to clarify.)

The timing is exquisite.  A new council, made up largely of newcomers to City Hall, will be looking to translate their promises and commitments to consultation into action, while at the same time learning about processes and resources available – in other words, the inevitable reality check.  What better way to take action, without having to spend a lot of political capital making precipitous real-world decisions in the short term, than committing to a long-term city-wide planning process?  (As far as responding to the housing crisis, much of the heavy lifting has already been done, especially with regard to senior-government commitments to new programs and billions in expenditures.  The results will soon be apparent.)

All the new council has to do is commit itself to the strategy that Kelley can quickly put in front of them (and vote for the dollars and staff to do it).

At this point I have to acknowledge that, as Patrick Condon can vouch, I have argued against the need for a city-wide plan.  Indeed, I and Geoff Meggs won an Urbanarium debate on this very subject against Patrick and Adrianne Carr back in April of 2016.

 

 

And I would still oppose any simultaneous revision of the Zoning and Development Bylaw for every neighbourhood in the city, hinged to a strategy of densification that would change scale and character throughout our 44 square miles.  Mainly, I don’t believe it is needed nor could actually happen, given the political capital required to overcome a unified opposition that could grind down any process which tried to move too quickly.

Kelley’s more incremental strategy has a better chance of success: first integrating existing plans and policies with a visionary exercise.  Then making the trade-offs needed, with council approval for further directions to take.  Assuming we get that far (not a slam dunk), we can then assess how expectations can be translated into changes on the ground.  I’ll bet that, rather than a city-wide rezoning, we’ll do it over time, neighbourhood by neighbourhood, once priorities have been set where needs are greatest and resources are available.

That’s the way we’ve been doing it since the seventies. Kelley’s version could provide a framework more appropriate for our times and the scale of our challenges. And it just might work.

 

30 Sep 05:10

Bessere Lösung gesucht

by Volker Weber

ZZ3715C8DB

Ich habe gestern ein Paket erwartet, musste aber in der Mittagszeit weg. Es gibt ein Gesetz, nachdem Kuriere nur dann kommen, wenn man mal eben weg ist, aber ich weiß nicht, wie es heißt. Also habe ich einen bunten Zettel mit der minimalen Anweisung 'Paket -> 5' beschrieben und über meine Klingel gehängt. Ich habe mittlerweile gelernt, keine langen Sätze zu schreiben, weil die Kuriere das oft nicht verstehen.

Nachdem ich weg war, kam der UPS-Laster. Die Arlo hat alles aufgezeichnet. Ein Mitarbeiter ging zur Tür und drückte auf die Klingel. Als keiner aufmachte, schaute er sich hilflos um und ging zum Laster. Ein zweiter Mitarbeiter stieg aus, guckte auf meinen Zettel und verstand die Anweisung. Dann schauten sie beide nach. 8, 10, 12. In der anderen Richtung 8, 6, 4. Dann schrieb einer eine Benachrichtigung und war sie in den Kasten. Das Paket nahmen sie wieder mit.

Der Vorteil der Arlo ist, dass ich wenige Minuten später sah, was die beiden Helden gemacht haben. Auf dem Rückweg habe ich den Laster deshalb an anderer Stelle abgepasst. Ja, das Paket hatten sie noch. Ich habe sie gefragt, warum sie es nicht bei der 5 abgegeben haben. Antwort: "Da ist keine 5". Auch wenn sie nur wenig deutsch verstanden, konnte ich ihnen erklären, dass in Deutschland die ungeraden Zahlen einfach auf der anderen Straßenseite zu finden sind.

"BESUHGER-32" soll übrigens "Bessunger Str. 32" heißen. Das kann ich verstehen, denn das ist der nächste UPS-Stützpunkt. Laut handschriftlicher Anweisung sollte das Paket "heute ab 10" da sein. Das wurde so ca. 13 Uhr geschrieben und um 22 Uhr hat der Stützpunkt definitiv zu. Ich hätte es halt spät am Nachmittag probiert. Oder den herumirrenden Laster irgendwo anders gefunden.

Letzte Woche hat der Postbriefträger einen Umschlag mit einem Produkt gebracht. Da er es nicht in den Briefkasten stecken konnte, hat er es einfach wieder mitgenommen und statt dessen einen Zettel eingeworfen. Ich sollte den Umschlag am anderen Ende der Stadt in "meiner Filiale" abholen, am nächsten Tag ab 11 Uhr. Dort war es aber nicht aufzufinden. Man zuckte mit den Schultern, meinte, ich solle morgen noch mal kommen, schrieb sich aber meine Handynummer auf. Am nächsten Morgen dann der Anruf. Der Briefträger sei krank, aber der Brief sei bestimmt morgen da. Ein weiterer Tag später dann der Anruf: "Wir haben ihren Brief". Nachmittags bin ich dann wieder zu "meiner" Filiale. Dort wollte ich mich ausweisen, um den Brief mitzunehmen. Nein, das sei nicht notwendig.

Ich habe die beiden Kuriere darüber aufgeklärt, wie man die Nummer 5 findet und dem freundlichen Briefträger erklärt, er möge in Zukunft Briefe einfach vor die (überdachte) Tür legen, sollte mal einer zu dick sein. Zumindest solle er mal versuchen, zu klingeln. Ich war nämlich da.

Und nächste Woche haben wir dann wieder andere Zusteller.

30 Sep 03:50

Special counsel Robert Mueller is now investigating his MacBook

by Jacob Kastrenakes

Robert Mueller surfaced from the Russia investigation this afternoon to ask for help — tech support, specifically. The special counsel and former FBI director was spotted in the Georgetown Apple store today, working with a Genius bar employee to get help with a MacBook Pro. The photo was taken by Meghan Pianta, who works in communications in DC; we initially caught it via Washington Examiner reporter Kelly Cohen.

In the photo, you can see a pained, confused look on the special counsel’s face. What is he looking at? We tried calling the Apple store to find out, but were stymied by endless phone trees. An investigator who went on location on behalf of The Verge said there was no sighting of either Mueller or the employee seen helping him.

...

Continue reading…

28 Sep 22:54

Google will reportedly name its 2-in-1 Chromebook, Pixel Slate

by Dean Daley
Google HQ

Google will reportedly name its upcoming, rumoured 2-in-1 Chromebook, the Pixel Slate, according to a tweet from Android Police’s David Ruddock. 

The upcoming detachable 2-in-1 is rumoured to feature two USB-C ports, a 3,000 x 2,000 pixel screen resolution, and no SD card slot. Additionally 9to5Google, points to there being two Pixel Slate devices, one of which will sport a powerful Kaby Lake processor.

Other rumours indicate the Pixel Slate will feature a built-in fingerprint scanner.

Google will reportedly unveil the Pixel Slate alongside another Pixelbook and two smartphones on October 9th at keynote in New York City. 

Source: David Ruddock, 9to5Google

The post Google will reportedly name its 2-in-1 Chromebook, Pixel Slate appeared first on MobileSyrup.

28 Sep 22:54

Oculus says it hasn’t forgotten about Samsung’s Gear VR

by Patrick O'Rourke
Gear VR

With Facebook-owned Oculus shifting its focus to the mid-range Oculus Quest, while still supporting the high-end Oculus Rift and the entry-level Oculus Go, it might make sense for the company to drop Samsung’s Gear VR from its lineup of VR headsets.

That doesn’t seem to be the case though, according to a recent The Verge interview with Nate Mitchell, Oculus’ vice president of product.

Mitchell told the publication that Oculus still views the Gear VR as a priority, stating that, “A lot of the work that we’re doing on Oculus Go carries right over to Gear VR — they’re binary compatible. So a lot of the improvements we’ve been making to the software side, a lot of the products we’ve been investing in, are coming to Gear VR as well.”

Still, the last major hardware update to the Gear VR — which shipped five million headsets as of January 2017 — was a new controller that released over a year ago. News related to the Gear VR has been scarce at Oculus Connect, too. Even Samsung neglected to mention the headset at its recent keynote for the Note 9.

While phone-powered VR is an affordable way to jump into the virtual reality space if you already own the phone that powers the headset in question, it doesn’t always result in the greatest experience.

Google also seems to be letting its Daydream project fall to the wayside, though the company could have something in store for its upcoming fall hardware event.

Source: The Verge 

The post Oculus says it hasn’t forgotten about Samsung’s Gear VR appeared first on MobileSyrup.

27 Sep 23:48

Photokina Observations

Photokina is a great place to observe, even though I often have to do that from afar. The press conferences, the press releases, the order of announcements, who's involved, small details that some gloss over, they all tell a story beyond just "Company X announced Y."

27 Sep 23:48

Acorn 6.2 With Mojave Dark Mode Is Out

On Monday I flipped some switches on the FM servers and Acorn 6.2 was released to the universe. You might also remember that Monday a little known operating system from Apple was updated, which includes a neat new feature known as Dark Mode.

I think Acorn looks pretty good in Dark Aqua, especially the icon refresh from Matthew Skiles.

To celebrate the new release, we've put Acorn on sale for 50% off. So go grab it at the insanely low price of $14.99. If you haven't already upgraded from previous versions of Acorn, now is a good time to do so.

We've also packed a bunch of little changes, bug fixes, and compatibility with Mojave in there. And of course, there's more to come in the future as always.