Two years ago I purchased a 5 oz. tube of yellow letterpress ink from NA Graphics. With the exception of a tiny experiment last year, I’d been afraid to use the ink; black and red are easy and substantial, whereas yellow is fickle and mysterious, and I feared that the result would be too light or too dark or just off.
Today, though, I decided to break through this mental logjam.
And to print.
In yellow.
I’m cooking up a skunkworks project for December that I need a poster for (details to follow soon!) and so I pulled out the Akzidenz Grotesk and set to work.
I had nothing to fear: the yellow is throaty and bold and deeply sunshiny. I love it.
Nobody likes spam. We’ve heard it loud and clear from our Flickr community—and we feel exactly the same way. It’s an unfortunate reality in modern times online, and it can be particularly challenging when you have a free website like Flickr. Still, we’ve been fighting spam for some time now. And thanks to our recent acquisition by SmugMug, we have new resources that are helping us take stronger action and make significant progress on this front.
Turning things around.
Right after we joined SmugMug, our new CEO introduced us to a helpful new partner called Sift—a company that’s uniquely equipped to track spam-like behavior. We’ve been working together for months now, complementing our team’s personal reviews of suspicious activity with Sift’s 24/7 machine learning to identify and stop activities that aren’t appropriate for our community. Our system is getting smarter over time and allowing us to not only take action against aggressive spammers, but also to clean up our search results by deleting accounts that violate our community guidelines.
We’ve thwarted several major spam attacks in the past few weeks alone, but we know that spammers are relentless, so we’ll continue working at this as their techniques evolve. Spam is a problem that may never be completely solved, but we’re confident you’ll be seeing far less spam on Flickr in the coming months.
More ways we protect our members.
Spam prevention is just one of the ways we keep you safe online. We’ve also partnered with cybersecurity company HackerOne to allow security researchers to constantly test the strength of Flickr’s defense systems. When they do discover a weakness, they tell us right away so we can address the issue. We then pay generous bounties to these researchers for helping us keep Flickr shipshape.
Keeping our focus on photography.
Flickr has always been a vibrant, photographer-focused community. And we want it to always be a safe place for our members to share and appreciate images they love. Flickr is not a place for people who have nothing to do with photography to sell things or otherwise influence our members. Our goal with all of the efforts described above is to create the best experience possible for you on Flickr.
Your help can make all the difference.
Our supportive community has been great at reporting spam and other abuses of the Flickr site. We know that our new partnerships and tools will make a big difference on this front as well. At the same time, if you happen to see members or content that feel wrong for the Flickr community, click on the Report Abuse link at the bottom of any page to report it immediately. Our Trust and Safety department will investigate and take action right away.
As always, thanks for all you do to make Flickr great. Ideas or suggestions? Share them here.
During my time as a developer, I have come across wildly differing opinions on testing. Many would say that you absolutely, without a doubt, should write tests around the code you write. Others felt that testing was a waste of the developer’s time and energy.
When I first began programming, I did not like writing tests. It was my least favorite part of programming. I mean, come on, my code is working, now I am supposed to go back and write tests too? No, thanks! Little did my naive self know that tests would soon make my life as a developer a whole lot easier.
When I learned to write tests before writing code (Test-Driven Development, or TDD) , I soon realized that by writing my tests first I was actually writing less code.
Does this mean I was previously writing unnecessary code? Was I making my code more complex than it needed to be? Yes. Yes, I was.
Test driven development to me is not just about testing and reliability, but about design. Writing a test first allows me to organize my thoughts, and focus on the things I need, and what I am trying to accomplish. Writing tests first naturally leads to less coupling in my code.
I have experienced benefits of having and writing tests around code, but still, many of my current coworkers had a less than warm fuzzy feeling when it came to testing. It was only a matter of time.
Recently, a new team was formed which included myself and two other developers. We had a new task. We were to add new functionality to two different systems. No members of our team had experience with the first system. So how do we get started? We need to implement the new functionality but not, I repeat, do not break the existing functionality.
How do we start implementing code safely if we don’t know what is already existing? I’ll tell you how: we studied the tests. In about 15 minutes we went from knowing just about nothing about the system to knowing the type of services it used and what it was supposed to do. This was enough knowledge to get us going and feel safe enough to change things!
All we had to do was read the test names. How easy was that? So simple, yet, so valuable! On to the second system. Now lucky for us, one of our team members had worked on the code for this system a year and some change ago. But wait, it had been so long that he really couldn’t remember how it really worked. That’s okay, we will just run the tests. Oh no! There are no tests. We could not possibly start working as we had no idea how things worked. We also were lacking the safety to experiment because how would we know if we broke something? Long story short, it took us a day and a half to read through the code, try to understand what was going on and where we needed to start. Something that had previously taken us 15 minutes, turned into over 10 hours. Spending all that time just trying to decipher code really took a lot out of us. By the time we could finally start coding we were so low energy, we weren’t really working at the best of our ability. We felt dragged down. Now where is the wasted energy?
I hope my story will inspire others to give test-driven development a fair chance. It is so helpful both in the process of developing, and also when you are dealing with existing code. We felt so much appreciation to those developers who came before us. They took the time to write tests. Those developers saved us time, effort, and energy that we were able to use in implementing new functionality. By the way, we ended up writing tests around the untested code we modified. We build our own safety net to allow us the space to experiment. This was the best way for us to feel confident and safe about making changes. Now, hopefully, those developers who will come next can benefit from those tests. Who knows, maybe the next developers will be us.
Which means the entire community is hosted on the same website but hidden behind a tab. If you click on the tab, the community appears. But the rest of the navigation options remain the same.
This means the community has to maintain the look and the feel of the rest of the site, but 9 out of 10 navigation options take people away from the community. To navigate around, members must hover over the tab and select where they want to go.
The upside of this is it likely helps with search engine rankings. The downside is it makes it really difficult for members to discover everything the community offers. That navigation bar really does matter.
The Alteryx community is a bit different. The community is behind a tab, but once you’re there the navigation reflects the best places to go within the community. The navigation bar appears on the left-hand side on every page. Members can easily browse around and find what they want.
The branding is the same, the community is on the same URL, but the navigation is a lot better. Members can click the Alteryx logo at the top to return to the website.
The Dropbox community and Airbnb community are similar but with one noticeable difference; they’re hosted on their own unique URLs. This enables both (in theory) to create slightly more distinct experiences, but at a cost of search traffic they would be likely to receive if on a subdomain of the main URL. In practice, however, they usually have to adhere to brand guidelines. Thus they get much of the same downsides but without the additional upside.
Unique URLs are often symptomatic of a community with limited internal support. The Dropbox community, for example, is the very last option from the navigation bar at the bottom of the homepage.
The Airbnb host community is almost impossible to find from the community homepage (I’m genuinely not sure how people stumble across it).
As a rule, you generally want the community to be as a subdomain prominently linked to from the main homepage. This subdomain should allow you to create a navigation structure that shows off the best of the community with an easy link back to the main homepage.
Last June, SFMOMA launched Send Me SFMOMA, a service where individuals could text a variety of requests – “send me love”, “send me hope”, “send me smiles” – and SFMOMA would respond with an artwork that best matched the request. They received over 5 million texts from hundreds of thousands of individuals over the course of a year.
And they’ve asked me to do something fun with that data.
Each tree represents a day, and each leaf or flower represents something that the service sent back.
Today, we’re announcing updates to our Free and Pro accounts that mark a new step forward for Flickr. To be candid, we’re driving toward the future of Flickr with one eye on the rearview mirror; we’re certain that Flickr’s brightest days lay ahead, but we remain acutely aware that past missteps have alienated some members of our community. We also recognize that many of the clues for how best to build the future of Flickr can be found in our own, rich history.
Many of today’s announcements are unequivocally positive things: a new, simplified login with any email you prefer; improvements to the Pro account; and additional partner perks. The changes to our Free accounts are significant, and I’d like to explain why these changes are necessary and why we’re confident they’re the right path forward for Flickr.
Beginning January 8, 2019, Free accounts will be limited to 1,000 photos and videos. If you need unlimited storage, you’ll need to upgrade to Flickr Pro.
In 2013, Yahoo lost sight of what makes Flickr truly special and responded to a changing landscape in online photo sharing by giving every Flickr user a staggering terabyte of free storage. This, and numerous related changes to the Flickr product during that time, had strongly negative consequences.
First, and most crucially, the free terabyte largely attracted members who were drawn by the free storage, not by engagement with other lovers of photography. This caused a significant tonal shift in our platform, away from the community interaction and exploration of shared interests that makes Flickr the best shared home for photographers in the world. We know those of you who value a vibrant community didn’t like this shift, and with this change we’re re-committing Flickr to focus on fostering this interaction.
Second, you can tell a lot about a product by how it makes money. Giving away vast amounts of storage creates data that can be sold to advertisers, with the inevitable result being that advertisers’ interests are prioritized over yours. Reducing the free storage offering ensures that we run Flickr on subscriptions, which guarantees that our focus is always on how to make your experience better. SmugMug, the photography company that recently acquired Flickr from Yahoo, has long had a saying that resonates deeply with the Flickr team and the way we believe we can best serve your needs: “You are not our product. You are our priority.” We want to build features and experiences that delight you, not our advertisers; ensuring that our members are also our customers makes this possible.
Third, making storage free had the unfortunate effect of signaling to an entire generation of Flickr members that storage—and even Flickr itself—isn’t worth paying for. Nothing could be further from the truth: there is no place like Flickr to share, to discover, to learn, and to interact around photography. And because storing tens of billions of Flickr members’ photos is staggeringly expensive, we need our most-active members to help us continue investing in Flickr’s stability, growth, and innovation.
How did we choose the 1,000 photo/video limit?
We started from the point of view that free members are essential to a vibrant, healthy Flickr, and we were determined to provide a free tier that allows anyone who is unable or unwilling to pay for a Pro account to meaningfully participate in, and contribute to, the Flickr community.
While most products today sell storage in megabytes or gigabytes, the photographers we spoke to all knew about how many photos they had shot in the past few days, but few knew how much storage those photos consume without doing tricky math. Counting photos is simpler and more intuitive. It’s also more closely aligned to Flickr’s past (before 2013, Free members were limited to 200 photos), and we liked the idea of returning to our roots but with free space for five times as many photos as before.
We also think that photographers should be able to upload full, uncompressed original images to Flickr without worrying about conserving space or worrying about a future where images continue to grow in size.
Lastly, we looked at our members and found a clear line between Free and Pro accounts: the overwhelming majority of Pros have more than 1,000 photos on Flickr, and more than 97% of Free members have fewer than 1,000. We believe we’ve landed on a fair and generous place to draw the line.
We know change can be overwhelming, no doubt. But we are committed to making sure Flickr’s focus stays on you, the photographer. We believe this is the right path forward to build a sustainable future for Flickr and our community so we can continue developing features and products that shape the world of photography for years to come.
Hi! My name is Don MacAskill, and I’m the Co-Founder, CEO, and Chief Geek at SmugMug. We’re the company that recently acquired Flickr from Yahoo. We love photography.
A Sharper Focus
Unlike most photo sharing services, SmugMug is photographer-focused and has been for more than 16 years. We are privately owned and operated. We never raised venture capital to grow our business and we don’t make money selling our customers or their data to advertisers. That’s kind of amazing.
Instead, we’ve focused on getting to know our customers, who are photographers around the world. We asked them what they wanted, listened carefully, built those things, then asked again. And we kept asking. That’s the big “secret” to our success. I’m excited to bring the same laser customer focus to Flickr.
At SmugMug, we also charged a fair price when others were pretending “free” was actually free. We work for you, not investors or advertisers. We don’t mine you or your photos for data to re-sell or advertise to you. Your data, and your photos, are yours. You’ve entrusted them to us to keep safe. We take that responsibility very seriously and so does Flickr.
The days of lurching from strategy to strategy at Flickr, chasing hot social media trends, are over. Photography and photographers are our strategy. You are our strategy. Period.
We Love Flickr
We bought Flickr because it’s the largest photographer-focused community in the world. I’ve been a fan for 14 years. There’s nothing else like it. It’s the best place to explore, discover, and connect with amazing photographers and their beautiful photography. Flickr is a priceless Internet treasure for everyone and we’re so excited to be investing in its future. Together, hand-in-hand with the the most amazing community on the planet, we can shape the future of photography.
“If you build it, they will come”
Over the summer, we hit the ground running, learning from a lot of Flickr members. We listened carefully, then got building. That’s what we do. In July, when asked by a member about Flickr’s roadmap, I tweeted:
Still doing a lot of listening to customers and learning from the @Flickr team, but briefly, the top priority things (subject to change) are: 1. Move off of Yahoo Login. 2. De-spam comments, follows, & etc. 3. Improve & enhance Flickr Pro. 4. Stabilize & speed up the service. https://t.co/oz9zYvCaJZ
We’re not done yet, far from it, but I’m excited to share some progress updates on all of those today.
New Simple Login
Easily the #1 most-requested feature has been an improved login system that removes the requirement to have a Yahoo email address. We heard you loud and clear, and we’ve been hard at work building it ever since. You’ll be able to bring your own email and password from anywhere you’d like. You won’t need to worry about moving your photos or losing your Flickr screen name – just login, give us your preferred new credentials, and we’ll handle the rest. If you’d like to keep your Yahoo login, you can even do that as well.
The great news is that internal testing is going well already. We hope to have the community help us finalize testing in December, and for everyone on Flickr to be loving it in January.
De-spamming Flickr
We also heard a lot about spam problems on Flickr. Spammy comments and follows were getting in the way of engaging with other photographers. That’s just terrible and we’re hard at work getting rid of it.
I’m here today to let you know that we’ve already made big improvements at Flickr and have put a serious dent in all sorts of spam. If you’ve used Flickr in the last month, you’ve probably already noticed the dramatic drop-off in bad follows and comments filled with spam.
We’re not done yet, we still have lots of work to do, but I hope you love the improvements we’ve already made, and I can’t wait for you to see what’s next.
Enhancing Flickr Pro
We heard, loud and clear, that Flickr Pro needed some more love. You gave us a big, long list: more storage, great customer support, better stats, additional partner discounts, and more.
I’m happy to announce that, as of today, Flickr Pro is better than ever. It’s easily the best deal in the world for photographers. Not only that, but we’re publishing our exciting roadmap for everything that’s coming in the next few months which will continue to make Flickr Pro an incredible experience.
Flickr Free members are also essential to a vibrant, healthy Flickr. So we are determined to provide a free tier that allows anyone who is unable or unwilling to pay for Flickr Pro to meaningfully participate in, and contribute to, the Flickr community. Free accounts on Flickr are now for sharing and displaying your 1,000 best photos and videos. When you fall in love with photography and Flickr, unlimited photos and so much more can be found with a Flickr Pro upgrade.
Faster and Stronger
Moving Flickr to a modern new platform is no small feat. With more than 100M accounts, tens of billions of photos, and many billions of page views, it’s a huge job, even for a team with deep experience like ours. We’re hard at work moving Flickr out of Yahoo’s datacenters and into an extremely modern software stack built on top of Amazon Web Services (AWS), and we’ve already made some serious progress.
When we’re finished, Flickr will be faster than ever before. It’ll be more stable, with error-free uploads, and even fewer downtime Pandas. We’ve already moved multiple Flickr services to run 100% in AWS, with many more coming over the next months.
I wrote just the other day about some of the improvements coming to Flickr’s photo ingestion, rendering, and delivery pipelines, which I’m personally involved in. I hope you’ll love the improvements as much as I do.
Bright Future
Flickr is in great hands with a team that truly loves it. We’re investing in building on your behalf and we have a proven track record of focusing on photographers.
Thank you so much for your trust and engagement. Flickr’s community is everything. We couldn’t do all of this without you and I’m honored to be part of this journey with all of you. I promise I will keep listening, so please, keep the feedback coming.
Don
PS – If you love photography and Flickr like I do, please sign up for Flickr Pro today. If you are already Flickr Pro, thank you so much. You helped get us here today and are bringing all of us into the future. You rock and we love you.
HomeRun is a simple, elegant utility for triggering HomeKit scenes from your Apple Watch. Through a combination of color and iconography, HomeRun developer Aaron Pearce, who is the creator of other excellent HomeKit apps like HomeCam and HomePass, creates an effective solution for accessing HomeKit scenes from your wrist. It’s a user-friendly approach that’s a fantastic alternative for HomeKit device users frustrated by Apple’s Home app.
Apple’s Home app is hard to use on the Apple Watch. First, when you open Home on the Watch, it’s not clear what you’re seeing. Home presents a series of card-like, monochrome scene and accessory buttons that you scroll through one or two at a time. Although the app doesn’t say so, these are the favorite scenes and accessories from the Home tab of the iOS app. That makes the list customizable, which is nice, but the app should do a better job identifying where the user is in relationship to the iOS app.
Second, although you can rearrange your Home favorites to reorder them on the Watch too, you can only see two scenes or one accessory at a time. Depending on how many favorites you have, that limits the Watch app’s utility because a long list of scenes and accessories requires a lot of swiping or scrolling with the Digital Crown.
HomeRun avoids this by eliminating text and relying on color and iconography to distinguish between scenes. The app is also limited to triggering scenes, reducing potential clutter further. The approach allows HomeRun to display up to 12 scenes on a single screen of a 44mm Apple Watch compared to the two scene buttons that Home can display. If you set up more than 12 scenes, they are accessible by scrolling.
Scenes are added to the Watch from HomeRun’s iOS app. The plus button allows you to choose from any of the scenes you have in the Home app. Once added, tap on a scene’s button to change its icon and color, or remove the scene from HomeRun. You can also rearrange the buttons by long pressing and dragging them around, creating a grid with rows of 1-3 buttons each. On supported devices, HomeRun responds with a satisfying bit of haptic feedback as you move buttons around. In my testing, the changes I made in HomeRun’s iOS app showed up almost immediately on my Apple Watch.
HomeRun also includes customizable Watch complications. Each of the different types of complications supported by HomeRun can be associated with a different scene. For example, on an Infograph face, I added a corner complication to run my ‘Outside On’ scene, which turns on lights on my garage and above the back door. A circular complication on the same face triggers a scene that turns the lights off. I could do the same thing with Siri, but tapping HomeRun’s complications has been more reliable and is fast, which makes it an excellent alternative.
You can also run scenes from the main Watch app. Although 12 buttons sounds like a lot to have on an Apple Watch screen at once, in practice, I’ve found the buttons are large enough to be easy to tap. Scenes that are currently activated are highlighted with an inverted color scheme. Force pressing on the Watch screen lets you switch homes if you have more than one location set up in the app.
To test HomeRun, I added a few of my most-used scenes. I have pairs of scenes for turning the lights on and off in my studio and outside my home. For those, I use green for turning the lights on and red for turning them off. The studio scenes are identified with a desk icon because that’s where I work, while the outdoor lights use a leaf icon. I also have a pair of scenes that turn the heat down when I’m away from home and return it to the usual setting – these use blue and orange with thermometer icons to show how each affects the heat. The final icon is a light bulb against a gray background for dimming the lights in our living room.
There’s no text on the buttons to tell me which is which, but the use of color, icons, and placement of the buttons is plenty to help me distinguish them from each other. Better yet, all seven of my scenes fit on my Watch’s screen at once with room to spare, and I can access my most frequently used scenes as complications, making HomeRun far more usable than Apple’s Home app.
HomeRun succeeds by not trying to do too much, and through a thoughtful design that allows users to handle scenes quickly and efficiently with minimal interaction. If you are using HomeKit devices and have an Apple Watch, try HomeRun; it’s become one of my favorite ways to trigger scenes.
Flickr was part of the Yahoo family for many years. At some point along the way, Yahoo decided to integrate the two sites more significantly and require a Yahoo account for logging into Flickr. This was perhaps an easy transition for people who were already members of both sites, but it presented a challenge to Flickr members who didn’t have or need a Yahoo account at the time.
We’ve heard you, loud and clear.
We know many of our members have had challenges with the Yahoo login requirement. Some have been bothered by having to get a new email account just to access Flickr. Others have been confused about which address was associated with their account, and missed important notifications as a result. Even Flickr’s CEO, Don MacAskill, has been locked out of his Flickr account due to confusion over login credentials. We understand the challenges our members have faced trying to access their Flickr accounts, and we want to make sure that every Flickr member has easy access to their content.
One seamless login, coming soon.
Here at Flickr, we’ve long wanted a simpler login solution that would allow each member to use the email address of their choice. And since SmugMug bought Flickr from Yahoo, we’ve been working toward this goal. By early 2019, our new login will be live on Flickr. At that point, you’ll be able to use any email address you choose to log into Flickr and will no longer need a Yahoo account in order to do so.
Adding an extra layer of security.
We’ve partnered with a team of experts at AWS to create this new, simpler login solution for Flickr. By leveraging AWS’s Machine Learning-based account protection, we’re working to keep your Flickr account more secure than ever. Also, you’ll have the ability to add your phone number and enable token-based, multi-factor authentication as an added layer of security. This way, we can get you back into your account quickly if you ever forget your login information or get locked out of your Flickr account. Our focus is, and will continue to be, keeping your photos safe and your account as secure as ever.
Thanks for your patience as we continue working on the new Flickr login. We’ll keep you posted on our progress, and let you know as soon as it’s ready in January 2019.
Over the next few months, I'll be looking more closely at the available visualization apps to see what works and what doesn't. In this issue, I start with Flourish. Read More
I was recently asked to moderate an academic panel on the role of universities in training the data science workforce. I preceded each question with opinionated introductions which I have fused into this blog post. These are weakly held opinions so please consider commenting if you disagree with anything.
To discuss data science education we first need to clearly state what it means. The panel organizers defined data science as “an emerging discipline that draws upon knowledge in statistical methodology and computer science to create impactful predictions and insights for a wide range of traditional scholarly fields.“
But is it an academic discipline? If so, what are the shared fundamental principles, expertise, skills, and knowledge-based shared by data scientists? Is there a core curriculum for Data Science? Providing a more detailed definition might help.
My attempt at defining Data Science
The term Data Sciece may have been coined in academia, but the proliferation of its use has been mostly driven by the tech industry. The term became prominent because recruiters needed to more specifically describe what they needed for data driven initiatives, a new type of project becoming more and more common. Post graduate degrees in Statistics or Computer Science did not guarantee the expertise needed to successfully complete these projects. Programming skills and experience analyzing messy, complex and large datasets were fundamental. But because you can obtain a PhD in Statistics without ever looking at a real datasetstatistician was not a specific enough job title. And because you can obtain a PhD in Computer Science without ever writing one line of codecomputer scientist was not specific enough either. Statisticians and computer scientist could be good hires, but not always. On the other hand,
some graduates from other areas, such as the social sciences and particle physics, had enough experience managing and analyzing data to be hired. So the credentials provided by universities did not provide a useful signal to these employers. The academic knowledge base offered by Statistics and Computer Science was necessary, but was not sufficient. The term data scientist therefore became useful for making the distinction between, for example, someone with experience analyzing data in all its messy glory versus someone that can prove an estimate is asymptotically normal or making the distinction between someone that knows how to write fast, efficient, interoperable, and reliable code to extract/insert data from a database versus someone that can prove if an algorithm is Np complete.
However, because the challenges posed by data driven enterprises vary greatly across different organizations, and even within organization, the term remains quite vague. As a result the best definition I can provide is that data science is an umbrella term used by organizations to describe the processes used to extract value from data.
The data science areas of expertise
So what falls under the data science umbrella? First, I make one big distinction between back-end and front-end data science. I define the back-end as the part that deals with hardware, efficient computing, and data storage infrastructure. I define the front end as the part geared more towards data analysis and can be further divided into data analysts and applied machine learners. The data analysts
explore, quality assess, wrangle, and fit models to data. The applied machine learners build and assess prediction algorithms. Domain knowledge is of course important for both these tasks. Often, to finish the project the front-end data scientist develops a prototype that the back-end data scientists convert into robust pipelines. As a result front-end data scientist tend to use R or Python, while back-end data scientist program in low-level languages such as C++ and database languages such as SQL.
Another data science area of expertise is defined by what I call the frontend software engineers. These are not necessarily involved in producing data science pipelines but instead develop the software tools that facilitate data science. They tend to have experience as front-end data scientist and use this experience to develop tools that many others find useful. Examples are the developers of Rstudio, iPython notebooks, tidyverse, Hadoop and D3 to name a few. Because academia tends to favor method developers (by methods I mean mathematical abstractions that permit data analysis ideas to be applied more widely than its original application) over software developers, this group tends to work outside of academia (with exceptions) and prefer being labeled data scientists even if they have PhDs in Statistics or Computer Science.
The implication for academic programs
Having the goal of training an individual to be an expert that can tackle all the challenges involved in the data science process is too ambitious. However, as the term Data Science became more and more fashionable, demand for Data Science education increased accordingly. Universities rushed to figure out how to meet this demand. Developing revenue generating masters programs was the first priority and, as a result, today we have dozens of universities offering these degrees. But what exactly are these students being prepared to do? What do these new programs offer that existing ones did not? Given that, with some exceptions, no new faculty were hired when creating these new programs and, in many cases, no new classes were developed, it is not clear that a masters degree in Data Science provides the signal employers are looking for.
Clearly, existing academic courses provide excellent ways of gaining some of the expertise listed above. These include courses on discrete math, probability, statistical inference an modeling, computer programming, software engineering principles, and machine learning. But this was true before Data Science programs emerged. So what can academia do to better prepare students from the data science workforce and to provide a better signal to industry. Here are my recommendations.
Realize that Data Science is an umbrella term and offer specific tracks targeted at the different aspects of data science listed above. Three tracks might be enough but one isn’t.
Adapting statistics and machine learning course to have applications in the forefront rather than a theoretical focus. Data scientist have to produce pipelines that work in the real-world and one needs training to learn this. Implementation is hard.
Provide learning experiences that expose students to long-term projects like those they will be tasked to work on in industry. For this, many universities will have to invest in new faculty, with real-world experience.
Oliver and I were in Catherine’s studio yesterday morning for an emergency Halloween costume alteration when I heard a commotion in the parking lot below her back window.
I looked out to see the November issue of The Buzz being unloaded.
I’m so used to seeing piles of the paper in coffee shops in quantities of 5 or 10 that being exposed to all of the copies gave me a new appreciation for how much The Buzz has grown since the early years.
This week’s Weekly chart is a small one. It’s the calm before the storm – we’re celebrating 52 weeks of Weekly Charts next week. So while everyone talks about the upcoming US election, we will talk about German students and where they live:
If we try, we can see the four big cities in Germany, each one in a different direction: Hamburg in the north, Berlin in the east, Munich in the south and Cologne in the west. These are the only four German cities with more than 1 million inhabitants. And that’s also where most students live, but they don’t make up a big share of the population. The biggest German cities have less than 10% students.
We can also see some dark spots, where more than one in five inhabitants is a student. All of them except Jena are in the former West German states. These are true “student cities”, with a high share of students: Tübingen, Gießen, Marburg, Göttingen, Würzburg, Erlangen.
Chart Choices
To get to the insights I just described, we need to do quite a bit of hovering over the circles. To compare population numbers with each other and to simply name the cities we need to see the content in the tooltips.
Let’s do an experiment. I prepared a scatterplot with the same data. How hard is it to have the same insights as above using this chart type?
Did you find it easier or harder? Personally, I find it easier. Instead of encoding the number and share of students with size and color, we encode it with position – the position on the y & x-axis. This makes it far easier to compare the values with each other. We can see the “big but small share of students”-cities in the bottom right, and the “small but big share of students”-cities in the upper left.
So why did I choose the map as the Weekly Chart and not the scatterplot? Because I can find myself on this map. Almost exactly ten years ago, I started studying in a green circle called Weimar. I can point to this circle and say: “I was there. That’s me.” And then I can point towards other circles and say: “Here were my friends. I visited them. I was there, too”. In a scatterplot, I can only find “my” circle if it has a search function. And even then: The circles around my green circle won’t be named after the cities that were close-by when I studied.
Close-by cities in a scatterplot are conceptually close, not physically close. Close-by cities in a scatterplot will tell me which cities had a similar share of students and number of students similar to Weimar. It will be interesting. But only close-by cities on a map will bring back memories of taking the train to them because Weimar was too small to go shopping properly. It won’t be new information, but it will give me warm feelings.
Both kinds of proximity are important, the conceptual and the physical one. It can be hard to decide between a map and a scatterplot. The great thing is that we can show both! And I’ll see you next week.
People often ask me what we can do about older people and online information literacy. Old people are not necessarily more confused than young people, but for various reasons they are positioned to do much more harm when they get things wrong. They also tend to be embedded in more ideological tribes whereas as young people form tribes around other interests.
My answer is this: teach the young people how to fact-check and then have them teach their parents. Young folks are already embarrassed about their parents’ cluelessness on the web, and my experience with young folks (in a middle class American context at least) is they have no trouble speaking up when your actions as a parent are embarrassing. So give young people the skills, and show them how to teach others.
A short example: I’m not a personal fan of the post-consumer recycling approach we’ve adopted to packaging in the U.S. But in the 1980s and 1990s we decided to teach a nation to recycle their trash. Did we go out and have massive education initiatives for adults on the recycling process and the importance of it? Nope. We educated the kids so that every parent who threw a yogurt cup into the wrong container had to endure the “why do you hate baby seals” stare of their fifth grader. And some folks got resentful, but for most it was easier just to learn how to do it.
There are many other examples. My Dad quit smoking partially because his recently educated grade schoolers guilted him into it. Children of the 1970s were often the ones teaching their parents to not throw trash out the car window. College students of the 1990s were often the ones showing their parents how to work the new computer, or get on the web.
People — of all ages — are already there in terms of desiring to curb misinformation’s spread, but they need to be able to teach the skills to their parents in a systemic way. I talked to a person in D.C. a month ago whose mother always shares those fake “missing kid” memes on Facebook. And she always would comment “Mom, it’s fake” (or old or whatever). But it never occurred to her that she could show her Mom how to check it herself. When we get these checks down to easily demonstrable 10 second checks, that changes.
Teach the children and give them the skills and tools to teach their parents, stopping them from sliding into conspiracy subcultures and alternate realities. Teach the interns to teach their Senators and policy makers how to check this stuff. The college students to navigate health information for their aunt or uncle. Graduate wave after wave of people who know how to navigate the web and are committed to helping other to do better with it too. That’s how you get this done.
At this rate, we are never going to be able to use Apple Maps in Europe.
Although Apple Maps have improved markedly (and worked OK during my recent trip to Edinburgh) there are still loads of things missing (landmarks, streets, transit info), and have stayed that way for years, whereas Google Maps are sometimes corrected within days.
I wish Google would bring back their Apple Watch app, which has been MIA for a year or so…
Despite the Mayor’s Council and its 10-year transportation plan that’s been around for a while now, along with a bunch of hard-to-get Federal and Provincial money, Surrey’s new mayor Doug McCallum wants to change it.
Mayor McCallum wants transit, but on a new route in Surrey to new destinations, using different (Skytrain) technology. Blow up the Mayor’s Council’s 10-year plan, and blow up the City of Surrey’s Community (Land Use) Plan.
And it’s sort of late in the game. More background HERE and HERE.
Not surprisingly, there has been reaction from several parties to this development:
Reaction to Mayor McCallum’s Surrey Skytrain:
Feds: looking at you, Mayor’s Council.
Plus behind-the-potted-palms, hinted-at whispers that the Feds may wish to get the $ 1.6B used-to-be-Surrey-LRT money spent before the 2019 Federal election.
In a recent interview, Ken Hardie, the Liberal MP for Fleetwood-Port Kells, said federal funding for transit projects can be used for whatever regional mayors decide is a priority.
Province: looking at you, Mayor’s Council. Um, about that extra billion-three . . .
B.C. premier John Horgan has said the province will support whatever plan the Mayors’ Council puts forward, but won’t be providing any additional money. . . .
“It would certainly not fit into our new capital plan,” said Horgan.
“The new mayors from around the region will get together and we are looking forward to hearing from them, but certainly we and the federal government have funded the plan put forward by the last council and the council before them.”
Speaking of the dog that caught the bus, or someone catching a tiger by the tail — maybe we should switch the analogy to “hornet’s nest”.
Jonathan Coté, New Westminster:
“I definitely think the City of Surrey is going to have to come to the table to talk about how the region could be reimbursed for costs that have been made so far on the light-rail project,” Jonathan Coté, the mayor of New Westminster, said in an interview this week.
“I definitely think the City of Surrey should be responsible for the costs.” . . .
Mr. Coté said the LRT cannot be forced on Surrey if voters, by electing Mr. McCallum and his team, have signalled they do not want it.
Richmond Mayor Malcolm Brodie said Monday he could not see himself supporting a request from Surrey for another billion dollars for a SkyTrain line to Langley, the easternmost suburb of Metro Vancouver, with regional taxpayers having to pay for likely a third of it.
He said if Mr. McCallum cancels the LRT lines, there will be many others in the region lined up to take the money for their projects.
“The first thing that’ll have to be done is they will need to find the extra money to pay for it. Surrey will be asking for a premium type of service, and they will have to pay the increment or find the money,” said Brodie. “I don’t think there will be regional sources for that money.”
. . . would not support giving Surrey the additional money for a SkyTrain Line.
“I don’t want to spend more in a region that’s already got $1.65-billion. There’s no more money going around,” she said. “If he wants to give that up, yes, I would like it to come to the North Shore.”
“I don’t think it’s a case of just switching technologies,” from light rail to SkyTrain in Surrey, Stewart said. “It will be interesting to see the argument put forward.
“I worry though that if someone succeeds in getting the current work cancelled, it could result in another decade of work to get SkyTrain for Surrey.
“It took a decade to get the current plan.”
. . . “I would caution (Surrey) not to cancel the project, because that is what they would be doing.”
With thanks to Gordon McIntyre in Postmedia outlet the Vancouver Sun
Kennedy Stewart, Vancouver
After voicing hearty support for Mr. McCallum’s position on Monday, Mr. Stewart added a caveat Tuesday: “At the same time, we cannot put in jeopardy any infrastructure dollars that have already been committed, including funds earmarked for the Broadway Subway line,” he said in a statement issued to The Globe and Mail.
Superficially, it might seem that offering riders a more convenient service—especially one that comes directly to their door—would increase ridership. And for individual riders who don’t use buses or rail for whatever reason, it might. But for a municipality with a fixed budget for service, shifting resources from fixed routes to microtransit is a way of lowering ridership overall, not increasing it. …
If cities want to move people faster than walking while allowing them to take up only their fair share of space, two options arise. One is to use a vehicle that’s not much bigger than the human body, such as bicycles and scooters. Those tools work well for certain people in particular circumstances, but not for everyone. The other option is to share the ride in a vehicle. If space is really scarce, that vehicle will have to carry lots of people. In most cases, riders will have to share a vehicle with strangers, people who are not traveling for the same purposes or even to the same places. That’s what public transit is.
Fixed public transit deploys large vehicles flowing along a set path, and riders gathering at stops to use them. That way, the vehicles can follow a fairly straight line, and they don’t need to stop once for every customer. That is what makes them worth walking to get to. It is one of the best ideas in the history of transportation.
… most U.S. cities have a large unmet demand for frequent bus service, which is why cities investing in more frequent service have seen ridership rise. Outside the largest metro areas, you can also verify this fact by comparing your city to the most similar one in Canada. There, you’ll usually find much more bus service in a city that looks a lot like yours, with rider numbers that are higher than your city’s and growing faster. Fewer people are forced to drive in those cities, too. Americans could share that benefit, and without the need for technology. Just run as much bus service as Canada does, and demand that it have the priority it needs to succeed. …
The technology industry’s marketers can mix these issues together, dangling an electric, autonomous future before the citizenry, but if their vision hasn’t solved the problem of sharing space, it is not a vision of a functional, inclusive city. They will try everything else first, but in the end, the only solution will be the bus.
This was the teaser: " Assuming this all works out, the image in this tweet is also a valid ZIP archive, containing a multipart RAR archive, containing the complete works of Shakespeare." It did work out, as the comments to this Twitter thread make clear. And this raises the interesting question of what else might be hiding in images across the web. And it suggests a new version of the old saying: "A picture is worth a thousand scenes." You have to use 7zip; regular Windows zip won't work. But it does work; I tested it. Use the image here. Via O'Reilly.
Advertising is the original fake news. "The vast majority of ads were not marked at all. Characters in children’s games gently pressured the kids to make purchases, a practice known as host-selling, banned in children’s TV programs in 1974 by the Federal Trade Commission. At other times an onscreen character would cry if the child did not buy something." There's no difference between this and the army of Twitter bots driving conversation about the caravan.
From the recent posting on Mastodon and it currently lacking a long tail, I want to highlight a specific notion, and that’s why I am posting it here separately. This is the notion that tool usage having a long tail is a measure of distribution, and as such a proxy for networked agency. [A long tail is defined as the bottom 80% of certain things making up over 50% of a ‘market’. The 80% least sold books in the world make up more than 50% of total book sales. The 80% smallest Mastodon instances on the other hand account for less than 15% of all Mastodon users, so it’s not a long tail].
To me being able to deploy and control your own tools (both technology and methods), as a small group of connected individuals, is a source of agency, of empowerment. I call this Networked Agency, as opposed to individual agency. Networked also means that running your own tool is useful in itself, and even more useful when connected to other instances of the same tool. It is useful for me to have this blog even if I am its only reader, but my blog is even more useful to me because it creates conversations with other bloggers, it creates relationships. That ‘more useful when connected’ is why distributed technology is important. It allows you to do your own thing while being connected to the wider world, but you’re not dependent on that wider world to be able to do your own thing.
Whether a technology or method supports a distributed mode, in other words is an important feature to look for when deciding to use it or not. Another aspect is the threshold to adoption of such a tool. If it is too high, it is unlikely that people will use it, and the actual distribution will be very low, even if in theory the tools support it. Looking at the distribution of usage of a tool is then a good measure of success of a tool. Are more people using it individually or in small groups, or are more people using it in a centralised way? That is what a long tail describes: at least 50% of usage takes place in the 80% of smallest occurrences.
In June I spoke at State of the Net in Trieste, where I talked about Networked Agency. One of the issues raised there in response was about scale, as in “what you propose will never scale”. I interpreted that as a ‘centralist’ remark, and not a ‘distributed’ view, as it implied somebody specific would do the scaling. In response I wrote about the ‘invisible hand of networks‘:
“Every node in a network is a scaler, by doing something because it is of value to themselves in the moment, changes them, and by extension adding themselves to the growing number of nodes doing it. Some nodes may take a stronger interest in spreading something, convincing others to adopt something, but that’s about it. You might say the source of scaling is the invisible hand of networks.”
In part it is a pun on the ‘invisible hand of markets’, but it is also a bit of hand waving, as I don’t actually had precise notions of how that would need to work at the time of writing. Thinking about the long tail that is missing in Mastodon, and thus Mastodon not yet building the distributed social networking experience that Mastodon is intended for, allows me to make the ‘invisible hand of networks’ a bit more visible I think.
If we want to see distributed tools get more traction, that really should not come from a central entity doing the scaling. It will create counter-productive effects. Most of the Mastodon promotion comes from the first few moderators that as a consequence now run large de-facto centralised services, where 77% of all participants are housed on 0,7% (25 of over 3400) of servers. In networks smartness needs to be at the edges goes the adagium, and that means that promoting adoption needs to come from those edges, not the core, to extend the edges, to expand the frontier. In the case of Mastodon that means the outreach needs to come from the smallest instances towards their immediate environment.
Long tail forming as an adoption pattern is a good way then to see if broad distribution is being achieved.
Likely elements in promoting from the edge, that form the ‘invisible hand of networks’ doing the scaling are I suspect:
Show and tell, how one instance of tool has value to you, how connected instances have more value
Being able to explain core concepts (distribution, federation, agency) in contextually meaningful ways
Being able to explain how you can discover others using the same tool, that you might want to connect to
Lower thresholds of adoption (technically, financially, socially, intellectually)
Reach out to groups and people close to you (geographically, socially, intellectually), that you think would derive value from adoption. Your contextual knowledge is key to adoption.
Help those you reach out to set up their own tools, or if that is still too hard, ‘take them in’ and allow them the use of your own tools (so they at least can experience if it has value to them, building motivation to do it themselves)
Document and share all you do. In Bruce Sterling’s words: it’s not experimenting if you’re not publishing about it.
An adoption-inducing setting: Frank Meeuwsen explaining his steps in leaving online silos like Facebook, Twitter, and doing more on the open web. In our living room, during my wife’s birthday party.
Who of you would want to try out Mastodon as an alternative to Twitter or Facebook? Would it help if I offer you a place to try it out? On my own instance, or rather a small group instance? Would you want to find a small group instance near you / around an interest you have? Do you have questions I can help with? See Sandro Hawke’s suggestion to do a once a month push for why I am asking, and this blogpost for why I think driving adoption from the edge matters.
At the end of this month my Flickr Pro account will come up for renewal. It turns out that they doubled the price last summer (from 25 to 50 USD/year). Recently Flickr also announced that free accounts will be limited starting January 1st, 2019. The new limit will be at 1000 photos, and accounts with more images will see their oldest images deleted.
I have been a Flickr Pro member since early 2005, and store some 25.000 photos there, making up 75GB. I took a paid account in 2005 because back then they had a 200 photo cap for free accounts, and I easily reached that limit.
With a rate change like this it is a good time to evaluate whether the service is still good enough for me to keep at the new rate.
How do I use Flickr?
It’s an off-site back-up of photos
I use it to find Creative Commons licensed material for my presentations
I publish photos under such a license myself, to enable others to use them
I embed Flickr photos in my blogposts, so I do not have to store them on my hosting account (which has much less storage, 3GB)
I use it to quickly find things back in my own photos, through its album structure and search. “Don’t I have a picture of that building from when I visited that conference in Copenhagen a few years ago?”
So if I would want to replace Flickr, e.g. by bringing it home to something more under my control, what would that need to look like?
For the off-site storage I could easily find cheaper alternatives, in fact I already run several of them where there’s still over 50TB of total storage available.
Finding CC images on Flickr is still possible if you’re not a registered member, but it misses some showing me photos by myself and those I’m connected to first. I have a preference for using photos from my network.
Contributing CC images is important to me, also as I feel reciprocity is important, as I do use CC images by others a lot too. I don’t know of any other place where I could add CC licenses to my photos that casually. I have seen places where you’d pool your curated images under CC but that is additional work. Part of the utility is to automatically add CC licenses to everything I store online. Maybe some of you know an alternative?
Embedding photos easily at various formats (using HTML only, as I strip out the javascript stuff Flickr also provides) is something I have no ideas for an alternative currently. Probably it would mean exposing a replacement storage to the public, but not sure how to replace resizing on the fly. I could also try and do what Peter did, replacing all currently embedded photos on my blog, the photos I made at least, with locally hosted ones. It would solve this for the past, but not for the future.
Search replacement like embedding replacement would depend on having public storage, and would require keeping the album structure, added titles, geo-locations etc. That added metadata (80% of my photos have tags and geotags) were all added manually during upload, geotags mostly added manually, some automatically)
The ‘cost of leaving’ is mostly sunk efforts like added titles, tags and locations. So even if it feels differently, that is not a rational consideration to keep an account. Especially not as you can download all Flickr material including that metadata, so it is more about how you would make that metadata useful in a new set-up. The decision to make is if I want to find and set-up a workable alternative in the coming three weeks to save 100 USD, or do I buy myself 2 years of time with those 100USD?
If you have left Flickr in the past few years, what does your current workflow around photos look like?
Some of us at Wirecutter like our bikes so much, we’ll ride them when it’s raining—or snowing, or sleeting, or freezing, or whatever the weather brings. Thanks to our experiences riding in many conditions through many cities, we’ve accumulated a few pieces of gear we think anyone should consider if, as we do, you like to show up on a bike when nobody expects it.
Nach zwei Jahren mit Windows 10 wird es mal wieder Zeit für einen Tapetenwechsel. Ich habe meine alten Weggefährten MacBook Pro (13 Zoll, Late 2013) und iPad Pro (9,7 Zoll, 2016) wieder aktiviert. Das MacBook läuft nun mit macOS Mojave, das iPad Pro mit iOS 12.1.
Mein eigentliches Ziel ist aber das neue iPad Pro in 12,9 Zoll. Wie bei dem Surface will ich versuchen, alle täglichen Workflows auf das iPad zu bringen. Für seltenere Workflows habe ich andere Hardware. Meine Buchhaltung mache ich zum Beispiel auf einem Lenovo.
Warum nicht das alte iPad Pro? Das hat ein paar Schwächen. Erstens ist es zu klein, zweitens ist die Tastatur zu klein und der Aufstellmechanismus zu schlecht. Das ändert sich alles mit dem neuen iPad Pro. Das neue Smart Keyboard Folio schließt Vor- und Rückseite ein. Und die Vorderseite sollte eine stabile Basis bieten, mit der man das iPad sogar auf dem Schoß nutzen kann. Das große Modell ist auch nicht mehr so riesig, weil der Rand kleiner ist. Das könnte für mich der perfekte Notebook-Ersatz sein. Aktuell bin ich dabei, viele Workflows mit Shortcuts zu automatisieren, zum Beispiel das Hochladen von Bildern auf diese Seite.
My readership is quite international (see small map in the sidebar), so I should write everything in English, right? Wrong. Most of my readers speak German as their first language and learned English in school. I learned half a dozen languages, but I can only speak and write two of those with confidence. So I like to write German in about half of my posts, to make my German readers feel at home here.
Thomas suggested today that I put a translate button on my site for those people who do not understand the languages I write in. Thomas is from Sweden so his native language is one that I do not understand at all. But technology helps. If I encounter a language on the web that I do not understand, I use technology to help me out. For the iPhone there is Google Translate and Microsoft Translator, and probably a whole lot more.
I suggested to Thomas to use a browser plugin instead of me putting in a translate link. Since he was on the iPhone he shared my German post with Microsoft Translator and got the same one in English. Exciting times we live in.
On one side, by creating a robust decentralized web, we could create a way for extremist movements to thrive. On another, by restricting hate speech, we could create overarching censorship that genuinely guts freedom of speech protections
I think this is a false dilemma, Bernd.
I’d say that it would be great if those extremists would see using a distributed tool like Mastodon as the only remaining viable platform for them. It would not suppress their speech. But it woud deny them any amplification, which they now enjoy by being very visible on mainstream platforms, giving them the illusion they are indeed mainstream. It will be much easier to convince, if at all needed, instance moderators to not federate with instances of those guys, reducing them ever more to their own bubble. They can spew hate amongst themselves for eternity, but without amplification it won’t thrive. Jotted down some thoughts on this earlier in “What does Gab’s demise mean for federation?“
Well, Walmart vs. Amazon is all about big business – a platform giant (Amazon) disrupting an OldBigCo (Walmart and its kin). Over the past two decades, Amazon bumped Walmart out of the race to a trillion-dollar market cap, and the OldCo from Bentonville had to reset and play the role of the upstart. The Token Act levels the playing field, forcing both to win where it really matters: In service to the customer.
But while BigCos are sexy and well known, it’s the small and medium-sized business ecosystem that determines whether or not we have an economy of mass flourishing. So let’s explore the Token Act from the point of view of a small business startup, in this case, a new neighborhood restaurant. I briefly touched upon this idea in my set up post, Don’t Break Up The Tech Oligarchs. Force Them To Share Instead. (If you haven’t already, you might want to read that post before this one, as I lay out the framework in which this scenario would play out.) What I envision below assumes the Token Act has passed, and we’re at least a year or two into its adoption by most major data players. Here we go…
***
Fresh off her $2,700 win from Walmart, Michelle decides she’s ready to lean into a lifelong dream: Starting a restaurant in her newly adopted neighborhood of Chelsea in New York City. Since moving to the area from California, she’s noticed two puzzling trends: First, a dearth of interesting mid- to high-end dinner spots walking distance from her new place, and second, what appears to be higher-than-average vacancy rates for the retail storefronts in the same general area. It appears to be a buyer’s market for retail restaurant space in Chelsea. So why aren’t new places launching? She read the Times’ piece on vacancies a few years ago (before the Token Act passed) and was left just as puzzled as before – seems like there’s no rhyme or reason to the market.
Michelle wants to start a high end American gastro pub – the kind of place she loved back when she lived in Northern California (she’s fond of Danny Meyers’ Gramercy Tavern, pictured above, but it’s a bit too far away from her new place). She has a strong hunch that such a place would be a hit in her new neighborhood, but she’s not sure her new neighbors will agree.
Now starting a restaurant requires a certain breed of insanity – they say the best way to make a small fortune in the business is to start with a large one. The truth is, launching restaurants has historically been a crap shoot – you might find the best talent, the best designer, and the best location – but if for some reason you don’t bring the je ne sai quois, the place will fail within months, leaving you and your partners millions of dollar poorer.
It’s that je ne sai quois that Michelle is determined to reveal. The tools she will leverage? The newly liberated resources of data tokens.
Before we continue, allow me to draw your attention back to the rise of search, indeed, the very era which begat Searchblog in the early 2000s. Google Adwords launched in 2000, and within a few years, the media world had been turned upside down by what I termed The Database of Intentions. As if by magic, people everywhere could suddenly ask new kinds of questions, finding themselves both surprised and delighted by the answers they received.
A Gates-Line compliant ecosystem quickly developed on top of this new platform, driven by an emerging industry of search engine marketing and optimization. SEO/SEM sprung into existence to help small and medium sized businesses take advantage of the Google platform – by 2006 the industry stood at nearly $10 billion in spend, growing more than 60 percent year on year. Adwords grew from zero to millions of advertisers by connecting to a long tail of small businesses that took advantage of an entirely new class of revealed information: The intents, desires, and needs of tens of millions of consumers, who relentlessly poured their queries into Google’s placid and unblinking search box.
Were you a limo service in the Bronx looking for new customers? It paid huge dividends to purchase Adwords like “car service bronx” and “best limo manhattan.” Were you a dry cleaner in West LA hoping to expand? Best be first in line when customers typed in “best cleaners Beverly Hills.” Selling heavy machinery to construction services in the midwest? If you don’t own keywords like “caterpillar dealer des moines” you’d lose, and quick, to whoever did optimize to phrases like that.
My point is simply this: Adwords was a freaking revolution, but it ain’t nothing compared to what will happen if we unleash data tokens on the world.
***
Ok, back to Michelle and her new restaurant. Of course Michelle will leverage Adwords, and Facebook, and any other advertising service to help her new business grow. But none of those services can help her figure out her je ne sai quois – for that, she needs something entirely novel. She needs a new question machine. And the ecosystem that develops around data tokens will offer it.
Thanks to her Walmart experience, Michelle has become aware of the power of personal data. She’s also read up on the Token Act, the new law requiring all data players at scale to allow individuals to create machine-readable data tokens that can be exchanged for value as directed by the consumer. After doing a bit of research, she stumbles across a startup called OfferExchange, which manages “Token Offers” on behalf of anyone who might want to query TokenLand. OfferExchange is a spinout from ProtocolLabs, a pioneer in secure blockchain software platforms like Filecoin. It’s still early in TokenLand, so an at-scale Google of the space hasn’t emerged. OfferExchange works more like a bespoke yet platform-based research outfit – the firm has a sophisticated website and impressive client list. It uses Facebook, Twitter, LiveRamp, and Instagram to identify potential token-creating consumers, then solicits those individuals with offers of cash or other value in exchange for said tokens.
Michelle does a Crunchbase search for OfferExchange and sees it’s backed by Union Square Ventures and Benchmark, which gives her some comfort – those firms don’t fund fly-by-night hucksters. And OfferExchange site is impressive – in less than five minutes, it guides her through the construction of an elegant query. Here’s how the process works:
First, the site asks Michelle what her goal is. “Starting a restaurant in New York City,” she responds. The site reconstructs around her answer, showing suggested data repositories she might mine. “Restaurants, New York City,” reads the top layer of a directory-like page. Underneath are several categories, each populated with familiar company names:
Restaurant Reservation and Review Services
OpenTable Google Resy Yelp Eat24 Facebook (more)
Food Delivery Services
GrubHub Uber Eats PostMates InstaCart (more)
Transportation Services
Uber Lyft Juno Via (more)
Real Estate Services (Commercial)
LoopNet DocuSign CompStak (more)
Location Services
Foursquare Uber Lyft Google NinthDecimal (more)
Financial Services
American Express Visa Mastercard Apple Pay Diners Club (more)
And so on – if she wished, Michelle could dig into dozens of categories related to her initial “restaurant New York City” search.
Michelle’s imagination sparks – the kinds of queries she could ask of these services is mind blowing. She could limit her query to people who live within walking distance of her neighborhood, asking her *actual neighbors* for tokens that tell her what restaurants they eat at, when they eat there, the size of their checks, related reviews, abandoned reservations, the works. She might discover that folks like Indian takeout on Mondays, that they rarely spend more than $100 on a meal on Tuesdays, but that they splurge on the weekends. She could discover the percentage of diners in Chelsea who travel more than two miles by car service to eat out at a place similar to the one she has in mind, and what the size of the check might be when they do. She can also check historical average rents for restaurants in her zip code, over time, which will certainly help with negotiating her lease. The possibilities are endless.
Put another way, with OfferExchange’s services, Michelle can litigate the merde out of her je ne sai quois.
***
This post is getting long, so I’ll stop here and pull back for a spot of Thinking Out Loud. I could continue the story, imagining the process of the token offer Michelle would put out through OfferExchange’s platform, but suffice to say, she’d be willing to pay upwards of $5-20 per potential customer for their data. The marketing benefit alone – alerting potential customers in the neighborhood that she’s exploring a new restaurant in the area – is worth tens of thousands already. And of course, OfferExchange can connect anyone who offers their tokens to Michelle’s new project a discount on their first meal at the restaurant, should it actually launch. Cool!
But let’s stop there and consider what happens when local entrepreneurs have access to the information currently silo’d across thousands of walled garden services like Uber, LoopNet, Resy, and of course Facebook and Google. While better data won’t insure that Michelle’s restaurant will succeed, it certainly increases the odds that it won’t fail. And it will give both Michelle and her investors – local banks, savvy friends and family members – much more conviction that her new enterprise is viable. Take this local restaurant example and apply it to all manner of small business – dry cleaners, hardware stores, bike shops – and this newly liberated class of information enables an explosion of efficiency, investment, and, well, flourishing in what has become, over the past four decades, a stagnant SMB environment.
Is this Money Ball for SMB? Perhaps. And yes, I can imagine any number of downsides to this new data economy. But I also believe the benefits would far outweigh the downsides. Under the Token Act as I envision it, co-creators of the data – the services like Uber, OpenTable, or Facebook – have the right to charge a vig for the data being monetized. Sure, it’d be possible for an entrepreneur to steal customers via tokens, but I’m going to guess the economic value of allowing your customers to discover new use cases for their data will dwarf the downside of possibly losing those customers to a new competitor. Plus, this new competitive force will drive everyone to play at a higher level, focusing not on moats built on data silos, but instead on what really matters: A highly satisfied customer. That’s certainly Michelle’s goal, and the goal of every successful local business. Why shouldn’t it also be the goal of the data giants?
Ben Werdmüller