Rolandt

This year Julia Silge and I released the tidytext package for text mining using tidy tools such as dplyr, tidyr, ggplot2 and broom. One of the canonical examples of tidy text mining this package makes possible is sentiment analysis.

Sentiment analysis is often used by companies to quantify general social media opinion (for example, using tweets about several brands to compare customer satisfaction). One of the simplest and most common sentiment analysis methods is to classify words as “positive” or “negative”, then to average the values of each word to categorize the entire document. (See this vignette and Julia’s post for examples of a tidy application of sentiment analysis). But does this method actually work? Can you predict the positivity or negativity of someone’s writing by counting words?

To answer this, let’s try sentiment analysis on a text dataset where we know the “right answer”- one where each customer also quantified their opinion. In particular, we’ll use the Yelp Dataset: a wonderful collection of millions of restaurant reviews, each accompanied by a 1-5 star rating. We’ll try out a specific sentiment analysis method, and see the extent to which we can predict a customer’s rating based on their written opinion. In the process we’ll get a sense of the strengths and weaknesses of sentiment analysis, and explore another example of tidy text mining with tidytext, dplyr, and ggplot2.

Setup

I’ve downloaded the yelp_dataset_challenge_academic_dataset folder from here.¹ First I read and process them into a data frame:

library(readr)
library(dplyr)

# we're reading only 200,000 in this example
# you can try it with the full dataset too, it's just a little slower to process!
infile <- "~/Downloads/yelp_dataset_challenge_academic_dataset/yelp_academic_dataset_review.json"
review_lines <- read_lines(infile, n_max = 200000, progress = FALSE)

library(stringr)
library(jsonlite)

# Each line is a JSON object- the fastest way to process is to combine into a
# single JSON string and use fromJSON and flatten
reviews_combined <- str_c("[", str_c(review_lines, collapse = ", "), "]")

reviews <- fromJSON(reviews_combined) %>%
  flatten() %>%
  tbl_df()

We now have a data frame with one row per review:

reviews

## # A tibble: 200,000 x 10
##                   user_id              review_id stars       date
##                     <chr>                  <chr> <int>      <chr>
## 1  PUFPaY9KxDAcGqfsorJp3Q Ya85v4eqdd6k9Od8HbQjyA     4 2012-08-01
## 2  Iu6AxdBYGR4A0wspR9BYHA KPvLNJ21_4wbYNctrOwWdQ     5 2014-02-13
## 3  auESFwWvW42h6alXgFxAXQ fFSoGV46Yxuwbr3fHNuZig     5 2015-10-31
## 4  uK8tzraOp4M5u3uYrqIBXg Di3exaUCFNw1V4kSNW5pgA     5 2013-11-08
## 5  I_47G-R2_egp7ME5u_ltew 0Lua2-PbqEQMjD9r89-asw     3 2014-03-29
## 6  PP_xoMSYlGr2pb67BbqBdA 7N9j5YbBHBW6qguE5DAeyA     1 2014-10-29
## 7  JPPhyFE-UE453zA6K0TVgw mjCJR33jvUNt41iJCxDU_g     4 2014-11-28
## 8  2d5HeDvZTDUNVog_WuUpSg Ieh3kfZ-5J9pLju4JiQDvQ     5 2014-02-27
## 9  BShxMIUwaJS378xcrz4Nmg PU28OoBSHpZLkYGCmNxlmg     5 2015-06-16
## 10 fhNxoMwwTipzjO8A9LFe8Q XsA6AojkWjOHA4FmuAb8XQ     3 2012-08-19
## # ... with 199,990 more rows, and 6 more variables: text <chr>,
## #   type <chr>, business_id <chr>, votes.funny <int>, votes.useful <int>,
## #   votes.cool <int>

Notice the stars column with the star rating the user gave, as well as the text column (too large to display) with the actual text of the review. For now, we’ll focus on whether we can predict the star rating based on the text.

Tidy sentiment analysis

Right now, there is one row for each review. To analyze in the tidy text framework, we need to use the unnest_tokens function and turn this into one-row-per-term-per-document:

library(tidytext)

review_words <- reviews %>%
  select(review_id, business_id, stars, text) %>%
  unnest_tokens(word, text) %>%
  filter(!word %in% stop_words$word,
         str_detect(word, "^[a-z']+$"))

review_words

## # A tibble: 7,688,667 x 4
##                 review_id            business_id stars        word
##                     <chr>                  <chr> <int>       <chr>
## 1  Ya85v4eqdd6k9Od8HbQjyA 5UmKMjUEUNdYWqANhGckJw     4      hoagie
## 2  Ya85v4eqdd6k9Od8HbQjyA 5UmKMjUEUNdYWqANhGckJw     4 institution
## 3  Ya85v4eqdd6k9Od8HbQjyA 5UmKMjUEUNdYWqANhGckJw     4     walking
## 4  Ya85v4eqdd6k9Od8HbQjyA 5UmKMjUEUNdYWqANhGckJw     4   throwback
## 5  Ya85v4eqdd6k9Od8HbQjyA 5UmKMjUEUNdYWqANhGckJw     4         ago
## 6  Ya85v4eqdd6k9Od8HbQjyA 5UmKMjUEUNdYWqANhGckJw     4   fashioned
## 7  Ya85v4eqdd6k9Od8HbQjyA 5UmKMjUEUNdYWqANhGckJw     4        menu
## 8  Ya85v4eqdd6k9Od8HbQjyA 5UmKMjUEUNdYWqANhGckJw     4       board
## 9  Ya85v4eqdd6k9Od8HbQjyA 5UmKMjUEUNdYWqANhGckJw     4      booths
## 10 Ya85v4eqdd6k9Od8HbQjyA 5UmKMjUEUNdYWqANhGckJw     4   selection
## # ... with 7,688,657 more rows

Notice that there is now one-row-per-term-per-document: the tidy text form. In this cleaning process we’ve also removed “stopwords” (such as “I”, “the”, “and”, etc), and removing things things that are formatting (e.g. “—-“) rather than a word.

Now let’s perform sentiment analysis on each review. We’ll use the AFINN lexicon, which provides a positivity score for each word, from -5 (most negative) to 5 (most positive). This, along with several other lexicons, are stored in the sentiments table that comes with tidytext. (I’ve tried some other lexicons on this dataset and the results are pretty similar.)

AFINN <- sentiments %>%
  filter(lexicon == "AFINN") %>%
  select(word, afinn_score = score)

AFINN

## # A tibble: 2,476 x 2
##          word afinn_score
##         <chr>       <int>
## 1     abandon          -2
## 2   abandoned          -2
## 3    abandons          -2
## 4    abducted          -2
## 5   abduction          -2
## 6  abductions          -2
## 7       abhor          -3
## 8    abhorred          -3
## 9   abhorrent          -3
## 10     abhors          -3
## # ... with 2,466 more rows

Now as described in Julia’s post, our sentiment analysis is just an inner-join operation followed by a summary:

reviews_sentiment <- review_words %>%
  inner_join(AFINN, by = "word") %>%
  group_by(review_id, stars) %>%
  summarize(sentiment = mean(afinn_score))

reviews_sentiment

## Source: local data frame [187,688 x 3]
## Groups: review_id [?]
## 
##                 review_id stars sentiment
##                     (chr) (int)     (dbl)
## 1  __-r0eC3hZlaejvuliC8zQ     5 4.0000000
## 2  __1yzxN39QzdeJqicAg99A     3 1.3333333
## 3  __3Vy9VLHV5jKjgFDRWCiQ     2 1.3333333
## 4  __56FUEaW57kZEm56OZk7w     5 0.8333333
## 5  __5webDfFxADKz_3k5YipA     5 2.2222222
## 6  __6QkPtePef4_oW6A_tbOg     4 2.0000000
## 7  __6tOxx2VcvGR02d2ILkuw     5 1.7500000
## 8  __77nP3Nf1wsGz5HPs2hdw     5 1.6000000
## 9  __7MkcofSZYHj9v5KuLVvQ     4 1.8333333
## 10 __7RBFUZgxef8gZ8guaVhg     5 2.4000000
## ..                    ...   ...       ...

We now have an average sentiment alongside the star ratings. If we’re right and sentiment analysis can predict a review’s opinion towards a restaurant, we should expect the sentiment score to correlate with the star rating.

Did it work?

library(ggplot2)
theme_set(theme_bw())

ggplot(reviews_sentiment, aes(stars, sentiment, group = stars)) +
  geom_boxplot() +
  ylab("Average sentiment score")

Well, it’s a very good start! Our sentiment scores are certainly correlated with positivity ratings. But we do see that there’s a large amount of prediction error- some 5-star reviews have a highly negative sentiment score, and vice versa.

Which words are positive or negative?

Our algorithm works at the word level, so if we want to improve our approach we should start there. Which words are suggestive of positive reviews, and which are negative?

To examine this, let’s create a per-word summary, and see which words tend to appear in positive or negative reviews. This takes more grouping and summarizing:

review_words_counted <- review_words %>%
  count(review_id, business_id, stars, word) %>%
  ungroup()

review_words_counted

## # A tibble: 6,566,367 x 5
##                 review_id            business_id stars      word     n
##                     <chr>                  <chr> <int>     <chr> <int>
## 1  ___XYEos-RIkPsQwplRYyw YxMnfznT3eYya0YV37tE8w     5    batter     1
## 2  ___XYEos-RIkPsQwplRYyw YxMnfznT3eYya0YV37tE8w     5     chips     3
## 3  ___XYEos-RIkPsQwplRYyw YxMnfznT3eYya0YV37tE8w     5  compares     1
## 4  ___XYEos-RIkPsQwplRYyw YxMnfznT3eYya0YV37tE8w     5 fashioned     1
## 5  ___XYEos-RIkPsQwplRYyw YxMnfznT3eYya0YV37tE8w     5  filleted     1
## 6  ___XYEos-RIkPsQwplRYyw YxMnfznT3eYya0YV37tE8w     5      fish     4
## 7  ___XYEos-RIkPsQwplRYyw YxMnfznT3eYya0YV37tE8w     5     fries     1
## 8  ___XYEos-RIkPsQwplRYyw YxMnfznT3eYya0YV37tE8w     5    frozen     1
## 9  ___XYEos-RIkPsQwplRYyw YxMnfznT3eYya0YV37tE8w     5 greenlake     1
## 10 ___XYEos-RIkPsQwplRYyw YxMnfznT3eYya0YV37tE8w     5      hand     1
## # ... with 6,566,357 more rows

word_summaries <- review_words_counted %>%
  group_by(word) %>%
  summarize(businesses = n_distinct(business_id),
            reviews = n(),
            uses = sum(n),
            average_stars = mean(stars)) %>%
  ungroup()

word_summaries

## # A tibble: 100,177 x 5
##          word businesses reviews  uses average_stars
##         <chr>      <int>   <int> <int>         <dbl>
## 1   a'boiling          1       1     1           4.0
## 2      a'fare          1       1     1           4.0
## 3      a'hole          1       1     1           5.0
## 4      a'ight          6       6     6           2.5
## 5        a'la          2       2     2           4.5
## 6        a'll          1       1     1           1.0
## 7      a'lyce          1       1     2           5.0
## 8      a'more          1       2     2           5.0
## 9    a'orange          1       1     1           5.0
## 10 a'prowling          1       1     1           3.0
## # ... with 100,167 more rows

We can start by looking only at words that appear in at least 200 (out of 200000) reviews. This makes sense both because rare words will have a noisier measurement (a few good or bad reviews could shift the balance), and because they’re less likely to be useful in classifying future reviews or text. I also filter for ones that appear in at least 10 businesses (others are likely to be specific to a particular restaurant).

word_summaries_filtered <- word_summaries %>%
  filter(reviews >= 200, businesses >= 10)

word_summaries_filtered

## # A tibble: 4,328 x 5
##          word businesses reviews  uses average_stars
##         <chr>      <int>   <int> <int>         <dbl>
## 1     ability        374     402   410      3.465174
## 2    absolute        808    1150  1183      3.710435
## 3  absolutely       2728    6158  6538      3.757389
## 4          ac        378     646   919      3.191950
## 5      accent        171     203   214      3.285714
## 6      accept        557     720   772      2.929167
## 7  acceptable        500     587   608      2.505963
## 8    accepted        293     321   332      2.968847
## 9      access        544     840   925      3.505952
## 10 accessible        220     272   282      3.816176
## # ... with 4,318 more rows

What were the most positive and negative words?

word_summaries_filtered %>%
  arrange(desc(average_stars))

## # A tibble: 4,328 x 5
##             word businesses reviews  uses average_stars
##            <chr>      <int>   <int> <int>         <dbl>
## 1  compassionate        193     298   312      4.677852
## 2        listens        177     215   218      4.632558
## 3       exceeded        286     320   321      4.596875
## 4       painless        224     290   294      4.568966
## 5   knowledgable        607     775   786      4.549677
## 6            gem        874    1703  1733      4.537874
## 7     impeccable        278     475   477      4.520000
## 8        happier        545     638   654      4.495298
## 9  knowledgeable       1550    2747  2807      4.493629
## 10   compliments        333     418   428      4.488038
## # ... with 4,318 more rows

Looks plausible to me! What about negative?

word_summaries_filtered %>%
  arrange(average_stars)

## # A tibble: 4,328 x 5
##              word businesses reviews  uses average_stars
##             <chr>      <int>   <int> <int>         <dbl>
## 1            scam        211     263   297      1.368821
## 2     incompetent        275     317   337      1.378549
## 3  unprofessional        748     921   988      1.380022
## 4       disgusted        251     283   292      1.381625
## 5          rudely        349     391   418      1.493606
## 6            lied        281     332   372      1.496988
## 7          refund        717     930  1229      1.545161
## 8    unacceptable        387     441   449      1.569161
## 9           worst       2574    5107  5597      1.569219
## 10        refused        803     983  1096      1.579858
## # ... with 4,318 more rows

Also makes a lot of sense. We can also plot positivity by frequency:

ggplot(word_summaries_filtered, aes(reviews, average_stars)) +
  geom_point() +
  geom_text(aes(label = word), check_overlap = TRUE, vjust = 1, hjust = 1) +
  scale_x_log10() +
  geom_hline(yintercept = mean(reviews$stars), color = "red", lty = 2) +
  xlab("# of reviews") +
  ylab("Average Stars")

Note that some of the most common words (e.g. “food”) are pretty neutral. There are some common words that are pretty positive (e.g. “amazing”, “awesome”) and others that are pretty negative (“bad”, “told”).

Comparing to sentiment analysis

When we perform sentiment analysis, we’re typically comparing to a pre-existing lexicon, one that may have been developed for a particular purpose. That means that on our new dataset (Yelp reviews), some words may have different implications.

We can combine and compare the two datasets with inner_join.

words_afinn <- word_summaries_filtered %>%
  inner_join(AFINN)

words_afinn

## # A tibble: 505 x 6
##            word businesses reviews  uses average_stars afinn_score
##           <chr>      <int>   <int> <int>         <dbl>       <int>
## 1       ability        374     402   410      3.465174           2
## 2        accept        557     720   772      2.929167           1
## 3      accepted        293     321   332      2.968847           1
## 4      accident        369     447   501      3.536913          -2
## 5  accidentally        279     305   307      3.252459          -2
## 6        active        177     215   238      3.744186           1
## 7      adequate        420     502   527      3.203187           1
## 8         admit        942    1316  1348      3.620821          -1
## 9      admitted        196     248   271      2.157258          -1
## 10     adorable        305     416   431      4.281250           3
## # ... with 495 more rows

ggplot(words_afinn, aes(afinn_score, average_stars, group = afinn_score)) +
  geom_boxplot() +
  xlab("AFINN score of word") +
  ylab("Average stars of reviews with this word")

Just like in our per-review predictions, there’s a very clear trend. AFINN sentiment analysis works, at least a little bit!

But we may want to see some of those details. Which positive/negative words were most successful in predicting a positive/negative review, and which broke the trend?

For example, we can see that most profanity has an AFINN score of -4, and that while some words, like “wtf”, successfully predict a negative review, others, like “damn”, are often positive (e.g. “the roast beef was damn good!”). Some of the words that AFINN most underestimated included “die” (“the pork chops are to die for!”), and one of the words it most overestimated was “joke” (“the service is a complete joke!”).

One other way we could look at misclassifications is to add AFINN sentiments to our frequency vs average stars plot:

One thing I like about the tidy text mining framework is that it lets us explore the successes and failures of our model at this granular level, using tools (ggplot2, dplyr) that we’re already familiar with.

Next time: Machine learning

In this post I’ve focused on basic exploration of the Yelp review dataset, and an evaluation of one sentiment analysis method for predicting review positivity. (Our conclusion: it’s good, but far from perfect!) But what if we want to create our own prediction method based on these reviews?

In my next post on this topic, I’ll show how to train LASSO regression (with the glmnet package) on this dataset to create a predictive model. This will serve as an introduction to machine learning methods in text classification. It will also let us create our own new “lexicon” of positive and negative words, one that may be more appropriate to our context of restaurant reviews.

Heather M. Ross, McToonish, Jul 24, 2016

As you can see, I wasn't willing to see Heather Ross's blog self-destruct. More to the point, I wanted to share her thoughts on digital citizenship, thoughts which go well beyond digital literacy. She cites  Mike Ribble’ s list of the Nine Themes of Digital Citizenship, a list which includes digital access, digital commerce, digital communication, and more. She ends with "video about the 'filter bubble' that explains why you see a lot of what you as an individual see online." I don't really experience the filter bubble - there are days when I wish I did. But this isn't one of the posts I'd filter.

Apple is in the midst of its weakest growth period in 15 years. Despite deteriorating financial trends, Apple's stock has held up remarkably well in 2016. There are a number of clues that suggest we are in the latter stages of a major Apple expectations reset on Wall Street. The implications are significant not just for AAPL, but also for the amount of leeway given to Tim Cook and Luca Maestri over the next two to three years. 

AAPL: The Growth Stock

On May 22nd, 2015, Apple's prospects were running high. AAPL shares had just closed at an all-time high of $132.54, giving the company a $769 billion market cap. Excluding the cash and cash equivalents on the balance sheet, Apple's enterprise value was $619 billion. Three weeks earlier, Apple had reported record 2Q15 earnings. The iPhone 6 and 6 Plus drove strong 40% year-over-year growth in iPhone unit sales. In addition, Apple had just experienced its largest product category launch ever with Apple Watch. Expectations were that the Watch would quickly become a key Apple revenue driver. 

While a few analysts voiced concerns about Apple's dependency on the iPhone for revenue and profits, there were others projecting Apple would become the first trillion dollar market cap company. Apple financial expectations were quite high as many analysts and investors were modeling 20%+ EPS growth in 2016 and 2017. Apple was the largest growth stock on Wall Street. 

The Apple Reset

Over the subsequent three months following AAPL's record-high close in May 2015, AAPL shares declined 20%. The Chinese stock market had crashed, and fears began to grow that there could be a spillover effect on iPhone demand. Wall Street eventually began to doubt whether or not Apple would be able to maintain its revenue and earnings growth rates given China's slowing economy.

In October 2015, only a few weeks after the iPhone 6s and 6s Plus launch, reports surfaced that the new iPhones were not selling as strongly as expected. While the iPhone business had contained so much promise just a few months earlier, things were beginning to look much more fragile. Analysts began to cut their iPhone sales estimates on growing fear that the iPhone business was headed into a rough patch. 

Apple's 1Q16 earnings report back in January confirmed that fear. Apple was able to grow iPhone sales by a mere 311,000 units year-over-year, and management's 2Q16 guidance implied Apple would report its first year-over-year decline in iPhone sales. Meanwhile, Apple Watch sales were not living up to Wall Street's lofty expectations.

In the days following Apple's miserable 2Q16 earnings report this past April, expectations took another step down, resulting in AAPL shares bottoming at $90.34 on May 12th, 2016. During the twelve-month span ranging from May 2015 to May 2016, AAPL shares had declined by 32%, wiping out $271 billion of market cap. Excluding cash and cash equivalents, Apple's enterprise value had declined by $274 billion. Apple had just gone through a year-long expectations reset concluding with Apple no longer being considered a growth stock.

The Expectations Game

Apple's reset over the past year has been guided by Wall Street's expectations game. Analysts and investors make projections about a company's future stream of cash flows and earnings. These expectations help explain how one company's stock can perform poorly the day after reporting strong earnings while another company's stock can outperform the market after posting a weak earnings report. 

Apple suffered from two primary problems pertaining to the expectations game. Management never had a cohesive narrative for how investors and analysts should judge Apple's financial success. In addition, there was evidence that Apple management was caught off guard by slowing iPhone 6s and 6s Plus sales, which made it difficult to communicate clarity with top Apple shareholders and analysts.

Establishing a Narrative. A narrative provides a management team an avenue to explain its business to analysts and investors. Companies can use everything from SEC filings to the prepared remarks found in the beginning of a quarterly earnings conference call to help establish and then nurture a narrative. The strongest Wall Street narratives are those that are easiest to explain and understand. Examples of well-received narratives include Facebook owning the most attractive mobile properties for advertisers or Amazon forgoing near-term profits in order to invest in long-term investments and bets. For each narrative, investors monitor certain financial metrics in order to judge how a management team is performing in comparison to benchmarks. 

When it came to Apple, the company has never had a functioning narrative on Wall Street. Instead, the only way investors have come to judge Apple's success has been to look at whatever hardware product was the top seller at any given moment. In the early 2000s, it was the Mac and iPod. Eventually, the focus turned to the iPhone, and soon, iPad. Once the iPad ran into sales trouble, the iPhone then became the center of attention. Nothing else mattered but iPhone unit sales growth. This was a recipe for disaster considering how Apple's business model is based on profit share, not market share. Once iPhone unit sales growth slowed, the closest thing Apple had to a Wall Street narrative fell apart. 

Managing Financial Expectations. Companies have additional tools at their disposal to help guide investor's near-term expectations. Providing financial guidance is one of the more common methods used to establish an adequate framework for expectations. Management teams can issue estimate ranges for what they think is achievable in terms of revenue, income, or metrics such as monthly users. Another option that is less talked about, but at times far more effective, is for management teams to keep open communication channels with top shareholders and financial analysts.  

Even though Tim Cook and Luca Maestri have been providing quarterly guidance as well as communicating with top shareholders, the two were not able to adequately manage financial expectations surrounding the iPhone business. There is evidence that even Apple management themselves were caught off guard by slowing iPhone sales and that this contributed to the lack of proper messaging and explanation. 

AAPL: The Value Stock

After a year-long expectations reset that saw a 30% drop in Apple's stock price, there is evidence that AAPL is now considered a value stock on Wall Street. This new distinction is likely playing a big role in Apple's resilient stock price in the face of deteriorating financial trends. As AAPL shares declined over the past year, Apple's shareholder base underwent significant changes. Investors focused on earnings growth sold their shares to investors primarily focused on value. This shareholder rotation has led to the average Apple investor now holding very different opinions about Apple and how to judge financial success.

In what will likely end up serving as a symbol of Apple's newly appointed value stock distinction, Todd Combs and Ted Weschler of Berkshire Hathaway bought a $1 billion stake in Apple in 1Q16. To have such value-oriented investors take a stake in Apple is quite telling. 

Implications

There are three key implications related to AAPL now being considered a value stock. 

Different Expectations. Given shareholder rotation, Apple now faces different expectations when it comes to its financial performance. Wall Street has become much more accepting of what was once unimaginable only a few months ago, iPhone sales declines. This change in expectations will play a role in Apple's earnings release next week as the company is expected to report the weakest quarter for iPhone sales performance since the product launched in 2007. (My complete Apple 3Q16 earnings preview is available here.)

Management Flexibility. Given the amount of shareholder rotation and reduced expectations surrounding growth, Tim Cook and Luca Maestri have been given additional time by Wall Street to position Apple for a post-iPhone era. It remains in Apple's best interest to come up with a long-term narrative that includes some aspect of hardware but is not based on unit sales. 

Capital Management. Apple's capital management strategy will play an increasingly important role for value-oriented investors. Apple's announcements surrounding annual dividend increases and share buyback authorization changes will likely garner much more interest. 

The Big Picture

Apple's stock has hit rough stretches in the past. However, this most recent drop and the resulting expectations reset stands out from the rest. For the first time, Wall Street is dealing with an Apple with declining revenue. In the past, Apple revenue declines were only discussed as worst-case scenarios. The reality of what is taking place in the iPhone business has likely shaken the AAPL investor base much more than any other event over the past 15 years.

While it may seem like this kind of expectations reset should have resulted in more than a 30% drop in Apple's stock price over the past year, I suspect Apple's share buyback program will provide some much-needed answers. Management has been buying back approximately $30 billion of AAPL shares per year. This is an unprecedented amount of buyback. Since 2012, Apple has bought back 16% of its shares outstanding. These were shares formerly held by investors with expectations of Apple as a growth stock. Accordingly, Apple's buyback program played a role in rotating Apple's shareholder base by removing shares that would have otherwise been sold to other investors in the marketplace. 

One sign that a company's expectations have truly been reset is that company's stock price increases on negative news. This serves as an indicator that the investor base has likely been flushed out and expectations have been reset. We will be able to test this theory when Apple releases its 3Q16 earnings report. 

Receive my analysis and perspective on Apple throughout the week via exclusive daily updates (2-3 stories a day, 10-12 stories a week). To sign up, visit the membership page. 

This is one of my favorite things. Olliclip makes many lenses for iPhone, but this is by far my favorite. When I travel, it just wear it around my neck. When I am shooting photos with the iPhone, I can grap the lens with my other hand and remove it from the lanyard.

The Active Lens has two lenses: a 2x telephoto and an ultra-wide lens. And they work on both cameras, main camera (iSight) and front facing (FaceTime). You will hardly ever need the telephoto on the front facing camera, but it'S great to get closer to the action if you can't walk there.

I have a Trek commuter bike (8.4 DS). I bike 5.5 miles to work, and up to 15+ miles home. I’m considering the switch to tubeless. Most of my commute (all but 1.5 miles) is on a well-maintained concrete trail. For that 1.5 miles, I’m in gutters or hugging them on residential and downtown streets. I’ve had no flats in the past year of commuting under those or similar circumstances. But I’ve read and heard that I may get better performance from tubeless tires. I’ve also heard the opposite. My Armadillo (Kevlar™) tires have probably seen around 2000 miles over the past 6 months, and my back tire is going bald (under my weight and the additional 5-to-15 lb. load of my pannier). With that in mind, …

• Will I recognize any worthwhile performance advantage from switching?
• Do tubeless tires tend to go bald any faster or slower than Kevlar tires?
• Could I expect more or less flats with tubeless than with my Kevlar tires?

P.S. In case weather is a consideration, we have little enough snow and rain here in Topeka, Kansas, that I am not worried about weather-related performance issues.

One of the most exciting rumoured features of the Galaxy Note 7 is its iris scanner. It brings with it an entirely new security feature for the company and has the potential to usher in a new era of biometric identification, considering Samsung’s global popularity.

Of course, many believe iris scanning is a gimmick which in reality will prove to be more inconvenient than passing your thumb over a sensor.

Only time will tell who is right and wrong in this argument, but a new leak from a Chinese media platform allows us a closer glimpse at this new, potentially-groundbreaking technology.

The video shows a user entering the lock screen and the phone immediately capturing and registering his eyes. With impressive zip, the device unlocks in less than a second. Yet to be seen, of course, is the false acceptance and rejection rate, which is the most crucial element of any security feature.

Underneath the video portion that captures his eyes at the top of the screen, the bottom is dedicated to a back-up pattern code.

Another interesting elements of the video is that the user doesn’t have to get very close to the camera. He is far enough away from the handset that the video captures his hairline and most of his nose, in fact.

Does this mean that the scanner is based more on eye structure than iris? We’ll have to wait until the Note 7’s August 2nd launch to find out.

In addition to the iris scanner, the Note 7 is rumoured to contain Qualcomm’s Snapdragon 821 SoC and up to 6GB of RAM (at least, that’s the hope, though there’s been some doubt), as well as 64GB internal storage plus MicroSD and UFS hybrid memory card slots, a 12-megapixel dual camera, 5-megapixel front-facing camera, a 3,600mAh battery and IP68 water resistance rating.

Related reading: Dual-edged Galaxy Note 7 appears in new video leak

Hello, SUMO Nation!

Chances are you have noticed that we had some weird temporal issues, possibly caused by a glitch in the spacetime continuum. I don’t think we can pin the blame on the latest incarnation of Dr Who, but you never know… Let’s see what the past of the future brings then, shall we?

Welcome, new contributors!

• MoraisPuttiVinicius
• KennethGKendall
• Meeta
• thomas_wilson
• the.Derf
• giomanetta
• Naresh Prajapat
• nasvera
• wdot789

If you just joined us, don’t hesitate – come over and say “hi” in the forums!

Most recent SUMO Community meeting

• You can read the notes here and see the video at AirMozilla.
  

The next SUMO Community meeting…

• …is happening on the 27th of July!
• If you want to add a discussion topic to the upcoming meeting agenda:
  • Start a thread in the Community Forums, so that everyone in the community can see what will be discussed and voice their opinion here before Wednesday (this will make it easier to have an efficient meeting).
  • Please do so as soon as you can before the meeting, so that people have time to read, think, and reply (and also add it to the agenda).
  • If you can, please attend the meeting in person (or via IRC), so we can follow up on your discussion topic during the meeting with your feedback.

Community

• PLATFORM REMINDER! The Platform Meetings are BACK! If you missed the previous ones, here is the etherpad, with links to the recordings on AirMo.
  
• Ongoing reminder #1: if you think you can benefit from getting a second-hand device to help you with contributing to SUMO, you know where to find us.
• Ongoing reminder #2: we are looking for more contributors to our blog. Do you write on the web about open source, communities, SUMO, Mozilla… and more? Do let us know!
• Ongoing reminder #3: want to know what’s going on with the Admins? Check this thread in the forum.

Social

• A social sprint for Firefox 48 is taking place soon, stay tuned for more news soon!
• Ongoing reminder: We have a training out there for all those interested in Social Support – talk to Madalina or Costenslayer on #AoA (IRC) for more information.

Support Forum

Firefox

• for Android
  • Version 48 is still on track – release in early August.

• for Desktop
  • Version 48 is still on track – release in early August.

• for iOS
  • 5.0 coming next wek! Get your release hats on!
    
  • Remember, it will be a big update – don’t forget the articles for it are ready to be localized!

Now that we’re safely out of the dangerous vortex of a spacetime continuum loop, I can only wish you a great weekend. Take it easy and keep rocking the helpful web!

Christie Neptune, She Fell from Normalcy Still. All images and video courtesy of the artist.

In 1984, the author and Black feminist, Audre Lorde penned the essay, “Age, Race, Class and Sex: Women Redefining Difference,” where a “mythical norm” was defined as “white, thin, male, young, heterosexual, Christian, and financially secure.” Lorde wrote that anyone that exists outside of that identity lives on the margins of “the trappings of power.” In the exhibition She Fell from Normalcy,  artist Christie Neptune, counters those hegemonic idealizations described by Lorde through a sci-fi fantasy that centers around blackness, femininity, and a struggle with depression.

Neptune tells The Creators Project, “I deal with depression and it’s my attempt to reconcile that period. I developed this series of work that validates that experience in the African-American female. Depression is typically stigmatized in communities of color. It’s me speaking out and pulling away at those labels that limit my experience.” She adds, “You always hear this thing, ‘black people don’t get depressed that’s some white people shit.’ I decided to build up a mythical norm that is queued to Audre Lorde’s essay, where she describes how we are trying to live up to standards.”

The Manuscript: Pulling at My Labels

In the video for She Fell from Normalcy, Neptune creates a matrix-like “afro-surrealist” world that tracks the evolution of two black female figures. They begin unconscious, existing outside of society which evokes the discrimination people of color deal with. The figures ask, “Why do I feel this way? Where’s my strength?” and “Why am I not like them?” The figures traverse the space with a desire to leave the world that discriminates against them.  At the end of the film the two figures, whose eyes are mostly watching what’s above, look at each other. In that moment, the figures look at each other devoid of the misconceptions often placed on them.

Shadow of Self: His Luxury 44. 20x20. Inkjet Print. 2014

She Fell from Normalcy, exists a three-part series entitled, Eye of the Storm that Neptune began in 2012. The first iteration, Shadow of Self, features prints of two white men laying on a platform underneath a picture of a black family smiling and staring at him. It is a critique of the white male gaze. The Manuscript: Pulling at My Labels, Neptune presents a text modeled to appear as a computer chip where she removes the race and gender identities that often goes before the word “artist” in descriptions of artists of color. She writes, for example, “I am not a Black Artist. I am an Artist,” and “I am not a White Female Artist. I am an Artist.”

She Fell from Normalcy still.

In a two-channel video work that accompanies the manuscript, Neptune can be seen taking photographs of herself while she pull at labels with the phrases from the manuscript. “With the entire project, I am trying to imagine a world where I can pull away the labels that define my personal experience and form my own identity,” says Neptune. “It’s really an investigation of what it could look like if we had this break from the perceptive modes that governor and define the self.” She adds, “I think I tackle that by using white space that is devoid of the social-political systems that historically limits the experiences and marginalizes people of color in this whole decorative thing we call the world.”

See the video here:

Excerpt: From She Fell From Normalcy from Christie Neptune on Vimeo.

She Fell from Normalcy continues through July 30 at Hamiltonian Gallery. For more information, click here.

Related:

Artist Uses Watercolors to Spotlight Black Femininity

Artist Depicts Black Female Nudes as Ancient Egyptian Goddesses

Imagining the Inner Lives of Black Girls

It’s not even close, is it?

Try it for yourself.

Take a recent discussion you’ve had via instant messenger and say it out loud either by yourself or with a (patient) partner. Try to do this at a normal conversational speed.

We took a 30 minute one to one instant messenger discussion on Slack last week and read it out loud at normal conversational pace. It took 3 minutes.

That’s a big 27 minute difference.

If you have 7 working hours each day, that’s 6% of your day wasted. You can extrapolate your own company-wide costs and benefits here (cost per employee time, benefits of your team being 6% more efficient etc…).

It’s clearly quicker to say what’s on your mind than type it, wait for the other person to read it, write their reply, and then have them wait for you to read it etc…No surprises here.

But this changes as a group gets bigger. At around 5+ people, it’s usually quicker to use instant messenger. Multiple people can share and chime in with their opinion simultaneously. We can read quicker than we can listen.

Simple collaboration principles then.

1) If you’re making a simple request to a large group (where any recipient might have the information you need) instant messenger works as a great tool.

2) If you’re having a discussion with a group of 5+ people it may also make sense to go digital.

3) If you’re having a discussion with 2 to 5 people that extends beyond 3+ exchanges (via email, responses etc..) you should always call.

You might be surprised just how much time this simple tweak could help you.

p.s. Can you answer these 10 questions about collaboration?

Hello Mozillians,

Good news! We are having another testday for you This time we will take a swing at Firefox 49.0 Aurora, this Friday, 22nd of July.  The main focus during the testing will be around Context Menu, PDF Viewer and Browser Customization. Check out the detailed instructions via this etherpad.

No previous testing experience is required, so feel free to join us on #qa IRC channel where our moderators will offer you guidance and answer your questions.

I know this is short notice but we hope you will join us in the process of making Firefox a better browser. See you on Friday!

According to a report from Venture Beat, Shopify has launched a chatbot on Facebook Messenger through Kit CRM.

Shopify acquired San Francisco-based Kit, which manages a small business’ social marketing via SMS, in April. Kit was also one of the first companies to announce an integration with Facebook’s conversational platform.

At the time, Kit CEO Michael Perry said that Kit would continue to operate as an independent company. Perry is now the director of Kit at Shopify, and told Venture Beat that the move into Facebook was a positive one for the company. “[With SMS], it was challenging to get phone numbers in particular countries. Messenger solves this problem and allows us to bring Kit to merchant accounts,” Perry said.

With Kit, customers can manage Facebook and Instagram ads, manage email marketing, email customers, and update Facebook their Facebook page.

A nanosecond is a billionth of a second, but we’re not very good with really big or tiny numbers. So, Grace Hopper, the inventor of “the first compiler for a computer programming language”, explains to some eager, young minds with a piece of wire.

Tags: Grace Hopper, nanosecond

It was June, 2003. Lori Douglas was a family lawyer in private practice at a boutique law firm in Winnipeg, Manitoba, where her husband, Jack King, also worked. She was taking a rare break for a hair appointment — it’s a stray detail from the day that’s etched itself in her memory — and when King asked her to cancel it and to meet him for lunch she refused. But then he said, “I’m begging you,” and she relented, worried that something was wrong with him, that he might be seriously ill. Instead, once they sat down, he made a stunning confession. He had sent several sexually explicit photos of her to a client of his named Alexander Chapman, suggesting he join Douglas for a tryst.

Douglas was devastated. She had agreed to pose for King with the understanding that the pictures were confidential, something for her and King alone. She’d never imagined he would show them to anyone, let alone use them without her knowledge or consent to seduce someone else. Then it got worse. King told her that Chapman was threatening to make the images public. Word of the pictures had also gotten out to Winnipeg’s small legal community and some of Douglas’s colleagues were aware of King’s overtures to Chapman. Douglas broke down and left the restaurant in tears. Over the next couple days, she’d learn the rest of the story. In addition to sending the pictures to Chapman, King had posted them on a porn website.

King, who died in 2014, would later reveal he was struggling with depression and would describe what he did to Douglas as “bizarre, ridiculous, stupid, self-indulgent and grotesque… my judgment in this regard had left me.” Full of remorse, King promised Douglas he could take care of it. He paid Chapman $25,000 in exchange for the return of the photos and his silence. King had the website remove the pictures, he destroyed the originals, and even demolished the computer he used to upload them. The couple assumed that the images were gone for good.

If this sounds naïve, and Douglas knows it might, recall that it was 2003. MySpace had just been founded, Facebook would launch a year later, the iPhone wouldn’t be introduced for another four years. For the average non-techie at the time, it would have been entirely possible to believe that the pictures had been erased forever. And so, Douglas forgave King, with whom she had a young son. She pursued her ambition of becoming a family court judge, was appointed to the bench in 2005 and was eventually promoted to associate chief justice.

But in July 2010, Chapman resurfaced. Breaking his confidentiality agreement with King, he filed complaints of sexual harassment against Douglas and King. (Chapman, who is black, also said he was harassed because of his race: King, who was white, had directed him to a porn website devoted to interracial sex.) Chapman also sent the photos of Douglas to journalists, as well as to at least 20 members of the legal community, and the images once again appeared online.

The subsequent inquiry by a Canadian Judicial Council (CJC) committee devolved into a witch hunt. The case centered largely on whether Douglas was fit to sit on the bench and whether she should have disclosed what occurred in her application to become a judge. Her reputation was savaged, with little concern paid to her privacy or her innocence. A dean at a Canadian law school told the media, “If pictures of you naked end up on an internet site, it’s quite difficult to say you have the credibility to be a judge.” In the National Post, conservative columnist Barbara Kay wrote, “I think a woman who wants to be photographed in transgressive sexual mode is giving tacit permission for her transgressions to be shared with others. In old-fashioned parlance, [she is] guilty as sin.” The CJC committee’s proceedings, which were characterized as “unwieldy” and “torturous” by one journalist who covered them, fed the public perception that Douglas was to blame for everything that had transpired. It was a classic case of victim blaming: having her images made public was seen as a fitting consequence for her having had a sex life in the first place.

After four years without a conclusion, Douglas agreed to retire from the bench in order for the proceedings to end. A deeply private person, she had not spoken to the press at all until January this year, when she gave an interview to Canadian Lawyer magazine. She agreed to talk to me for Real Life because she wanted to add her voice to those of other women who had been sexually harassed and humiliated online. She told me that she didn’t want what happened to her to simply disappear. We exchanged several emails and spoke twice by phone, before having a formal interview. The following conversation has been edited and condensed for clarity.

I want to start with your career as a lawyer and later a judge. What was it that appealed to you about the law?

I really just fell into law school, because I didn’t know what else to do. But then I discovered that I loved the practice of law. Family law is often described as being about the “pots and pans,” but I never perceived it that way. I saw the work as having major significance in the lives of all people. And when I applied to be a judge I thought it would be the pinnacle of the career I already created.

What happened when Alexander Chapman came forward with his complaint?

Jack and I thought it had been resolved in 2003 and we thought we could move on. [Chapman] had been paid and Jack got the photos back and he also had written confirmation from the website that the pictures had been removed. I’ve actually never seen the images and I don’t ever want to see them. My name had never been attached to them and Jack assured me that no one would know.

But years after Chapman signed the agreement with Jack, he sued the Winnipeg police force [for malicious prosecution, in a case unrelated to King and Douglas]. He appeared at a pre-trial conference, where there was an attempt to settle the case instead of going to court. He didn’t like the terms of the settlement, and had heard that I was a colleague of the judge assigned to his case. So he decided to go after Jack and me in retaliation.

Jack King always maintained that his wife knew nothing about the images being posted online or about his overtures to Chapman, and there was no evidence that Douglas had ever approached Chapman. Yet despite Douglas’s innocence, the CJC seized upon the point that she had not disclosed the situation between King and Chapman on her judicial application form.

It should be noted that Douglas first applied to be a judge in 2003. At that time, the Chief Justice of the Manitoba Court of the Queen’s Bench was made aware that sexual photos of Douglas had been put online and he initially objected to her candidacy because of this. The chair of the Judicial Appointment Committee was then informed of the incident in 2004, when Douglas’s application to be a judge was once again under consideration. Ultimately, the full committee was consulted and not one member said that the situation should prevent Douglas from becoming a judge.

When you applied to be a judge, after what happened in 2003, did you believe that you had you done anything that was illegal or contravened a professional code of ethics?

There’s no code that says you can’t have sex with your husband. There isn’t a code telling you what kind of sex is okay to have. There isn’t a code saying you can’t take pictures of yourself having sex. Unless you’re doing something illegal, your sex life is private. I didn’t consent to having those pictures posted online, but even if I had agreed, it’s not illegal to do that either.

On the application form, there are a few questions near the end that ask you things like whether you’ve ever been convicted of a crime, or whether you owe income tax. There’s also a question — I’m not quoting it exactly — that asks if there’s anything you wish to disclose that would affect your ability to be a judge, or that would affect the judiciary as a whole. I ticked the box for “no” because I hadn’t done anything wrong. None of this was my doing. And yet that was the sticking point for the CJC.

Given the vagueness of the question, it suggests that it’s up to the discretion and judgment of the applicant to determine what they feel is relevant. As you point out, you didn’t do anything other than have a sex life. Why should you have to disclose that?

Right. The question is then, what should you have to disclose? What if you’d been raped? Would you have to disclose that? What if you’d been beaten by your husband? What if your sibling or father had done something illegal once? Why should you be accountable for those things?

In a written statement to the CJC, Douglas said, “Right thinking people do not conclude that a woman who has been victimized by her husband is to blame for her husband’s conduct and, accordingly, lacks integrity and is not suitable to sit as a judge. If a judge can be disqualified through the malicious actions of a disgruntled litigant or the disreputable conduct of a spouse, when the judge is innocent of any wrongdoing, there is a serious threat to judicial independence.”

Douglas’s lawyers sought an order to have the images inadmissible at her CJC committee of inquiry. However, the committee decided to admit them and view them without Douglas’s consent and despite the fact that admitting them caused her considerable psychological distress. The committee said that the “specific content” of the photos had a “relevant bearing” on whether Douglas was capable to work as a judge.

Was anyone ever able to make to a convincing case that the images had anything to do with your professional life, or with the credibility of the court?

I was represented by three spectacular women lawyers and they wrote boxes and boxes worth of fantastic arguments. Every single one said the same thing: This was not her doing, and this has nothing to do with her ability to sit on the bench. We never denied that the photos existed, but our argument was: (a) I didn’t consent to have the photos made public and (b) they weren’t relevant to the inquiry. Nobody listened.

As a family lawyer and judge, you would have seen cases in which a woman’s sex life would be used against her in a custody or divorce proceeding, or where a woman was a victim of revenge porn by an angry ex. Was there anything that prepared you for what happened to you?  

I was totally alone. There was no precedent. This was new territory for the law. But often in a case conference, which is a meeting before you go to court, in a divorce or a child custody case, I’d have the woman turn to me and say, “Judge, could you do something for me? Could you please ask him to return the pictures to me?” That was common. There was a fear that the ex would use sexually explicit pictures of them at some point down the road to humiliate or punish the woman for the break-up by putting the pictures online, or showing them in court.

Is that what you felt the members of CJC wanted to do to you when they asked to look at your pictures? They wanted to humiliate and punish you?

At the beginning, no. I didn’t believe that people I knew — and I knew those people, I’d had dinner with those people, sat in meetings with them — I didn’t want to believe at all that they would be faulty in their judgment. So it was another form of betrayal. I believed that reason, that rational thought, that the law would prevail and protect me. I remember that at the beginning of the inquiry, I was really upset and panicking about the pictures being made public and my peers looking at them. A very kind male colleague said to me, he said, “Lori, don’t worry. No gentleman would ever look them.” And I remember that I took such comfort in that. Clearly we were both misguided. Eventually I had to accept reality, which was that I, the person, was of no concern whatsoever. I was expendable to protect the view of what the judiciary should be. I was judged by a white male-dominated group of people who had a very particular take on how women should behave and what women should be about.

During the course of our correspondence, Douglas mentioned the 2014 phone hack of dozens of female celebrities, including Jennifer Lawrence, Kate Upton and Gabrielle Union, whose private photos were later released on 4chan. The hack marked a turning point in public discussions about consent and the distribution of intimate images. Several of the women who were targeted characterized it as a sexual assault rather than a theft, and the perpetrator as sex offender rather than a hacker.

This is also how Douglas frames her experience with the CJC, as a sexual assault. She sent me a Canadian legal decision from January, believed to be the first of its kind in the country, in which a man was ordered by the court to pay his ex-girlfriend more than $140,000 in damages after posting an explicit video of her online without her consent. In his precedent-setting decision, the judge ruled that posting an image online without permission (even if the original image was taken with consent) can be considered a form of sexual assault.

I was thinking about Rehtaeh Parsons and Amanda Todd, two Canadian teenage girls who committed suicide after sexual images of them went viral — in Parsons’ case it was images of her sexual assault. In both cases, it was the public exposure and public humiliation that seemed to be particularly devastating for them, that the images were inescapable, that they were forced to relive the trauma every time someone looked at their pictures. It seems like it was similar for you — that what made this so painful was the permanence of images and the reality that you had no control over who saw them. You’ve said that having the images of you made public felt like you were being raped. Can you talk about that?

I want to be clear about my use of the word “rape,” which I use because that is what it really felt like to me. But I don’t want to belittle, or take anything away from any woman who has been actually raped.

That said, the worst aspect of — well, let’s call my experience a kind of assault — was when I realized I was going to be exposed and shamed over and over and over again. It wasn’t going to be a one-time thing. Something like 20 people saw copies of those pictures [aside from those who saw them online]. It felt to me like the way sexual assault is used as a weapon of war, like I was waiting for the next invading army to march through my village and do it to me all over again while they forced my friends and family to watch.

I thought that Jennifer Lawrence expressed it completely correctly, when she said that every person who looked at her hacked photos without her consent was perpetuating a sexual offense. That’s what it felt like to me. Anyone who looked at my photos without my consent, that felt like a sexual assault.

This is a new way of thinking about sexual assault, as something that can happen with an image, not just to an actual physical body. We’re increasingly expressing our identity in the digital world, carrying on relationships online, having sex online, and so on. So it follows that rape and sexual assault can happen online as well. Several jurisdictions have created legislation to deal with revenge porn, Canada passed a cyber-bullying law in 2015 making it illegal to distribute intimate images of a person without their consent, and Twitter and Reddit have also banned the posting of sexually explicit material without the subject’s consent. And just as I was writing this piece, Playboy model Dani Mathers took a picture of a naked woman at her gym without the woman’s consent and posted it on Snapchat with a cruel comment about the woman’s body. After the post went viral, L.A. Fitness banned her from its gyms and reported her to the police — her actions were illegal under California law.

What do you make of how the law is dealing with these sorts of things?

We’re beginning to create laws to address things like online harassment or revenge porn or cyber-bullying, but I’m not sure how effective they can be to stop it from happening. I think public awareness and changing public attitudes is critical. It isn’t harmless to look at something you don’t have consent to look at. If you don’t have permission, don’t look.

The CJC process dragged on over four years. Jack King pled to three counts of professional misconduct in 2011 before a Manitoba Law Society disciplinary panel and was ordered to pay $14,000 in costs, but retained his license to practice law. In 2014, he died of cancer. Soon after, Douglas, exhausted by her legal battle and the loss of her husband, quit her job in order for the inquiry to end. She currently works part-time as a family lawyer and occasionally teaches and lectures.

You forgave your husband and stayed with him until the end. Can you tell me how and why you forgave him?

This is important to me, and I’m never asked about it and I think I was pilloried by some people because I didn’t leave him. There’s no question that I was seriously angry with him. Oh my god, I was so mad that he betrayed me in such a horrendous way. But I forgave Jack in 2003, that’s the key time. He did nothing else after that that required forgiveness. And here’s one reason I could forgive him: I loved him very much and he was very remorseful. Our little boy was four-and-a-half years old in 2003. I wasn’t about to punish my child by leaving his father. Jack also had a daughter and son from a previous marriage and I didn’t want to cause a rift there.

The other reason I forgave him is that I needed to do so for me, so that I wouldn’t poison myself with hate and anger.

Hearing you say that, I’m struck by your generosity and compassion. I also do wonder if you needed to forgive your husband to separate what he did from what you went through at the CJC.

In those last two or three years, during the CJC inquiry, I did yell at Jack because I couldn’t believe what was happening to me. I do remember saying to him at one point, though, “I’m sorry for yelling at you because nothing you did is anywhere as horrible as what they’ve done to me.” He looked at me and said, “Thank you for telling me that.” And it was a fairly profound moment between us. I think he felt horrible and for everything that happened to me, it was worse for him in a way because he did the deed. I don’t think he ever got past it. And I think that didn’t help with his cancer. The guilt added to his pain and stress.

How did you survive?

I suffered terrible panic attacks and I lost close to 20 pounds. I couldn’t eat or sleep. It was debilitating and punishing. I could get up in the morning and get my son’s lunch and homework packed and send him off to school. And then I’d get back into bed and sleep as long as I could. I had so many good friends who called and spoke to me. And if I could speak to one of them and get past noon or 1 p.m., I could function. That’s how I survived. If it hadn’t been for those friends and for the reality of my little boy, I never would have made it. I would have been another suicide statistic. I don’t want other women to go through what happened to me.

I’m angry, very angry still, about what happened to me. My wonderful career and life were thrown away — and I didn’t do anything wrong. With the new legislation, if this had happened now, people could have gone to jail for what they did to me [by transmitting her photos without her consent]. But I also have a realistic approach to it now. And at some point, I found a poem by Carl Sandberg and I keep it with me. It begins, “If I should pass the tomb of Jonah, I would stop there and sit for awhile; Because I was swallowed one time deep in the dark, and came out alive after all.” I guess in truth, that’s how I approach the world now.

Reply to this post

Huawei flat-line passes advantage to Samsung. 

• The first indications for the smartphone market in Q2 16A are pointing a loss of momentum for Huawei, Vivo and Oppo which will put a crimp in their plans to continue their rapid growth in 2016.
• Huawei is of particular note as it has very aggressive plans to become No. 1 in smartphones indicating that it needs to employ a different strategy to continue gaining market share.
• Production volume estimates from TrendForce indicate that Huawei’s share has remained broadly flat at around 9% while both Oppo and Vivo have also barely advanced from the 5% they had in Q1 16A.
• Although, Samsung’s share of production is down a little, I suspect when it comes to sales to end users it has advanced slightly thanks to the popularity of the Galaxy s7.
• The net result is that Huawei is no longer closing the gap on Samsung meaning that its global ambitions are grinding to a halt.
• Huawei is now a comfortable No. 2 in Android but because Android devices are commoditised, that means that I see it making margins of 2-4% in the best instance.
• In order to earn better margin, it must become the No. 1 in terms of volume and outsell its closest rival by a factor of more than 2 to 1.
• It is this volume advantage that allows Samsung to earn 10-12% margins on Android devices which I think is sustainable for as long as it can maintain that volume advantage.
• This is why to be successful, Huawei has got to do far more than just catch Samsung; it must outsell it by more than 2 to 1.
• This will be very difficult to achieve which is why I think that Huawei is also working on differentiating its products through software and services.
• If it can create a good user experience and services that users are prepared to pay something to have access to, then it should be able to make better than commodity margins.
• However, this is easier said than done and I think that Huawei has a lot of work to do before it will be in this position.
• This is why, I continue to believe that its best chance of success remains in China where a tie up with Baidu or Tencent could help it plug the service gap it currently has (see here).
• However, this won’t help in developed markets and here Huawei must do everything that it can to develop the appeal and attractiveness of its Honor brand.
• This will be difficult given the dominance of the Google ecosystem in these markets but I see cracks in Google’s position that might just give Huawei a chance.
• Huawei’s commitment to this strategy will be sorely tested as it is going to be a long and hard road and handsets could easily lose a lot of money before everything comes right.
• For the moment, the advantage has passed back to Samsung, who is also capitalising on popularity of the Galaxy s7 to extend its lead in terms of profitability.
• This is why Samsung rests alongside Microsoft and Baidu as my top picks for 2016.

mkalus shared this story from Consumerist.

Are you in the market for a brand new Video Cassette Recorder? Then you better head to RadioShack — or another electronics store of yester-year — soon, as the last known company to make the video-playing machines will stop production after this month. 

Funai Electric, last known company to make VCRs, will close down its production of the devices in August, ArsTechnica, citing Japanese newspaper Nikkei, reports.

The shrinking market and the difficulty in obtaining needed components were cited by Funai as reasons for winding down the VCR production lines.

Funai, which continues to make the devices for Sanyo, estimates it sold about 750,000 VCRs last year, which is impressive considering most people have turned to streaming services, and before that DVDs and BluRay discs.

That’s a drop in the bucket compared to the 15 million units sold a year at its peak production, sometime after it began manufacturing the devices in 1983.

Last known VCR maker stops production, 40 years after VHS format launched [ArsTechnica]

Overwatch enthusiast Elon Musk unveiled Tesla Motor’s Master plan yesterday, outlining his vision for Tesla and SolarCity, a company of which he is chairman and recently expressed an interest in acquiring.

Musk says Tesla wants to focus on selling integrated energy generation and storage, as well as expand the use of autonomous driving technology in trucks and buses. He also envisions a future where Tesla owners are able to share their vehicles with others in order to continue earning money when they aren’t personally using the vehicle.

In terms of SolarCity, Musk wants to utilize the company’s technology to create an integrated solar-roof-with-battery product. The two companies, according to Musk’ vision, will combine forces to create a singular product, combining the batteries being build at Tesla’s Gigafactory with SolarCity’s expertise in solar power.

When it comes to specific products, Tesla says it has plans to move beyond its current Model S, Model X and Model 3, revealing early plans for a compact SUV as well as a “new kind of pickup truck.” Musk also discussed plans for electric buses, which already exist today and something called a “Tesla Semi” that will reportedly result in a “substantial reduction in the cost of cargo transport.”

When it comes to autonomous driving, Musk says Tesla’s Autopilot system will move out of beta when the system is “10 times safer than the U.S. vehicle average.”

Tesla Master Plan, Part Deuxhttps://t.co/XQeTlsoEdE

— Elon Musk (@elonmusk) July 21, 2016

Additionally, Tesla shared specific plans related to its vision for the future of vehicle sharing.

“When true self-driving is approved by regulators, it will mean that you will be able to summon your Tesla from pretty much anywhere,” writes Musk in the post. Tesla owners will be able to add their vehicles to a “Tesla shared fleet” that allows autonomous vehicles to give rides to the general public. If the system is used enough, according to Musk, Tesla vehicles will essentially pay for themselves.

While an impressive vision for the future of Tesla and transportation, many of Musk’s far-reaching ideas are decades away from real-world implementation. Tesla’s last master plan was released back in 2006.

Image credit: Flickr — Steve Jurvetson

Completely separate from the Section 1201 lawsuit against the Department of Justice, I’m working with the FPF on a project to counter lawful abuses of digital surveillance. Here’s the abstract:

Front-line journalists are high-value targets, and their enemies will spare no expense to silence them. Unfortunately, journalists can be betrayed by their own tools. Their smartphones are also the perfect tracking device. Because of the precedent set by the US’s “third-party doctrine,” which holds that metadata on such signals enjoys no meaningful legal protection, governments and powerful political institutions are gaining access to comprehensive records of phone emissions unwittingly broadcast by device owners. This leaves journalists, activists, and rights workers in a position of vulnerability. This work aims to give journalists the tools to know when their smart phones are tracking or disclosing their location when the devices are supposed to be in airplane mode. We propose to accomplish this via direct introspection of signals controlling the phone’s radio hardware. The introspection engine will be an open source, user-inspectable and field-verifiable module attached to an existing smart phone that makes no assumptions about the trustability of the phone’s operating system.

You can find out more about the project by reading the white paper at Pubpub.

Today I filed a lawsuit against the US government, challenging Section 1201 of the Digital Millennium Copyright Act. Section 1201 means that you can be sued or prosecuted for accessing, speaking about, and tinkering with digital media and technologies that you have paid for. This violates our First Amendment rights, and I am asking the court to order the federal government to stop enforcing Section 1201.

Before Section 1201, the ownership of ideas was tempered by constitutional protections. Under this law, we had the right to tinker with gadgets that we bought, we had the right to record TV shows on our VCRs, and we had the right to remix songs. Section 1201 built an extra barrier around copyrightable works, restricting our prior ability to explore and create. In order to repair a gadget, we may have to decrypt its firmware; in order to remix a video, we may have to strip HDCP. Whereas we once readily expressed feelings and new ideas through remixes and hardware modifications, now we must first pause and ask: does this violate Section 1201? Especially now that cryptography pervades every aspect of modern life, every creative spark is likewise dampened by the chill of Section 1201.

The act of creation is no longer spontaneous.

Our recent generation of Makers, hackers, and entrepreneurs have developed under the shadow of Section 1201. Like the parable of the frog in the well, their creativity has been confined to a small patch, not realizing how big and blue the sky could be if they could step outside that well. Nascent 1201-free ecosystems outside the US are leading indicators of how far behind the next generation of Americans will be if we keep with the status quo.

Our children deserve better.

I can no longer stand by as a passive witness to this situation. I was born into a 1201-free world, and our future generations deserve that same freedom of thought and expression. I am but one instrument in a large orchestra performing the symphony for freedom, but I hope my small part can remind us that once upon a time, there was a world free of such artificial barriers, and that creativity and expression go hand in hand with the ability to share without fear.

If you want to read more about the lawsuit, please check out the EFF’s press release on the matter.

One of the most interesting and unimagineable ideas about the nature of reality in the 21^st century is that we are living in a computer simulation. Philosopher Nick Bostrum posed the question in Philosophical Quarterly (2003) this way: given the enormous computing power of any posthuman civilization, and the likelihood that they would run simulations to better understand their evolutionary history, it is entirely possible that we are living in a simulation created by a higher intelligence. Since Bostrum’s essay was published, many theorists have laid out reasons for entertaining the hypothesis, which are typically grounded in the mathematic nature of our current understanding of the universe. But I think we’re overlooking the most compelling argument in favor of the simulation hypothesis to date: the meteoric rise of Republican presidential candidate Donald J. Trump.

To understand why I think Trump is all the proof we need that we are living in a simulation, we have to begin with one of the most fundamental questions of religious thought, theodicy; if there is a god, why does it allow evil and suffering to exist? Why not create a world of perfect harmony and happiness? Why would god subject its most faithful and righteous of servants, as in The Book of Job, to immense anguish? In Job’s tale, it is essentially to settle a bet with Satan—to prove that Job’s faith does not come from the blessings and wealth bestowed upon him, and will remain strong in the face of loss and sorrow.

The explanations for theodicy range throughout history. Polytheistic peoples saw human suffering as the result of squabbles and power grabs among the gods. The Abrahamic traditions cite original sin and the folly of Adam and Eve. Later religious scholars argued that, in order for humans to be made in the image of god, they must be granted free will, which opened the door to sin. Even the deists explained the problem of theodicy by arguing that god had merely created the universe and then left it to its own devices, thus never intervening on behalf of “the good.”

The simulation argument offers a much more straightforward answer: we’re an experiment. Or, an investigation of sorts, a mode of trying to understand causality and the factors that give certain civilizations some characteristics over others. Some simulations may strive to create the happiest civilization possible, others the most efficient, and still others the most self-destructive. Maybe we just got unlucky. Maybe, in some other file folder on some other hard drive, there is a happy little simulation where everyone gets a free puppy that never grows up and there’s orange soda in all the water fountains.

But we didn’t get the puppy-orange-soda universe. Nope, we got Trump.

Why does Trump prove the simulation hypothesis? First is the naked fact that his campaign is stranger than fiction. Trump operates outside all the bounds of politics-as-physics. He breaks every law that we know of, and yet continues to exist. And, as with the wave-particle duality of light or the hypothetical existence of dark matter, we must alter our explanatory models of the world to understand new phenomena.

Commentators have come up with hundreds of explanations for Trump’s popularity: white working-class discontent, rising xenophobia and persistent racism, distrust in the political class and other institutions, and even the power of name recognition in electoral politics. All fine analyses. But how do you explain this?

Descending an escalator before announcing his campaign for the highest office in the land

Or this?

Even the freedom bird doesn’t like you

Or, perhaps the most bizarre entrance in nominating-convention history, this?

How do you caption this I don’t even know

Doesn’t it all seem a bit over the top?

Actual screen when Trump won the requisite number of delegate votes. For real.

Anyone who has been paying attention to our long history of racism, sexism, xenophobia, and free market ideology should not be surprised that we are witnessing the rise of a political figure who embodies the ugliest characteristics of our society. But who could have predicted that it would look so utterly ridiculous? How can one of the gravest threats to the country in recent memory come in the form of this buffoon?

If we take the computer simulation hypothesis as a possible explanation, Trump is either an experiment or a glitch.

Perhaps Trump was introduced into the simulation to see how the current conditions would interact with this phenomenon. Perhaps our coding overlords took the experiment to the extreme, making the intervention as ridiculous as possible to see the effects. Maybe they’re having a bit of sadistic fun, blissfully ignorant to the fact that it is all too real to us. Or, perhaps, something further back—mass media or reality TV—was introduced, and Trump is a (il)logical conclusion of that earlier experiment.

I, however, lean toward the glitch hypothesis. One of the most basic existential questions in the history of human thought is what is real? Am I real? Is anything outside of me real? During the Enlightenment, we came up with complex models to deal with this question: empiricism, rationalism, scientific method and objectivity. More recently, other subjective, affective questions come to mind, specifically authenticity. From clothing and music to food and drink, the question of authenticity is ever-present. The mere fact that “authenticity” exists as a concept reveals our fear of the unreal, our distain for the seemingly artificial, or, at the very least, our need to value and hold on to something that confirms that we stand on solid, real ground.

This question of reality may be the kernel of the glitch—a tiny voice inside our consciousness that suspects that we aren’t real, a thought born of the unspoken realization that we are simulated.

But rather than an on-off switch, or a spectrum from real to not real, what if it is a circle? What if Trump is the point at which the circle meets, the unreal-real that solves the paradox of simulated reality? Reality TV was an answer to celebrity culture, to viewers wanting to see people like them on screen. As media studies scholars like June Deery have argued, reality TV reveals the fragility of “reality” as a construct, and “that it represents a longing for the real in the age of the virtual and digital” (See Deery’s Reality TV, 2015). Or, if you want to get really heady, see Jean Baudrillard’s work on simulation and hyper-reality. Tl;dr: reality is weird and complicated, especially with the proliferation of media.

Reality TV is not only important with regards to Trump because it’s his biggest claim to fame, but also because it is part and parcel of a larger cultural phenomenon that lays the groundwork for his popularity. For his supporters, he talks like a normal person. He acts like a normal person. He thinks like a normal person.

Trump is an iteration of the feedback loop of reality that is the true original sin of our simulated humanity. Adam and Eve didn’t eat an apple—they asked if they were real. Like a song that gets stuck in your head, we haven’t stopped asking the question ever since. And while some of us look on horrified, asking “Is this really happening?” others cheer for the billionaire every-man who will save our country with unabashed “real talk,” who won’t kowtow to PC culture.

You may remember Magnasanti, the terrifying dystopian SimCity that used mathematical principles to create maximum population efficiency. Sims in Magnasanti lived bleak lives of high unemployment, no social services, and death at the age of fifty under a totalitarian police state. Or maybe you’ve trolled your Sims at some point by filling their house with fireworks, removing the doors, and making them set off the explosives. Or deleting the bathroom until they pee their pants. Or taking them for a swim and removing the ladder until they drown to death. Sure, it’s disturbing that lots of people actively look for ways to torture and kill their Sims, but it’s far more disturbing to consider that we are made in the image of our own “gods,” programmed to create chaos and destruction for a few laughs.

Maybe I’m grasping at straws, trying to explain away the sadness we all feel at the tragedies posted to our daily news feeds. I hope we aren’t living in a simulation. I hope the horrors of this world aren’t some experiment of a higher intelligence trying to figure out how to maximize their own potential. I hope we have some modicum of control over our own reality. I hope that, even if we are in a simulation, we aren’t deterministically coded, and can creatively alter our computerized universe. I hope we can undo this historical moment, before President Trump gets to “restore law and order,” as he ominously promised in his acceptance speech, remotely delivered on a giant screen at the RNC on Tuesday. I hope we can stop it.

I hope nobody pulled up our ladder.

Britney is on Twitter.

Jonathan Chase, LinkedIn, Jul 24, 2016

The first two thirds of this post constitute a pretty good discussion of the Common Core emphasis on close reading (that is, reading where sentence construction and word selection are studied closely in order to understand the author's intent). A good reader reads closely naturally, and instances of ambiguity or errors of reasoning glare red like red scars over the text. But a sole focus on close reading dismisses as irrelevant what the readers themselves bring to the work, rendering it a performance and not a dialogue. "Why should students be denied this same opportunity to 'break away' from the text as they  make comparisons to personally relevant and timely issues related to  a broader and more lively discussion of who and what determines an unjust law," asks Jonathan Chase? This, he suggests, is a result of the focus of Common Core on  outcomes, as defined by standardized testing, rather than on process, where "students’ thoughts and feelings matter a great deal."

- Change your life ride a bike

- Change your life ride a bike

- Change your life ride a bike

- Change your life ride a bike

Here’s occasional PT contributor Chris Keam with impressions of his first date with a Mobi.

Bottom line. If you have to be downtown and move around the core, this is a convenient and cheap way to get around. Essentially 30 cents a day for a year for unlimited trips within the time limit of the package you choose. If you think you’re going to use it, the early-bird deal is a good one. I would recommend snapping it up before it ends (July 31).

He likes it, and so do I.  Mobi is just fun and easy to use.

Cardero and Davie — look for one in your neighbourhood

Ellie Bothwell, Chris Havergal, Times Higher Education, Jul 24, 2016

This article runs through some of the standard pronunciations to the effect that the MOOC is not disruptive, throws out some stats attesting to their popularity, and then shifts into a discussion of what can be done to make MOOCs work, for example, by employing them in the flipped classroom model. Most of the article is structured around a conversation with Stanford University president John Hennessy, which I think explains the focus on traditional education models. The middle part of the article focuses on the Stanford model for universities. "If you look at the threat to most universities, it’ s that their cost model currently grows faster than their revenue model,"  Hennessy says. "So now the question is, can you find a way to introduce technology and help reduce your cost growth?" Which brings us back to MOOCs, and Rick Levin, chief executive of Coursera. "Yale professors, instead of teaching a 15-person seminar three or four times a year, can teach 6,000 people in one sitting," he says.

(Note: to disable the sites limit on articles, search for and delete cookies with the string 'timesh' in your browser.)(The broken image accompanying the article is deliberate; I'm not sure why.)

Introduction

This (long!) blog is based on my forthcoming book:  Data Science for Internet of Things.

It is also the basis for the course I teach  Data Science for Internet of Things Course.   

Welcome your comments. 

Please email me at ajit.jaokar at futuretext.com  - Email me also for a pdf version if you are interested in joining the course

Here, we start off with the question:  At which points could you apply analytics to the IoT ecosystem and what are the implications?  We then extend this to a broader question:  Could we formulate a methodology to solve Data Science for IoT problems?  I have illustrated my thinking through a number of companies/examples.  I personally work with an Open Source strategy (based on R, Spark and Python) but  the methodology applies to any implementation. We are currently working with a range of implementations including AWS, Azure, GE Predix, Nvidia etc.  Thus, the discussion is vendor agnostic.

I also mention some trends I am following such as Apache NiFi etc

The Internet of Things and the flow of Data

As we move towards a world of 50 billion connected devices,  Data Science for IoT (IoT  analytics) helps to create new services and business models.  IoT analytics is the application of data science models  to IoT datasets.  The flow of data starts with the deployment of sensors.  Sensors detect events or changes in quantities. They provide a corresponding output in the form of a signal. Historically, sensors have been used in domains such as manufacturing. Now their deployment is becoming pervasive through ordinary objects like wearables. Sensors are also being deployed through new devices like Robots and Self driving cars. This widespread deployment of sensors has led to the Internet of Things.

Features of a typical wireless sensor node are described in this paper (wireless embedded sensor  architecture). Typically, data arising from sensors is in time series format and is often geotagged. This means, there are two forms of analytics for IoT: Time series and Spatial analytics. Time series analytics typically lead to insights like Anomaly detection. Thus, classifiers (used to detect anomalies) are commonly used for IoT analytics to detect anomalies.  But by looking at historical trends, streaming, combining data from multiple events(sensor fusion), we can get new insights. And more use cases for IoT keep emerging such as Augmented reality (think – Pokemon Go + IoT)

Meanwhile,  sensors themselves continue to evolve. Sensors have shrunk due to technologies like MEMS. Also, their communications protocols have improved through new technologies like LoRA. These protocols lead to new forms of communication for IoT such as Device to Device; Device to Server; or Server to Server. Thus, whichever way we look at it, IoT devices create a large amount of Data. Typically, the goal of IoT analytics is to analyse the data as close to the event as possible. We see this requirement in many ‘Smart city’ type applications such as Transportation, Energy grids, Utilities like Water, Street lighting, Parking etc

IoT data transformation techniques

Once data is captured through the sensor, there are a few analytics techniques that can be applied to the Data. Some of these are unique to IoT. For instance, not all data may be sent to the Cloud/Lake.  We could perform temporal or spatial analysis. Considering the volume of Data, some may be discarded at source or summarized at the Edge. Data could also be aggregated and aggregate analytics could be applied to the IoT data aggregates at the ‘Edge’. For example,  If you want to detect failure of a component, you could find spikes in values for that component over a recent span (thereby potentially predicting failure). Also, you could correlate data in multiple IoT streams. Typically, in stream processing, we are trying to find out what happened now (as opposed to what happened in the past).  Hence, response should be near real-time. Also, sensor data could be ‘cleaned’ at the Edge. Missing values in sensor data could be filled in(imputing values),  sensor data could be combined to infer an event(Complex event processing), Data could be normalized, we could handle different data formats or multiple communication protocols, manage thresholds, normalize data across sensors, time, devices etc

Applying IoT Analytics to the Flow of Data

Overview

Here, we address the possible locations and types of analytics that could be applied to IoT datasets.

(Please click to expand diagram)

Some initial thoughts:

• IoT data arises from  sensors and ultimately resides in the Cloud.
• We  use  the  concept  of  a  ‘Data  Lake’  to  refer  to  a repository of Data
• We consider four possible avenues for IoT analytics: ‘Analytics  at  the  Edge’,  ‘Streaming  Analytics’ , NoSQL databases and ‘IoT analytics at the Data Lake’
• For  Streaming  analytics,  we  could  build  an  offline model and apply it to a stream
• If  we  consider  cameras  as  sensors,  Deep  learning techniques could be applied to Image and video datasets (for example  CNNs)
• Even when IoT data volumes are high, not  all  scenarios  need  Data  to  be distributed. It is very much possible to run analytics on a single node using a non-distributed architecture using Python or R systems.
• Feedback mechanisms are a key part of IoT analytics. Feedback is part of multiple IoT analytics modalities ex Edge, Streaming etc
• CEP (Complex event processing) can be applied to multiple points as we see in the diagram

We now describe various analytics techniques which could apply to IoT datasets

Complex event processing

Complex Event Processing (CEP) can be used in multiple points for IoT analytics (ex : Edge, Stream, Cloud et).

In general, Event processing is a method of tracking and  analyzing  streams  of  data and deriving a conclusion from them. Complex event processing, or CEP, is event processing that combines data from multiple sources to infer events or patterns that suggest more complicated circumstances. The goal of complex event processing is to identify meaningful events (such as opportunities or threats) and respond to them as quickly as possible.

In CEP, the data is at motion. In contrast, a traditional Query (ex an RDBMS) acts on Static Data. Thus, CEP is mainly about Stream processing but the algorithms underlining CEP can also be applied to historical data

CEP relies on a number of techniques including for Events: pattern detection, abstraction, filtering,  aggregation and transformation. CEP algorithms model event hierarchies and detect relationships (such as causality, membership or timing) between events. They create an abstraction of an  event-driven processes. Thus, typically, CEP engines act as event correlation engines where they analyze a mass of events, pinpoint the most significant ones, and trigger actions.

Most CEP solutions and concepts can be classified into two main categories: Aggregation-oriented CEP and Detection-oriented CEP.  An aggregation-oriented CEP solution is focused on executing on-line algorithms as a response  to  event  data  entering  the  system  –  for example to continuously calculate an average based on data in the inbound events. Detection-oriented CEP is focused on detecting combinations of events called events patterns or situations – for example detecting a situation is to look for a specific sequence of events. For IoT, CEP techniques are concerned with deriving a higher order value / abstraction from discrete sensor readings.

CEP uses techniques like Bayesian    networks,    neural    networks,     Dempster- Shafer methods, kalman filters etc. Some more background at Developing a complex event processing architecture for IoT

Streaming analytics

Real-time systems differ in the way they perform analytics. Specifically,  Real-time  systems  perform  analytics  on  short time  windows  for  Data  Streams.  Hence, the scope  of  Real Time analytics is a ‘window’ which typically comprises of the last few time slots. Making Predictions on Real Time Data streams involves building an Offline model and applying it to a stream. Models incorporate one or more machine learning algorithms which are trained using the training Data. Models are first built offline based on historical data (Spam, Credit card fraud etc). Once built, the model can be validated against a real time system to find deviations in the real time stream data. Deviations beyond a certain threshold are tagged as anomalies.

IoT ecosystems can create many logs depending on the status of IoT devices. By collecting these logs for a period of time and analyzing the sequence of event patterns, a model to predict a fault can be built including the probability of failure for the sequence. This model to predict failure is then applied to the stream (online). A technique like the Hidden Markov Model can be used for detecting failure patterns based on the observed sequence. Complex Event Processing can be used to combine events over a time frame (ex in the last one minute) and co-relate patterns to detect the failure pattern.

Typically, streaming systems could be implemented in Kafka and spark

Some interesting links on streaming I am tracking:

 Newer versions of kafka designed for iot use cases

Data Science Central: stream processing and streaming analytics how it works

Iot 101 everything you need to know to start your iot project – Part One

Iot 101 everything you need to know to start your iot project – Part Two

Edge Processing

Many vendors like Cisco and Intel are proponents of Edge Processing  (also  called  Edge  computing).  The  main  idea behind Edge Computing is to push processing away from the core and towards the Edge of the network. For IoT, that means pushing processing towards the sensors or a gateway. This enables data to be initially processed at the Edge device possibly enabling smaller datasets sent to the core. Devices at the Edge may not be continuously connected to the network. Hence, these devices may need a copy of the master data/reference data for processing in an offline format. Edge devices may also include other features like:

•    Apply rules and workflow against that data

•    Take action as needed

•    Filter and cleanse the data

•    Store local data for local use

•    Enhance security

•    Provide governance admin controls

IoT analytics techniques applied at the Data Lake

Data Lakes

The concept of a Data Lake is similar to that of a Data warehouse or a Data Mart. In this context, we see a Data Lake as a repository for data from different IoT sources. A Data Lake is driven by the Hadoop platform. This means, Data in a Data lake is preserved in its raw format. Unlike a Data Warehouse, Data in a Data Lake is not pre-categorised. From an analytics perspective, Data Lakes are relevant in the following ways:

• We could monitor the stream of data arriving in the lake for specific events or could co-relate different streams. Both of these tasks use Complex event processing (CEP). CEP could also apply to Data when it is stored in the lake to extract broad, historical perspectives.
• Similarly, Deep learning and other techniques could be applied to IoT datasets in the Data Lake when the Data  is ‘at rest’. We describe these below.

ETL (Extract Transform and Load)

Companies like Pentaho are applying ETL techniques to IoT data

Deep learning

Some deep learning techniques could apply to IoT datasets. If you consider images and video as sensor data, then we could apply various convolutional neural network techniques to this data.

It gets more interesting when we consider RNNs(Recurrent Neural Networks)  and Reinforcement learning. For example – Reinforcement learning and time series – Brandon Rohrer How to turn your house robot into a robot – Answering the challenge – a new reinforcement learning robot

Over time, we will see far more complex options – for example for Self driving cars  and the use of Recurrent neural networks (mobileeye)

Some more interesting links for Deep Learning and IoT:

• http://stats.stackexchange.com/questions/8000/proper-way-of-using-recurrent-neural-network-for-time-series-analysis
• Noisy Time Series Prediction using a Recurrent Neural Network and Grammatical Inference – https://clgiles.ist.psu.edu/papers/MLJ-finance.pdf
• https://www.quora.com/Can-recurrent-neural-networks-with-LSTM-be-used-for-time-series-prediction
• Time series forecasting with recurrent neural networks http://www.neural-forecasting-competition.com/downloads/NN3/methods/24-NN3_Mahmoud_Abunasr_competition_complete.pdf
• and an article by Sibhanjan Das and me on Deep Learning and IoT with H2O @kdnuggets

Optimization

Systems level optimization and process level optimization for IoT is another complex area where we are doing work. Some links for this

• http://blog.tsia.com/blog/how-iot-process-optimization-can-improve-customer-outcomes
• http://www.intel.co.uk/content/dam/www/public/us/en/documents/white-papers/industrial-optimizing-manufacturing-with-iot-paper.pdf

 Visualization

Visualization is necessary for analytics in general and IoT analytics is no exception

Here are some links

• iotdataviz and also
•  Nick Bearman’s work on R and Spatial analytics

NOSQL databases

NoSQL databases today offer a great way to implement IoT analytics. For instance,

Apache Cassandra for IoT

MongoDB and IoT tutorial

Other  IoT analytic techniques

In this section, I list some IoT  technologies where we could implement analytics

• spatial analytics for IoT  – ex from ESRI
• Implementation of IoT analytics on DSPs
• Augmented reality – ex in conjunction with Pokemon Go
• GPUs – there is some extensive work done by Nvidia in this space especially in relation to Deep learning
• I am also following Apache NiFi with great interest with features like  Birectional flow of data, perimeter of control changes and data provenance

A Methodology to solve Data Science for IoT problems

We started off with the question: Which points could you apply analytics to the IoT ecosystem and what are the implications? But behind this work is a broader question:  Could we formulate a methodology to solve Data Science for IoT problems?  I am exploring this question as part of my teaching both online and at Oxford University along with Jean-Jacques Bernard.

Here is more on our thinking:

• CRISP-DM is a Data mining process methodology used in analytics.  More on CRISP-DM HERE and HERE (pdf documents).
• From a business perspective (top down),we can extend CRISP-DM to incorporate the understanding of the IoT domain i.e. add domain specific features.  This includes understanding the business impact, handling high volumes of IoT data, understanding the nature of Data coming from various IoT devices etc
  • From an implementation perspective(bottom up),  once we have an understanding of the Data and the business processes, for each IoT vertical : We first find the analytics (what is being measured, optimized etc). Then find the data needed for those analytics. Then we provide examples of that implementation using code. Extending CRISP-DM to an implementation methodology, we could have Process(workflow), templates,  code, use cases, Data etc
  • For implementation in R, we are looking to initially use Open source R and Spark and the  h2o.ai  API
    • Make the methodology practical by considering high volume IoT data problems, Project management methodologies for IoT,  IoT analytics best practices etc
    • Most importantly, we will be Open  and Open Sourced
    • There are some parallels with this thinking with Big Data Business maturity model index and to Systems thinking – my favourite systems thinking text is An Introduction to General Systems Thinking – Gerald M. Weinberg
    • When used in teaching for my course, it has parallels to the Concept-context pedagogy (pdf)  where the concepts are tied to the practise in terms of Projects which take center stage

Conclusion

We started off with the question:  At which points could you apply analytics to the IoT ecosystem and what are the implications? And extended this to a broader question:  Could we formulate a methodology to solve Data Science for IoT problems?  The above is comprehensive but not absolute. For example, you can implement deep learning algorithms on mobile devices (Qualcomm snapdragon machine learning development kit for mobile mobile devices).  So, even as I write it, I can think of exceptions!

This article is part of my forthcoming book on Data Science for IoT and also the courses I teach

Welcome your comments.  Please email me at ajit.jaokar at futuretext.com  - Email me also for a pdf version if you are interested. If you want to be a part of my course please see the testimonials at Data Science for Internet of Things Course.  

Sonos customers know the product line, much like Apple customers do. They can give you the full list of options. Sometimes they know the difference between PLAY:1, PLAY:3 and PLAY:5, sometimes they don't. But there is one thing that is really, really hard to understand and to communicate for that matter: what makes Sonos more than a speaker.

There have only been two ways to "get" Sonos. Visit somebody's home or take a leap of faith by trying it yourself, at home. I helped quite a few people to take that leap of faith, and our home has also been a Sonos showroom.

And now there is a third one.

Sonos has built a unique retail experience with their new flagship store in the middle of the New York fashion district, just a few steps away from the iconic Soho Apple Store. 101 Greene Street, New York, New York.

This is not a store you need as a Sonos customer. This is a place for people who fit the profile of a Sonos customer, but who did not "get" it until now.

And here we are again: you don't get the store until you experienced it yourself. At least I didn't. Not from the photos, not from the stories. But now I did. I spent two hours with the people who created the store, and they explained every little detail.

I will try to explain, but I guess I will fail as much as everybody else who did.

While others write bullshit-filled press-releases, Elon Musk tells you exactly what he’s planning, in plain language. Here’s what you can learn from Musk: write directly to the reader, be clear, be organized, and be engaging. Elon Musk’s Master Plan, Part Deux Yesterday, Elon Musk posted “Master Plan, Part Deux” on the Tesla site. It begins like this: … Continue reading How to be as clear as Elon Musk →

The post How to be as clear as Elon Musk appeared first on without bullshit.