Rolandt

When it was revealed that every Government in the world was using technology to spy unlawfully on every other country, there was an arugment that this was acceptable "as long as you have nothing to hide". That argument is, of course, complete rubbish and individual freedoms and rights to privacy are extremely important.

But we've just witnessed in the US another problem with this argument. It's acceptable "as long as you have nothing to hide from every single government and authority that is going to possibly come after". For example, the current Canadian Liberal government might be "ok", but who's going to get elected next. And next? What happens if one of those governments is like the one coming into the US?

There is a chilling example of this in history. In the Netherlands they recorded peoples information in a census and that included peoples religion. In 1941 there were about 140,000 Dutch Jews living in the Netherlands. After the Nazi invasion there were an estimated 35,000 left. Because the Government had census data for them, it was easy for the Nazis to find and eliminate them using that data.

Some 75% of the Dutch-Jewish population perished, an unusually high percentage compared with other occupied countries in western Europe.

Wikipedia

As Cory Doctrow said today:

All those times we said, "When you build mass surveillance, you don't just put power in the hands of those you trust, but also the next guy"

— Cory Doctorow (@doctorow) November 9, 2016

Most of my (admittedly small) readership lives in North America or Europe. That means there's likely a Government surveillance operation monitoring you. It could be internal (in my case CSIS) or external (in my case the NSA). Even worse, they share their information with each other through Five Eyes to get around laws. There is a:

"supra-national intelligence organisation that doesn't answer to the known laws of its own countries"

Edward Snowden, via Wikipedia

The US has just elected someone who repeatedly spews lies, rascism, sexism, hatred and profiling and has said he will go after his enemies [1]. That person will soon oversee the largest surveillance operation in the history of mankind. A surveillance operation that spys on the majority of people reading this blog. That should worry you.

---

[1] By enemies, that's the press, women who accuse him of doing the things he says he does and so on. But you know all this right?

How to make a more readable and more visually accurate map, before you dive into the big transitions. Read More

At RubyKaigi I caught up with Matz, Koichi, and Aaron Patterson aka Tenderlove to talk about Ruby 3x3 and our path so far to reach that goal. We discussed Koichi’s guild proposal, just-in-time compilation and the future of Ruby performance.

Jonan: Welcome everyone. Today we are doing an interview to talk about new features coming in Ruby 3. I am here with my coworkers from Heroku, Sasada Koichi and Yukihiro Matsumoto, along with Aaron Patterson from GitHub.

Jonan: So, last year at RubyKaigi you announced an initiative to speed up Ruby by three times by the release of version three. Tell us more about Ruby 3x3.

Matz: In the design of the Ruby language we have been primarily focused on productivity and the joy of programming. As a result, Ruby was too slow, because we focused on run-time efficiency, so we’ve tried to do many things to make Ruby faster. For example the engine in Ruby 1.8 was very slow, it was written by me. Then Koichi came in and we replaced the virtual machine. The new virtual machine runs many times faster. Ruby and the Ruby community have continued to grow, and some people still complain about the performance. So we are trying to do new things to boost the performance of the virtual machine. Even though we are an open source project and not a business, I felt it was important for us to set some kind of goal, so I named it Ruby 3x3. The goal is to make Ruby 3 run three times faster as compared to Ruby 2.0. Other languages, for example Java, use the JIT technique, just in time compilation; we don't use that yet in Ruby. So by using that kind of technology and with some other improvements, I think we can accomplish the three times boost.

Aaron: So it’s called Ruby 3x3, three times three is nine and JRuby is at nine thousand. Should we just use JRuby?

Jonan: Maybe we should. So Ruby 3x3 will be three times faster. How are you measuring your progress towards that goal? How do we know? How do you check that?

Matz: Yes, that's an important point. So in the Ruby 3x3 project, we are comparing the speed of Ruby 3.0 with the speed of Ruby 2.0. We have completed many performance improvements in Ruby 2.1 and 2.3, so we want to include that effort in Ruby 3x3. The baseline is Ruby 2.0. This is the classification.

Aaron: So your Rails app will not likely be three times faster on Ruby 3?

Matz: Yeah. Our simple micro-benchmark may run three times faster but we are worried that a real-world application may be slower, it could happen. So we are going to set up some benchmarks to measure Ruby 3x3. We will measure our progress towards this three times goal using those benchmark suites. We haven't set them all up yet but they likely include at least optcarrot (an NES emulator) and some small Rails applications, because Rails is the major application framework for the Ruby language. We’ll include several other types of benchmarks as well. So we have to set that up, we are going to set up the benchmark suites.

Jonan: So, Koichi recently made some changes to GC in Ruby.We now use a generational garbage collector. Beyond the improvements that have been made already to GC, what possibility is there for more improvement that could get us closer to Ruby 3x3? Do you think the GC changes are going to be part of our progress there?

Koichi: As Matz says Ruby’s GC is an important program, it has a huge overhead. However, the recent generational garbage collector I don't think has nearly as much overhead. Maybe only ten percent of Ruby’s time is spent in GC, or something like that. If we can speed up garbage collection an additional ten times, it's still only ten percent of the overall time. So sure we should do more for garbage collection, but we have lots of other more impactful ideas. If we have time and specific requests for GC changes, we will certainly consider those.

Aaron: … and resources...

Koichi: Yes.

Aaron: The problem is, since, for us at GitHub we do out-of-band garbage collections, garbage collection time makes no difference on the performance of the requests anyway. So even if garbage collection time is only ten percent of the program and we reduce that to zero, say garbage collection takes no time at all, that's not three times faster so we wouldn't make our goal anyway. So, maybe, GC isn't a good place to focus for the Ruby 3x3 improvements.

Matz: Yeah we have already added the generational garbage collector and incremental garbage collection. So in some cases, some applications, large web applications for example, may no longer need to do that out-of-band garbage collection.

Aaron: Yeah, I think the only reason we are doing it is because we are running Ruby 2.1 in production but we're actually on the path to upgrading. We did a lot of work to get us to a point where we could update to Ruby 2.3, it may be in production already. My team and I did the work, somebody else is doing the deployment of it, so I am not sure if it is in production yet but we may soon be able to get rid of out-of-band collection anyway.

Matz: Yes in my friend's site, out-of-band collection wasn’t necessary after the deployment of Ruby 2.3.

Jonan: So the GC situation right now is that GC is only maybe about ten percent of the time it takes to run any Ruby program anyway. So, even if we cut that time by half, we're not going to win that much progress.

Matz: It's no longer a bottleneck so the priority is lower now.

Jonan: At RailsConf, Aaron talked about memory and memory fragmentation in Ruby. If I remember correctly it looked to me like we were defragging memory, which is addressed, so in my understanding that means that we just point to it by the address; we don't need to put those pieces of memory close together. I'm sure there's a reason we we might want to do that; maybe you can explain it Aaron.

Aaron: Sure. So, one of the issues that we had at, well, we have this issue at GitHub too, is that our heap gets fragmented. We use forking processes, our web server forks, and eventually it means that all of the memory pages get copied out at some point. This is due to fragmentation. When you have a fragmented heap, when we allocate objects, we are allocating into those free slots and so since we're doing writes into those slots, it will copy those pages to child processes. So, what would be nice, is if we could eliminate that fragmentation or reduce the fragmentation and maybe we wouldn't copy the child pages so much. Doing that, reducing the fragmentation like that, can improve locality but not necessarily. If it does, if you are able to improve the locality by storing those objects close to each other in memory, they will be able to hit caches more easily. If they hit those caches, you get faster access, but you can't predict that. That may or may not be a thing, and it definitely won't get us to Ruby 3x3.

Jonan: Alright.

Matz: Do you have any proof on this? Or a plan?

Aaron: Any plan? Well yes, I prepared a patch that...

Matz: Making it easier to separate the heap.

Aaron: Yes, two separate heaps. For example with classes or whatever types with classes, we’ll allocate them into a separate heap, because we know that classes are probably not going to get garbage collected so we can put those into a specific location.

Koichi: Do you have plans to use threads at GitHub?

Aaron: Do I have plans to use threads at GitHub? Honestly, I don't know. I doubt it. Probably not. We'll probably continue to use unicorn in production. Well I mean we could but I don't see why. I mean we're working pretty well and we're pretty happy using unicorn in production so I don't think we would switch. Honestly, I like the presentation that you gave about guilds, if we could use a web server based on guilds, that would be, in my opinion, the best way.

Matz: Yes, I think it's promising.

Jonan: So these guilds you mentioned (Koichi spoke about guilds at RubyKaigi), maybe now is a good time to discuss that. Do you want to tell us about guilds? What they are and how that affect plans for Ruby 3x3?

Matz: We have three major goals in Ruby 3. One of them is performance, which is that our program is really running three times faster. The second goal is the concurrency model, which is implemented by something like Ruby guilds.

Koichi: So concurrency and parallelism utilize some CPU cores.

Matz: Yeah, I say concurrency just because the guild is the concurrency model from the programmer's view. Implementation-wise it should be parallelism.

Koichi: I'm asking about the motivation of the concurrency.

Matz: Motivation of the concurrency?

Koichi: Not only the performance but also the model.

Matz: Well we already have threads. Threads are mostly ok but it doesn't run parallel, due to the existing GIL. So guilds are a performance optimization. Concurrency by guilds may make the threading program or the concurrency runtime program faster, but the main topic is the data abstraction for concurrent projects.

Jonan: OK. So while we are on the topic of threads I am curious. I've heard people talk about how it might be valuable to have a higher level of abstraction on top of threads because threads are quite difficult to use safely. Have you all thought about adding something in addition to threads that maybe protects us from ourselves a little bit around some of those problems? Is that what guilds are?

Aaron: Yes, that's essentially what the guild is, it's a higher level abstraction so you can do parallel programming safely versus threads where it's not safe at all. It's just...

Koichi: Yes, so it's a problem with concurrency in Ruby now; sharing mutable objects between threads. The idea of guilds, the abstraction more than guilds specifically, is to prohibit sharing of mutable objects.

Jonan: So when I create an object how would I get it into a guild? If I understand correctly, you have two guilds - A and B - and they both contain mutable objects. With the objects in A, you could run a thread that used only those objects, and run a different thread that used only objects in B, and then you would eliminate this problem and that's why guilds will exist. But how do I put my objects into guilds or move them between guilds? Have you thought about it that far yet?

Matz: Yeah, a guild is like some kind of bin, a container with objects. With it, you cannot access the objects inside the guild from outside, because the objects are members of the guild. However, you can transfer the objects from one guild to another. So, by transferring, the new objects can be accessed in the destination guild.

Jonan: I see, OK. So the objects that are in a guild can't be accessed from outside that guild; other guilds can't get access to them. Then immutable objects are not members of guilds. They are outside.

Koichi: So immutable objects are something like freelance objects. Freelance objects are immutable, so any guild can access them because there are no read-write conflicts.

Jonan: So you would just use pointers to point to those immutable objects?

Koichi: Yes. Also, I want to note that immutable doesn't mean frozen object. Frozen objects can contain mutable objects. So I mean those immutable objects which only contain children that point to immutable objects.

Jonan: So if we had a nested hash, some large data structure, we would need to freeze every object in that in order to reference it this way. Is there a facility in Ruby right now to do that? I think I would have to iterate over that structure freezing objects manually today.

Matz: Not yet.

Jonan: So there might be?

Matz: We need to provide something to freeze these objects.

Aaron: A deep freeze.

Matz: Yes, deep freeze.

Jonan: Deep Freeze is the name of this feature maybe? I think that would be an excellent name for it.

Aaron: I like deep freeze. (Koichi would like to note that the name for this feature has not yet been determined)

Jonan: I think you mentioned it earlier but maybe you could tell us a little more about just in time compilation, the JIT, and how we might approach that in Ruby 3.

Matz: The JIT is a pretty attractive technology for gaining performance. You know, as part of the Ruby 3x3 effort we are probably going to introduce some kind of JIT. Many other virtual machines have introduced the LLVM JIT. However, personally, I don't want to use the LLVM JIT for Ruby 3, just because the LLVM itself is a huge project, and it's much younger than Ruby. Ruby is more than twenty years old. It's possibly going to live for twenty more years, or even longer, so relying on other huge projects is kind of dangerous.

Aaron: What do you think of Shyouhei’s stuff?

Matz: The optimizer?

Aaron: Yeah.

Matz: Yeah, it's quite interesting, but its application is kind of limited. We have to measure it.

Koichi: I think Shyouhei’s project is a good first step, but we need more time to consider it.

Jonan: Can you explain what it is?

Aaron: Yeah, so Shouhei, what he did was he...

Matz: De-optimization.

Aaron: Yeah he introduced a de-optimization framework that essentially lets us copy old instructions, or de-optimized instructions, into the existing instruction sequences. So he can optimize instructions and if anything happens that would… well, I guess I should step back a little bit. So if you write, in Ruby, 2 + 4, typically the plus operator is not overwritten. So if you can make that assumption then maybe we can collapse that down and replace it with just six. Right?

Jonan: I see.

Aaron: But if somebody were to override the plus method, we would have to not do that class because we wouldn't know what the plus does. And in order to do that, we have to de-optimize and go back to the original instructions that we had before. So, what Shouhei did was he introduced this de-optimization framework. It would allow us to take those old instructions and copy them back in, in case someone were to do something like what I described, overriding plus.

Matz: JRuby people implement very nice de-optimization technologies. They made just such a de-optimization framework on the Java Virtual Machine, so on this topic at least they are a bit ahead of us.

Aaron: Well the one thing, the other thing that I don't know; if you watch the JRuby or JRuby Plus Truffle stuff, if you read any of the papers about it, there are tradeoffs, the JIT isn't free. I mean we have to take into consideration how much memory usage that will require. People hearing this shouldn't think "oh well let's just add a JIT that's all we have to do and then it will be done". It’s much harder, there are more tradeoffs than just simply add a JIT.

Jonan: Yes. So there was an older implementation, RuJIT, the Ruby JIT, but RuJIT had some memory issues didn't it?

Koichi: Yes, quite severe. It needed a lot of memory. Such memory consumption is controllable, however, so we can configure how much memory they can use.

Jonan: OK, so you just set a limit for how much the JIT uses and then it would do the best it could with what you had given it, basically?

Koichi: Yeah.

Jonan: OK.

Koichi: RuJIT can improve the performance of micro-benchmarks but I’m not sure about the performance in larger applications.

Jonan: So, for Rails applications maybe we should call it "Ruby 1.2x3" or something.

Aaron: I think that's an interesting question to bring up because if a Rails application is part of the base benchmarks, are we really going to make a Rails application three times faster?

Matz: We need to make our performance number calculations pretty soon. This is a big problem I think. So maybe some kind of operation such as concatenating...

Aaron: Concatenation, yeah.

Matz: … or temporary variable creation or something like that, we can improve the performance.

Aaron: So, I think it's interesting if we come up with a benchmark that's using string concatenation. I mean we could use an implementation for that. For example, what if we used ropes instead. If we did that, maybe string concatenation would become very fast, but we didn't really improve the virtual machine at all, right? So, how do we balance, does that make sense? How do we balance those things?

Matz: So unlike the typical application, the language can be applied anywhere, so it can be used to write Rails applications, or science applications, or games, so I don't think improving that one thing will necessarily change that situation. So we have to do everything, maybe introducing ropes, introducing a JIT in some form, introducing some other mechanisms as well to see that improvement. We have to do it.

Aaron: So maybe the key is in the benchmarks that we have. We have something doing a lot of string concatenations, something doing a lot of math, maybe something doing, I don't know, I/O. Something like that?

Matz: Yeah. We have to. We cannot be measured by one single application, we need several.

Aaron: Right.

Matz: And then in the Rails application we have to avoid the database access. Just because, you know, access to the database is slow, can be very slow. That we cannot improve.

Jonan: So, along with the JIT, you've also talked about some type changes to coming to Ruby 3 and the optional static types. Can you tell us about that?

Matz: Yeah, the third major goal of the Ruby 3 is adding some kind of static typing whille keeping the duck typing, so some kind of structure for soft-typing or something like that. The main goal of the type system is to detect errors early. So adding this kind of static type check or type interfaces does not affect runtime.

Matz: It’s just a compile time check. Maybe you can use that kind of information in IDEs so that the editors can use that data for their code completion or something like that, but not for performance improvement.

Aaron: You missed out on a really good opportunity for a pun.

Jonan: Did I? What was the pun?

Aaron: You should have said, "What type of changes will those be?"

Jonan: What type of type changes would those be? Yes. I've been one-upped once again, by pun-master Aaron here.

Aaron: I was holding it in, I really wanted to say something.

Jonan: You looked up there suddenly and I thought, did I move on too early from the JIT discussion? No, it was a pun. That was the pun alert face that happened there, good. I'm sorry that we missed the pun. So, to summarize then, the static type system is not something that will necessarily improve performance...

Koichi: Yes.

Jonan: ...but it would be an optional static type system, and it would allow you to check some things before you're running your program and actually running into errors.

Matz: Yeah, and if you catch those errors early you can improve your productivity.

Jonan: Yes, developer productivity.

Matz: Yeah.

Jonan: Which is, of course, the primary goal of Ruby, or developer happiness rather, not necessarily productivity. So, the JIT, this just in time compiler, right now Ruby has ahead of time compilation (AOT) optionally? There's some kind of AOT stuff that you can do in Ruby?

Matz: I don't code with it.

Aaron: “Some, kind of”.

Jonan: OK.

Aaron: It has a framework built in to allow you to build your own AOT compiler. It has the tools in there to let you build an AOT compiler, and I think you wrote a gem, the...

Koichi: Yeah, Yomikomu.

Aaron: Yeah.

Jonan: OK. Yomikomu is an AOT compiler for Ruby. Can you describe just a little bit what that means? What ahead of time compilation would mean in this case? What does it do?

Koichi: Ruby compiles at runtime, so we could store the compiled binary to the file system or something, some database or somewhere. The Yomikomu gem uses this feature, writing out instruction sequences to the file system at runtime, so we can skip the compiler tool in the future. It’s only a small improvement, I think, maybe 30%.

Aaron: 30%?

Matz: 30% is huge.

Aaron: Yeah!

Jonan: That seems like a pretty good improvement to me.

Koichi: I do think so.

Aaron: We just need a few more 30% improvements then Ruby 3x3 is done.

Matz: Yeah, that means 30% of the time is spent in the compiler.

Koichi: Yeah, in 2.3.

Matz: That’s huge!

Aaron: That's what I said!

Jonan: So, rather than JIT, have you thought about maybe like a little too late compiler? We could just compile after the program runs and we don't need to compile it all then. Maybe wouldn’t be as popular as a just in time compiler.

Aaron: One thing I think would be interesting, one thing that I'd like to try someday, is to take the bytecode that's been written out and analyze it. So we could know for example that we can use this trick that Shyouhei’s doing with constant folding. Since we have all of the bytecode written out, you should be able to tell by analyzing the bytecode whether or not... actually maybe you couldn't tell that. I was going to say we could analyze the bytecode and optimize it with code, rewriting an optimized version to disk. But since you can do so much stuff at runtime, I don't know if it would work in all cases.

Koichi: This is exactly what the JIT or some kind of comparable approach aims to do.

Aaron: Yeah.

Jonan: So, cases like you were talking about earlier where this plus can be overridden in Ruby, so what you would do is assume the plus is not overridden and you would just put six, you would actually write that into the bytecode, just the result of this value. Then this framework would allow you to later, if someone did overwrite the plus method dynamically while the program was running, to swap it out again for the old implementation.

Aaron: Yes.

Jonan: OK.

Aaron: So basically the public service announcement is: "don't do that."

Jonan: Don't do that. Don't override plus.

Aaron: Just stop it.

Jonan: Just stop it. You're going to make the Ruby team's life harder.

Koichi: Yes, lots harder.

Jonan: OK. Is there anything else you would like to add about Ruby 3? Anything we didn't touch on today that might be coming?

Matz: You know, we’ve been working on Ruby 3 for maybe two years right now, but we are not expecting to release in a year or even two. Maybe by 2020?

Aaron: Does that mean that we have to wait, are we really going to wait for Ruby 3 to introduce guilds? Or are we going to introduce that before Ruby 3?

Matz: Before Ruby 3 I guess.

Aaron: OK.

Matz: Yeah, we still have a lot of things to do to implement guilds.

Aaron: Of course.

Matz: For example, the garbage collection is pretty difficult. The isolated threads can't access the same objects in that space, so it will be very difficult to implement garbage collection. I think we’ve had a lot of issues with that in the past, so that could take years. But if we’re done, we are happy to introduce guilds into maybe Ruby 2... 6?.

Aaron: 2.6, yeah.

Matz: So this is because we don't want to break compatibility. So if a program isn’t using guilds it should run the same way.

Jonan: So this is how we are able to use immutable objects in Ruby, but they’re frozen objects. They can’t be unfrozen.

Matz: No.

Jonan: OK.

Koichi: Freezing is a one-way operation.

Aaron: Yes.

Jonan: OK. So then, a friend asked me when I described guilds, he writes a lot of Haskell, he asked me when we are we going to have "real immutable objects", and I don't quite know what he means. Is there some distinction between an immutable object in Ruby and an immutable object in a different language that’s important?

Matz: For example in Haskell, everything is immutable, it’s that kind of language, everything is immutable from day one.

Jonan: Yes.

Matz: But in Ruby we have mutable objects, so under that kind of situation we need a whole new construct.

Aaron: Frozen objects should really be immutable. It's really immutable.

Jonan: OK.

Aaron: I don't...

Jonan: You don't know what this person who air-quoted me "real immutable" was saying?

Aaron: Yeah I don't know why they would say "real immutable".

Jonan: Should I unfriend him on Facebook? I think I'm going to after this.

Matz: At least tell him if you want "real immutable" go ahead and use Haskell.

Jonan: I think that's an excellent option, yeah.

Aaron: You just to need to say to them quit "Haskelling" me.

Jonan: I should, I’ll just tell them to quit "Haskelling" me about immutable objects. Well, it has been a pleasure. Thank you very much for taking the time. We've run a little bit longer than promised but I think it was very informative, so hopefully people get a lot out of it. Thank you so much for being here.

Firefox crashes sometimes. This bothers users, so a large amount of time, money, and developer effort is devoted to keep it from happening.

That’s why I like that image of Firefox Aurora’s crash rate from my post about Firefox’s release model. It clearly demonstrates the result of those efforts to reduce crash events:

So how do we measure crashes?

That picture I like so much comes from this dashboard I’m developing, and uses a very specific measure of both what a crash is, and what we normalize it by so we can use it as a measure of Firefox’s quality.

Specifically, we count the number of times Firefox or the web page content disappears from the user’s view without warning. Unfortunately, this simple count of crash events doesn’t give us a full picture of Firefox’s quality, unless you think Firefox is miraculously 30% less crashy on weekends:

So we need to normalize it based on some measure of how much Firefox is being used. We choose to normalize it by thousands of “usage hours” where a usage hour is one hour Firefox was open and running for a user without crashing.

Unfortunately, this choice of crashes per thousand usage hours as our metric, and how we collect data to support it, has its problems. Most significant amongst these problems is the delay between when a new build is released and when this measure can tell you if it is a good build or not.

Crashes tend to come in quickly. Generally speaking, when a user’s Firefox disappears out from under them, they are quick to restart it. This means this new Firefox instance is able to send us information about that crash usually within minutes of it happening. So for now, we can ignore the delay between a crash happening and our servers being told about it.

The second part is harder: when should users tell us that everything is fine?

We can introduce code into Firefox that would tell us every minute that nothing bad happened… but could you imagine the bandwidth costs? Even every hour might be too often. Presently we record this information when the user closes their browser (or if the user doesn’t close their browser, at the user’s local midnight).

The difference between the user experiencing an hour of un-crashing Firefox and that data being recorded is recording delay. This tends to not exceed 24 hours.

If the user shuts down their browser for the day, there isn’t an active Firefox instance to send us the data for collection. This means we have to wait for the next time the user starts up Firefox to send us their “usage hours” number. If this was a Friday’s record, it could easily take until Monday to be sent.

The difference between the data being recorded and the data being sent is the submission delay. This can take an arbitrary length of time, but we tend to see a decent chunk of the data within two days’ time.

This data is being sent in throughout each and every day. Somewhere at this very moment (or very soon) a user is starting up Firefox and that Firefox will send us some Telemetry. We have the facilities to calculate at any given time the usage hours and the crash numbers for each and every part of each and every day… but this would be a wasteful approach. Instead, a scheduled task performs an aggregation of crash figures and usage hour records per day. This happens once per day and the result is put in the CrashAggregates dataset.

The difference between a crash or usage hour record being submitted and it being present in this daily derived dataset is aggregation delay. This can be anywhere from 0 to 23 hours.

This dataset is stored in one format (parquet), but queried in another (prestodb fronted by re:dash). This migration task is performed once per day some time after the dataset is derived.

The difference between the aggregate dataset being derived and its appearance in the query interface is migration delay. This is roughly an hour or two.

Many queries run against this dataset and are scheduled sequentially or on an ad hoc basis. The query that supplies the data to the telemetry crash dashboard runs once per day at 2pm UTC.

The difference between the dataset being present in the query interface and the query running is query scheduling delay. This is about an hour.

This provides us with a handy equation:

latency = reporting delay + submission delay + aggregation delay + migration delay + query scheduling delay

With typical values, we’re seeing:

latency = 6 hours + 24 hours + 12 hours + 1 hour + 1 hour

latency = 2 days

And since submission delay is unbounded (and tends to be longer than 24 hours on weekends and over holidays), the latency is actually a range of probable values. We’re never really sure when we’ve heard from everyone.

So what’s to blame, and what can we do about it?

The design of Firefox’s Telemetry data reporting system is responsible for reporting delay and submission delay: two of the worst offenders. submission delay could be radically improved if we devoted engineering resources to submitting Telemetry (both crash numbers and “usage hour” reports) without an active Firefox running (using, say, a small executable that runs as soon as Firefox crashes or closes). reporting delay will probably not be adjusted very much as we don’t want to saturate our users’ bandwidth (or our own).

We can improve aggregation delay simply by running the aggregation, migration, and query multiple times a day, as information is coming in. Proper scheduling infrastructure can remove all the non-processing overhead from migration delay and query scheduling delay which can bring them easily down below a single hour, combined.

In conclusion, even given a clear and specific metric and a data collection mechanism with which to collect all the data necessary to measure it, there are still problems when you try to use it to make timely decisions. There are technical solutions to these technical problems, but they require a focused approach to improve the timeliness of reported data.

:chutten

This is an update to the guide I wrote in 2009, which as it turns out, is now mostly outdated. So, 2016. Here we go. Read More

Content and Commerce: a big opportunity for Drupal Dries Thu, 11/10/2016 - 16:08

Comments

Add new comment

• Add new comment

• Read more about Apigee Is Now Part of Google

Editor’s Note: This essay originally ran on November 9, 2016 and included a call to politics of affinity. On November 10th, I added to the essay by applying the framework to ongoing protests. 

As the reality of the 2016 election results sunk in, my echo-chamber of a leftist newsfeed was full of two key things: heartbreak and I told you so’s. The heartbroken expressed disbelief that the U.S. would elect a person with an impressive record of bigotry coupled with an appalling record of incompetence. The I told you so’s said they already knew. Not knowing was a sign of privilege, naivety, foolish trust in big data. We should have nominated Bernie, they said. You should have voted, but not for Jill Stein.

Donna Haraway, so keen on blurring boundaries, promotes what she calls affinity politics, vis-à-vis identity politics. Identity politics mobilize around identity labels and the interests associated therein. Affinity politics intersect identities that share the same general agendas. Affinity politics circumvent identity boundaries and conjoin those who dream and work for similar causes. From Haraway’s  Cyborg Manifesto:

Identities seem contradictory, partial, and strategic. With the hard-won recognition of their social and historical constitution, gender, race, and class cannot provide the basis for belief in ‘essential’ unity…

The recent history for much of the US left and US feminism has been a response to this kind of crisis by endless splitting and searches for a new essential unity. But there has also been a growing recognition of another response through coalition — affinity, not identity

The I told you so’s represent identity politics, but so too do some of the heartbroken. In this moment of microcelebrity and performative social media, we are both people and brands. Politics, though always connected to sense of self, becomes a high stakes marker when sharing and posting are both expected and always up for comment and critique.

Today’s critical leftism is about being smart, owning the sharpest and most unexpected take, creating self-distinction and in so doing, creating fissures among those who grapple after the progressive label—fissures between pragmatists and hard-liners, anarchists and party members, collaborators and confrontationists. Identity distinction serves those who tack such distinction to themselves, but aren’t effective in the business of doing. And as is now spectacularly clear, the business of doing is serious.

Understanding the U.S. (and global) hard rightwing pushback makes for a complex puzzle. But let’s start with ourselves, the progressive left. What did we do and how can we do better? Politics of identity are a tangible culprit. The palpable elitism cited by those on the right is not off base. We’re snarky, and self-satisfied, and so caught up in “getting it” that we engage in esoteric conversations of relevance only to ourselves. For those who don’t understand, or who disagree, or who feel ridiculed and left out, Donald Trump is a validating force.

If we make it less about who we are and more about what we need to do, could we approach the business of doing, differently? If we root our movements in politics of affinity, might we recognize that our goals are the same even if the means and the tools might be quite different? I think so. Because affinity politics are animated less by ego and more by shared goals. If we begin at the hoped for end—obtaining national healthcare, ending the police state and its sharp effects within communities of color, replenishing environmental resources, distributing wealth in ways that are both equitable and humane, disconnecting moneyed elites from political elites, establishing a standard of global diplomacy, preserving women’s reproductive rights, and insisting upon safety and full inclusion for LGBQT* persons—we can perhaps have more sober discussions and flexible positions on how best to get there.

So what does affinity politics look like? It looks like meeting those who share similar goals where they are, appreciating their positions and their courses of action, and figuring out how to incorporate those positions and those actions into the tangible change that we all want. It means offering gentle critique and remaining humble and open to those who challenge your perspective. It means remaining firm and also, pliable.

Lest I deal in abstractions, let’s go now to a concerete case: the post-election protests.  Presented in generalities, those protesting represent radical leftism with goals of upturning existing structures. Progressives who oppose the protests represent a more moderate position that maintains value in the democratic system. The latter is a pragmatist approach to change from within. “Let’s be reasonable,” the pragmatists say, while the radicals claim that the time for reason has past. Okay. But we need to do something, so let’s figure it out.

Each of these are valuable positions and important tactics. Both of these groups share the goals of equality and inclusion. Neither group’s positions or strategies negate the other’s. Let’s think about how we can reconcile these divergent logics in a productive way.

My proposal, subject to the tweaks and critiques of memeification in its most collaborative form, is to maintain the protests but reframe them symbolically and tie their symbolism to action items.

Call it a demonstration and/or a unity march instead of a protest.

What’s so valuable about a gathering like this is the symbolism of strength against oppressive powers. The demonstrations across the country make clear that although our electoral system produced a right wing president, that decision far from reflects the sentiment of all the people. This symbolism can be a key source of strength for all of us, but especially those who have been harshly cast into the role of target. A unifying show doesn’t need specific demands. It doesn’t need to protest something. It is instead a demonstration of strength and caring that communicates a powerful message: We are here and we are together and we are ready to get to work.

Insist on accountability instead of calling for obstruction.

Michael Moore’s day after to-do list includes a call to obstruct all legislative action in ways that mirror Obama’s experiences with conservative lawmakers. This sentiment can be remolded into accountability. Rather than preemptively stopping everything (a distinctly political move), we can construct positions on actual policy proposals once we know their content. It is far more effective to make specific demands upon specific legislative documents. Doing so creates an opportunity to collaborate across political identities (even conservative ones) in ways that keep end goals at the center. Plus, maybe Trump will propose something good. I could get behind congressional term limits and would hate to already commit to opposition.

Draft a list of legislative priorities and link them to existing and potential lawmakers who would enact these policies. Support those politicians.

What do we care about and how can we make those things happen? Which senators and representatives support bolstering the ACA? Who has expressed support for Black Lives Matter? Who wants to raise the minimum wage? Bernie Sanders and Elizabeth Warren are obvious allies, but others are too. Kai Degner just lost in my district, but he’s pro-union, pro-LGBQT*, active in campaign finance reform, and an environmental protectionist. I’m certain we can identify others like Degner at the local and state levels and work to lift them up.

Draft a list of legislative redlines and link them to existing and potential lawmakers who would enact these policies. Push back against those politicians and support their adversaries in upcoming elections.

What are we most worried about and who do we think will put those worries into action? Who fought the hardest against marriage equality? Who wants less regulation? Who is adamant about law and order? Mitch McConnell, Sarah Palin, and Rudy Giuliani are known culprits, along with the lesser knowns like Bob Goodlatte, who beat Degner in my district. Just as we lift up those who share progressive values, we can take down those who traffic in oppression.

 

 

*Special thanks to James Chouinard who should probably have shared authorship and was especially helpful thinking through leftist fissures and their tangible consequences. And special thanks to the whole Cyborgology team, whose thoughtful discussion and debate helped me formulate the second half of this essay.

Jenny Davis is on Twitter @Jenny_L_Davis

Headline Pic Via: Source

My Facebook feed is full of people angry at election forecaster Nate Silver of fivethirtyeight.com, declaring him a failure. They’re wrong. Here’s why he was right . . . and why you’re blaming the wrong guy. My argument has three simple points: Nate Silver’s forecast wasn’t wrong, because he didn’t predict a Clinton win. Polling … Continued

The post Nate Silver wasn’t wrong appeared first on without bullshit.

Earlier this week, Samsung started its Nougat beta test for the Galaxy S7 and Galaxy S7 edge. With early registrants of the program getting the beta Nougat OTA update and the firmware itself leaking, screenshots showing the new Grace UI on the S7 have been posted by the folks over at SamMobile.  Continue reading →

The CBC recently stirred up some press by sending a legal threat to the developers of Podcast Republic stating that using its publicly accessible podcast RSS feeds constitutes “commercial use” and violates copyright, adding that the developer could discuss a license fee model with the organization, if interested.

This move baffled many, including Michael Geist, Canada Research Chair in Internet and E-Commerce Law at the University of Ottawa, who told the Financial Post: “I don’t get the claim, an app that simply brings in a publicly available feed that you have made public is not something I’m convinced is a commercial use.”

CBC has now issued a statement in an attempt to set the record straight, however.

“Providing RSS feeds doesn’t mean that CBC/Radio-Canada gives a blanket permission for other apps or aggregators to distribute our podcasts, without condition,” stated the Mother Corp.

The organization then went on to name its two main sticking points: advertising revenue and its perceived loss of control over the content it creates.

“CBC/Radio-Canada relies on advertising revenue; we try to monetize our podcasts,” writes the CBC.

“Apps that sell premium accounts, or use in-stream or banner advertising for example, are earning revenue through the content they offer. As a general principle, we think it’s fair for podcast creators to be compensated in some way for the content they create.”

The CBC goes on to state that it’s not all about the money, however, noting that it has no idea what third-party podcast app developers do with its content.

“Are they copying our content to their servers and serving cached versions? What’s their business model? Do our analytics capture their traffic? Is there a competitive conflict between our ads and the ads being displayed on their apps?”

The CBC highlights one instance of their podcast content appearing alongside “ads for pornography” as a particularly undesirable situation, and concludes the statement by saying that while it welcomes conversations with third-party podcast apps, it is within its right to define appropriate or inappropriate use.

Related: Ontario-based Caddle receives investments from four dragons on CBC’s Dragons’ Den

Going into Microsoft’s fall Windows 10 event in New York City on October 26th, the company was not expected to announce a new Surface Book or Surface Pro. However, in something of a surprise announcement, Panos Panay, corporate vice president of devices at Microsoft and head of the Surface team, unveiled the Surface Book i7, a new high-end version of the company’s existing Surface Book two-in-one.

On the surface (ahem), the new laptop, available in Canada starting today, looks exactly like the Surface Book Microsoft announced more than a year ago. Inside, however, it features a number of improvements over its predecessor. Most notably, it adds a sixth generation Intel Skylake Core i7 processor, discrete Nvidia Geforce GTX 965M GPU, up to 1TB of internal storage, a higher capacity battery and a more efficient cooling system.

Microsoft sent MobileSyrup a Surface Book i7 to review. Given that we’ve had our review unit for less than a week and the fact that the Surface Book i7 represents, at best, a minor refresh of the Surface Book lineup, we thought it best to keep our thoughts on the device brief. We’ll have a lot more to say when Microsoft releases the Surface Book 2 and Surface Pro 5.

Specs

• Windows 10 Pro
• Sixth generation Intel Core i7 processor
• 256GB, 512GB or 1TB SSD
• 8GB or 16GB of RAM
• 13.5-inch PixelSense Display, 3000 x 2000 resolution
• Nvidia GeForce 965M GPU with 2GB of GDDR5 RAM
• 802.11ac Wi-Fi wireless
• Two full-size USB 3.0 ports, as well as a SD card reader and headset jack
• Surface Pen included

When the usually charismatic Panos Panay unveiled the Surface Book i7, he was more sarcastic than his usual self. Speaking to the two-in-one’s more capable Nvidia GPU, he said, “Gamers, they want more frame rates… I don’t really get it, but they want it.” Later, talking about mechanical engineers and their needs, he stated, “Engineers, they want to spin more parts in CAD and they don’t want any lag. I don’t get that either, but they do.”

In hindsight, I think he was trying to say Surface Book i7 has far more computing hardware than the average person will ever need.

Panay called Surface Book i7 “the most performative” 2-in-1 on the market, and while I haven’t had an opportunity to test it against the field, I can say it’s overkill for most computing tasks — though it’s overkill in the best way possible.

Surface Book i7 makes using apps like Adobe Photoshop and Lightroom a pleasure. It’s also able to play modern games like Overwatch with relative ease — though given the amount of heat generated by the computer when playing said games, it’s probably best to look to a desktop solution for more serious gaming.

All that performance comes at a significant cost, however.

In Canada, the Surface Book i7 starts at $3,129. The most expensive model, which comes with 16GB of RAM and a 1TB SSD, costs a staggering $4,379. Moreover, despite their expensive price tag, Surface Book i7 models do not come Intel’s new Kaby Lake processors nor one of Nvidia’s 1000 series GPUs.

Both these components are a significant improvement over their predecessors. Nvidia’s new graphics cards, in particular, deliver the best price-per-performance ratio of any 3D cards on the market. Taken together, it feels like the Surface Book i7 is a something of missed opportunity in this respect.

It’s also a disappointment in terms of battery life. Microsoft boasted about the computer’s improved battery life, touting up to 16 hours of up time. While it may be possible to get 16 hours of battery life out of the Surface Book i7 by aggressively managing its power consumption, at best I was able to get four to five hours of battery life while doing things like surfing the web, listening to music on Spotify and editing photos in Lightroom.

That said, its less than stellar battery life performance is at least understandable given its many high-end components.

One other issue I will note — and its a minor one at that — is that while the Surface Book i7 possess an excellent display, with brilliant colour reproduction, its 60Hz refresh rate can be a bit jarring. Surface Book i7 is a powerful machine and yet switching between virtual desktops and windows can feel sluggish. That said, I’m attuned to these difference because I own and use a 144Hz monitor at home. Most consumers, having not used a 120Hz or 144Hz monitor before, will be happy with the Surface Book i7’s display.

Performance aside, this is the same laptop Microsoft released more than a year ago. While nowhere near as divisive as Apple’s new MacBook Pro lineup, the Surface Book does have its detractors, thanks to its unique hinge mechanism.

Microsoft didn’t mention this during its keynote, but the Surface design team was able to narrow the gap between the top of the device and its base. It’s still there, but based on what I’ve seen with my unit, the Surface Book i7 should be better at keeping dust and dirt away while closed.

Surface Book i7 on top; 2015 Surface Book on the bottom. Notice the thicker base of the Surface Book i7.

Look closely and you’ll also notice that the Surface Book i7 features a slightly thicker base than the 2015 Surface Book. The former is also heavier than the latter: 3.63 pounds compared to the original’s 3.48 pounds.

Expensive price tag and minor issues aside, the Surface Book i7 is a phenomenal computer.

If you find yourself in its target market and are in desperate need of a new machine, then you could do far worse than the Surface Book i7. However, if you can wait about a year, then it’s probably you bide your time. Given just how much the Surface Pro has improved in the short time it’s been on the market, I imagine the Surface Book 2 will improve upon the Surface Book in a number of significant ways.

Indeed, for everyone else this computer is most exciting for the engineering feat it represents. Using almost exactly the same frame as the 2015 Surface Book, the Surface team was able to cram several new components. It’s likely that experience will help inform the team’s future products, devices that will be more affordable to the average consumer.

Pros

• Great typing experience
• Beautiful display
• Monster performance

Cons

• Average battery life
• Very expensive

Starting today, iPhone and iPad users in Canada can send PayPal money transfers to their friends and family members by simply talking to Siri.

The functionality, available in both English and French, is limited to iPhones and iPads running iOS 10. To initiate a money transfer, all you need to say is something along the lines of “Hey Siri, send Bill $50 using PayPal.”

The added functionality comes ahead of a month in which PayPal predicts people around the world will complete more than 17 million peer-to-peer transactions.

In October, PayPal signed a deal with Facebook to allow its users to send peer-to-peer transfers via the social media giant’s Messenger app.

Download the PayPal app from the iTunes App Store.

Related: PayPal rolls out support for mobile payments via Facebook Messenger

Canada’s LTE networks remain among the most readily accessible in the world, according to OpenSignal’s latestT State of LTE report.

OpenSignal’s network quality report comes three months after its Global State of Mobile Networks report was released in August. Compared to the latter, which ranked network speeds across the world, this new report looks at global LTE penetration.

While South Korea still has the highest 4G network availability at 95.71 percent, Canada was able to sneak into the top 20 with 75.42 percent nationwide LTE availability. For the sake of comparison, the United States almost misses the top ten countries for LTE availability in a list of 78 nations.

However, when it comes to the speed of those networks, the numbers change significantly. For instance, with network speeds averaging 45.86 Mbps, Singapore dethrones South Korea from its top spot, and the U.S falls to eleventh from the bottom with speeds of 13.03 Mbps.

Canada, on the other hand, narrowly misses the top 20 in this category with recorded speeds of 26.55 Mbps. While the previous LTE network quality report released by OpenSignal in February, 2016 included carrier ratings, this one does not.

In February, Rogers was ranked the 29th carrier in the world for comprehensive LTE coverage, whereas Videotron was ranked 31st, Telus was ranked 48th and Bell was ranked 58th.

Looking at a map of the world truly puts into perspective the rates at which some countries are powering ahead, whereas others, such as the U.S, are still pulling unexpectedly low numbers.

The report also includes a global map that rates the 78 countries on a colour scale that runs from light green (slow/inconsistent coverage) to dark blue (fast/consistent coverage). When the toggle is set to measure speed, Canada falls near the middle of the scale with a grainy blue filling, whereas in regards to availability, Canada falls closer to the high end of the scale with a dark blue filling.

This report makes clear that while 4G technology is progressing around the world, speed doesn’t mean consistency and vice versa. The U.S. is a prime example of these drastic parallels with near perfect consistency and yet among the poorest network speeds of the countries listed.

Globally, however, LTE speeds average approximately 17.4 Mbps, with Costa Rica ranking the slowest at 5.82 Mbps and Singapore ranking the fastest at 45.86 Mbps.The last 30 countries on the list recorded speeds of below 20 Mbps.

In terms of availability, Sri Lanka recorded the least consistent coverage at 40.27 percent, while South Korea recorded the best coverage at 95.71 percent. Just the last eight countries on the list recorded coverage levels of less than 50 percent.

The methodology for this report was collected from regular consumer smartphones under conditions of normal usage. For this particular report, 17,065,628,107 data points were collected from 500,687 users during the three-month period between July 1st  and Sep. 30th, 2016.

OpenSignal predicts that by the time the next LTE report is released, several countries in Africa and Asia could be added to the list. For countries already on the map, the next report could see countries pushing speed beyond 50 Mbps.

Related: Canada just misses top ten world’s fastest mobile networks, says OpenSignal

Images courtesy the artist

The Creators Project team thought we’d give you some much needed R&R from the headlines, so we’ve roped together a regimen of healing and happy stories to help get you through the day.

Neon colors explode and transmute abstract patterns in the expressive psychedelic prints of digital artist Zouassi. The massive archive of work by this mysterious glitch artist ranges from the abstract and decorative to the expressive and naturalistic. However, it's hard to categorize his work as a whole because no two canvases look the same. Each colorscape is contained within a distinct pattern or mixture pool that explores different modes of texture and composition.

Faced with no budget for software or formal training, Zouassi leverages the power of the internet to help establish an avenue for his creative expression. “I try to utilize the public domain, free software, and social sharing to get the best work possible in front of as many eyes as I can, without an art background and not much more in terms of resources than an iPhone,” he tells The Creators Project.

“App developers in the past few years have really spoiled us with some powerful tools. The ability to create renderings and take high res photos, edit and modify them in countless ways, and immediately share them all over the world from a tiny little device still blows my mind.” The artist praises public domain galleries like Getty Images for the exposure and access to resources they provide prospective image makers. Check out some more of our favorite Zouassi prints below:

The artist has a stacked Society6 page where he pedals everything from throw pillows to iPhone cases stamped with his unique prints. You can check out his online store here, and be sure to follow him on Instagram and Facebook.  

Related:

Psychedelic Neon Fluids Float in a Stunning 4K Video

The Sober Psychedelic World of Archan Nair | High Art

The Sacred and Psychedelic Multimedia Art of Lala Abaddon

Image courtesy of the artist

Earlier this year it was revealed that sculptor and color-hoarder Anish Kapoor had been given exclusive rights to the blackest black in the world. Called Vantablack it was developed by British company NanoSystem—specialists in nanomaterials—who created it for military and scientific uses. However, after Kapoor contacted the company he was allowed to be the only artist in the world given permission to paint with it. Created with carbon nanotubes it is able to absorb 99.96% of visible light.

So if you wanted to get hold of some yourself, possibly to turn your living room into an abyss, that wasn't happening. Kapoor also told The Guardian there's an issue with producing sufficient quantities to paint with anyway because it's so dense. Still, if you felt slighted by the exclusivity bestowed on Kapoor, then you might enjoy this retort by British artist Stuart Semple. 

Image courtesy of the artist

Semple has released his own brand of pink paint called PINK. It's not just any old pink paint though, it's the world's pinkest pigment and is available to everyone...except Anish Kapoor.

"When I first heard that Anish had the exclusive rights to the blackest black I was really disappointed," Semple tells The Creators Project. "I was desperate to have a play with it in my own work and I knew lots of other artists who wanted to use it too. It just seemed really mean-spirited and against the spirit of generosity that most artists who make and share their work are driven by. I thought a good comment would be if I made a paint that was available to everyone but exclude him from using it. That way he can have a taste of his own medicine!"

When you go to purchase the paint, which is available at culturehustle.com for £3.99 ($4.95) for 50g of powder, you are required to make a legal declaration at the checkout stating that "you are not Anish Kapoor, you are in no way affiliated to Anish Kapoor, you are not purchasing this item on behalf of Anish Kapoor or an associate of Anish Kapoor. To the best of your knowledge, information and belief this paint will not make its way into the hands of Anish Kapoor.”

Image courtesy of the artist

As for why the color pink, Semple says he uses very vibrant colors in his work and this pink is his most distinctive and brightest, so thought that would be the best one to share. "It's the best one I've got," he says.

Semple developed the color (among others) over the last decade working with paint manufacturers from across the globe, looking to create the most vibrant colors for his work. The more a paint reflects light the more fluorescent it becomes, and Semple's PINK is very reflective, meaning it gives off a very powerful fluorescence. "The pink already chucks out bucket loads of light but then we made it water soluble so it can be used as a paint and that took it up a notch." explains the artist. "Basically, now it pretty much glows. Whilst Anish's absorbs nearly all light, mine reflects it, kind of the antithesis of what his is doing."

So what does Anish Kapoor think of all this? Has Semple told him? Will there be some kind of paint-off between the two colors? "One of my friends is good friends with a friend of Anish's and apparently he's desperate to get his hands on some PINK, " notes Semple. "But I'm not budging till he shares the black. He started it!"

Image courtesy of the artist

Visit Stuart Semple's website here to learn more about the artist.

Related:

The World's Blackest Black Meets the Vacuum of Space

Anish Kapoor Sculpture Censored—In Gold

Spray Paint Transforms an Abandoned Beach House into a Painting

Grind TV has a nice summary about the Roam Co-working and co-living operation that currently have three properties up and running in Madrid, Miami and Bali. The company, which is promising more locations in the near future, has raised over $3 million dollars to fuel property acquisition and growth around the globe. 

Far more than a hostel or hotel, Roam is creating workspace/livespace communities for the digital nomadic community at relatively modest prices. Given the locations, $500 a week or $1800 a month for a private room and all the co-working facilities is fairly reasonable, especially given the promise of blazing fast, business grade Wi-Fi and Internet connectivity, and a full-time on site manager.

The winds of change in work, travel and residential living are here now, and properties like Roam and WeLive from WeWork are going to become the next phase in work, life, reside. It's only a matter of time and place.

mkalus shared this story from Lunarbaboon.

---

Turncoats* #3: We Got This

The technocrats have triumphed! More than ever, cities are shaped by an expanding army of specialists who, confounded by an onslaught of economic, demographic and environmental pressures, robotically approach city making with a kit of technical solutions. When was the last time you were inspired to linger in one of these formulaic places? What happened to the vision? It’s time to put the responsibility for city making back in the hands of the original generalists: the architects!

The Panel

Leslie Van Duzer, Bruce Carscadden, Neal Lamontagne, Mitchell Reardon

 

Thursday, November 17

6 to 9 pm

Dudoc Vancouver
1445 West Georgia Street – View Map

Attend Event

* Turncoats is a shot in the arm. Framed by theatrically provocative opening gambits, a series of debates will rugby tackle fundamental issues facing contemporary practice with a playful and combative format designed to foment open and critical discussion, turning conventional consensus on its head.

(Note: My “stay positive” attempt)

The most brilliant observation of the state of the nation was the SNL Black Jeopardy piece with Tom Hanks. People have described the election as being one of “insiders vs. outsiders” or “urban vs. rural.” There is truth to both of these observations (and the skit captures the urban vs. rural — spot on).

We live in a land of great diversity. It’s our strength. It’s one of the things that  makes us special.

But when you live in a land of diversity, it’s far too easy to hang labels on people because of our clichéd perceptions of them based on where they live or what they wear or the lifestyle they exhibit.

If I were running a major political party, I’d be looking at the world through the glasses of the Black Jeopardy / Tom Hanks filter and try to figure out how we can take the strength of America’s diversity to address the negative aspects of that diversity.

Clinton lost the election at the point where her frustration led her to say something that confirmed what the people who ultimately elected Trump believe: She views them as deplorable. (Granted, Trump said more heinous things in a day that most people will in a lifetime.)

Below, is a link to a recent Hammock Idea Email that explains the “backfire effect,” how presenting people with the facts can result in them not only disbelieving the facts, but will cause the result opposite of your intention.

Building firewalls around small groups is never the pathway to success. Finding ways to communicate how we are similar, when everything in our gut tells us we’re not, is the pathway to success.

Idea Email: What the Loyalty of Voters Teaches Us About Marketing

As part of its guidelines on malware, phishing and social engineering, Google flags websites that have the potential to harm web users.

However, thanks to a loophole in those guidelines, malicious websites were able to continue harming users, even after getting flagged by Google. They did this by complying with the company’s safe browsing guidelines for a short time, thus lifting any warnings associated with their site, before relapsing into their harmful behaviour.

Google is closing that loophole today.

Moving forward, the company will flag such websites as “repeat offenders,” a categorization the company’s search and browser users will see anytime said websites shows up in a search result.

Google defines repeat offenders as websites that “repeatedly switch between compliant and policy-violating behaviour for the purpose of having a successful review and having warnings removed.” The company does note hacked websites will not be classified as repeat offenders.

Lastly, once a website is flagged a repeat offender, it cannot request a review for 30 days.

Tesla is the main company leading the electric vehicle charge but that is quickly changing.

This shift from gas to battery power is gaining in popularity, encouraging traditional manufacturers to develop new products and offer customers additional choices.

The latest is the Elettrica, a scooter powered by just battery from the Piaggio Group, the makers of the Vespa. The company says it plans to “redesign the mobility of the future” and its two-wheel electric bike will build off 70-years of design history and expertise.

“The Vespa Elettrica will, first and foremost, be a true Vespa, bringing the very latest in electric technology to a vehicle that will retain all the features that have been vital to its success,” notes Vespa on its site. “A perfect mix of excitement, technology and environmental friendliness that only Vespa can create.”

Unfortunately Piaggio has not released additional details related to an official release, pricing, availability, or the range of the vehicle’s battery.

Four years ago we posted on Nate Silver’s, and other forecasters’, triumph over pundits. In contrast, after yesterday’s presidential election, results contradicted most polls and data-driven forecasters, several news articles came out wondering how this happened. It is important to point out that not all forecasters got it wrong. Statistically speaking, Nate Silver, once again, got it right.

To show this, below I include a plot showing the expected margin of victory for Clinton versus the actual results for the most competitive states provided by 538. It includes the uncertainty bands provided by 538 in this site (I eyeballed the band sizes to make the plot in R, so they are not exactly like 538’s).

Note that if these are 95% confidence/credible intervals, 538 got 1 wrong. This is exactly what we expect since 15/16 is about 95%. Furthermore, judging by the plot here, 538 estimated the popular vote margin to be 3.6% with a confidence/credible interval of about 5%. This too was an accurate prediction since Clinton is going to win the popular vote by about 1% 0.5% (note this final result is in the margin of error of several traditional polls as well). Finally, when other forecasters were giving Trump between 14% and 0.1% chances of winning, 538 gave him about a 30% chance which is slightly more than what a team has when down 3-2 in the World Series. In contrast, in 2012 538 gave Romney only a 9% chance of winning. Also, remember, if in ten election cycles you call it for someone with a 70% chance, you should get it wrong 3 times. If you get it right every time then your 70% statement was wrong.

So how did 538 outperform all other forecasters? First, as far as I can tell they model the possibility of an overall bias, modeled as a random effect, that affects every state. This bias can be introduced by systematic lying to pollsters or under sampling some group. Note that this bias can’t be estimated from data from one election cycle but it’s variability can be estimated from historical data. 538 appear to estimate the standard error of this term to be about 2%. More details on this are included here. In 2016 we saw this bias and you can see it in the plot above (more points are above the line than below). The confidence bands account for this source of variabilty and furthermore their simulations account for the strong correlation you will see across states: the chance of seeing an upset in Pennsylvania, Wisconsin, and Michigan is not the product of an upset in each. In fact it’s much higher. Another advantage 538 had is that they somehow were able to predict a systematic, not random, bias against Trump. You can see this by comparing their adjusted data to the raw data (the adjustment favored Trump about 1.5 on average). We can clearly see this when comparing the 538 estimates to The Upshots’:

The fact that 538 did so much better than other forecasters should remind us how hard it is to do data analysis in real life. Knowing math, statistics and programming is not enough. It requires experience and a deep understanding of the nuances related to the specific problem at hand. Nate Silver and the 538 team seem to understand this more than others.

Update: Jason Merkin points out (via Twitter) that 538 provides 80% credible intervals.

mkalus shared this story from Consumerist.

---

You’ve seen it in movies, you’ve seen it on TV shows, and soon, you may see it plugged in to charge: Vespa’s parent company has announced that it’s making an electric version of its classic scooter.

…And that’s about the extent of the information offered by Piaggio Group in its announcement: the Vespa Elettrica project will be a “new mobility solution” that “will, first and foremost, be a true Vespa, bringing the very latest in electric technology to a vehicle that will retain all the features that have been vital to its success,” the company said.

The scooter will have the same “style, agility, ease of use, and riding pleasure,” the company says, just with the addition of some new technology.

As for when the scooter will be available, where it will be sold, and what it will cost, there’s no word yet. Whether it will come in multiple styles and colors or just one, like the brushed metal version accented in blue shown in the photo accompanying Piaggio’s announcement, is also unclear.

I can’t really leave Twitter, because I don’t want someone else to take my username. But I did make my account private and I unfollowed everybody. If you @-mention me, I’ll never know.

Twitter was always a 51-49 thing for me — barely worth it. The company has not dealt with harassment. It has treated its third-party developers shabbily. And, at best, it was just quips and outrages — a diet of candy.

And then it was part of the system that helped elect a fascist President. This tipped it over for me: it’s no longer worth my participation. The shitheads can have it.

Update 2:30 pm: If you want to reach me, you can. My email address is my Twitter username plus the domain for Apple’s email service.

There’s no pretending that this is okay. More and bigger shocks are coming.

Our electorate just repudiated the American dream — the real dream, the idea that we can build a nation dedicated to liberty where all people are created equal.

Instead we decided that we’re a nation like any other. A nation with a ruling ethnic tribe. Nothing special anymore.

It feels like finally, in the end, we lost World War II.

And certainly the post-war order is over: the world shouldn’t trust a country that’s never more than four years away from electing a know-nothing sociopath. Even if Trump has only one term, that we could elect him at all won’t be forgotten.

The free world is unraveling.

* * *

I wanted to write about how I’m done: I’m going to do my best to take care of my family, and that’s it.

I’m 48 years old, and the last couple years have been difficult: we’ve lost family members we were extremely close to, both to slow and difficult illnesses.

I was already exhausted, before this.

But I can’t check out, and neither can you, because we have a responsibility to help other people, however we can. Now more than ever.

We’re going to have to figure out how to blunt the worst of this, and then rebuild the American dream afterward.

But right now it starts with kindness toward other people.

Images courtesy Guided Meditation VR

The Creators Project team thought we’d give you some much needed R&R from the headlines, so we’ve roped together a regimen of healing and happy stories to help get you through the day.

If you're still struggling with your lizard brain instinct to crawl into a deep dark hole, virtual reality might be the cave you're looking for. Cubicle Ninjas is one of several companies offering VR mediation programs promising a break from the harshness of reality. Whether you want to relax on a tropical beach or in a crystalline cavern, their Guided Meditation VR program has you covered. The GMVR website offers, "mind vacations in exotic locations across the universe," that, "bring peace, joy, and calm back into your daily life." 

Josh Farkas, who leads the GMVR team, has made a job of calming nerves and stifling jitters. He does everything from designing virtual environments to answering user questions in online forums. He lent us his expertise  to help us chill:

The Creators Project: How did you become interested in meditation?

Josh Farkas: About 10 years ago I became an entrepreneur full-time. It was such a happy time, but as any one self-employed knows, it can also be a mind-numbingly terrifying rollercoaster of emotions. And when working from home, your place of comfort gets infected with this underlying dread of uncertainty. One day, by sheer chance, I stumbled into meditation, and for the first time in years, I felt completely at peace. Meditation changed my life.

What are the advantages of VR meditation over IRL?

When I found meditation I was so excited to share this with my friends and family. But they all sort of scoffed, "I've tried that and it doesn't work for me." After hearing this repeatedly, I was bummed to realize that often the people who need meditation the most are also those who find it the most difficult: the type-A, ultra competitive, hyper efficient, self-critical, folks who are focused on perfection. They'd tried meditation, didn't receive clear feedback on success of failure, and gave up.

We also heard, a person's home was too distracting to meditate within. With bills in the corner, and all of your preferred technology at an arm's length, it is overflowing with disruptions.

VR has a few built-in advantages to tackle these common problems:

-Virtual Reality is like a vacation for the mind, so the household clutter can truly be left behind. Taking a 10-minute breather in virtual Hawaii or Japan is joyful.

-Real-time biofeedback is available for some devices, so users can visualize the positive results meditation has on their body. Living in your heartbeat is so intimate, it helped me respect my body in a fresh way.

-Users can select from multiple meditation practices, helping them to understand that if a meditation doesn't work it isn't their fault. We can then help them find which techniques work best for them.

How have you personally coped with the stressfulness of this election?

Building new VR environments is pretty soothing! Pretending to be a digital Bob Ross and hiding happy little trees never gets old. Our research has found that people find the opposite of their normal life as most relaxing. If you live on a beach you'll find forests or mountains most effective. This same research applies to temperature, weather, or even accent of the spoken meditation teacher. The mysterious other is comforting. So, if you're feeling stressed, experiencing something new and unique may just help.

Has any of your company's research pointed to the detriments of such a stressful political environment, and the resulting benefits of meditation?

A recent study by the American Pyschological Association identified the top three sources for stress: money, work, and the economy. The results of this election are important, because all three of these sources could be heavily impacted. Millennials and GenX-ers report the highest stress levels of all age groups, and are more likely to engage in unhealthy behaviors due to uncontrolled stress levels.

All stress, whether from politics, family, or even work, is more than just a nuisance. It can cause physical ailments, like headaches, muscle tension, or general fatigue. And it can be even worse for mental health, fueling anxiety, anger, or depression.Meditation is recommended by leading organizations such as the Mayo Clinic because it has repeatedly shown to reduce stress. 

Nature is also shown to restore our focus (known as attention restoration theory), our mood, our self-esteem, our memory, decrease aggression, reduce stress, and even reduce pain. What is remarkable, is that the same restoration effects that we see in real-world nature spaces are also found in virtual reality spaces!

One more question: How am I supposed to stay calm today?

I can't help but think of the quote:

“Holding onto anger is like drinking poison and expecting the other person to die.” - Anonymous

When we realize we're not in control of today's news, but the only true thing we can control is our reaction to this news, then nothing can hurt us.

Learn more about Guided Meditation VR on the official website. Click here for more relaxing artwork.

Related:

I Tried Virtual Reality Meditation Inside a Marijuana Grow Tent

Miami Artist Offers a VR-Inspired Guided Pool Meditation

I Tried the 'Abramović Method' (And I Think I Passed Out)

I could not sleep last night at all. So I organized my notes I’ve been taking over the last year on the problem of doing politics in advertising funded stream-based systems.

I know this election was about so much more than that (so much more), and our problems are so much deeper. But I remain convinced that even if social media is not the fire or the fuel of Breitbartian racism it is in fact the oxygen that helps it thrive and spread.

There are 537 pages of notes in this PDF, and it may not be immediately clear what each has to do with the book, but in my head at least they all relate. They are worth a read.

notes-for-lost-in-the-stream-how-social-media-destroyed-our-democracy