Ashar

PoisonTap is an impressive hacking tool that can compromise computers via the USB port, even when they are password-protected. What's interesting is the chain of vulnerabilities the tool exploits. No individual vulnerability is a problem, but together they create a big problem.

Kamkar's trick works by chaining together a long, complex series of seemingly innocuous software security oversights that only together add up to a full-blown threat. When PoisonTap -- a tiny $5 Raspberry Pi microcomputer loaded with Kamkar's code and attached to a USB adapter -- is plugged into a computer's USB drive, it starts impersonating a new ethernet connection. Even if the computer is already connected to Wifi, PoisonTap is programmed to tell the victim's computer that any IP address accessed through that connection is actually on the computer's local network rather than the internet, fooling the machine into prioritizing its network connection to PoisonTap over that of the Wifi network.

With that interception point established, the malicious USB device waits for any request from the user's browser for new web content; if you leave your browser open when you walk away from your machine, chances are there's at least one tab in your browser that's still periodically loading new bits of HTTP data like ads or news updates. When PoisonTap sees that request, it spoofs a response and feeds your browser its own payload: a page that contains a collection of iframes -- a technique for invisibly loading content from one website inside another­that consist of carefully crafted versions of virtually every popular website address on the internet. (Kamkar pulled his list from web-popularity ranking service Alexa's top one million sites.)

As it loads that long list of site addresses, PoisonTap tricks your browser into sharing any cookies it's stored from visiting them, and writes all of that cookie data to a text file on the USB stick. Sites use cookies to check if a visitor has recently logged into the page, allowing visitors to avoid doing so repeatedly. So that list of cookies allows any hacker who walks away with the PoisonTap and its stored text file to access the user's accounts on those sites.

There's more. Here's another article with more details. Also note that HTTPS is a protection.

Yesterday, I testified about this at a joint hearing of the Subcommittee on Communications and Technology, and the Subcommittee on Commerce, Manufacturing, and Trade -- both part of the Committee on Energy and Commerce of the US House of Representatives. Here's the video; my testimony starts around 1:10:10.

The topic was the Dyn attacks and the Internet of Things. I talked about different market failures that will affect security on the Internet of Things. One of them was this problem of emergent vulnerabilities. I worry that as we continue to connect things to the Internet, we're going to be seeing a lot of these sorts of attacks: chains of tiny vulnerabilities that combine into a massive security risk. It'll be hard to defend against these types of attacks. If no one product or process is to blame, no one has responsibility to fix the problem. So I gave a mostly Republican audience a pro-regulation message. They were surprisingly polite and receptive.

Last fall, the Interior Minister of Ukraine announced the creation of a national Cyberpolice (Кіберполіцію) to protect the country from everything from credit card fraud to malware.  Here’s something that would be great to add to their list: fraudulent BGP routing out of Ukraine.  Last year, we reported on an incident in which Ukrainian ISP Vega hijacked routes from British Telecom (including that of the UK’s Atomic Weapons Establishment), an event that could perhaps be chalked up to an innocent mistake.  However, the fraudulent routing we’re now seeing from Ukraine is deliberately designed to go unnoticed.  We’ll review some of this new behavior in this blog.

Governments take note

The profile of this issue has grown in the past year as governments have had to respond to their address space being fraudulently used.  Last July, the Dutch Minister of Foreign Affairs (pictured right) was confronted with parliamentary questions concerning an incident where “attackers” had commandeered IP address space belonging to the Ministry of Foreign Affairs the previous year.  In that incident, on 18 November 2014, Decision Marketing (AS62228) out of Sofia, Bulgaria began globally announcing eleven BGP routes that did not belong to them.

These routes included the following:
159.100.0.0/17      Transport Research Laboratory                   GB
171.25.0.0/17       Swisscom IT Services AG Sankt Gallen            CH
193.177.64.0/18     Ministerie van Buitenlandse Zaken               NL
193.201.243.0/24    MA3X Ltd.       Sofiya  Sofiya-Grad             BG
193.202.128.0/18    Bayer Business Services GmbH Nordrhein-Westfalen DE
193.243.0.0/17      Cable & Wireless UK P.U.C.                      GB
194.38.0.0/18       RIPE Network Coordination Centre                AU
210.79.128.0/18     Mediatti Communications Inc.                    JP
210.87.64.0/18      Asia Pacific Network Information Centre         AU
80.114.192.0/18     Ziggo B.V.      Amsterdam       Noord-Holland   NL
83.175.0.0/18       Telecom Italia S.p.a.                           IT

The one that caught the attention of the Dutch was 193.177.64.0/18.  Its propagation profile is shown below on the left – note it never was circulated to more than 40% of our peering base.  Decision Marketing (clearly a spamming operation) impressively embeds a Bulgarian accent into their logo with the slogan “We are email marketing company.”

In the following month, the Swiss Governmental CERT announced that it had (with the assistance of Spamhaus) recovered IP address space belonging to a Swiss regional government but being used by spammers.  The graphic below shows the route being originated by the spamming operation (AS62741) on the left disappears on 25 June and returns on 29 June, being announced by its rightful owners, the canton of Fribourg.

In the Dutch Minister’s defense, there isn’t much one can do to completely prevent any entity from announcing the address space of another entity as the routing system is based on trust.  Also the hijacking of unused address space is undoubtedly lower on the priority list than other things facing European governments these days.  Perhaps with available IPv4 address space drying up, might it be getting harder for spammers to squat on unused IP address space without someone noticing?

A Problem in Ukraine

Last October, Dyn’s Scientist Emeritus Jim Cowie was the keynote speaker at ENOG 10 in Odessa, Ukraine.  The ENOG (Eurasia Network Operators Group) covers the Russian Federation, CIS and Eastern Europe and a video of Jim’s presentation is posted below – and is advanced to the portion which covers the fraudulent routing we spotted coming out of the Ukraine.

At the beginning of last year, we published a blog entitled The Vast World of Fraudulent Routing which detailed six different entities deliberately announcing address space that didn’t belong to them.  In Case 5 from that post, we described a perpetrator attempting to mask his fraudulent routing by forging the AS Path to contain what would otherwise appear to be a believable origin for the address space being announced.

In that case, we observed things like unused British Telecom address space being announced by AS5400 (British Telecom’s ASN) according to the AS Paths in BGP data.  To the lay observer this would appear legitimate, however, it was being exclusively transited through a small ISP in Ufa, Russia – a city unlikely to house a branch office of BT.

The activity described in Case 5 disappeared in November 2014, but the next month in December we started seeing something similar out of Kiev, Ukraine, i.e., a new instance of phony, yet plausible AS origins for bogus routes.

200.202.64.0/19 (Brazil Home Shopping Ltd) was one of those routes.  It was routed along the following path:

   ... 9002 8438 18739 10495 11295

If we investigate this route, we can see that it is originated by the rightmost AS on the path AS11295 (Brazil Home Shopping Ltd).  Well that seems to check out — good so far.  Then it goes through AS18739 and AS10495, which are both Brazilian ASNs.  Ok, still looks plausible, right?  But then it exclusively goes through Ukrainian provider Hetman Soft (AS8434) and on to Russian fixed-line carrier RETN (AS9002).  Routes along paths like these are only circulated to a limited set of mostly Russian carriers.

In the past year, we observed this entity announcing the following phony, yet plausible origins (it seems to have a preference for LACNIC resources):

Prefix 187.239.0.0/16 (Uninet, MX) 177.90.0.0/16 (Universidade De Sao Paulo, BR) 200.200.0.0/16 (Embratel, BR) 181.56.0.0/16 (Telmex Colombia, CO) 161.255.0.0/16 (Movistar (Telcel), VE) 177.21.128.0/20 (Netdigit Telecom, BR) 196.3.16.0/20 (Net Uno, C.A., VE) 186.189.224.0/20 (FastBee Argentina S.A.) 186.236.240.0/20 (Prefeitura de Cuiabá, BR) 191.102.224.0/20 (DirecTV Colombia) 177.8.80.0/20 (Centro Int. de Telemática do Exército,BR) ... many more

Plausible, but Phoney Origin AS8151 (Uninet, MX) AS28571 (Univ De Sao Paulo, BR) AS4230 (Embratel, BR) AS10620 (Telmex Colombia, CO) AS6306 (Movistar (Telcel), VE) AS28245 (Netdigit Telecomunicacoes, BR) AS11562 (Net Uno, C.A., VE) AS28028 (FastBee Argentina S.A) AS263638 (Prefeitura de Cuiabá, BR) AS262928 (DirecTV Colombia) AS52890 (Centro Int. de Telemática do Exército,BR)

In case we needed additional confirmation of the location of where these routes were coming from, one could run traceroutes into this address space and get times and paths that were consistent with Ukraine, not Brazil. Such as 20ms from Moscow:

trace from Moscow, RU to 200.202.64.1
1 *                                                                          0.0
2 87.245.229.46  ReTN external interconnections  Moscow          Russia    0.478
3 87.245.233.26  ReTN's Backbone                 Kiev            Ukraine  19.717
4 *                                                                          0.0
5 200.202.64.1   BR HOME SHOPPING LTDA           Belo Horizonte  Brazil   20.419

And 12ms from Minsk:

trace from Minsk, BY to 200.202.64.1
1 *                                                                             0.0
2 *                                                                             0.0
3 93.84.125.194   BELTELECOM                        Minsk           Belarus   4.343
4 93.85.80.54     Republican Unitary Telecommunica  Minsk           Belarus   4.425
5 93.85.80.126    Republican Unitary Telecommunica  Minsk           Belarus   0.984
6 87.245.237.21   ReTN external interconnections    Kiev            Ukraine  12.405
7 87.245.232.173  ReTN's Backbone                   Kiev            Ukraine  12.511
8 *                                                                             0.0
9 200.202.64.1    BR HOME SHOPPING LTDA             Belo Horizonte  Brazil    12.67

As of Friday last week, 200.202.64.0/19 (Brazil Home Shopping Ltd) was still being fraudulently announced out of Ukraine, although the AS path has changed slightly (AS41331 has taken the place of AS8434):

   ... 9002 41331 18739 10495 11295

The individuals involved in this type of activity can be quite brazen.  Aside from having the audacity to announce the address space of the Brazilian Military (Centro Int. de Telemática do Exército) from the example above, earlier this year a new IP squatting operation began hijacking address space of APRICOT 2016, just weeks before the conference was set to begin.  APRICOT is APNIC‘s technical conference which focuses on topics like routing security.  We alerted the conference organizers, who were able to fend off the hijack by getting the perpetrator’s (AS260) upstream (GTT, AS3257) to drop the bad routes.  Dyn’s Director of Infrastructure, Joe Abley then described the entire incident in a lightning talk at APRICOT 2016:

.@ableyjoe presents at #APRICOT2016 on BGP hijacking of #APRICOT2016 itself! Video: https://t.co/TdzU8ZS4JB pic.twitter.com/No9OBfPHgO

— Dyn Research (@DynResearch) February 24, 2016

Although GTT blocked the specific routes hijacking APRICOT 2016 IP space from its customer AS260 (Xconnect24), this entity continues to announce bogus routes via GTT out to parts of the Internet using bogus origins, just as it did with APRICOT.

Unfortunately, fighting this type of activity is difficult because the perpetrators are getting more advanced at hiding their activity from basic BGP analysis, but also because even when nefarious activity is identified and upstream providers are alerted, the fraudulent routes continue to be circulated.  This is why we support the Internet Society‘s Mutually Agreed Norms for Routing Security (MANRS) project and recommend that companies monitor their IP address space (routed and unrouted) with tools like those found in Dyn’s Internet Intelligence family of products. For more information about this type of phenomenon, see last year’s coverage of our analysis in the Washington Post and the Wall Street Journal.

More Fraudulent Routing = More Need for #MANRS https://t.co/fgLdry1Ljq

— Routing Manifesto (@RtgManifesto) March 22, 2016

Update:

This blog post has been translated into Russian and published in the online magazine Internet Inside. (Pg 33)

The post Ukraine Emerges as Bogus Routing Source appeared first on Dyn Research.

There were times when all data were wild... and if it was stored at all, it was committed to a memory of an individual which tended to fade away.  To facilitate this transient storage the data was wrapped in protocols of rhymes and vivid symbolic images; the pictures were drawn, stories were told. Then, about 5,000 years ago the writing systems begun develop – ideographic, symbols and, finally, letters. The data was tamed. The letters made up words, the words made up a sentence, and the sentence, hopefully, made sense. The data was written on clay tablets, animal skins, recorded on papyrus, vellum, paper, magnetic and laser disks… We got quite skillful at butchering the data into neatly organized chunks, and devising ever more sophisticated structures to hold it – scrolls, books, databases.

And then Internet happened. They say that we are creating more data in a year than in the previous thousand years, that 90% of the world’s data have been created in the past two years (though, according to one interpretation of the Law of Information Conservation, we only engage in recycling the information redistributed from existing sources) We are swamped with information, and the old, tried, trusted and true approach of organizing information into a palatable chunks is no longer working. Facing information deluge, we are forced to go back to basics – raw data – and find ways to make use of it without forcing it into a Procrustean bed  of some structure that might have seemed like a bright idea once. Hence the resurrection of an old idea of hierarchical databases – the ones before the advent of SQL – under the guise of NoSQL movement, and much hyped Big Data... In a sense, Big Data is nothing new, it’s been around us as long as humanity itself but just as with the proverbial iceberg, most of it was hidden from our conscious use – which by no means does not mean that we haven’t used it! No, it was always there for us, seeping in from traditions, proverbs, legends – something that we use without consciously thinking about it, the gut feeling, the social norms. The modern Big Data but extends this concept to the computers.

And I believe that the data can take care of itself, if only humans stopped telling it what to do – but we do have to arrange the meeting J

Instead of thinking how to accommodate the new data source or new data formats (e.g  video, mp3 files, text of various degrees of structural and semantic complexity), the humans can let the data figure out how to interpret it by itself.

The new data format could be analyzed, its structure inferred from background information/metadata, its usage from countless examples of similar (or not) data… This will require enormous computing power but we are getting close to it with likes of crowdsourcing, probability scores and machine learning, Hadoop infrastructure, variety of NoSQL and RDBMS data working together to produce insights from the data in the wild, the data over which we have no control, unreliable, inherently “dirty” data  ( and the degree of “dirtiness” itself is a valuable piece of information!)

It is nice to have smart ontology all figured out for the information we are using but it would be hundred times nicer not to pay any attention to any given ontology, and still being able to make meaningful use of the data!

The plan seems bound to further raise tensions in Xinjiang.
Ordinary Chinese social media users have reacted with nonchalance and even some rejoicing to news that Xinjiang, a far western region beset with bloody ethnic unrest, may tighten its family planning policies to curb population growth among minority Uighurs.

China News | 0 Comments

Last week saw the first full release of Qlik Sense. The new product from the people that brought you QlikView. There is already a lot of information about the product out there and this post seeks to guide you through this.

This Is Not QlikView 12

Qlik Sense has been quite a long time in gestation. It has for the last couple of years been referred to as QlikView.next, which has caused some confusion over whether it is superseding the existing product or not. The recent re-branding of QlikTech to Qlik allowed them to have scope for various streams to their offerings under a single company umbrella and the first major example of this is Qlik Sense.

The important thing to remember here is that QlikView will have separate yet parallel existences. Support for and development of QlikView is going to continue for some time yet. A clear vision for how the products differ has now been conveyed, with QlikView still being the product of choice for “guided analytics” where Qlik Sense is more for allowing users to freely explore data and create their own visualisations.

This has been set out clearly by Henric Cronström of Qlik in his recent blog post:
http://community.qlik.com/blogs/qlikviewdesignblog/2014/07/29/view-or-sense

It is good to see that whilst these are separate products that each product will benefit in future from having features moved across from the other – that we will be seeing some of the advanced charting options in QlikView is excellent news.

Great! Where Do I Start?

First of all you will want to be downloading a copy of Qlik Sense Desktop and getting that installed. Presently this is the only version of Sense available, and it is free to use.

You can download Qlik Sense from this link:
http://www.qlik.com/us/explore/products/sense/desktop.aspx

This version is a Windows Desktop pre-cursor to the Qlik Sense Server product that is due out next month. Details of how Qlik Sense will be licenced will be made available next month, and at this point details of the licencing model will be made available.

The download is a simple Windows installer and Sense can be installed by double clicking the EXE and following the prompts. Once installed Qlik Sense will exist on your Windows menu and, if you selected it to do so, your Dekstop.

Qlik are keen that everyone is given the assistance they need to be able to use this app. On downloading you will be directed to a help screen, you will be emailed some helpful links and at the bottom of the main screen in Qlik Sense Desktop is a Getting Started link. You can’t go far wrong here.

Step By Step Help

The links you go to have two main sources of help. The first is an excellent series of videos created by Michael Tarralo and Josh Good, the complete list of which can be found here:
http://www.qlik.com/us/explore/products/sense/desktop/get-started/how-to-videos

If you work your way through these you will know everything you need to get up and running. The new Quick Data Load, you will notice, is a vastly simplified way of getting data into the app (just drag and drop), and there is a video dedicated to this process.

The other source of help for Qlik Sense (as it is for QlikView) is the Qlik Community. There is a new area dedicated to Qlik Sense, and indeed the main Community page is now split between QlikView and Qlik Sense content. There are many discussions already started on there, with users asking questions about the product and Qlik staff being quick at this important stage to respond and clarify any questions. If you want a deeper dive into what is happening with the product reading these threads is a good idea:

http://community.qlik.com/community/new-to-qlik-sense

The whole point of Qlik Sense though is that it is intuitive, so the best way to find out how it works is to dive in and get creating.

Everybody’s Talking

Since Sense landed last week there has been a lot of discussion about the product and many QlikView bloggers have turned their focus to Qlik Sense. A few of these posts are listed here:

http://www.qliktips.com/2014/07/qlik-sense.html
http://qlikthinking.co.uk/2014/qlik-sense-the-next-qlikview-has-arrived/
http://blog.axc.net/?p=1575

I’m sure there will be more and more Qlik Sense specific posts soon, and these should all appear via the AskQV.com news feed:
http://www.askqv.com/news/

The Same, But Different

As Henric points out there are many things which have been carried across from QlikView in to Qlik Sense, and the architecture it sits upon is largely the same. I remember being very pleased at the Business Discovery Tour last year when I saw that the load script in .Next looked much like it does in QlikView – that is still the case in Qlik Sense. It is possible to import a load script from a QlikView document into Qlik Sense – but not any charts etc. – as these are fundamentally different between the platforms.

This means that much of the ecosystem that has grown up around QlikView will continue to work and grow with Qlik Sense. Extensions created for QlikView will work in Sense, so some visualisations are portable. I was also very pleased to learn that Qlik Sense will be able to load from QVSource based sources, as Chris reported on the ICB blog:
http://blog.qvsource.com/post/2014/07/25/QVSource-Works-With-Qlik-Sense

Seeing how these two new siblings grow up together is going to be interesting. It is good to know that the new product is very much built on the solid foundations of its big brother. It is just not restrained by some of the things which perhaps hold QlikView back, which has been backwardly compatible for many, many versions.

Shiny New Toys

There are lots of things to be excited about with Qlik Sense, and I am very much looking forward to rolling this out with customers over the coming months and years. All of the charts and graphs feel very fluid to use and there are some great UI touches. The way the display flexes as you resize the window in Desktop gives a good glimpse into what Qlik Sense will be like on Mobile. If usage of Qlik Sense Desktop feels a bit “webby” then that is because it is entirely browser based, with the Windows app providing a shell onto this. When Qlik Sense Server lands and is rolled out, users on any platform (including tablets and Macs) will be able to connect to the server and create apps in exactly the same way as their Windows counterparts.

Personally, I have been most impressed this past week or so with the ease of dropping geographical objects into a Qlik Sense app. There is native support for KML mapping files, which allow you to create data connected maps. I have already found that the NHS provide KML definitions for their CCG areas, so this provides a great new way for displaying data for my UK pharma clients.

Another enticing prospect is hinted at with the cloud icon nestled at the top right of the Sense Desktop Hub. My understanding is that there will be a service where you can upload your Qlik Sense apps to the cloud for public consumption. Whilst this is perhaps limited in scope due to data that is less than public, the ability for people to publish data in a Sense app on their website means the product will get a whole lot more exposure. We could be seeing a lot more Qlik out there, and it can regain some of the ground lost to other products in this regard. Exactly how the Qlik Cloud will work in practice remains to be seen. As the website says, it is “Coming soon”.

A Very Bright Future

There is a lot to be excited about with Qlik Sense, and I am happy to be along for the ride here. These are still early days and the potential of the product is even more exciting than what we have now. How things pan out between now and the end of the year, with the Qlik Sense Server release and details of the new licencing model, is crucial for how the product is received and the in-roads it can make in the market place.

For now I am happy that I can show off a tool that is truly next generation and has lots of new toys to play with.

The post Qlik Sense – A Quick Primer appeared first on .

curiosamathematica:

Tom Beddard constructed this pyramid (and variations) as a recursive structure. Be sure to check out his awesome website.

curiosamathematica:

Construct a chain of six circles, all tangent to a seventh circle and each tangent to its two neighbors. The seven circles theorem guarantees that the three lines drawn between opposite pairs of the points of tangency on the seventh circle, all pass through the same point.

Several variations are possible, such as circles externally tangent to the central one, and other variations.

There is a sentence I have heard or read multiple times in my journey into (academic) visualization: visualization is a tool people use when they don’t know what question to ask to their data. I have always taken this sentence as a given and accepted it as it is. Good, I thought, we have a […]

Some weeks ago the chart type “Slope Graph”, invented by Eward Tufte in 1983, was first brought to my attention.

When I first saw it I thought that it would be quite easy to bring this chart also to the QlikView world. But after some research I realized that Slopegrahps have some drawbacks which are quite tricky to solve.

I have found quite a dozen of nice implementations out there but none of them was really satisfying me. So this little project was a perfect playground for me to understand challenges when creating chart-visualizations better. It really was the first project where I did not only adapt existing visualizations based on D3 to be used in QlikView, instead I created a new implementation of Slopegraphs in D3 (by combining several existing approaches).

Here is the result:

Some Theory about Slopegraphs

Edward Tufte states: “Slopegraphs compare changes over time for a list of nouns located on an ordinal or interval scale” and he also first published and example in his book “The Visual Display of Quantitative Information” in 1983:

Advantages and Scenarios for Using SlopeGraph

• Slopegraphs are very minimal visualizations, there is no unnecessary information
• It is easy to capture changes over time
• The same type of data is shown on the left and right side using the same units of measurement
• “Any time you’d use a line chart to show a progression of univariate data among multiple actors over time, you might have a good candidate for a SlopeGraph” (source)

Drawback

The most important and relevant drawback of using Slopegraphs is that if you have a lot of dimensions with similar values the visualization becomes messy very quickly (see some screenshots below demonstrating this).

My Approach for Slopegrahps

To overcome the above described drawback I have chosen a combination of several approaches based on the default visualization:

Handling Conflicting Label
In the above displayed example you’ll see that three dimensions on the left side share the same value for 2012. Instead of displaying them upon each other the dimensions are shown one below the other.

Emphasizing Dimensions
If you hover a dimension, the current dimension will be emphasized / highlighted

Displaying a Summary
While hovering a dimension a tooltip will be displayed with a short summary. This is especially useful if you have a lot of dimensions with little movement from one to the other time period:

Text Fadout for Colliding Labels
But there is still the issue that if you have a lot of dimensions with similar values that they will collide:

There are many suggestions how this can be solved, I have chosen an interactive one where conflicting labels will be faded out if you hover a dimension:

Still to Many Dimensions & Messy Visualization
There are situation where all the above described solutions are not sufficient. Especially if you are using a lot of dimensions this can happen.

There are two possible solutions:

• Use a different chart type
• Increase either the size of the chart object (extension) or increase the size of the plotting area. By doing so you’ll see a vertical scroll-bar, but certainly, the user has to scroll …

Conclusion

In my opinion it is really worth giving this type of chart a try (and I would not be surprised if we’ll see Slopegraphs much more often in the future). But this type of chart is also not a “generalist”, usage and configuration has to be considered with caution.

Download Extension, Demo + Source Code

The solution called “D3SlopeGraph QlikView Extension” is published under MIT-license and free to use (but still, I am happy to hear your feedback!!!)

• Documentation and Repository
• Download of D3SlopeGraph QlikView Extension (.qar file)
• Demo QlikView application using D3SlopeGraph
• Some instructions how to install and deploy QlikView Extensions can be found here
• Licensing Information

Issues, Bugs, Wishes

If you have any issues please post them here.

Additional Articles on the Web

• Slopegraphs for comparing gradients: Slopegraph theory and practice, (Dec 2013) by Edward Tufte
• Edward Tufte’s “Slopegraphs”, an article by Charlie Park
• A Slopegraph Update, by Charlie Park

I have recently published a free calendar heatmap extension to compare values on a day-per-day basis over a long period of time (several years). The extension is developed using D3 and an existing visualization called “Calendar View” by Mike Bostok.

The data passed to the extension is displayed in a diverging color scale. The values are visualized as colored cells per day. Days are arranged into columns by week, then grouped by month and years.

Screenshots

Usage Scenarios

There are several scenarios where using this extension seems to be a helpful kind of visualization:

• Comparing changes of stock exchange prices
• Making out of stock trends visible
• etc.

Download, Configuration & Source Code

The source code, a demo-application and the .qar files are published on GitHub, where you will also find a short description how to configure the extension.

GitHub-Repository to post issues, fork the source-code, etc.:

• https://github.com/stefanwalther/QlikView-Extension-D3CalendarView

I love slopegraphs. I’m happy to nail my colours to the mast and declare it. I’d probably not go as far as to wear a t-shirt with such a slogan but I feel a need to express my praise for the still-underused slopegraph and try help continue spread the word of its worth. The typical […]

Editor's note: This post was written by Nick Carchedi, a Master's degree student in the Department of Biostatistics at Johns Hopkins. He is working with us to develop software for interactive learning of R and statistics. 

Inspired by the relative lack of computer-based platforms for learning statistics and the R programming language, we at Johns Hopkins Biostatistics have created a new R package designed to teach both topics simultaneously and interactively. Accordingly, we’ve named the package swirl, which stands for “Statistics with Interactive R Learning”. We sought to model swirl after other highly successful interactive learning platforms such as Codecademy, Code School, and Khan Academy, but with a specific focus on teaching statistics and R. Additionally, we wanted users to learn these topics within the same environment in which they would be applying them, namely the R console.

If you’re reading this article, then you probably already have an appreciation for the R language and there’s no need to beat that drum any further. Staying true to the R culture, the swirl package is totally open-source and free for anyone to use, modify, or improve. Furthermore, anyone with something to teach can use the platform to create their own interactive content for the world to use.

A typical swirl session has a user load the package from the R console, choose from a menu of options the course he or she would like to take, then work through 10-15 minute interactive modules, each covering a particular topic. A module generally alternates between instructional text output to the user and prompts for the user to answer questions. One question may ask for the result of a simple numerical calculation, while another requires the user to enter an actual R command (which is parsed and executed, if correct) to perform a requested task. Multiple choice, text-based and approximate numerical answers are also fair game. Whenever the user answers a question incorrectly, immediate feedback is given in the form of a hint before prompting her to try again. Finally, plots, figures, and even videos may be incorporated into a module for the sake of reinforcing the methods or concepts being taught.

We believe that this form of interactive learning, or learning by doing, is essential for true mastery of topics as challenging and complex as statistics and statistical computing. While we are aware of a handful of other platforms for learning R interactively, our goal was to focus on the teaching of R and statistics simultaneously. As far as we know, swirl is the only platform of its kind and almost certainly the only one that takes place within the R console.

When we developed the swirl package, we wanted from the start to allow other people to extend and customize it to their particular needs. The beauty of the swirl platform is that anyone can create their own content and have it included in the package for all users to access. We have designed pre-formatted templates (color-coded spreadsheets) that instructors can fill out with their own content according to a fairly simple set of instructions. Once instructors send us the completed templates, we then load the content into the package so that anyone with the most recent version of swirl on their computer can access the content. We’ve tried to make the process of content creation as simple and painless as possible so that the statistics and computing communities are encouraged to share their knowledge with the world through our platform.

The package currently includes only a few sample modules that we’ve created in-house, primarily serving as demonstrations of how the platform works and how a typical module may appear to users. In the future, we envision a vibrant and dynamic collection of full courses and short modules that users can vote up or down based on the quality of their experience with each. In such a scenario, the very best courses would naturally float to the top and the less effective courses would fall out of favor and perhaps be recommended for revision.

In addition to making more content available to future users, we hope to one day transition swirl from being an interactive learning environment to one that is truly adaptive to the individual needs of each user. Perhaps this future version of our software would support a more intricate web of content, intelligently navigating users among topics based on a dynamic, data-driven interpretation of their strengths, weaknesses, competencies, and knowledge gaps. With the right people on board, this could become a reality.

We’ve created this package with the hope that the statistics and computing communities find it to be a valuable educational tool. We’ve got the basic infrastructure in place, but we recognize that there is a great deal of room for improvement. The swirl package is still very much in development and we are actively seeking feedback on how we can make it better. Please visit the swirl website to download the package or for more information on the project. We’d love for you to give it a try and let us know what you think.

Go to swirl website: http://ncarchedi.github.io/swirl/

Exciting news: My third 'Introduction to Infographics and Visualization' Massive Open Online Course (MOOC), offered through the Knight Center for Journalism in the Americas, will begin October 6. It'll be shorter than the two previous courses —four weeks, instead of six. Registration is free and space is limited to a few thousand people. The video above will give you an idea of what to expect.

According to Rosental Calmon Alves, the first edition was the first MOOC about journalism ever organized in the world. It was also an unexpected success: As I conceived it as a little experiment at first, we were hoping to attract just a few hundred people. We ended up with 2,000 in the first edition, and 5,000 in the second one, coming from more than 100 countries.

We had also assumed that a course that focuses on how to communicate with charts, maps, and diagrams —and not so much on how to use them to analyze data— would appeal mainly to journalists and designers. We were wrong again. Students came from several scientific disciplines, statistics, cartography, education, business intelligence, etc. This variety of backgrounds —professional and cultural— enlivened the discussions a lot.

Here you have an excerpt from the introduction to the third edition:

"Previous experience in information graphics and visualization is not needed to take this course. With the readings, video lectures and tutorials available, participants will acquire enough skills to start producing compelling, simple infographics almost immediately. Participants can expect to spend 4-6 hours per week on the course. Although the MOOC was initially conceived with journalists and designers in mind, others are welcome as well and can benefit from the very practical skills that will be taught."

You can also read some reviews of the previous courses: 1, 2, 3, 4. I hope that you will consider joining us, particularly if you didn't participate on any of these courses before.

REGISTER HERE

This summer I’ve been spending most of my time working on a new book. The current working title is Signal. As the title suggests, this book will focus on analytical techniques for detecting signals in the midst of noisy data. And guess what? All data sets are noisy. In fact, at any given moment, most of the data that we collect are noise. This will always be true, because signals in data are the exception, not the rule.

Signal detection is actually getting harder with the advent of so-called Big Data. By its very nature, most Big Data will never be anything but noise. Collecting everything possible, based on the Big Data argument that the costs of doing so are negligible and that even data that you can’t imagine as useful today could become useful tomorrow, is a dangerous premise. The costs of collecting and storing everything extend far beyond the hardware that’s used to store it. People already struggle to use data effectively. This will become dramatically harder as the volume of data grows. Finding a needle in a haystack doesn’t get easier as you’re tossing more and more hay on the pile.

Most people who are responsible for data analysis in organizations have never been trained to do this work. An insidious assumption exists, promoted by software vendors, that knowing how to use a particular data analysis software product “auto-magically” imbues one with the skills of a data analyst. Even with good software—something that’s rare—this is far from true. Just as with any area of expertise, data analysis requires training and practice, practice, practice. Because few people whose work involves data analysis possess the required skills, much time is wasted and money lost as analysts pore over data without knowing what to look for. They end up chasing patterns that mean nothing and missing those that are gold. Essentially, data analysis is the process of signal detection.

Data that do not convey useful knowledge are noise. When data are displayed, noise can exist both as data that don’t provide useful knowledge and also as useless non-data elements of the display (e.g., irrelevant visual attributes, such as a third dimension of depth in bars, meaningless color variation, and effects of light and shadow). Both sources of noise must be filtered to find and focus on the signals.

When we rely on data for decision making, what qualifies as a signal and what is merely noise? In and of themselves, data are neither. Data are merely facts. When facts are useful, they serve as signals. When they aren’t useful, data clutter the environment with distracting noise.

For data to be useful, they must:

• Address something that matters
• Promote understanding
• Provide an opportunity for action to achieve or maintain a desired state

When any of these qualities are missing, data remain noise.

Signals are always signs of something in particular. In a sense, a signal is not a thing but a relationship. Data become useful knowledge of something that matters when they connect understanding to a question to form an answer. This connection (relationship) is the signal.

As I work on this book to define the nature of signals and to describe techniques for detecting them, I could benefit from your thoughts on the matter. In your experience, what data qualify as signals? How do you find them? What do you do to understand them? What do you do about them once found? What examples have you seen in your own organization or others of time wasted chasing noise. What can we do to reduce noise? Please share with me any thoughts that you have along these lines.

Take care

Notepad++ version 6.4.1 was released recently and contains some exciting new additions.Top of the list is the new function list which has been awaited for a while now and is a popular feature in many other code editors (including UltraEdit). This is another significant update and includes: New function list support allowing Notepad++ to show subroutines, qualify and unqaulify statements, table loads, info loads, mapping loads and store statements all identified within the function...

Read the full post at QlikViewAddict.com

Investment in dynamic, growing areas such as micro-energy and synthetic biology could yield new fields of employment by 2030

Politicians know full well that there is a serious unemployment problem in Europe and many of them also know there is a relationship between youth unemployment, alienation and violence. But they don't seem to know what kind of jobs we should prepare the next generation for. Here are a few suggestions for secure employment in the year 2030.

Micro-energy traders

In the future, much of our energy will be generated, transmitted and stored locally. Interest will therefore grow in what will essentially be an "energy internet", where households and small businesses buy, sell and trade energy within their own local communities. For example, if you have an electric car but don't drive it very often, you could rent out its battery space to store energy for others on a temporary basis. Anyone with the necessary time and technical skills will therefore have the potential to become an energy trader.

Synthetic biologists

In the future we will grow energy or, more precisely, we will harvest algae that secretes bio-diesel. Synthetic biology, which is the next big thing which hardly anyone has heard of, is capable of this and a whole lot more. Of course, scientists in other countries are capable of exploiting new technologies too, but will they? Putting aside the inherent fragility of emerging economies, there's the question of whether regions outside the US and Europe, with their long traditions of free speech and creative destruction, will be able to replicate the curiosity and frank exchange of views required to commercialise such thinking.

Sharia financiers

With Muslim populations expanding in Europe, one thing we are likely to see is further growth of Sharia or Islamic finance: for example, Sharia-compliant financial planning or Sharia-compliant venture capital, where making money from money is forbidden. Given current suspicions about traditional British banking, we might also see the development of related forms of ethical banking, where bankers refuse to lend money for purely speculative purposes, preferring instead to lend to individuals or institutions that create or build things with inherent social value. Graduates flocking to get into the next generation of banking? It's not impossible.

Teachers

It may not sound very futuristic, but if Europe is to compete with the likes of China and India it needs to become more competitive and education is the key. You might expect education to be virtual by 2030, and parts of it will be, but a fully outsourced, automated, massively open, online future is unlikely to be the case, especially in early years education where socialisation and empathy are key ingredients. Of course, the challenge here is to persuade European governments to spend more on education when their natural inclination will be to downplay their debt by spending less. Let's not forget also that Europe is ageing faster than any other region, so there will be political pressure from voters to switch spending to aged care going forward. We should resist the temptation.

Doctors and nurses

We've already got e-health, telemedicine, medical tourism and even robotic surgery, so what is it about healthcare, especially in Europe, that's future proof? The answer is trust. Do you really want life-saving surgery to take place in a country where you don't speak the language? And have you ever met a robot with a good bedside manner that can gently hold your hand and tell you that everything will be all right? No. Thought not.

• This article was amended on 2 July to correct two spelling mistakes in the standfirst