Shared posts

05 Mar 07:54

.NET Core Data Access

by Bertrand Le Roy

.NET Core was released a few months ago, and data access libraries for most databases, both relational and NoSQL are now available. In this post, I’ll detail what client libraries are available, as well as show code samples for each of them.

ORM

EF Core

Entity Framework is Microsoft’s Object-Relational Mapper for .NET, and as such is one of the most-used data access technologies for .NET. EF Core, released simultaneously with .NET Core, is a lightweight and extensible version of Entity Framework that works on both .NET Core and .NET Framework. It has support for Microsoft SQL Server, SQLite, PostgreSQL, MySQL, Microsoft SQL Server Compact Edition, with more, such as DB2 and Oracle, to come.

What follows is an example of EF Core code accessing a blog’s database. The full tutorial can be found on the EF documentation site.

Dapper

Dapper is a micro-ORM built and maintained by StackExchange engineers. It focuses on performance, and can map the results of a query to a strongly-typed list, or to dynamic objects. .NET Core support is available.

NPoco

NPoco is a micro-ORM that also works on .NET Core.

Relational databases

SQL Server

The Microsoft SQL Server client library is built into .NET Core. You don’t have to use an ORM, and can instead go directly to the metal and talk to a SQL Server instance or to an Azure SQL database using the same APIs from the System.Data.SqlClient package.

PostgreSQL

PostgreSQL is an open source relational database with a devoted following. The Npgsql client library supports .NET Core.

Another interesting library for PostgreSQL that is compatible with .NET Core is Marten. Marten uses PostgreSQL storage to implements a document database.

MySQL

MySQL is one of the most commonly used relational databases on the market and is open source. Support for .NET Core is now available, both through EF Core and directly through the MySQL Connector for .NET Core.

SQLite

SQLite is a self-contained, embedded relational database that is released in the public domain. SQLite is lightweight (less than 1MB), cross-platform, and is extremely easy to embed and deploy with an application, which explains how it quietly became the most widely deployed database in the world. It’s commonly used as an application file format.

You can use SQLite with EF Core, or you can talk to a SQLite database directly using the Microsoft.Data.Sqlite library that is maintained by the ASP.NET team.

There’s another SQLite package that’s compatible with .NET Core called SQLitePCL.raw.

Firebird

Firebird is a mature relational database with a small footprint. It now has a .NET Core compatible client library.

NoSQL

Azure DocumentDB

Azure DocumentDB is a NoSQL database service built for fast and predictable performance, high availability, automatic scaling, and ease of development. Its flexible data model, consistent low latencies, and rich query capabilities make it a great fit for web, mobile, gaming, IoT, and many other applications that need seamless scale. Read more in the DocumentDB introduction. DocumentDB databases can now be used as the data store for apps written for MongoDB. Using existing drivers for MongoDB, applications can easily and transparently communicate with DocumentDB, in many cases by simply changing a connection string. The next version of the DocumentDB client library, which will be available around the Connect event, supports .NET Core. The latest version of the DocumentDB SDK supports .NET Core.

MongoDB

MongoDB is a document database with an official .NET driver that supports .NET Core.

RavenDB

RavenDB is a document database that is not only compatible with .NET Core, it’s also built with it.

Redis

Redis is one of the most popular key-value stores.

StackExchange.Redis is a high performance Redis client that is maintained by the StackExchange team.

ServiceStack has its own Redis client library, that is compatible with .NET Core, like the rest of ServiceStack.

Cassandra

Apache Cassandra is a highly scalable and fault-tolerant NoSQL database. DataStax is a C# driver for Cassandra with built-in support for mapping Cassandra data to CLR objects. The latest version is compatible with .NET Core.

CouchBase

CouchBase is an open source document database that is popular in mobile applications. The offical Couchbase client library is compatible with .NET Core.

CouchDB

CouchDB is a document database that I personally like a lot for its simplicity. It can scale from small devices such as a Raspberry Pi to cloud applications. It uses a very simple HTTP and JSON-based API, which limits the need for a client library. C# client libraries do exist, but none of them support .NET Core today as far as I can tell except for Kanapa which hasn’t been updated for a while. It’s very easy to interact with the database through its REST API nonetheless.

Neo4j

Neo4j is a graph database, which means that it establishes relationships not between tables but between data nodes, and treats these relationships as first class semantically rich entities that carry data and can be queried on. Readify has a .NET Core-compatible client for Neo4j called Neo4jClient.

The full sample can be found in Tugberk Ugurlu’s blog post “Getting Started with Neo4j in .NET with Neo4jClient Library”

RethinkDB

RethinkDB is a document database that can stream real-time result updates to a query. RethinkDb.Driver is a .NET Core driver for RethinkDB that covers 100% of the features of the official Java driver. One should exercise caution for new project however, as the company behind RetinkDB is shutting down.

YesSql

YesSql is an interesting library that implements a transactional document database on top of relational stores such as SQL Server.

Lucene.NET

Finally, I want to mention Lucene.NET. It’s not technically a database, but it’s so useful in setting up full-text search on a data-driven project that a post on data access wouldn’t be complete without it. The team has put a lot of work into the new version of Lucene to implement new features and major improvements, and they also made it compatible with .NET Core. It’s still early, but prerelease packages will soon be available.

What about OLE DB?

OLE DB has been a great way to access various data sources in a uniform manner, but it was based on COM, which is a Windows-only technology, and as such was not the best fit for a cross-platform technology such as .NET Core. It is also unsupported in SQL Server versions 2014 and later. For those reasons, OLE DB won’t be supported by .NET Core.

Keeping track

More database support for .NET Core will no doubt become available in the future, and we’ll make sure to highlight new client libraries in our Week in .NET posts as they get announced. In the meantime, I hope this post helps get you started with .NET Core application development.

22 Nov 13:18

Bliki: AliasingBug

Aliasing occurs when the same memory location is accessed through more than one reference. Often this is a good thing, but frequently it occurs in an unexpected way, which leads to confusing bugs.

Here's a simple example of the bug.

Date retirementDate = new Date(Date.parse("Tue 1 Nov 2016"));

// this means we need a retirement party
Date partyDate = retirementDate;

// but that date is a Tuesday, let's party on the weekend
partyDate.setDate(5);

assertEquals(new Date(Date.parse("Sat 5 Nov 2016")), retirementDate);
// oops, now I have to work three more days :-(

What's happening here is that when we do the assignment, the partyDate variable is assigned a reference to the same object that the retirement data refers to. If I then alter the internals of that object (with setDate) then both variables are updated, since they refer to the same thing.

Although aliasing is a problem in that example, in other contexts it's what I expect.

Person me = new Person("Martin");
me.setPhoneNumber("1234");
Person articleAuthor = me;
me.setPhoneNumber("999");
assertEquals("999", articleAuthor.getPhoneNumber());

It's common to want to share records like this, and then if it changes, it changes for all references. This is why it's useful to think of reference objects, which we deliberately share [1], and Value Objects that we don't want this kind of shared update behavior. A good way to avoid shared updates of value objects is to make value objects immutable.

Functional languages, of course, prefer everything to be immutable. So if we want changes to be shared, we need to handle that as the exception rather than the rule. Immutability is a handy property, one that makes it harder to create several kinds of bugs. But when things do need to change, immutability can introduce complexity, so it's by no means a free breakfast.

Acknowledgements

Graham Brooks and James Birnie's comments on our internal mailing list led me to write this post.

Further Reading

The term aliasing bug has been around for a while. It appears in Eric Raymond's Jargon file in the context of the C language where the raw memory accesses make it even more unpleasant.

Notes

1: The Evans Classification has the notion of Entity, which I see as a common form of reference object.

Share:
if you found this article useful, please share it. I appreciate the feedback and encouragement
20 Nov 22:19

Bliki: ValueObject

When programming, I often find it's useful to represent things as a compound. A 2D coordinate consists of an x value and y value. An amount of money consists of a number and a currency. A date range consists of start and end dates, which themselves can be compounds of year, month, and day.

As I do this, I run into the question of whether two compound objects are the same. If I have two point objects that both represent the Cartesian coordinates of (2,3), it makes sense to treat them as equal. Objects that are equal due to the value of their properties, in this case their x and y coordinates, are called value objects.

But unless I'm careful when programming, I may not get that behavior in my programs

Say I want to represent a point in JavaScript.

const p1 = {x: 2, y: 3};
const p2 = {x: 2, y: 3};
assert(p1 !== p2);  // NOT what I want

Sadly that test passes. It does so because JavaScript tests equality for js objects by looking at their references, ignoring the values they contain.

In many situations using references rather than values makes sense. If I'm loading and manipulating a bunch of sales orders, it makes sense to load each order into a single place. If I then need to see if the Alice's latest order is in the next delivery, I can take the memory reference, or identity, of Alice's order and see if that reference is in the list of orders in the delivery. For this test, I don't have to worry about what's in the order. Similarly I might rely on a unique order number, testing to see if Alice's order number is on the delivery list.

Therefore I find it useful to think of two classes of object: value objects and reference objects, depending on how I tell them apart [1]. I need to ensure that I know how I expect each object to handle equality and to program them so they behave according to my expectations. How I do that depends on the programming language I'm working in.

Some languages treat all compound data as values. If I make a simple compound in Clojure, it looks like this.

> (= {:x 2, :y 3} {:x 2, :y 3})
true

That's the functional style - treating everything as immutable values.

But if I'm not in a functional language, I can still often create value objects. In Java for example, the default point class behaves how I'd like.

assertEquals(new Point(2, 3), new Point(2, 3)); // Java

The way this works is that the point class overrides the default equals method with the tests for the values. [2] [3]

I can do something similar in JavaScript.

class Point {
  constructor(x, y) {
    this.x = x;
    this.y = y;
  }
  equals (other) {
    return this.x === other.x && this.y === other.y;
  }
}
const p1 = new Point(2,3);
const p2 = new Point(2,3);
assert(p1.equals(p2));

The problem with JavaScript here is that this equals method I defined is a mystery to any other JavaScript library.

const somePoints = [new Point(2,3)];
const p = new Point(2,3);
assert.isFalse(somePoints.includes(p)); // not what I want

//so I have to do this
assert(somePoints.some(i => i.equals(p)));

This isn't an issue in Java because Object.equals is defined in the core library and all other libraries use it for comparisons (== is usually used only for primitives).

One of the nice consequences of value objects is that I don't need to care about whether I have a reference to the same object in memory or a different reference with an equal value. However if I'm not careful that happy ignorance can lead to a problem, which I'll illustrate with a bit of Java.

Date retirementDate = new Date(Date.parse("Tue 1 Nov 2016"));

// this means we need a retirement party
Date partyDate = retirementDate;

// but that date is a Tuesday, let's party on the weekend
partyDate.setDate(5);

assertEquals(new Date(Date.parse("Sat 5 Nov 2016")), retirementDate);
// oops, now I have to work three more days :-(

This is an example of an Aliasing Bug, I change a date in one place and it has consequences beyond what I expected [4]. To avoid aliasing bugs I follow a simple but important rule: value objects should be immutable. If I want to change my party date, I create a new object instead.

Date retirementDate = new Date(Date.parse("Tue 1 Nov 2016"));
Date partyDate = retirementDate;

// treat date as immutable
partyDate = new Date(Date.parse("Sat 5 Nov 2016"));

// and I still retire on Tuesday
assertEquals(new Date(Date.parse("Tue 1 Nov 2016")), retirementDate);

Of course, it makes it much easier to treat value objects as immutable if they really are immutable. With objects I can usually do this by simply not providing any setting methods. So my earlier JavaScript class would look like this: [5]

class Point {
  constructor(x, y) {
    this._data = {x: x, y: y};
  }
  get x() {return this._data.x;}
  get y() {return this._data.y;}
  equals (other) {
    return this.x === other.x && this.y === other.y;
  }
}

While immutability is my favorite technique to avoid aliasing bugs, it's also possible to avoid them by ensuring assignments always make a copy. Some languages provide this ability, such as structs in C#.

Whether to treat a concept as a reference object or value object depends on your context. In many situations it's worth treating a postal address as a simple structure of text with value equality. But a more sophisticated mapping system might link postal addresses into a sophisticated hierarchic model where references make more sense. As with most modeling problems, different contexts lead to different solutions. [6]

It's often a good idea to replace common primitives, such as strings, with appropriate value objects. While I can represent a telephone number as a string, turning into a telephone number object makes variables and parameters more explicit (with type checking when the language supports it), a natural focus for validation, and avoiding inapplicable behaviors (such as doing arithmetic on integer id numbers).

Small objects, such as points, monies, or ranges, are good examples of value objects. But larger structures can often be programmed as value objects if they don't have any conceptual identity or don't need share references around a program. This is a more natural fit with functional languages that default to immutability. [7]

I find that value objects, particularly small ones, are often overlooked - seen as too trivial to be worth thinking about. But once I've spotted a good set of value objects, I find I can create a rich behavior over them. For taste of this try using a Range class and see how it prevents all sorts of duplicate fiddling with start and end attributes by using richer behaviors. I often run into code bases where domain-specific value objects like this can act as a focus for refactoring, leading to a drastic simplification of a system. Such a simplification often surprises people, until they've seen it a few times - by then it is a good friend.

Acknowledgements

James Shore, Beth Andres-Beck, and Pete Hodgson shared their experiences of using value objects in JavaScript.

Graham Brooks, James Birnie, Jeroen Soeters, Mariano Giuffrida, Matteo Vaccari, Ricardo Cavalcanti, and Steven Lowe provided valuable comments on our internal mailing lists.

Further Reading

Vaughn Vernon's description is probably the best in-depth discussion of value objects from a DDD perspective. He covers how to decide between values and entities, implementation tips, and the techniques for persisting value objects.

The term started gaining traction in the early noughties. Two books that talk about them from that time are PoEAA and DDD. There was also some interesting discussion on Ward's Wiki.

One source of terminological confusion is that around the turn of the century some J2EE literature used "value object" for Data Transfer Object. That usage has mostly disappeared by now, but you might run into it.

Notes

1: In Domain-Driven Design the Evans Classification contrasts value objects with entities. I consider entities to be a common form of reference object, but use the term "entity" only within domain models while the reference/value object dichotomy is useful for all code.

2: Strictly this is done in awt.geom.Point2D, which is a superclass of awt.Point

3: Most object comparisons in Java are done with equals - which is itself a bit awkward since I have to remember to use that rather than the equals operator ==. This is annoying, but Java programmers soon get used to it since String behaves the same way. Other OO languages can avoid this - Ruby uses the == operator, but allows it to be overridden.

4: There is robust competition for the worst feature of the pre-Java-8 date and time system - but my vote would be this one. Thankfully we can avoid most of this now with Java 8's java.time package

5: This isn't strictly immutable since a client can manipulate the _data property. But a suitably disciplined team can make it immutable in practice. If I was concerned that a team wouldn't be disciplined enough I might use freeze. Indeed I could just use freeze on a simple JavaScript object, but I prefer the explicitness of a class with declared accessors.

6: There is more discussion of this in Evans's DDD book.

7: Immutability is valuable for reference objects too - if a sales order doesn't change during a get request, then making it immutable is valuable; and that would make it safe to copy it, if that were useful. But that wouldn't make the sales order be a value object if I'm determining equality based on a unique order number.

Translations: Chinese
Share:
if you found this article useful, please share it. I appreciate the feedback and encouragement
20 Nov 07:53

Bearer Token Authentication in ASP.NET Core

by Jeffrey T. Fritz

This is a guest post from Mike Rousos

Introduction

ASP.NET Core Identity automatically supports cookie authentication. It is also straightforward to support authentication by external providers using the Google, Facebook, or Twitter ASP.NET Core authentication packages. One authentication scenario that requires a little bit more work, though, is to authenticate via bearer tokens. I recently worked with a customer who was interested in using JWT bearer tokens for authentication in mobile apps that worked with an ASP.NET Core back-end. Because some of their customers don’t have reliable internet connections, they also wanted to be able to validate the tokens without having to communicate with the issuing server.

In this article, I offer a quick look at how to issue JWT bearer tokens in ASP.NET Core. In subsequent posts, I’ll show how those same tokens can be used for authentication and authorization (even without access to the authentication server or the identity data store).

Offline Token Validation Considerations

First, here’s a quick diagram of the desired architecture.authentication-architecture

 

The customer has a local server with business information which will need to be accessed and updated periodically by client devices. Rather than store user names and hashed passwords locally, the customer prefers to use a common authentication micro-service which is hosted in Azure and used in many scenarios beyond just this specific one. This particular scenario is interesting, though, because the connection between the customer’s location (where the server and clients reside) and the internet is not reliable. Therefore, they would like a user to be able to authenticate at some point in the morning when the connection is up and have a token that will be valid throughout that user’s work shift. The local server, therefore, needs to be able to validate the token without access to the Azure authentication service.

This local validation is easily accomplished with JWT tokens. A JWT token typically contains a body with information about the authenticated user (subject identifier, claims, etc.), the issuer of the token, the audience (recipient) the token is intended for, and an expiration time (after which the token is invalid). The token also contains a cryptographic signature as detailed in RFC 7518. This signature is generated by a private key known only to the authentication server, but can be validated by anyone in possession of the corresponding public key. One JWT validation work flow (used by AD and some identity providers) involves requesting the public key from the issuing server and using it to validate the token’s signature. In our offline scenario, though, the local server can be prepared with the necessary public key ahead of time. The challenge with this architecture is that the local server will need to be given an updated public key anytime the private key used by the cloud service changes, but this inconvenience means that no internet connection is needed at the time the JWT tokens are validated.

Issuing Authentication Tokens

As mentioned previously, Microsoft.AspNetCore.* libraries don’t have support for issuing JWT tokens. There are, however, several other good options available.

First, Azure Active Directory Authentication provides identity and authentication as a service. Using Azure AD is a quick way to get identity in an ASP.NET Core app without having to write authentication server code.

Alternatively, if a developer wishes to write the authentication service themselves, there are a couple third-party libraries available to handle this scenario. IdentityServer4 is a flexible OpenID Connect framework for ASP.NET Core. Another good option is OpenIddict. Like IdentityServer4, OpenIddict offers OpenID Connect server functionality for ASP.NET Core. Both OpenIddict and IdentityServer4 work well with ASP.NET Identity 3.

For this demo, I will use OpenIddict. There is excellent documentation on accomplishing the same tasks with IdentityServer4 available in the IdentityServer4 documentation, which I would encourage you to take a look at, as well.

A Disclaimer

Please note that both IdentityServer4 and OpenIddict are pre-release packages currently. OpenIddict is currently released as a beta and IdentityServer4 as an RC, so both are still in development and subject to change!

Setup the User Store

In this scenario, we will use a common ASP.NET Identity 3-based user store, accessed via Entity Framework Core. Because this is a common scenario, setting it up is as easy as creating a new ASP.NET Core web app from new project templates and selecting ‘individual user accounts’ for the authentication mode.new-web-app

 

This template will provide a default ApplicationUser type and Entity Framework Core connections to manage users. The connection string in appsettings.json can be modifier to point at the database where you want this data stored.

Because JWT tokens can encapsulate claims, it’s interesting to include some claims for users other than just the defaults of user name or email address. For demo purposes, let’s include two different types of claims.

Adding Roles

ASP.NET Identity 3 includes the concept of roles. To take advantage of this, we need to create some roles which users can be assigned to. In a real application, this would likely be done by managing roles through a web interface. For this short sample, though, I just seeded the database with sample roles by adding this code to startup.cs:

// Initialize some test roles. In the real world, these would be setup explicitly by a role manager
private string[] roles = new[] { "User", "Manager", "Administrator" };
private async Task InitializeRoles(RoleManager<IdentityRole> roleManager)
{
    foreach (var role in roles)
    {
        if (!await roleManager.RoleExistsAsync(role))
        {
            var newRole = new IdentityRole(role);
            await roleManager.CreateAsync(newRole);
            // In the real world, there might be claims associated with roles
            // _roleManager.AddClaimAsync(newRole, new )
        }
    }
}

I then call InitializeRoles from my app’s Startup.Configure method. The RoleManager needed as a parameter to InitializeRoles can be retrieved by IoC (just add a RoleManager parameter to your Startup.Configure method).

Because roles are already part of ASP.NET Identity, there’s no need to modify models or our database schema.

Adding Custom Claims to the Data Model

It’s also possible to encode completely custom claims in JWT tokens. To demonstrate that, I added an extra property to my ApplicationUser type. For sample purposes, I added an integer called OfficeNumber:

public virtual int OfficeNumber { get; set; }

This is not something that would likely be a useful claim in the real world, but I added it in my sample specifically because it’s not the sort of claim that’s already handled by any of the frameworks we’re using.

I also updated the view models and controllers associated with creating a new user to allow specifying role and office number when creating new users.

I added the following properties to the RegisterViewModel type:

[Display(Name = "Is administrator")]
public bool IsAdministrator { get; set; }

[Display(Name = "Is manager")]
public bool IsManager { get; set; }

[Required]
[Display(Name = "Office Number")]
public int OfficeNumber { get; set; }

I also added cshtml for gathering this information to the registration view:

<div class="form-group">
    <label asp-for="OfficeNumber" class="col-md-2 control-label"></label>
    <div class="col-md-10">
        <input asp-for="OfficeNumber" class="form-control" />
        <span asp-validation-for="OfficeNumber" class="text-danger"></span>
    </div>
</div>
<div class="form-group">
    <label asp-for="IsManager" class="col-md-2 control-label"></label>
    <div class="col-md-10">
        <input asp-for="IsManager" class="form-control" />
        <span asp-validation-for="IsManager" class="text-danger"></span>
    </div>
</div>
<div class="form-group">
    <label asp-for="IsAdministrator" class="col-md-2 control-label"></label>
    <div class="col-md-10">
        <input asp-for="IsAdministrator" class="form-control" />
        <span asp-validation-for="IsAdministrator" class="text-danger"></span>
    </div>
</div>

Finally, I updated the AccountController.Register action to set role and office number information when creating users in the database. Notice that we add a custom claim for the office number. This takes advantage of ASP.NET Identity’s custom claim tracking. Be aware that ASP.NET Identity doesn’t store claim value types, so even in cases where the claim is always an integer (as in this example), it will be stored and returned as a string. Later in this post, I explain how non-string claims can be included in JWT tokens.

var user = new ApplicationUser { UserName = model.Email, Email = model.Email, OfficeNumber = model.OfficeNumber };
var result = await _userManager.CreateAsync(user, model.Password);
if (result.Succeeded)
{
    if (model.IsAdministrator)
    {
        await _userManager.AddToRoleAsync(user, "Administrator");
    }
    else if (model.IsManager)
    {
        await _userManager.AddToRoleAsync(user, "Manager");
    }

    var officeClaim = new Claim("office", user.OfficeNumber.ToString(), ClaimValueTypes.Integer);
    await _userManager.AddClaimAsync(user, officeClaim);
    ...

Updating the Database Schema

After making these changes, we can use Entity Framework’s migration tooling to easily update the database to match (the only change to the database should be to add an OfficeNumber column to the users table). To migrate, simply run dotnet ef migrations add OfficeNumberMigration and dotnet ef database update from the command line.

At this point, the authentication server should allow registering new users. If you’re following along in code, go ahead and add some sample users at this point.

Issuing Tokens with OpenIddict

The OpenIddict package is still pre-release, so it’s not yet available on NuGet.org. Instead, the package is available on the aspnet-contrib MyGet feed.

To restore it, we need to add that feed to our solution’s NuGet.config. If you don’t yet have a NuGet.config file in your solution, you can add one that looks like this:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
    <add key="aspnet-contrib" value="https://www.myget.org/F/aspnet-contrib/api/v3/index.json" />
  </packageSources>
</configuration>

Once that’s done, add a reference to "OpenIddict": "1.0.0-beta1-*" and "OpenIddict.Mvc": "1.0.0-beta1-*" in your project.json file’s dependencies section. OpenIddict.Mvc contains some helpful extensions that allow OpenIddict to automatically bind OpenID Connect requests to MVC action parameters.

There are only a few steps needed to enable OpenIddict endpoints.

Use OpenIddict Model Types

The first change is to update your ApplicationDBContext model type to inherit from OpenIddictDbContext instead of IdentityDbContext.

After making this change, migrate the database to update it, as well (dotnet ef migrations add OpenIddictMigration and dotnet ef database update).

Configure OpenIddict

Next, it’s necessary to register OpenIddict types in our ConfigureServices method in our Startup type. This can be done with a call like this:

services.AddOpenIddict<ApplicationDbContext>()
    .AddMvcBinders()
    .EnableTokenEndpoint("/connect/token")
    .UseJsonWebTokens()
    .AllowPasswordFlow()
    .AddSigningCertificate(jwtSigningCert);

The specific methods called on the OpenIddictBuilder here are important to understand.

  • AddMvcBinders. This method registers custom model binders that will populate OpenIdConnectRequest parameters in MVC actions with OpenID Connect requests read from incoming HTTP request’s context. This isn’t required, since the OpenID Connect requests can be read manually, but it’s a helpful convenience.
  • EnableTokenEndpoint. This method allows you to specify the endpoint which will be serving authentication tokens. The endpoint shown above (/connect/token) is a pretty common default endpoint for token issuance. OpenIddict needs to know the location of this endpoint so that it can be included in responses when a client queries for information about how to connect (using the .well-known/openid-configuration endpoint, which OpenIddict automatically provides). OpenIddict will also validate requests to this endpoint to be sure they are valid OpenID Connect requests. If a request is invalid (if it’s missing mandatory parameters like grant_type, for example), then OpenIddict will reject the request before it even reaches the app’s controllers.
  • UseJsonWebTokens. This instructs OpenIddict to use JWT as the format for bearer tokens it produces.
  • AllowPasswordFlow. This enables the password grant type when logging on a user. The different OpenID Connect authorization flows are documented in RFC and OpenID Connect specs. The password flow means that client authorization is performed based on user credentials (name and password) which are provided from the client. This is the flow that best matches our sample scenario.
  • AddSigningCertificate. This API specifies the certificate which should be used to sign JWT tokens. In my sample code, I produce the jwtSigningCert argument from a pfx file on disk (var jwtSigningCert = new X509Certificate2(certLocation, certPassword);). In a real-world scenario, the certificate would more likely be loaded from the authentication server’s certificate store, in which case a different overload of AddSigningCertificate would be used (one which takes the cert’s thumbprint and store name/location).
    • If you need a self-signed certificate for testing purposes, one can be produced with the makecert and pvk2pfx command line tools (which should be on the path in a Visual Studio Developer Command prompt).
      • makecert -n "CN=AuthSample" -a sha256 -sv AuthSample.pvk -r AuthSample.cer This will create a new self-signed test certificate with its public key in AuthSample.cer and it’s private key in AuthSample.pvk.
      • pvk2pfx -pvk AuthSample.pvk -spc AuthSample.cer -pfx AuthSample.pfx -pi [A password] This will combine the pvk and cer files into a single pfx file containing both the public and private keys for the certificate (protected by a password).
      • This pfx file is what needs to be loaded by OpenIddict (since the private key is necessary to sign tokens). Note that this private key (and any files containing it) must be kept secure.
  • DisableHttpsRequirement. The code snippet above doesn’t include a call to DisableHttpsRequirement(), but such a call may be useful during testing to disable the requirement that authentication calls be made over HTTPS. Of course, this should never be used outside of testing as it would allow authentication tokens to be observed in transit and, therefore, enable malicious parties to impersonate legitimate users.

Enable OpenIddict Endpoints

Once AddOpenIddict has been used to configure OpenIddict services, a call to app.UseOpenIddict(); (which should come after the existing call to UseIdentity) should be added to Startup.Configure to actually enable OpenIddict in the app’s HTTP request processing pipeline.

Implementing the Connect/Token Endpoint

The final step necessary to enable the authentication server is to implement the connect/token endpoint. The EnableTokenEndpoint call made during OpenIddict configuration indicates where the token-issuing endpoint will be (and allows OpenIddict to validate incoming OIDC requests), but the endpoint still needs to be implemented.

OpenIddict’s owner, Kévin Chalet, gives a good example of how to implement a token endpoint supporting a password flow in this sample. I’ve restated the gist of how to create a simple token endpoint here.

First, create a new controller called ConnectController and give it a Token post action. Of course, the specific names are not important, but it is important that the route matches the one given to EnableTokenEndpoint.

Give the action method an OpenIdConnectRequest parameter. Because we are using the OpenIddict MVC binder, this parameter will be supplied by OpenIddict. Alternatively (without using the OpenIddict model binder), the GetOpenIdConnectRequest extension method could be used to retrieve the OpenID Connect request.

Based on the contents of the request, you should validate that the request is valid.

  1. Confirm that the grant type is as expected (‘Password’ for this authentication server).
  2. Confirm that the requested user exists (using the ASP.NET Identity UserManager).
  3. Confirm that the requested user is able to sign in (since ASP.NET Identity allows for accounts that are locked or not yet confirmed).
  4. Confirm that the password provided is correct (again, using a UserManager).

If everything in the request checks out, then a ClaimsPrincipal can be created using SignInManager.CreateUserPrincipalAsync.

Roles and custom claims known to ASP.NET identity will automatically be present in the ClaimsPrincipal. If any changes are needed to the claims, those can be made now.

One set of claims updates that will be important is to attach destinations to claims. A claim is only included in a token if that claim includes a destination for that token type. So, even though the ClaimsPrincipal will contain all ASP.NET Identity claims, they will only be included in tokens if they have appropriate destinations. This allows some claims to be kept private and others to be included only in particular token types (access or identity tokens) or if particular scopes are requested. For the purposes of this simple demo, I am including all claims for all token types.

This is also an opportunity to add additional custom claims to the ClaimsPrincipal. Typically, tracking the claims with ASP.NET Identity is sufficient but, as mentioned earlier, ASP.NET Identity does not remember claim value types. So, if it was important that the office claim be an integer (rather than a string), we could instead add it here based on data in the ApplicationUser object returned from the UserManager. Claims cannot be added to a ClaimsPrincipal directly, but the underlying identity can be retrieved and modified. For example, if the office claim was created here (instead of at user registration), it could be added like this:

var identity = (ClaimsIdentity)principal.Identity;
var officeClaim = new Claim("office", user.OfficeNumber.ToString(), ClaimValueTypes.Integer);
officeClaim.SetDestinations(OpenIdConnectConstants.Destinations.AccessToken, OpenIdConnectConstants.Destinations.IdentityToken);
identity.AddClaim(officeClaim);

Finally, an AuthenticationTicket can be created from the claims principal and used to sign in the user. The ticket object allows us to use helpful OpenID Connect extension methods to specify scopes and resources to be granted access. In my sample, I pass the requested scopes filtered by those the server is able to provide. For resources, I provide a hard-coded string indicating the resource this token should be used to access. In more complex scenarios, the requested resources (request.GetResources()) might be considered when determining which resource claims to include in the ticket. Note that resources (which map to the audience element of a JWT) are not mandatory according to the JWT specification, though many JWT consumers expect them.

Put all together, here’s a simple implementation of a connect/token endpoint:

[HttpPost]
public async Task<IActionResult> Token(OpenIdConnectRequest request)
{
    if (!request.IsPasswordGrantType())
    {
        // Return bad request if the request is not for password grant type
        return BadRequest(new OpenIdConnectResponse
        {
            Error = OpenIdConnectConstants.Errors.UnsupportedGrantType,
            ErrorDescription = "The specified grant type is not supported."
        });
    }

    var user = await _userManager.FindByNameAsync(request.Username);
    if (user == null)
    {
        // Return bad request if the user doesn't exist
        return BadRequest(new OpenIdConnectResponse 
        {
            Error = OpenIdConnectConstants.Errors.InvalidGrant,
            ErrorDescription = "Invalid username or password"
        });
    }

    // Check that the user can sign in and is not locked out.
    // If two-factor authentication is supported, it would also be appropriate to check that 2FA is enabled for the user
    if (!await _signInManager.CanSignInAsync(user) || (_userManager.SupportsUserLockout && await _userManager.IsLockedOutAsync(user)))
    {
        // Return bad request is the user can't sign in
        return BadRequest(new OpenIdConnectResponse
        {
            Error = OpenIdConnectConstants.Errors.InvalidGrant,
            ErrorDescription = "The specified user cannot sign in."
        });
    }

    if (!await _userManager.CheckPasswordAsync(user, request.Password))
    {
        // Return bad request if the password is invalid
        return BadRequest(new OpenIdConnectResponse
        {
            Error = OpenIdConnectConstants.Errors.InvalidGrant,
            ErrorDescription = "Invalid username or password"
        });
    }

    // The user is now validated, so reset lockout counts, if necessary
    if (_userManager.SupportsUserLockout)
    {
        await _userManager.ResetAccessFailedCountAsync(user);
    }

    // Create the principal
    var principal = await _signInManager.CreateUserPrincipalAsync(user);

    // Claims will not be associated with specific destinations by default, so we must indicate whether they should
    // be included or not in access and identity tokens.
    foreach (var claim in principal.Claims)
    {
        // For this sample, just include all claims in all token types.
        // In reality, claims' destinations would probably differ by token type and depending on the scopes requested.
        claim.SetDestinations(OpenIdConnectConstants.Destinations.AccessToken, OpenIdConnectConstants.Destinations.IdentityToken);
    }

    // Create a new authentication ticket for the user's principal
    var ticket = new AuthenticationTicket(
        principal,
        new AuthenticationProperties(),
        OpenIdConnectServerDefaults.AuthenticationScheme);

    // Include resources and scopes, as appropriate
    var scope = new[]
    {
        OpenIdConnectConstants.Scopes.OpenId,
        OpenIdConnectConstants.Scopes.Email,
        OpenIdConnectConstants.Scopes.Profile,
        OpenIdConnectConstants.Scopes.OfflineAccess,
        OpenIddictConstants.Scopes.Roles
    }.Intersect(request.GetScopes());

    ticket.SetResources("http://localhost:5000/");
    ticket.SetScopes(scope);

    // Sign in the user
    return SignIn(ticket.Principal, ticket.Properties, ticket.AuthenticationScheme);
}

Testing the Authentication Server

At this point, our simple authentication server is done and should work to issue JWT bearer tokens for the users in our database.

OpenIddict implements OpenID Connect, so our sample should support a standard /.well-known/openid-configuration endpoint with information about how to authenticate with the server.

If you’ve followed along building the sample, launch the app and navigate to that endpoint. You should get a json response similar to this:

{
  "issuer": "http://localhost:5000/",
  "jwks_uri": "http://localhost:5000/.well-known/jwks",
  "token_endpoint": "http://localhost:5000/connect/token",
  "code_challenge_methods_supported": [ "S256" ],
  "grant_types_supported": [ "password" ],
  "subject_types_supported": [ "public" ],
  "scopes_supported": [ "openid", "profile", "email", "phone", "roles" ],
  "id_token_signing_alg_values_supported": [ "RS256" ]
}

This gives clients information about our authentication server. Some of the interesting values include:

  • The jwks_uri property is the endpoint that clients can use to retrieve public keys for validating token signatures from the issuer.
  • token_endpoint gives the endpoint that should be used for authentication requests.
  • The grant_types_supported property is a list of the grant types supported by the server. In the case of this sample, that is only password.
  • scopes_supported is a list of the scopes that a client can request access to.

If you’d like to check that the correct certificate is being used, you can navigate to the jwks_uri endpoint to see the public keys used by the server. The x5t property of the response should be the certificate thumbprint. You can check this against the thumbprint of the certificate you expect to be using to confirm that they’re the same.

Finally, we can test the authentication server by attempting to login! This is done via a POST to the token_endpoint. You can use a tool like Postman to put together a test request. The address for the post should be the token_endpoint URI and the body of the post should be x-www-form-urlencoded and include the following items:

  • grant_type must be ‘password’ for this scenario.
  • username should be the username to login.
  • password should be the user’s password.
  • scope should be the scopes that access is desired for.
  • resource is an optional parameter which can specify the resource the token is meant to access. Using this can help to make sure that a token issued to access one resource isn’t reused to access a different one.

Here are the complete request and response from me testing the connect/token API:

Request

POST /connect/token HTTP/1.1
Host: localhost:5000
Cache-Control: no-cache
Postman-Token: f1bb8681-a963-2282-bc94-03fdaea5da78
Content-Type: application/x-www-form-urlencoded

grant_type=password&username=Mike%40Fabrikam.com&password=MikePassword1!&scope=openid+email+name+profile+roles

Response

{
  "token_type": "Bearer",
  "access_token": "eyJhbGciOiJSUzI1NiIsImtpZCI6IkU1N0RBRTRBMzU5NDhGODhBQTg2NThFQkExMUZFOUIxMkI5Qzk5NjIiLCJ0eXAiOiJKV1QifQ.eyJ1bmlxdWVfbmFtZSI6Ik1pa2VAQ29udG9zby5jb20iLCJBc3BOZXQuSWRlbnRpdHkuU2VjdXJpdHlTdGFtcCI6ImMzM2U4NzQ5LTEyODAtNGQ5OS05OTMxLTI1Mzk1MzY3NDEzMiIsInJvbGUiOiJBZG1pbmlzdHJhdG9yIiwib2ZmaWNlIjoiMzAwIiwianRpIjoiY2UwOWVlMGUtNWQxMi00NmUyLWJhZGUtMjUyYTZhMGY3YTBlIiwidXNhZ2UiOiJhY2Nlc3NfdG9rZW4iLCJzY29wZSI6WyJlbWFpbCIsInByb2ZpbGUiLCJyb2xlcyJdLCJzdWIiOiJjMDM1YmU5OS0yMjQ3LTQ3NjktOWRjZC01NGJkYWRlZWY2MDEiLCJhdWQiOiJodHRwOi8vbG9jYWxob3N0OjUwMDEvIiwibmJmIjoxNDc2OTk3MDI5LCJleHAiOjE0NzY5OTg4MjksImlhdCI6MTQ3Njk5NzAyOSwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo1MDAwLyJ9.q-c6Ld1b7c77td8B-0LcppUbL4a8JvObiru4FDQWrJ_DZ4_zKn6_0ud7BSijj4CV3d3tseEM-3WHgxjjz0e8aa4Axm55y4Utf6kkjGjuYyen7bl9TpeObnG81ied9NFJTy5HGYW4ysq4DkB2IEOFu4031rcQsUonM1chtz14mr3wWHohCi7NJY0APVPnCoc6ae4bivqxcYxbXlTN4p6bfBQhr71kZzP0AU_BlGHJ1N8k4GpijHVz2lT-2ahYaVSvaWtqjlqLfM_8uphNH3V7T7smaMpomQvA6u-CTZNJOZKalx99GNL4JwGk13MlikdaMFXhcPiamhnKtfQEsoNauA",
  "expires_in": 1800
}

The access_token is the JWT and is nothing more than a base64-encoded string in three parts ([header].[body].[signature]). A number of websites offer JWT decoding functionality.

The access token above has these contents:

{
  "alg": "RS256",
  "kid": "E57DAE4A35948F88AA8658EBA11FE9B12B9C9962",
  "typ": "JWT"
}.
{
  "unique_name": "Mike@Contoso.com",
  "AspNet.Identity.SecurityStamp": "c33e8749-1280-4d99-9931-253953674132",
  "role": "Administrator",
  "office": "300",
  "jti": "ce09ee0e-5d12-46e2-bade-252a6a0f7a0e",
  "usage": "access_token",
  "scope": [
    "email",
    "profile",
    "roles"
  ],
  "sub": "c035be99-2247-4769-9dcd-54bdadeef601",
  "aud": "http://localhost:5001/",
  "nbf": 1476997029,
  "exp": 1476998829,
  "iat": 1476997029,
  "iss": "http://localhost:5000/"
}.
[signature]

Important fields in the token include:

  • kid is the key ID that can be use to look-up the key needed to validate the token’s signature.
    • x5t, similarly, is the signing certificate’s thumbprint.
  • role and office capture our custom claims.
  • exp is a timestamp for when the token should expire and no longer be considered valid.
  • iss is the issuing server’s address.

These fields can be used to validate the token.

Conclusion and Next Steps

Hopefully this article has provided a useful overview of how ASP.NET Core apps can issue JWT bearer tokens. The in-box abilities to authenticate with cookies or third-party social providers are sufficient for many scenarios, but in other cases (especially when supporting mobile clients), bearer authentication is more convenient.

Look for a follow-up to this post coming soon covering how to validate the token in ASP.NET Core so that it can be used to authenticate and signon a user automatically. And in keeping with the original scenario I ran into with a customer, we’ll make sure the validation can all be done without access to the authentication server or identity database.

Resources

19 Nov 07:50

Real-Time Feature Engineering for Machine Learning with DocumentDB

by Emily Lawton

Ever wanted to take advantage of your data stored in DocumentDB for machine learning solutions? This blog post demonstrates how to get started with event modeling, featurizing, and maintaining feature data for machine learning applications in Azure DocumentDB.

Machine Learning and RFM

The field of machine learning is pervasive – it is difficult to pinpoint all the ways in which machine learning affects our day-to-day lives. From the recommendation engines that power streaming music services to the models that forecast crop yields, machine learning is employed all around us to make predictions. Machine learning, a method for teaching computers how to think and recognize patterns in data, is increasingly being used to help garner insights from colossal datasets – feats humans do not have the memory capacity and computational power to perform.

In the world of event modeling and machine learning, RFM is no strange concept. Driven by three dimensions (Recency, Frequency, Monetary), RFM is a simple yet powerful method for segmenting customers used often in machine learning models. The reasoning behind RFM is intuitive and consistent across most scenarios: a customer who bought something yesterday is more likely to make another purchase than a customer who has not bought anything in a year. In addition, spendy customers who frequently make purchases are also categorized as valuable using the RFM technique.

Properties of RFM features:

  • RFM feature values can be calculated using basic database operations.
  • Raw values can be updated online as new events arrive.
  • RFM features are valuable in machine learning models.

Because insights drawn from raw data become less useful over time, being able to calculate RFM features in near real-time to aid in decision-making is important [1]. Thus, a general solution that enables one to send event logs and automatically featurize them in near real-time so the RFM features can be employed in a variety of problems is ideal.

Where does DocumentDB fit in?

Azure DocumentDB is a blazing fast, planet-scale NoSQL database service for highly available, globally distributed apps that scales seamlessly with guaranteed low latency and high throughput. Its language integrated, transactional execution of JavaScript permits developers to write stored procedures, triggers, and user defined functions (UDFs) natively in JavaScript.

Thanks to these capabilities, DocumentDB is able to meet the aforementioned time constraints and fill in the missing piece between gathering event logs and arriving at a dataset composed of RFM features in a format suitable for training a machine learning model that accurately segments customers. Because we implemented the featurization logic and probabilistic data structures used to aid the calculation of the RFM features with JavaScript stored procedures, this logic is shipped and executed directly on the database storage partitions. The rest of this post will demonstrate how to get started with event modeling and maintaining feature data in DocumentDB for a churn prediction scenario.

The end-to-end code sample of how to upload and featurize a list of documents to DocumentDB and update RFM feature metadata is hosted on our GitHub.

Scenario

The first scenario we chose to tackle to begin our dive into the machine learning and event modeling space is the problem from the 2015 KDD Cup, an annual Data Mining and Knowledge Discovery competition. The goal of the competition was to predict whether a student will drop out of a course based on his or her prior activities on XuetangX, one of the largest massive open course (MOOC) platforms in China.

The dataset is structured as follows:

Real-Time Feature Engineering for Machine Learning with DocumentDB

Figure 1. We would like to gratefully acknowledge the organizers of KDD Cup 2015 as well as XuetangX for making the datasets available.

Each event details an action a student completed. Examples include watching a video or answering a particular question. All events consist of a timestamp, a course ID (cid), student ID (uid), and enrollment ID (eid) which is unique for each course-student pair.

Approach

Modeling Event Logs

The first step was to determine how to model the event logs as documents in DocumentDB. We considered two main approaches. In the first approach, we used the combination of <entity name, entity value, feature name> as the primary key for each document. An example primary key with this strategy is <”eid”, 1, “cat”>. This means that we created a separate document for each feature we wanted to keep track of when the student enrollment id is 1. In the case of a large number of features, this can result in a multitude of documents to insert. We took a bulk approach in the second iteration, using <entity name, entity value> instead as the primary key. An example primary key with this strategy is <”eid”, 1>. In this approach, we used a single document to keep track of all the feature data when the student enrollment id is 1.

The first approach minimizes the number of conflicts during insertion because there is the additional feature name attribute, making the primary key more unique. The resulting throughput is not optimal, however, in the case of a large number of features because an additional document needs to be inserted for each feature. The second approach maximizes throughput by featurizing and inserting event logs in a bulk manner, increasing the probability of conflicts. For this blog post, we chose to walk through the first approach, which provides for simpler code and fewer conflits.

Step 1

Create the stored procedure responsible for updating the RFM feature metadata.

private static async Task CreateSproc()
{
    string scriptFileName = @"updateFeature.js";
    string scriptName = "updateFeature";
    string scriptId = Path.GetFileNameWithoutExtension(scriptFileName);

    var client = new DocumentClient(new Uri(Endpoint), AuthKey);
    Uri collectionLink = UriFactory.CreateDocumentCollectionUri(DbName, CollectionName);

    var sproc = new StoredProcedure
    {
        Id = scriptId,
        Body = File.ReadAllText(scriptFileName)
    };
    Uri sprocUri = UriFactory.CreateStoredProcedureUri(DbName, CollectionName, scriptName);

    bool needToCreate = false;

    try
    {
        await client.ReadStoredProcedureAsync(sprocUri);
    }
    catch (DocumentClientException de)
    {
        if (de.StatusCode != HttpStatusCode.NotFound)
        {
            throw;
        }
        else
        {
            needToCreate = true;
        }
    }

    if (needToCreate)
    {
        await client.CreateStoredProcedureAsync(collectionLink, sproc);
    }
}

Step 2

Featurize each event. In this example, each student action expands into 12 rows of the form { entity: { name: “ “, value: …}, feature: { name: “ “, value: …} } that must be inserted in your DocumentDB collection with the previously created stored procedure. We did this process in batches, the size of which can be configured.

private static string[] Featurize(RfmDoc doc)
{
    List<string> result = new List<string>();

    var entities = new Tuple<string, object>[] { new Tuple<string, object>("eid", doc.Eid), new Tuple<string, object>("cid", doc.Cid), 
        new Tuple<string, object>("uid", doc.Uid) };
    var features = new Tuple<string, object>[] { new Tuple<string, object>("time", doc.Time), new Tuple<string, object>("src_evt", doc.SourceEvent), 
        new Tuple<string, object>("cat", doc.Cat), new Tuple<string, object>("obj", doc.Obj) };

    foreach (var entity in entities)
    {
        foreach (var feature in features)
        {
            StringBuilder eb = new StringBuilder();
            StringBuilder fb = new StringBuilder();
            StringWriter eWriter = new StringWriter(eb);
            StringWriter fWriter = new StringWriter(fb);

            JsonSerializer s = new JsonSerializer();
            s.Serialize(eWriter, entity.Item2);
            string eValue = eb.ToString();

            s.Serialize(fWriter, feature.Item2);
            string fValue = fb.ToString();

            var value = string.Format(CultureInfo.InvariantCulture, "{{\"entity\":{{\"name\":\"{0}\",\"value\":{1}}},\"feature\":{{\"name\":\"{2}\",\"value\":{3}}}}}",
                entity.Item1, eValue, feature.Item1, fValue);
            result.Add(value);
        }
    }

    return result.ToArray();
}

Step 3

Execute the stored procedure created in step 1.

private static async Task<StoredProcedureResponse<string>> UpdateRFMMetadata(DocumentClient client, string metaDoc)
{
    object metaDocObj = JsonConvert.DeserializeObject(metaDoc);

    int retryCount = 100;
    while (retryCount > 0)
    {
        try
        {
            Uri sprocUri = UriFactory.CreateStoredProcedureUri(DbName, CollectionName, "updateFeature");
            var task = client.ExecuteStoredProcedureAsync<string>(
                sprocUri,
                metaDocObj);
            return await task;
        }
        catch (DocumentClientException ex)

The stored procedure takes as input a row of the form { entity: { name: “ ”, value: …}, feature: { name: “ ”, value: …} } and updates the relevant feature metadata to produce a document of the form { entity: { name: "", value: "" }, feature: { name: "", value: ...}, isMetadata: true, aggregates: { "count": ..., "min": ... } }. Depending on the name of the feature in the document that is being inserted into DocumentDB, a subset of predefined aggregates is updated. For example, if the feature name of the document is “cat” (category), the count_unique_hll aggregate is employed to keep track of the unique count of categories. Alternatively, if the feature name of the document is “time”, the minimum and maximum aggregates are utilized. The following code snippet demonstrates how the distinct count and minimum aggregates are updated. See the next section for a more detailed description of the data structures that we are using to maintain these aggregates.

case AGGREGATE.count_unique_hll:
    if (aggData === undefined) aggData = metaDoc.aggregates[agg] = new CountUniqueHLLData();
    aggData.hll = new HyperLogLog(aggData.hll.std_error, murmurhash3_32_gc, aggData.hll.M);

    let oldValue = aggData.value = aggData.hll.count();
    aggData.hll.count(doc.feature.value); // add entity to hll
    aggData.value = aggData.hll.count();

    if (aggData.value !== oldValue && !isUpdated) isUpdated = true;
    break;
case AGGREGATE.min:
    if (aggData === undefined) aggData = metaDoc.aggregates[agg] = new AggregateData();
    if (aggData.value === undefined) aggData.value = doc.feature.value;
    else if (doc.feature.value < aggData.value) {
        aggData.value = doc.feature.value;
        if (!isUpdated) isUpdated = true;
    }
    break;

Probabilistic Data Structures

We implemented the following three probabilistic data structures in JavaScript, each of which can be updated conditionally as part of the stored procedure created in the previous section.

HyperLogLog

Approximates the number of unique elements in a multiset by applying a hash function to each element in the multiset (obtaining a new multiset of uniformly distributed random numbers with the same cardinality as the original set) and calculating the maximum number of leading zeros in the binary representation of each number in the new set n. The estimated cardinality is 2^n [2].

BloomFilter

Tests whether an element is a member of a set. While false positives are possible, false negatives are not. Rather, a bloom filter either returns maybe in the set or definitely not in the set when asked if an element is a member of a set. To add an element to a bloom filter, the element is fed into k hash functions to arrive at k array positions. The bits at each of those positions are set to 1. To test whether an element is in the set, the element is again fed to each of the k hash functions to arrive at k array positions. If any one of the bits is 0, the element is definitely not in the set [3].

Count-Min Sketch

Ingests a stream of events and counts the frequency of distinct members in the set. The sketch may be queried for the frequency of a specific event type. Similar to the bloom filter, this data structure uses some number of hash functions to map events to values – however, it uses these hash functions to keep track of event frequencies instead of whether or not the event exists in the dataset [4].

Each of the above data structures returns an estimate within a certain range of the true value, with a certain probability. These probabilities are tunable, depending on how much memory you are willing to sacrifice. The following snippet shows how to retrieve the HyperLogLog approximation for the number of unique objects for the student with eid = 1.

private static void OutputResults()
{
    var client = new DocumentClient(new Uri(Endpoint), AuthKey);
    Uri collectionLink = UriFactory.CreateDocumentCollectionUri(DbName, CollectionName);

    string queryText = "select c.aggregates.count_unique_hll[\"value\"] from c where c.id = \"_en=eid.ev=1.fn=obj\"";
    var query = client.CreateDocumentQuery(collectionLink, queryText);

    Console.WriteLine("Result: {0}", query.ToList()[0]);
}

Conclusion

The range of scenarios where RFM features can have a positive impact extends far beyond churn prediction. Time and time again, a small number of RFM features have proven to be successful when used in a wide variety of machine learning competitions and customer scenarios.

Combining the power of RFM with DocumentDB’s server-side programming capabilities produces a synergistic effect. In this post, we demonstrate how to get started with event modeling and maintaining feature data with DocumentDB stored procedures. It is our hope that developers are now equipped with the tools to add functionality to our samples hosted on GitHub to maintain additional feature metadata on a case by case basis. Stay tuned for a future post that details how to integrate this type of solution with Azure Machine Learning where you can experiment with a wide variety of machine learning models on your data featurized by DocumentDB.

To learn more about how to write database program application logic that can be shipped and executed directly on the database storage partitions in DocumentDB, see DocumentDB server-side programming: Stored procedures, database triggers, and UDFs. Stay up-to-date on the latest DocumentDB news and features by following us on twitter @DocumentDB.

Lastly, please reach out to us at askdocdb@microsoft.com or leave a comment below for inquiries about additional ML support and to show us how you’re using DocumentDB for machine learning.

References

[1] Oshri, Gal. “RFM: A Simple and Powerful Approach to Event Modeling.” Cortana Intelligence and Machine Learning Blog (2016). https://blogs.technet.microsoft.com/machinelearning/2016/05/31/rfm-a-simple-and-powerful-approach-to-event-modeling/

[2] https://gist.github.com/terrancesnyder/3398489, http://stackoverflow.com/questions/5990713/loglog-and-hyperloglog-algorithms-for-counting-of-large-cardinalities

[3] https://github.com/jasondavies/bloomfilter.js, Copyright © 2011, Jason Davies

[4] https://github.com/mikolalysenko/count-min-sketch, The MIT License (MIT), Copyright © 2013 Mikola Lysenko

19 Nov 07:48

Azure IoT loves Open Source

by Sam George

I’m pleased to announce the release of a new open source library to connect to Azure IoT Hub. IoTHubReact is an Akka Stream library now available on GitHub that allows Java and Scala developers to read telemetry data from devices connected to Azure IoT Hub. This library represents another step toward building an open platform that enable developers to use various open source technologies along with Azure services such as Azure IoT Hub.

We plan to release more libraries, samples and demo applications in the next few months as part of our ongoing commitment to an open IoT ecosystem.

19 Nov 07:47

What’s brewing in Visual Studio Team Services: October 2016 Digest

by Buck Hodges

This post series provides the latest updates and news for Visual Studio Team Services and is a great way for Azure users to keep up-to-date with new features being released every three weeks. Visual Studio Team Services offers the best DevOps tooling to create an efficient continuous integration and release pipeline to Azure. With the rapidly expanding list of features in Team Services, teams can start to leverage it more efficiently for all areas of their Azure workflow, for apps written in any language and deployed to any OS.

Git best practice with Team Services: Branch Policies

How can you ensure you are finding bugs before they’re introduced into your codebase while still ensuring you have the right people reviewing? Branch policies can go a long way to enhancing your Pull Requests workflow.

Configure branch policies from the branches page.

Becoming more productive with Git: Tower and Team Services

Working with Git in Visual Studio Team Services and Team Foundation Server just became even easier: the popular Git desktop client Tower now comes with dedicated integrations for these services.

Team Services Git repositories in Tower

One-click Import of Git repositories into Team Services

Teams can now import a Git repository from GitHub, BitBucket, GitLab, or other locations. You can import into either a new or an existing empty repository.

import private repo

Enable continuous deployment to App Stores with Team Services

Whether you build apps for the iOS, Android or Windows, Team Services has app store extensions that make it easy to publish your app and set up continuous deployment

New features released in September 2016

Two rounds of Team Services updates in September empower your team to get stuff done faster so you can enjoy those pumpkin spice lattes and the crisp autumn air. Custom work item types, more static analysis options in builds and a new feedback option in the Exploratory Testing extension are just a few among many new delightful updates.

 Deployment widget

Build and Release pricing update

Release Management is coming out of trial mode in Team Foundation Server “15”. Learn more to see how teams get billed for releases in Team Services and TFS post TFS “15” release.

New build queue tab

A redesign of the Queued builds experience in the Build hub brings richer details of your queued/running builds in a more intuitive way.

Redesigned Queued Builds page

Changes to the way you log into Team Services

The new screens simplify login for users in organizations that use Azure Active Directory, bringing the experience more in line with the way you login to Azure, Office 365, etc.

New Azure SignIn Flow

19 Nov 07:45

Azure IoT Gateway SDK integrates support for Azure Functions

by Sam George

At Microsoft, we believe the edge of the network plays a critical role in IoT, not only in IoT devices themselves but also in intermediate field gateways.  Earlier this year, we announced the open source Azure IoT Gateway SDK, our approach to accelerating the development of IoT edge scenarios such as supporting legacy devices, minimizing latency, conserving network bandwidth, and addressing security concerns.  Since then, we’ve been busy improving and enhancing our SDK completely out in the open.

Today, we are happy to announce an exciting new capability we’ve added to the IoT Gateway SDK: Support for Azure Functions.  With Azure Functions integration, developers can easily call cloud-based logic from their IoT gateway. Just write an Azure Function and you can quickly call it from a Function Module in the Azure IoT Gateway SDK.

For example: if something goes wrong in your field gateway environment, such as local devices that can’t connect or misbehave, and you want to upload diagnostic information to your Azure IoT solution for inspection by operations, our new Functions integrations makes this simple. Just create an Azure Function that takes this data, stores it and alerts operations – and then call it from your gateway running the Azure IoT Gateway SDK when you encounter a problem.

The Azure IoT Gateway SDK supports everything from low-level modules written in C to connect the broad variety of deployed devices, to high level modules for productivity such as our new Azure Functions support.  The best part of the Azure IoT Gateway SDK is how easy it is to chain these modules together to create reusable processing pipelines that suit your needs.

We’ve already seen some great success stories from businesses benefitting from our approach to edge intelligence, and we’re looking forward to seeing what our customers and partners will create with this exciting new capability.

19 Nov 07:37

Azure Stream Analytics query testing now available in the new portal

by Ryan CrawCour

Azure Stream Analytics is a fully managed service allowing you to gain insights and run analytics in near real-time on your big data streaming workloads. The service was first deployed more than 2 years ago, long before the “new” Azure management portal, http://portal.azure.com, even existed.

For the past few months we’ve been hard at work adding exciting new features to the service as well as transitioning the management user interface from the old https://manage.windowsazure.com to the new portal

Today we want to announce that we’ve just added the ability to test queries in the “new” portal without needing to start or stop a job. Here’s a quick look at how this works.

Setup

You can setup a Stream Analytics by following this simple tutorial - How to create a Stream Analytics job

Once you have created a new Stream Analytics job you would typically Create Inputs and then Create Outputs. Or you can just skip ahead to building the query and once your query is working then go back and define the Inputs and Outputs to match those used in the query. Both ways work, giving you the flexibility to decide how you wish to work.

For the purposes of this blog post I have defined a job with 1 data stream input, called StreamInput and 1 output, called Output. You can see these in the query editor blade above.

0

Open the Query editor blade from the job details screen by clicking on the query in the “Query” lens. Or in our case the < > placeholder because there is no query yet.

1

You will be presented with the rich editor as before where you create your query. This blade has now been enhanced with a new pane on the left. This new pane shows the Inputs and Outputs used by the Query, and those defined for this job.

2

There is also 1 additional Input and Output shown which I did not define. These come from the new query template that we start off with. These will change, or even disappear all together, as we edit the query. You can safely ignore them for now.

A key requirement and a common ask from our customers while writing a query is being able to test, and test often, to ensure that the output is what it is expected to be, given some input data. Having to save the query after every edit, start the job, wait for incoming data, check the results, and then stop the job again each time you make a small change to the query would be slow and is sometimes not even possible. A way to test changes to a query quickly was needed.

I am happy to announce that with today’s latest release in the portal you can now test the query without going through this stop/start process. Here's how ...

Sample data and testing queries

To test with sample input data, right click on any of your Inputs and choose to Upload sample data from file.

3

Once the upload has completes you can then use the Test button to test this query against the sample data you have just provided.

4

The output of your query is displayed in the browser, with a link to Download results should you wish to save the test output for later use. You can now easily and iteratively modify your query, and test repeatedly to see how the output changes.

5

In the diagram above you can see how I have changed the query inline to have a 2nd output, called HighAvgTempOutput where I am only writing a subset of the data being received.
With multiple outputs used in a query you can see the results for both outputs separately and easily toggle between them.
Once you are happy with the results in the browser, then you can save your query, start your job, sit back and watch the magic of Stream Analytics happen for you.

Feature Parity and the road ahead

With the long awaited addition of sample data and query testing in the new portal we are happy to announce that we have reached feature parity between the portals. Everything you could do before, and more, is now in the new portal. Going forward all new development efforts will be concentrated on the new portal. The old portal will continue to work and existing functionality will remain until end of the calendar year when we place to completely retire support for Stream Analytics in the old portal.
If you have not tried Stream Analytics in the new portal we encourage you to head over and give it a try.

Next Steps

We’re really excited to bring local testing to the new portal and take this final step to reaching feature parity across the two portals. We hope this makes your life much easier as you go about developing (and testing) your queries.

We invite you to provide feedback on our User Voice page about what you want added next to the service!

If you are new to either Microsoft Azure or Stream Analytics, try it out by signing up for a free Azure trial account and create your first Stream Analytics job.

If you need help or have questions, please reach out to us through the MSDN or Stackoverflow forums, email the product team directly.

19 Nov 07:29

Temporal Tables are generally available in Azure SQL Database

by Borko Novakovic

Temporal Tables allow you to track the full history of data changes directly in Azure SQL Database, without the need for custom coding. With Temporal Tables you can see your data as of any point in time in the past and use declarative cleanup policy to control retention for the historical data.

When to use Temporal Tables?

Quite often you may be in the situation to ask yourself fundamental questions: How did important information look yesterday, a month ago, a year ago, etc. What changes have been made since the beginning of the year? What were the dominant trends during a specific period of time?  Without proper support in the database, however, questions like these have never been easy to answer.
Temporal Tables are designed to improve your productivity when you develop applications that work with ever-changing data and when you want to derive important insights from the changes.
Use Temporal Tables to:

  1. Support data auditing in your applications
  2. Analyze trends or detect anomalies over time
  3. Easily implement slowly changing dimension pattern
  4. Perform fine-grained row repairs in case of accidental data errors made by humans or applications

Manage historical data with easy-to-use retention policy

Keeping history of changes tends to increase database size, especially if historical data is retained for a longer period of time. Hence, retention policy for historical data is an important aspect of planning and managing the lifecycle of every temporal table.  Temporal Tables in Azure SQL Database come with an extremely easy-to-use retention mechanism. Applying retention policy is very simple: it requires users to set single parameter during the table creation or table schema change, like shown in the following example.

ALTER TABLE [WebSiteClicks]
SET 
(
	SYSTEM_VERSIONING = ON 
	(
		HISTORY_TABLE = dbo. WebSiteClicks_History, 
                HISTORY_RETENTION_PERIOD = 3 MONTHS  
	)
);

You can alter retention policy at any moment and your change will be effective immediately.

Why you should consider Temporal Tables?

If you have requirements for tracking data changes, using Temporal Tables will give you multiple benefits over any custom solution. Temporal Tables will simplify every phase in the development lifecycle: object creation, schema evolution, data modification, point-in-time analysis and data aging.

Temporal Benefits

Next steps

To learn how to integrate Temporal Tables in your application, read the following article with the step-by-step instructions. To utilize temporal retention, check out  Manage temporal history with retention policy article on Azure.com.
Visit Channel 9 to hear a real customer story and watch a live presentation with the demo. For more information, check out MSDN documentation.

19 Nov 07:28

Simpler Azure Management Libraries for .NET

by Asir Selvasingh

One C# statement to authenticate. One statement to create a virtual machine. One statement to modify an existing virtual network, etc. No more guessing about what is required vs. optional vs. non-modifiable.

https://github.com/Azure/azure-sdk-for-net/tree/Fluent

We are announcing the first developer preview release of the new, simplified Azure management libraries for .NET. Our goal is to improve the developer experience by providing a higher-level, object-oriented API, optimized for readability and writability. These libraries are built on the lower-level, request-response style auto generated clients and can run side-by-side with auto generated clients.

Azure Authentication

One statement to authenticate and choose a subscription. The Azure class is the simplest entry point for creating and interacting with Azure resources.

Azure azure = Azure.Authenticate(credFile).WithDefaultSubscription();

Create a Virtual Machine

You can create a virtual machine instance by using a Define() … Create() method chain.

Console.WriteLine("Creating a Windows VM");

var windowsVM = azure.VirtualMachines.Define("myWindowsVM")
    .WithRegion(Region.US_EAST)
    .WithNewResourceGroup(rgName)
    .WithNewPrimaryNetwork("10.0.0.0/28")
    .WithPrimaryPrivateIpAddressDynamic()
    .WithNewPrimaryPublicIpAddress("mywindowsvmdns")
    .WithPopularWindowsImage(KnownWindowsVirtualMachineImage.WINDOWS_SERVER_2012_R2_DATACENTER)
    .WithAdminUserName("tirekicker")
    .WithPassword(password)
    .WithSize(VirtualMachineSizeTypes.StandardD3V2)
    .Create();

Console.WriteLine("Created a Windows VM: " + windowsVM.Id);

Update a Virtual Machine

You can update a virtual machine instance by using an Update() … Apply() method chain.

windowsVM.Update()
     .WithNewDataDisk(10)
     .DefineNewDataDisk(dataDiskName)
          .WithSizeInGB(20)
          .WithCaching(CachingTypes.ReadWrite)
          .Attach()
     .Apply();

Management libraries unleash the power of IntelliSense in Visual Studio

Fluent interface-inspired method chains in combination with IntelliSense deliver a wizard-like developer experience by presenting required and optional methods in the right sequence. For example, once you choose a Windows virtual machine image, IntelliSense will prompt for an admin password and nothing else.

create-vm

Then, IntelliSense will prompt for a password and nothing else. This will continue until you reach the minimum required to call create().

create-vm-create

As another example, if you were to choose a Linux virtual machine image, IntelliSense would prompt for a root user name and then SSH key.

Samples

You can find plenty of sample code that illustrates key management scenarios in Azure Virtual Machines, Virtual Machine Scale Sets, Storage, Networking, Resource Manager, Key Vault and Batch …

Service

Management Scenario

Virtual Machines

Virtual Machines - parallel execution

Virtual Machine Scale Sets

Storage

Network

Resource Groups

Key Vault

Batch

Give it a try

This is a developer preview that supports major parts of Azure Virtual Machines, Virtual Machine Scale Sets, Storage, Networking, Resource Manager, Key Vault and Batch. You can run the samples above or go straight to our GitHub repo.

Give it a try and let us know what do you think (via e-mail or comments below), particularly -

  • Usability and effectiveness of the new management libraries for .NET?
  • What Azure services you would like to see supported soon?
  • What additional scenarios should be illustrated as sample code?

The next preview version of the Azure Management Libraries for .NET is a work in-progress. We will be adding support for more Azure services and tweaking the API over the next few months.

14 Nov 13:18

Las Vegas gets “kinetic tiles” that power lights with foot traffic

by Megan Geuss

Enlarge (credit: EnGoPlanet)

A New York-based startup called EnGoPlanet has installed four streetlights in a plaza off the Las Vegas Strip that are powered exclusively by solar and kinetic energy. The installations aren't mere streetlights though—they also power a variety of environmental monitors, support video surveillance, and, for the masses, offer USB ports for device charging.

The streetlights are topped by a solar panel crest, and have “kinetic tiles” on the ground below them. These panels reportedly can generate 4 to 8 watts from people walking on them, depending on the pressure of the step. The renewable energy is then collected by a battery for use at night. The solar-plus-kinetic energy design is useful on those rare Vegas days without too much sun—as long as there is still plenty of foot traffic.

The four streetlights have a host of sensors that collect information, and details on what kind of information is collected are sparse. In EnGoPlanet’s promotional video, a quick slide lists the streetlights’ additional capabilities: environmental monitoring, air quality monitoring, video surveillance, and the ever-vague “smart analytics.” If the bright side of progress is more environmentally-friendly streetlights, the dark side is that as you replace those old analog streetlights you get the addition of video surveillance from a private company.

Read 2 remaining paragraphs | Comments

13 Nov 08:26

See how the Contoso Corporation is implementing Azure services

by Joe Davies

The new Contoso in the Microsoft Cloud poster shows how the Contoso Corporation, a fictional but representative global organization, is embracing a cloud-inclusive IT infrastructure that includes all of Microsoft’s cloud offerings (Microsoft Azure, Office 365, Dynamics 365, and Microsoft Intune).

This new poster contains the following:

  • An overview of the fictional Contoso Corporation, its three-tier design for its offices, and key elements of Contoso’s implementation of the Microsoft cloud.

  • A review of Contoso’s on-premises IT infrastructure and how its business needs map to Microsoft’s cloud offerings, including opportunities for Azure PaaS and IaaS.

  • Contoso’s private WAN infrastructure, its app design for its offices, its use of Azure ExpressRoute Premium, and its step-by-step path to cloud networking readiness.

  • Contoso’s on-premises Windows Server Active Directory (AD) forest, how it geographically distributes authentication traffic, and how it uses Azure IaaS to provide authentication infrastructure redundancy for the 15,000 workers in its Paris headquarters.

  • Contoso’s structure for Microsoft cloud offering subscriptions, licenses, its common Azure AD tenant, and user accounts synched with Azure AD from its on-premises Windows Server AD forest.

  • Contoso’s security requirements in the cloud, its data sensitivity classification and mapping of cloud features to each level, and its step-by-step path to cloud security readiness.

Download this multi-page poster in PDF or Visio format, or see it online on Docs.com. You can easily print the 9 pages of this poster on tabloid format paper (also known as 11x17, ledger, or A3 format).

Future additions to this poster will include end-to-end configurations of key cloud scenarios, such as an Azure PaaS app that replaces their current document access system and SQL Server Stretch Database for their historical transaction data. Stay tuned for the updates as we evolve the Contoso Corporation from an on-premises IT infrastructure to a cloud-enabled and agile IT infrastructure.

If there is a scenario you would like to see in a future update to this poster, send a quick email to CloudAdopt@microsoft.com.

Here are the additional posters in the Microsoft Cloud for Enterprise Architects Series.

Microsoft Cloud Services and Platform Options

Microsoft Cloud Identity for Enterprise Architects

Microsoft Cloud Security for Enterprise Architects

Microsoft Hybrid Cloud for Enterprise Architects

Microsoft Cloud Networking for Enterprise Architects

Microsoft Cloud Storage for Enterprise Architects

Microsoft Cloud Mobility and Security for Enterprise Architects

13 Nov 08:24

Use actors to talk to millions

by James Staten

You have hundreds, thousands, perhaps even millions of sensors in your stores, factories, pipelines. And something goes wrong. You need software that can make intelligent decisions in real time, perhaps coordinating hundreds of devices: stop the assembly line, reroute the flow.

Or you’ve created a SaaS service that’s suddenly become wildly successful, with tens of thousands of users online at any given time. Your site is groaning under the strain. Your users demand real-time response, you need resiliency, you can’t wait on SQL databases – and you need it to be easy to program so your developers can get it to market fast.

Halo5

There’s a solution, and it’s been “battle-tested” in a surprising place: online video games, in particular, the best-selling “Halo” franchise from Microsoft. “Actor frameworks” make massive scale for millions of interacting users (or devices) a straightforward thing.

Think about it: when you log on to Halo online, there’s a little piece of code running in the cloud – it represents you; and there may be hundreds of thousands of other players online simultaneously, all with their own little piece of code. That little code object is your “actor,” and it keeps track of where you are in the game, your score, your weapons.

It’s you in the cloud.

The same concept can be used for managing IoT sensors, cars, customers – any scenario where you have to keep track of lots of things at once. Little code objects that represent something real. Not you (or your sensor or your customer): but code in the cloud that acts on your behalf. An actor.

Which server is your actor running on? Where is your opponent’s? What if the server fails? How do you scale? Your developers don’t have to care: the framework takes care of all the low-level infrastructure, scale, networking, and failover.

The point: massive, distributed scale – which used to be a Hard Problem – isn’t anymore. Your developers focus on business logic – adding value to your business.

The “actor frameworks” in Halo were originally developed by Microsoft Research and 343 Industries. But there’s been such demand for a “productized” version that we’ve incorporated the actor model into our comprehensive next-generation microservice cloud application framework, called Service Fabric, available in Azure (and on-premises as well) today. Service Fabric is equally battle-hardened, being the evolution of the application infrastructure used in mission-critical applications like Skype for Business, Cortana, and a number of Azure services. It includes auto-scaling, integrated health monitoring, service healing, orchestration and automation for microservices of all kinds – actors, as well as containerized (like Docker) applications.

Because developers focus only on business logic, not infrastructure, actors accelerate your time to market. Dr. Gregory Athas, principal software architect at BMW, who implemented Service Fabric actors for their BMW Connected application, says, “We’ve found actors to be a natural way to model users in our system. They allow us to focus on our core functionality while inherently supporting persistence, scalability, and resiliency.” Similarly, Stephen Berard of Schneider Electric adds, “Service Fabric reliable actors enabled us to build a scalable solution for implementing our device logic within EcoStruxure.io.”

You may have detected a theme in our recent posts: that increasingly the cloud permits developers and analysts to focus on business logic, and not infrastructure. Actor-model is a great example of this: the cloud provides hyperscale, failover, and all sorts of other benefits, while your developers focus on adding value to your business.

That’s what enterprise computing in the 21st century is all about.

*  *  *

Oh, and a postscript: You can do all sorts of things with actors. One of my colleagues, Barry Briggs, wrote an actor-based spreadsheet – where each cell is an actor -- a few years ago as a technology demonstration running in Azure. Using several hundred cores he was able to load and model the world’s historical weather records – into one sheet.

Here he is demonstrating (with a nice technical description of actor model):

13 Nov 08:15

Microsoft Azure SQL Database provides unparalleled performance with In-Memory technologies

by Rohan Kumar

Azure SQL Database built-in In-Memory technologies are now generally available for the Premium database tier including Premium pools. In-memory technology helps optimize the performance of transactional (OLTP), analytics (OLAP), as well as mixed workloads (HTAP). These technologies allow you to achieve phenomenal performance with Azure SQL Database – 75,000 transactions per second for order processing (11X perf gain) and reduced query execution time from 15 seconds to 0.26 (57X perf). You can also use them to reduce cost – on a P2 database obtain 9X perf gain for transactions or 10X perf gain for analytics queries by implementing In-Memory technologies, without any additional cost! See below for details about these performance and cost savings results.

  • In-Memory OLTP increases throughput and reduces latency for transaction processing. Scenarios such as trading and gaming really see the performance benefits. Another common scenario is data ingestion from events or IoT devices. You can also use it to speed up caching, data load, and temp table and table variable scenarios.
  • Clustered Columnstore Indexes reduce storage footprint (up to 10X) and improve performance for reporting and analytics queries. Use it with fact tables in your data marts to fit more data in your database and improve performance. Use it with historical data in your operational database to archive and be able to query up to 10 times more data.
  • Non-clustered Columnstore Indexes for Hybrid Transactional and Analytical Processing (HTAP) gain real-time insights into your business by querying the operational database directly, without the need to run an expensive ETL process and wait for the data warehouse to be populated. Non-clustered Columnstore indexes allow very fast execution of analytics queries on the OLTP database, while reducing the impact on the operational workload.
  • In-Memory OLTP and Columnstore can also be combined: you can have a memory-optimized table with a columnstore index, allowing you to both perform very fast transaction processing and run analytics queries very fast on the same data.

Quorum Business Solutions provides innovative software as a service (SaaS) solutions for field operations in the Oil and Gas industry running on Microsoft Azure. With In-Memory OLTP they were able to grow their business by onboarding new customers and supporting organizations at much larger scale, without spending more on more database throughput.

clip_image002 “Scalable performance is critical with our IoT platform for oil and gas that must run 24/7/365.  The addition of In-Memory OLTP tables and native-compiled stored procedures on Azure SQL Database for a few key operations immediately reduced our overall DTU consumption by seventy percent.  Without in-memory tables, our growth would have required significant effort to multiple areas of the platform to maintain performance.  For data-centric services, in-memory support provides instant scale to existing applications with little to no changes outside of the database.” Mark Freydl, solution architect, Quorum Business Solutions 

For more details about Quorum Business Solutions’ use of Azure SQL Database and the benefits they see with In-Memory OLTP, read this case study.

Spotlight is a solution provided by Quest for the monitoring of SQL Server deployments in their customers’ data centers and in the cloud. They leverage In-Memory OLTP to speed up the processing of requests and events such that Quest’s customers learn about any potential issues in their environments very quickly.

quest-logo“Quest’s customers rely on Spotlight to tell them in a timely manner about issues in their SQL Server environments. Quest relies on In-Memory OLTP in Azure SQL Database to provide a robust base on which to build Spotlight features quickly and deliver them to customers. With In-Memory OLTP, we get extremely high throughput and all the goodness of a flexible and well understood programming model based on T-SQL – this means we can deliver a high quality experience to our Spotlight customers very quickly.” Patrick O’Keeffe, executive director, software engineering (information management)

M-Files provides enterprise information management solutions that eliminate information silos and provide quick and easy access to the right content from any core business system and device. Non-clustered columnstore indexes allow running very fast analytical queries on the document data, in turn allowing M-Files’ customers to gain insights and find information much faster.

clip_image006“Lightning fast analytical queries as well as online transactions are a critical success factor in M-Files implementations. The addition of updatable non-clustered columnstore indexes in Azure SQL Database and in Microsoft SQL Server 2016 helped us to achieve over 10x faster queries in critical workloads at our customers without significant performance cost on online transaction processing. The change required no code changes in our application and provided instant performance boost both to our clients on-premises as well as to our M-Files Cloud Vault users in Azure.” Antti Nivala, founder and CTO

Performance and cost savings benefits

Both In-Memory OLTP and Columnstore Indexes achieve their respective performance benefits by utilizing resources such as CPU and IO more efficiently. Each Azure SQL Database pricing tier has a certain amount of resources allocated to it. Since In-Memory technologies are more resource-efficient, this means that you can achieve performance gains without having to increase your pricing tier. On the flip side, you can achieve the same performance level with a lower pricing tier when adopting In-Memory technologies in your database.

Some performance results for In-Memory OLTP in Azure SQL Database:

  • Order processing benchmark (scale factor 100, with 400 clients) on P15 (the highest tier at the time of writing): 75,000 transactions per second (TPS) with In-Memory OLTP, compared with 6,800 TPS with traditional tables and stored procedures, which translates to 11X performance gain with In-Memory OLTP.
  • Also lower performance tiers show impressive results. For a P2 with the same workload (scale factor 5, with 200 clients): 8,900 TPS with In-Memory OLTP, compared with 1,000.
  • 30-40% performance gain just by replacing traditional table-valued parameters (TVPs) with memory-optimized TVPs: High Speed IoT Data Ingestion Using In-Memory OLTP in Azure

Performance results for Columnstore indexes, using the sample on this page:

  • Using a P15 database (the highest performance tier at the time of writing), the query runs in 0.26 seconds with Columnstore, while it runs in 15 seconds when using traditional indexes with page compression. This translates to a performance gain of 58X!
  • Even with lower pricing tiers you see significant performance benefits: using a P1 database, the query runs in 4.8 seconds with Columnstore, while it runs in 27 seconds using traditional indexes with page compression. A performance gain of 5.6X without increasing the pricing tier!

Try out In-Memory technologies in Azure SQL Database today

For more information about In-Memory technologies in Azure SQL Database, tips on how to get started, as well as samples, check out the SQL In-Memory documentation.

 

Stay tuned on this blog for more detailed information about the individual In-Memory technologies over the next few weeks!

13 Nov 08:12

Azure big data services host Ask Me Anything session

by Maxim Lukiyanov

The big data teams on Azure will host a special Ask Me Anything session on /r/Azure, Thursday, November 17, 2016 from 10:00 am to 2:00 pm PST.

What's an AMA session?

We'll have folks from the Azure big data engineering teams available to answer any questions you have. You can ask us anything about our products, services or even our teams!

Why are you doing an AMA?

We like reaching out and learning from our customers and the community. We want to know how you use big data in the Azure cloud and how your experience has been on Azure Data Lake, HDInsight, Data Factory and Stream Analytics services. Your questions provide insights into how we can make the service better. If this AMA session turns out to be useful, we may start doing this on a regular schedule.

Who will be there?

You, of course! We'll also have PMs and Developers from the Azure Data Lake, HDInsight, R Server, Data Factory and Stream Analytics teams participating throughout the day.

Have any questions about the following topics? Bring them to the AMA.

  • Azure Data Lake Analytics, U-SQL
  • Azure Data Lake Store
  • Azure HDInsight
  • Microsoft R Server on HDInsight or standalone
  • Azure Data Factory
  • Azure Stream Analytics
  • Spark, Hadoop, HBase, Storm, Interactive Hive (LLAP), Jupyter, Zeppelin, Livy on HDInsight

Why should I ask questions here instead of StackOverflow, MSDN or Twitter? Can I really ask anything?

An AMA is a great place to ask us anything. StackOverflow and MSDN have restrictions on which questions can be asked while Twitter only allows 140 characters. With an AMA, you’ll get answers directly from the team and have a conversation with the people who build these products and services.

Here are some question ideas:

  • What is HDInsight?
  • What is the difference between Data Lake Analytics and Store?
  • What is the meaning of U in U-SQL?
  • How can I build my R regression model on 2 TB dataset?
  • What are the pros/cons of using Spark R vs R Server?
  • What is the easiest way to find driver logs of my Spark app?
  • What are the “gotchas” of geo-replication in HDInsight HBase?

Go ahead, ask us anything about our public products or the team. Please note, we cannot comment on unreleased features and future plans.

Join us! We're looking forward to having a conversation with you!

12 Nov 09:33

Stateless 3.0 - A State Machine library for .NET Core

by Scott Hanselman

.NET StandardState Machines and business processes that describe a series of states seem like they'll be easy to code but you'll eventually regret trying to do it yourself. Sure, you'll start with a boolean, then two, then you'll need to manage three states and there will be an invalid state to avoid then you'll just consider quitting all together. ;)

"Stateless" is a simple library for creating state machines in C# code. It's recently been updated to support .NET Core 1.0. They achieved this not by targeting .NET Core but by writing to the .NET Standard. Just like API levels in Android abstract away the many underlying versions of Android, .NET Standard is a set of APIs that all .NET platforms have to implement. Even better, the folks who wrote Stateless 3.0 targeted .NET Standard 1.0, which is the broadest and most compatible standard - it basically works everywhere and is portable across the .NET Framework on Windows, .NET Core on Windows, Mac, and LInux, as well as Windows Store apps and all phones.

Sure, there's Windows Workflow, but it may be overkill for some projects. In Nicholas Blumhardt's words:

...over time, the logic that decided which actions were allowed in each state, and what the state resulting from an action should be, grew into a tangle of if and switch. Inspired by Simple State Machine, I eventually refactored this out into a little state machine class that was configured declaratively: in this state, allow this trigger, transition to this other state, and so-on.

A state machine diagram describing the states a Bug can go throughYou can use state machines for anything. You can certainly describe high-level business state machines, but you can also easily model IoT device state, user interfaces, and more.

Even better, Stateless also serialize your state machine to a standard text-based "DOT Graph" format that can then be generated into an SVG or PNG like this with http://www.webgraphviz.com. It's super nice to be able to visualize state machines at runtime.

Modeling a Simple State Machine with Stateless

Let's look at a few code examples. You start by describing some finite states as an enum, and some finite "triggers" that cause a state to change. Like a switch could have On and Off as states and Toggle as a trigger.

A more useful example is the Bug Tracker included in the Stateless source on GitHub. To start with here are the states of a Bug and the Triggers that cause state to change:

enum State { Open, Assigned, Deferred, Resolved, Closed }

enum Trigger { Assign, Defer, Resolve, Close }

You then have your initial state, define your StateMachine, and if you like, you can pass Parameters when a state is trigger. For example, if a Bug is triggered with Assign you can pass in "Scott" so the bug goes into the Assigned state - assigned to Scott.

State _state = State.Open;

StateMachine<State, Trigger> _machine;
StateMachine<State, Trigger>.TriggerWithParameters<string> _assignTrigger;

string _title;
string _assignee;

Then, in this example, the Bug constructor describes the state machine using a fluent interface that reads rather nicely.

public Bug(string title)

{
_title = title;

_machine = new StateMachine<State, Trigger>(() => _state, s => _state = s);

_assignTrigger = _machine.SetTriggerParameters<string>(Trigger.Assign);

_machine.Configure(State.Open)
.Permit(Trigger.Assign, State.Assigned);

_machine.Configure(State.Assigned)
.SubstateOf(State.Open)
.OnEntryFrom(_assignTrigger, assignee => OnAssigned(assignee))
.PermitReentry(Trigger.Assign)
.Permit(Trigger.Close, State.Closed)
.Permit(Trigger.Defer, State.Deferred)
.OnExit(() => OnDeassigned());

_machine.Configure(State.Deferred)
.OnEntry(() => _assignee = null)
.Permit(Trigger.Assign, State.Assigned);
}

For example, when the State is Open, it can be Assigned. But as this is written (you can change it) you can't close a Bug that is Open but not Assigned. Make sense?

When the Bug is Assigned, you can Close it, Defer it, or Assign it again. That's PermitReentry(). Also, notice that Assigned is a Substate of Open.

You can have events that are fired as states change. Those events can take actions as you like.

void OnAssigned(string assignee)

{
if (_assignee != null && assignee != _assignee)
SendEmailToAssignee("Don't forget to help the new employee.");

_assignee = assignee;
SendEmailToAssignee("You own it.");
}

void OnDeassigned()
{
SendEmailToAssignee("You're off the hook.");
}

void SendEmailToAssignee(string message)
{
Console.WriteLine("{0}, RE {1}: {2}", _assignee, _title, message);
}

With a nice State Machine library like Stateless you can quickly model states that you'd ordinarily do with a "big ol' switch statement."

What have you used for state machines like this in your projects?


Sponsor: Big thanks to Telerik! They recently published a comprehensive whitepaper on The State of C#, discussing the history of C#, what’s new in C# 7 and whether C# is still a viable language. Check it out!



© 2016 Scott Hanselman. All rights reserved.
     
09 Nov 13:49

TypeScript 2.1 RC: Better Inference, Async Functions, and More

by Daniel Rosenwasser

Today we’re happy to announce our release candidate for TypeScript 2.1! If you aren’t familiar with it, TypeScript is a language that adds optional static types to JavaScript, and brings new features from ES6 and later to whatever JavaScript runtime you’re using.

As usual you can get the RC through NuGet, or just by running

npm install -g typescript@rc

You can then easily use the RC release with Visual Studio Code or our Sublime Text Plugin.

You can also grab the TypeScript 2.1 installer for Visual Studio 2015 after getting Update 3.

While TypeScript 2.1 has a lot of great features coming up, we’d like to highlight how much more powerful TypeScript 2.1’s inference will be, as well as how much easier it will be to write asynchronous code in all runtimes.

Smarter Inference

TypeScript 2.1 now makes it easier to model scenarios where you might incrementally initialize variables. Since a lot of code is written like this in JavaScript, this makes it even easier to migrate existing codebases to TypeScript.

To understand better, let’s start off by talking about the any type.

Most of the time, if TypeScript can’t figure out the type of a variable, it will choose the any type to be as flexible as possible without troubling you. We often call this an implicit any (as opposed to an explicit one, where you would have written out the type).

let x;      // implicitly 'any'
let y = []; // implicitly 'any[]'

let z: any; // explicitly 'any'.

From that point on, you can do anything you want with those values. For many people, that behavior was too loose, which is why the --noImplicitAny option will warn you whenever a type couldn’t be inferred.

With TypeScript 2.0 we built out a foundation of using control flow analysis to track the flow of types throughout your program. Because that analysis examines the assignments of every variable, we’ve leveraged that same foundation in TypeScript 2.1 to more deeply examine the type of any variable that seems like it’s destined for a better type. Instead of just choosing any, TypeScript will infer types based on what you end up assigning later on.

Let’s take the following example.

let x;

// We can still assign anything we want to 'x'.
x = () => 42;

// After that last assignment, TypeScript 2.1 knows that 'x'
// has the type '() => number', so it can be called.
x();

// But now we'll get an error that we can't add a number to a function!
console.log(x + 42);
//          ~~~~~~
// error: Operator '+' cannot be applied to types '() => number' and 'number'.

// TypeScript still allows us to assign anything we want to 'x'.
x = "Hello world!";

// But now it also knows that now 'x' is a 'string'!
x.toLowerCase();

When it comes to assignments, TypeScript will still trust you and allow you to assign anything you want to x. However, for any other uses, the type checker will know better by climbing up and looking at whatever you’ve actually done with x.

The same sort of tracking is now also done for empty arrays! This means better completions:

let puppies = [];

puppies.push(new Puppy());

for (let pup of puppies) {
    pup.bark();
    //  ^^^^ Get completion on 'bark'
}

And it also means that TypeScript can catch more obvious errors:

puppies[1] = new Kitty();

for (let pup of puppies) {
    pup.bark();
    //  ~~~~ error: 'bark' does not exist on type 'Kitty'
}

The end result of all this is that you’ll see way fewer implicit any errors in the future, and get much better tooling support.

Downlevel Async Functions

Support for down-level asynchronous functions (or async/await) is coming in 2.1, and you can use it in today’s release candidate! async/await is a new feature in ECMAScript 2017 that allows users to write code around promises without needing to use callbacks. async functions can be written in a style that looks synchronous, but acts asynchronously, using the await keyword.

This feature was supported before TypeScript 2.1, but only when targeting ES6/ES2015. TypeScript 2.1 brings the capability to ES3 and ES5 runtimes, meaning you’ll be free to take advantage of it no matter what environment you’re using.

For example, let’s take the following function named delay, which returns a promise and waits for a certain amount of time before resolving:

function delay(milliseconds: number) {
    return new Promise<void>(resolve => {
      setTimeout(resolve, milliseconds);
    });
}

Let’s try to work on a simple-sounding task. We want to write a program that prints "Hello", three dots, and then "World!".

function welcome() {
    console.log("Hello");

    for (let i = 0; i < 3; i++) {
        console.log(".");
    }

    console.log("World!");
}

This turned out to be about as simple as it sounded.

Now let’s say we want to use our delay function to pause before each dot.

Without async/await, we’d have to write something like the following:

function dramaticWelcome() {
    console.log("Hello");

    (function loop(i){
        if (i < 3) {
            delay(500).then(() => {
                console.log(".");
                loop(i + 1);
            });
        }
        else {
            console.log("World!");
        }
    })(0);
}

This doesn’t look quite as simple any more! What about if we tried using async functions to make this code more readable?

First, we need to make sure our runtime has an ECMAScript-compliant Promise available globally. That might involve grabbing a polyfill for Promise, or relying on one that you might have in the runtime that you’re targeting. We also need to make sure that TypeScript knows Promise exists by setting our lib flag to something like "dom", "es2015" or "dom", "es2015.promise", "es5":

{
    "compilerOptions": {
        "lib": ["dom", "es2015.promise", "es5"]
    }
}

Now we can rewrite our code to use async and await:

async function dramaticWelcome() {
    console.log("Hello");

    for (let i = 0; i < 3; i++) {
        await delay(500);
        console.log(".");
    }

    console.log("World!");
}

Notice how similar this is compared to our synchronous version! Despite its looks, this function is actually asynchronous, and won’t block other code from running in between each pause. In fact, the two versions of dramaticWelcome basically boil down to the same code, but with async & await, TypeScript does the heavy lifting for us.

Next Steps

TypeScript 2.1 RC has plenty of other features, and we’ll have even more coming for 2.1 proper. You can take a look at our roadmap to see what else is in store. We hope you give it a try and enjoy it!

07 Nov 13:22

FBI clears Clinton over new e-mails

by David Kravets

(credit: Matt Johnson)

James Comey, the director of the Federal Bureau of Investigation, told lawmakers in a letter Sunday that a review of new e-mails found connected to former Secretary of State Hillary Clinton uncovered nothing to change the bureau's view that Clinton had broken no law when she used a private e-mail server.

The announcement comes two weeks after the FBI director informed the chairmen and ranking members of Congressional committees in a previous letter that the bureau had found new e-mails related to the Clinton investigation and was reviewing them. Comey's ill-timed revelation, just over a week before Election Day, came under intense criticism from Democrats, including from Clinton, the Democrats' nominee for president, as well as some Republicans. Other Republicans, including Donald Trump, the GOP nominee, had seized on that news.

"Since my letter," Comey told lawmakers Sunday about his original October 28 note to them, "the FBI investigative team has been working around the clock to process and review a large volume of emails from a device obtained in connection with an unrelated criminal investigation.

Read 6 remaining paragraphs | Comments

06 Nov 13:06

Automated notifications from Azure Monitor for Atlassian JIRA

by Paul Bouwer

The public preview of Azure Monitor was recently announced at Ignite. This new platform service builds on some of the existing monitoring capabilities to provide a consolidated and inbuilt monitoring experience to all Azure users.

From within the Azure portal, you can use Azure Monitor to query across Activity Logs, Metrics and Diagnostic Logs. If you need the advanced monitoring and analytics tools like Application Insights, Azure Log Analytics and Operations Management Suite (OMS), the Azure Monitor blade contains quick links. You can also leverage the dashboard experience in the portal to visualize your monitoring data and share it with others in your team.

The consolidated Azure Monitor blade in the portal allows you to quickly and centrally manage alerts from the following sources:

  • Metrics
  • Events (eg. Autoscale events)
  • Locations (Application Insights Web Tests)
  • Proactive diagnostics (Application Insights)

These alerts can be configured to send an email and also in the case of the Metrics and Web Tests POST to a webhook. This allows for easy integration with external platforms.

Azure Monitor - List of Alerts

Integrating Azure Monitor with Atlassian JIRA

Atlassian JIRA is a familiar solution to many IT, software and business teams. It's an ideal candidate for connecting to the Azure Monitor service via the webhook mechanism in order to create JIRA Issues from Metric and Web Test Alerts.

"Azure Notifications with JIRA marries critical operational events with JIRA issues to help teams stay on top of app performance, move faster, and streamline their DevOps processes," said Bryant Lee, head of product partnerships and integrations at Atlassian.

Add-ons can be built for JIRA, Confluence, HipChat and BitBucket to extend their capabilities. In order to make the process of deploying the add-on as easy as possible, we've built this Azure Notifications add-on to be deployed and hosted on an Azure Web App which is connected to your JIRA instance via the Manage add-ons functionality in the JIRA Administration screen. The add-on establishes a secret, key exchange and other private details with JIRA that is used to secure, sign and verify all future communication between the two. All of this security information is stored in Azure Key Vault.

The add-on exposes token secured endpoints that can be configured in Azure Monitor against the webhooks exposed for various alerting mechanisms. Alerts will flow from Azure Monitor into the token secured endpoints. The add-on will then transform the payloads from the Azure Monitor alerts and securely create the appropriate Issue in JIRA.

Atlassian-Architecture

Relevant information is extracted from the Azure Monitor alerts and highlighted in the Issue. The full Azure Monitor alert payload is included for reference.

Atlassian JIRA Issue

Deploying the add-on

The Azure Notifications for Atlassian JIRA add-on is available today in Bitbucket for you to deploy and connect your JIRA instance and Azure Monitor alerts. The overview section of the add-on's repository provides documentation on the add-on and how to install it and all its associated infrastructure in Azure.

Once installed, the add-on will appear in your add-ons list in JIRA. You can then configure your Azure Monitor Alerts to send alerts to the add-on.

Atlassian JIRA - add-on installed

If you have resources deployed in Azure and are using JIRA, then this add-on has just made it really simple for you to start creating issues from your Azure Monitor alerts today!

For more information

06 Nov 13:04

Azure SQL Database: Now supporting up to 10 years of Backup Retention (Public Preview)

by Alexander (Sasha) Nosov

Does your application have compliance requirements to retain data for a long period of time? Or do you need to extend the built-in backup retention for oops recovery past 35 days? Now, with just a few clicks, you can easily enable your databases to have long-term retention. Azure SQL Database now supports backups stored in your own Azure Backup Service Vault. This allows you easily extend the built-it retention period from 35 days to up to 10 years.

Now supporting your data retention requirements is much simpler. Today Azure SQL Database automatically creates a full backup every week for each of your databases. Once you add the LTR policy to a database using Azure Portal or API, these weekly backups will be automatically copied to your own Azure Backup Service Vault. If your databases are encrypted with TDE, that's no problem -- the backups are automatically encrypted at rest. The Services Vault will automatically delete your expired backups based on their timestamp, so there's no need to to manage the backup schedule or worry about the cleanup of the old files. The following diagram shows how to add LTR policy in the Portal.

Blog screen shot

Get started with the Azure SQL Database long-term backup retention preview by simply selecting Azure Backup Service Vault for your SQL server in the Azure Portal and creating a retention policy for your database. The database backups will show up in the vault within seven days.

Learn more about Azure SQL Database backups

05 Nov 13:17

Exploring ASP.NET Core with Docker in both Linux and Windows Containers

by Scott Hanselman

In May of last year doing things with ASP.NET and Docker was in its infancy. But cool stuff was afoot. I wrote a blog post showing how to publish an ASP.NET 5 (5 at the time, now Core 1.0) app to Docker. Later in December of 2015 new tools like Docker Toolbox and Kitematic made things even easier. In May of 2016 Docker for Windows Beta continued to move the ball forward nicely.

I wanted to see how things are looking with ASP.NET Core, Docker, and Windows here in October of 2016.

I installed these things:

Docker for Windows is really nice as it automates setting up Hyper-V for you and creates the Docker host OS and gets it all running. This is a big time saver.

Hyper-V manager

There's my Linux host that I don't really have to think about. I'll do everything from the command line or from Visual Studio.

I'll say File | New Project and make a new ASP.NET Core application running on .NET Core.

Then I right click and Add | Docker Support. This menu comes from the Visual Studio Tools for Docker extension. This adds a basic Dockerfile and some docker-compose files. Out of the box, I'm all setup to deploy my ASP.NET Core app to a Docker Linux container.

ASP.NET Core in a Docker Linux Container

Starting from my ASP.NET Core app, I'll make sure my base image (that's the FROM in the Dockerfile) is the base ASP.NET Core image for Linux.

FROM microsoft/aspnetcore:1.0.1

ENTRYPOINT ["dotnet", "WebApplication4.dll"]
ARG source=.
WORKDIR /app
EXPOSE 80
COPY $source .

Next, since I don't want Docker to do the building of my application yet, I'll publish it locally. Be sure to read Steve Lasker's blog post "Building Optimized Docker Images with ASP.NET Core" to learn how to have one docker container build your app and the other run it it. This optimizes server density and resource.

I'll publish, then build the images, and run it.

>dotnet publish


>docker build bin\Debug\netcoreapp1.0\publish -t aspnetcoreonlinux

>docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
aspnetcoreonlinux latest dab2bff7e4a6 28 seconds ago 276.2 MB
microsoft/aspnetcore 1.0.1 2e781d03cb22 44 hours ago 266.7 MB

>docker run -it -d -p 85:80 aspnetcoreonlinux
1cfcc8e8e7d4e6257995f8b64505ce25ae80e05fe1962d4312b2e2fe33420413

>docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1cfcc8e8e7d4 aspnetcoreonlinux "dotnet WebApplicatio" 2 seconds ago Up 1 seconds 0.0.0.0:85->80/tcp clever_archimedes

And there's my ASP.NET Core app running in Docker. So I'm running Windows, running Hyper-V, running a Linux host that is hosting Docker containers.

What else can I do?

ASP.NET Core in a Docker Windows Container running Windows Nano Server

There's Windows Server, there's Windows Server Core that removes the UI among other things and there's Windows Nano Server which gets Windows down to like hundreds of megs instead of many gigs. This means there's a lot of great choices depending on what you need for functionality and server density. Ship as little as possible.

Let me see if I can get ASP.NET Core running on Kestrel under Windows Nano Server. Certainly, since Nano is very capable, I could run IIS within the container and there's docs on that.

Michael Friis from Docker has a great blog post on building and running your first Docker Windows Server Container. With the new Docker for Windows you can just right click on it and switch between Linux and Windows Containers.

Docker switches between Mac and Windows easily

So now I'm using Docker with Windows Containers. You may not know that you likely already have Windows Containers! It was shipped inside Windows 10 Anniversary Edition. You can check for Containers in Features:

Add Containers in Windows 10

I'll change my Dockerfile to use the Windows Nano Server image. I can also control the ports that ASP.NET talks on if I like with an Environment Variable and Expose that within Docker.

FROM microsoft/dotnet:nanoserver

ENTRYPOINT ["dotnet", "WebApplication4.dll"]
ARG source=.
WORKDIR /app
ENV ASPNETCORE_URLS http://+:82
EXPOSE 82
COPY $source .

Then I'll publish and build...

>dotnet publish

>docker build bin\Debug\netcoreapp1.0\publish -t aspnetcoreonnano

Then I'll run it, mapping the ports from Windows outside to the Windows container inside!

NOTE: There's a bug as of this writing that affects how Windows 10 talks to Containers via "NAT" (Network Address Translation) such that you can't easily go http://localhost:82 like you (and I) want to. Today you have to hit the IP of the container directly. I'll report back once I hear more about this bug and how it gets fixed. It'll show up in Windows Update one day. The workaround is to get the IP address of the container from docker like this:  docker inspect -f "{{ .NetworkSettings.Networks.nat.IPAddress }}" HASH

So I'll run my ASP.NET Core app on Windows Nano Server (again, to be clear, this is running on Windows 10 and Nano Server is inside a Container!)

>docker run -it -d -p 88:82 aspnetcoreonnano

afafdbead8b04205841a81d974545f033dcc9ba7f761ff7e6cc0ec8f3ecce215

>docker inspect -f "{{ .NetworkSettings.Networks.nat.IPAddress }}" afa
172.16.240.197

Now I can hit that site with 172.16.240.197:82. Once that bug above is fixed, it'll get hit and routed like any container.

The best part about Windows Containers is that they are fast and lightweight. Once the image is downloaded and build on your machine, you're starting and stopping them in seconds with Docker.

BUT, you can also isolate Windows Containers using Docker like this:

docker run --isolation=hyperv -it -d -p 86:82 aspnetcoreonnano

So now this instance is running fully isolated within Hyper-V itself. You get the best of all worlds. Speed and convenient deployment plus optional and easy isolation.

ASP.NET Core in a Docker Windows Container running Windows Server Core 2016

I can then change the Dockerfile to use the full Windows Server Core image. This is 8 gigs so be ready as it'll take a bit to download and extract but it is really Windows. You can also choose to run this as a container or as an isolated Hyper-V container.

Here I just change the FROM to get a Windows Sever Core with .NET Core included.

FROM microsoft/dotnet:1.0.0-preview2-windowsservercore-sdk

ENTRYPOINT ["dotnet", "WebApplication4.dll"]
ARG source=.
WORKDIR /app
ENV ASPNETCORE_URLS http://+:82
EXPOSE 82
COPY $source .

NOTE: I hear it's likely that the the .NET Core on Windows Server Core images will likely go away. It makes more sense for .NET Core to run on Windows Nano Server or other lightweight images. You'll use Server Core for heavier stuff, and Server is nice because it means you can run "full" .NET Framework apps in containers! If you REALLY want to have .NET Core on Server Core you can make your own Dockerfile and easily build and image that has the things you want.

Then I'll publish, build, and run again.

>docker images

REPOSITORY TAG IMAGE ID CREATED SIZE
aspnetcoreonnano latest 7e02d6800acf 24 minutes ago 1.113 GB
aspnetcoreonservercore latest a11d9a9ba0c2 28 minutes ago 7.751 GB

Since containers are so fast to start and stop I can have a complete web farm running with Redis in a Container, SQL in another, and my web stack in a third. Or mix and match.

>docker ps

CONTAINER ID IMAGE COMMAND PORTS NAMES
d32a981ceabb aspnetcoreonwindows "dotnet WebApplicatio" 0.0.0.0:87->82/tcp compassionate_blackwell
a179a48ca9f6 aspnetcoreonnano "dotnet WebApplicatio" 0.0.0.0:86->82/tcp determined_stallman
170a8afa1b8b aspnetcoreonnano "dotnet WebApplicatio" 0.0.0.0:89->82/tcp agitated_northcutt
afafdbead8b0 aspnetcoreonnano "dotnet WebApplicatio" 0.0.0.0:88->82/tcp naughty_ramanujan
2cf45ea2f008 a7fa77b6f1d4 "dotnet WebApplicatio" 0.0.0.0:97->82/tcp sleepy_hodgkin

Conclusion

Again, go check out Michael's article where he uses Docker Compose to bring up the ASP.NET Music Store sample with SQL Express in one Windows Container and ASP.NET Core in another as well as Steve Lasker's blog (in fact his whole blog is gold) on making optimized Docker images with ASP.NET Core.

IMAGE ID            RESPOSITORY                   TAG                 SIZE

0ec4274c5571 web optimized 276.2 MB
f9f196304c95 web single 583.8 MB
f450043e0a44 microsoft/aspnetcore 1.0.1 266.7 MB
706045865622 microsoft/aspnetcore-build 1.0.1 896.6 MB

Steve points out a number of techniques that will allow you to get the most out of Docker and ASP.NET Core.

The result of all this means (IMHO) that you can use ASP.NET Core:

  • ASP.NET Core on Linux
    • within Docker containers
    • in any Cloud
  • ASP.NET Core on Windows, Windows Server, Server Core, and Nano Server.
    • within Docker windows containers
    • within Docker isolated Hyper-V containers

This means you can choose the level of feature support and size to optimize for server density and convenience. Once all the tooling (the Docker folks with Docker for Windows and the VS folks with Visual Studio Docker Tools) is baked, we'll have nice debugging and workflows from dev to production.

What have you been doing with Docker, Containers, and ASP.NET Core? Sound off in the comments.


Sponsor: Thanks to Redgate this week! Discover the world’s most trusted SQL Server comparison tool. Enjoy a free trial of SQL Compare, the industry standard for comparing and deploying SQL Server schemas.



© 2016 Scott Hanselman. All rights reserved.
     
05 Nov 12:15

Azure DocumentDB updates: quick start experience, backup and restore, and firewall support

by Kirill Gavrylyuk

Over the past couple weeks we released a number of improvements to the developer experience and capabilities of Azure DocumentDB. We added a new quick start experience helping you to get up and running with a working app on DocumentDB in seconds. We launched a preview for backup/restore and inbound firewall capabilities, as well as released numerous runtime improvements including expanded support for geospatial types.

Quick start

One important characteristic of any service is the time it takes to get a working app up and running. We released a new quick start experience that gives you a personalized ready-to-run sample app connected to your newly created DocumentDB account in seconds. Create a new DocumentDB account and click on the quick start menu item in your existing account and give it a try.

image

For accounts with MongoDB API support, we had added a handy code snippet for all major platforms, with all the necessary configuration to get you started, including what connection string to use.

image

Backup and restore

DocumentDB is built with high availability and global distribution at its core – it allows you to scale throughput across multiple Azure regions along with policy driven failover and transparent multi-homing APIs. As a database system offering 99.99 availability SLAs, all the writes in DocumentDB are durably committed by a quorum of replicas within a local data center, and replicated across all the regions associated with your DocumentDB account.

DocumentDB also automatically takes backup of all your data at regular intervals. The backups are taken without affecting the performance or availability of your database operations. All your backups are stored separately in another storage service and are further globally replicated for resiliency against regional disasters. Customers can now request to restore their databases and collections from a backup by contacting Azure support. Below is an illustration of periodic backups to GRS Azure Storage performed by DocumentDB for all entities.

Periodic full backups of all DocumentDB entities in GRS Azure Storage

Learn more about backup and restore for DocumentDB.

Firewall support

Due to popular customer request, we recently offered support for IP filtering and firewall rules in DocumentDB for both DocumentDB and MongoDB APIs. Customers can configure their DocumentDB account to allow traffic only from a specified list of the individual IP addresses and IP address ranges. Once this configuration is applied, all requests originating from machines outside this allowed list will be blocked by DocumentDB. The connection processing flow for the IP-based access control is described in the following diagram.

Diagram showing the connection process for IP-based access control

Learn more about DocumentDB firewall support.

Expanded geospatial support

With a recent service update, Azure DocumentDB now supports geospatial indexing and querying of Polygon and LineString objects, in addition to the Point object. DocumentDB can automatically detect GeoJSON fragments that contain Polygon and LineString objects within your documents, and index them for efficient spatial proximity queries.

Spatial querying of Polygon and LineString objects is commonly used to detect "geo-fences" in IoT, telematics, gaming, and mobile applications. You can enable or disable spatial indexing by changing the indexing policy on a per-collection basis. Read more about working with geospatial data in DocumentDB.

 

We hope you find this new functionality useful as you build your solutions on top of DocumentDB. As always, let us know how we are doing and what improvements you'd like to see going forward through UserVoice,  StackOverflow #azure-documentdb, or Twitter @DocumentDB.

05 Nov 08:49

Using dotnet watch test for continuous testing with .NET Core and XUnit.net

by Scott Hanselman

When teaching .NET Core I do a lot of "dotnet new" Hello World demos to folks who've never seen it before. That has it's place, but I also wanted to show how easy it is to get setup with Unit Testing on .NET Core.

For this blog post I'm going to use the command line so you know there's nothing hidden, but you can also use Visual Studio or Visual Studio Code, of course. I'll start the command prompt then briefly move to Code.

Starting from an empty folder, I'll make a SomeApp folder and a SomeTests folder.

C:\example\someapp> dotnet new

C:\example\someapp> md ..\sometests && cd ..\sometests
C:\example\sometests> dotnet new -t xunittest

At this point I've got a HelloWorld app and a basic test but the two aren't related - They aren't attached and nothing real is being tested.

Tests are run with dotnet test, not dotnet run. Tests are libraries and don't have an entry point, so dotnet run isn't what you want.

c:\example>dotnet test SomeTests

Project SomeTests (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.
xUnit.net .NET CLI test runner (64-bit win10-x64)
Discovering: SomeTests
Discovered: SomeTests
Starting: SomeTests
Finished: SomeTests
=== TEST EXECUTION SUMMARY ===
SomeTests Total: 1, Errors: 0, Failed: 0, Skipped: 0, Time: 0.197s
SUMMARY: Total: 1 targets, Passed: 1, Failed: 0.

I'll open my test project's project.json and add a reference to my other project.

{

"version": "1.0.0-*",
"buildOptions": {
"debugType": "portable"
},
"dependencies": {
"System.Runtime.Serialization.Primitives": "4.1.1",
"xunit": "2.1.0",
"dotnet-test-xunit": "1.0.0-rc2-*"
},
"testRunner": "xunit",
"frameworks": {
"netcoreapp1.0": {
"dependencies": {
"Microsoft.NETCore.App": {
"type": "platform",
"version": "1.0.1"
},
"SomeApp": "1.0.0-*"
},
"imports": [
"dotnet5.4",
"portable-net451+win8"
]
}
}
}

I'll make a little thing to test in my App.

public class Calc {

public int Add(int x, int y) => x + y;
}

And add some tests.

public class Tests

{
[Fact]
public void TwoAndTwoIsFour()
{
var c = new Calc();
Assert.Equal(4, c.Add(2, 2));
}

[Fact]
public void TwoAndThreeIsFive()
{
var c = new Calc();
Assert.Equal(4, c.Add(2, 3));
}
}

Because the Test app references the other app/library, I can just make changes and run "dotnet test" from the command line. It will build both dependencies and run the tests all at once.

Here's the full output inclding both build and test.

c:\example> dotnet test SomeTests

Project SomeApp (.NETCoreApp,Version=v1.0) will be compiled because inputs were modified
Compiling SomeApp for .NETCoreApp,Version=v1.0

Compilation succeeded.
0 Warning(s)
0 Error(s)

Time elapsed 00:00:00.9814887
Project SomeTests (.NETCoreApp,Version=v1.0) will be compiled because dependencies changed
Compiling SomeTests for .NETCoreApp,Version=v1.0

Compilation succeeded.
0 Warning(s)
0 Error(s)

Time elapsed 00:00:01.0266293


xUnit.net .NET CLI test runner (64-bit win10-x64)
Discovering: SomeTests
Discovered: SomeTests
Starting: SomeTests
Tests.Tests.TwoAndThreeIsFive [FAIL]
Assert.Equal() Failure
Expected: 4
Actual: 5
Stack Trace:
c:\Users\scott\Desktop\testtest\SomeTests\Tests.cs(20,0): at Tests.Tests.TwoAndThreeIsFive()
Finished: SomeTests
=== TEST EXECUTION SUMMARY ===
SomeTests Total: 2, Errors: 0, Failed: 1, Skipped: 0, Time: 0.177s
SUMMARY: Total: 1 targets, Passed: 0, Failed: 1.

Oops, I made a mistake. I'll fix that test and run "dotnet test" again.

c:\example> dotnet test SomeTests

xUnit.net .NET CLI test runner (64-bit .NET Core win10-x64)
Discovering: SomeTests
Discovered: SomeTests
Starting: SomeTests
Finished: SomeTests
=== TEST EXECUTION SUMMARY ===
SomeTests Total: 2, Errors: 0, Failed: 0, Skipped: 0, Time: 0.145s
SUMMARY: Total: 1 targets, Passed: 1, Failed: 0.

I can keep changing code and running "dotnet test" but that's tedious. I'll add dotnet watch as a tool in my Test project's project.json.

{

"version": "1.0.0-*",
"buildOptions": {
"debugType": "portable"
},
"dependencies": {
"System.Runtime.Serialization.Primitives": "4.1.1",
"xunit": "2.1.0",
"dotnet-test-xunit": "1.0.0-rc2-*"
},
"tools": {
"Microsoft.DotNet.Watcher.Tools": "1.0.0-preview2-final"
},
"testRunner": "xunit",
"frameworks": {
"netcoreapp1.0": {
"dependencies": {
"Microsoft.NETCore.App": {
"type": "platform",
"version": "1.0.1"
},
"SomeApp": "1.0.0-*"
},

"imports": [
"dotnet5.4",
"portable-net451+win8"
]
}
}
}

Then I'll go back and rather than typing  "dotnet test" I'll type "dotnet watch test."

c:\example> dotnet watch test

[DotNetWatcher] info: Running dotnet with the following arguments: test
[DotNetWatcher] info: dotnet process id: 14064
Project SomeApp (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.
Project SomeTests (.NETCoreApp,Version=v1.0) will be compiled because inputs were modified
Compiling SomeTests for .NETCoreApp,Version=v1.0
Compilation succeeded.
0 Warning(s)
0 Error(s)
Time elapsed 00:00:01.1479348

xUnit.net .NET CLI test runner (64-bit .NET Core win10-x64)
Discovering: SomeTests
Discovered: SomeTests
Starting: SomeTests
Finished: SomeTests
=== TEST EXECUTION SUMMARY ===
SomeTests Total: 2, Errors: 0, Failed: 0, Skipped: 0, Time: 0.146s
SUMMARY: Total: 1 targets, Passed: 1, Failed: 0.
[DotNetWatcher] info: dotnet exit code: 0
[DotNetWatcher] info: Waiting for a file to change before restarting dotnet...

Now if I make a change to either the Tests or the projects under test it will automatically recompile and run the tests!

[DotNetWatcher] info: File changed: c:\example\SomeApp\Program.cs

[DotNetWatcher] info: Running dotnet with the following arguments: test
[DotNetWatcher] info: dotnet process id: 5492
Project SomeApp (.NETCoreApp,Version=v1.0) will be compiled because inputs were modified
Compiling SomeApp for .NETCoreApp,Version=v1.0

I'm able to do all of this with any text editor and a command prompt.

How do YOU test?


Sponsor: Do you deploy the same application multiple times for each of your end customers? The team at Octopus have taken the pain out of multi-tenant deployments. Check out their latest 3.4 release!



© 2016 Scott Hanselman. All rights reserved.
     
05 Nov 08:43

ASP.NET Core RESTful Web API versioning made easy

by Scott Hanselman

Pic by WoCTechChat used under Creative Commons There's a LOT of interesting and intense arguments that have been made around how you should version your Web API. As soon as you say RESTful it turns into a religious argument where folks may just well quote from the original text. ;)

Regardless of how you personally version your Web APIs, and side-stepping any arguments one way or the other, there's great new repository by Chris Martinez that Jon Galloway turned me on to at https://github.com/Microsoft/aspnet-api-versioning. There's ASP.NET 4.x Web API, ODATA with ASP.NET Web APIs, and now ASP.NET Core 1.x. Fortunately Chris has assembled a nicely factored set of libraries called "ASP.NET API Versioning" that add service API versioning in a very convenient way.

As Chris points out:

The default API versioning configuration is compliant with the versioning semantics outlined by the Microsoft REST Guidelines. There are also a number of customization and extension points available to support transitioning services that may not have supported API versioning in the past or supported API versioning with semantics that are different from the Microsoft REST versioning guidelines.

It's also worth pointing out how great the documentation is given it's mostly a one-contributor project. I'm sure Chris would appreciate your help though, even if you're a first timer.

Chris has NuGet packages for three flavors of Web APIs on ASP.NET:

But you should really clone the repo and check out his excellent samples.

When versioning services there's a few schools of thought and with ASP.NET Core it's super easy to get started:

public void ConfigureServices( IServiceCollection services )

{
services.AddMvc();
services.AddApiVersioning();

// remaining other stuff omitted for brevity
}

Oh, but you already have an API that's not versioned yet?

services.AddApiVersioning(

o =>
{
o.AssumeDefaultVersionWhenUnspecified = true );
o.DefaultApiVersion = new ApiVersion( new DateTime( 2016, 7, 1 ) );
} );

Your versions can look however'd you like them to:

  • /api/foo?api-version=1.0
  • /api/foo?api-version=2.0-Alpha
  • /api/foo?api-version=2015-05-01.3.0
  • /api/v1/foo
  • /api/v2.0-Alpha/foo
  • /api/v2015-05-01.3.0/foo

QueryString Parameter Versioning

I'm not a fan of this one, but here's the general idea:

[ApiVersion( "2.0" )]

[Route( "api/helloworld" )]
public class HelloWorld2Controller : Controller {
[HttpGet]
public string Get() => "Hello world!";
}

So this means to get 2.0 over 1.0 in another Controller with the same route, you'd go here:

/api/helloworld?api-version=2.0

Also, don't worry, you can use namespaces to have multiple HelloWorldControllers without having to have any numbers in the class names. ;)

URL Path Segment Versioning

This happens to be my first choice (yes I know Headers are "better," more on that later). You put the version in the route like this.

Here we're throwing in a little curveball. There's three versions but just two controllers.

[ApiVersion( "1.0" )]

[Route( "api/v{version:apiVersion}/[controller]" )]
public class HelloWorldController : Controller {
public string Get() => "Hello world!";
}

[ApiVersion( "2.0" )]
[ApiVersion( "3.0" )]
[Route( "api/v{version:apiVersion}/helloworld" )]
public class HelloWorld2Controller : Controller {
[HttpGet]
public string Get() => "Hello world v2!";

[HttpGet, MapToApiVersion( "3.0" )]
public string GetV3() => "Hello world v3!";
}

To be clear, you have total control, but the result from the outside is quite clean with /api/v[1|2|3]/helloworld. In fact, you can see with this example where this is more sophisticated than what you can do with routing out of the box. (I know some of you are thinking "meh I'll just make a route table." I think the semantics are much clearer and cleaner this way.

Header Versioning

Or, the hardest way (and the one that a lot of people this is best, but I disagree) is HTTP Headers. Set this up in ConfigureServices for ASP.NET Core:

public void ConfigureServices( IServiceCollection services )

{
services.AddMvc();
services.AddApiVersioning(o => o.ApiVersionReader = new HeaderApiVersionReader("api-version"));
}

When you do HeaderApiVersioning you won't be able to just do a GET in your browser, so I'll use Postman to add the header (or I could use Curl, or WGet, or PowerShell, or a Unit Test):

image

Deprecating

Speaking of semantics, here's a nice one. Let's say an API is going away in the next 6 months.

[ApiVersion( "2.0" )]

[ApiVersion( "1.0", Deprecated = true )]

This advertises that 1.0 is going away soon and that folks should consider 2.0. Where does it advertise this fact? The response headers!

api-supported-versions: 2.0, api-deprecated-versions: 1.0

All in all, this is very useful stuff and I'm happy to add it to my personal toolbox. Should this be built in? I don't know but I sure appreciate that it exists.

SIDE NOTE: There is/was the start of a VersionRoute over in the AspNet.Mvc repo. Maybe these folks need to join forces?

How do YOU version your Web APIs and Services?


Sponsor: Big thanks to Telerik! They recently launched their UI toolset for ASP.NET Core so feel free to check it out or learn more about ASP.NET Core development in their recent whitepaper.



© 2016 Scott Hanselman. All rights reserved.
     
05 Nov 08:31

The mystery of dotnet watch and 'Microsoft.NETCore.App', version '1.1.0-preview1-001100-00' was not found

by Scott Hanselman
dotnet watch says "specified framework not found"

WARNING: This post is full of internal technical stuff. I think it's interesting and useful. You may not.

I had an interesting Error/Warning happen when showing some folks .NET Core recently and I thought I'd deconstruct it here for you, Dear Reader, because it's somewhat multi-layered and it'll likely help you. It's not just about Core, but also NuGet, Versioning, Package Management in general, version pinning, "Tools" in .NET Core, as well as how .NET Runtimes work and version. That's a lot! All that from this little warning. Let's see what's up.

First, let's say you have .NET Core installed. You likely got it from http://dot.net and you have either 1.0.0 or the 1.0.1 update.

Then say you have a website, or any app at all. I made one with "dotnet new -t web" in an empty folder.

I added "dotnet watch" as a tool in the project.json like this. NOTE the "1.0.0-*" there.

"tools": {

"Microsoft.DotNet.Watcher.Tools": "1.0.0-*"
}

dotnet watch is nice because it watches the source code underneath it while running your app. If you change your code files, dotnet-watch will notice, and exit out, then launch "dotnet run" (or whatever, even test, etc) and your app will pick up the changes. It's a nice developer convenience.

I tested this out on last weekend and it worked great. I went to show some folks on Monday that same week and got this error when I typed "dotnet watch."

C:\Users\scott\Desktop\foofoo>dotnet watch

The specified framework 'Microsoft.NETCore.App', version '1.1.0-preview1-001100-00' was not found.
- Check application dependencies and target a framework version installed at:
C:\Program Files\dotnet\shared\Microsoft.NETCore.App
- The following versions are installed:
1.0.0
1.0.1
- Alternatively, install the framework version '1.1.0-preview1-001100-00'.

Let's really look at this. It says "the specified framework...1.1.0" was not found. That's weird, I'm not using that one. I check my project.json and I see:

"Microsoft.NETCore.App": {

"version": "1.0.1",
"type": "platform"
},

So who wants 1.1.0? I typed "dotnet watch." Can I "dotnet run?"

C:\Users\scott\Desktop\foofoo>dotnet run

Project foofoo (.NETCoreApp,Version=v1.0) will be compiled because expected outputs are missing
Compiling foofoo for .NETCoreApp,Version=v1.0
Hosting environment: Production
Content root path: C:\Users\scott\Desktop\foofoo
Now listening on: http://localhost:5000
Application started. Press Ctrl+C to shut down.

Hey, my app runs fine. But if I "dotnet watch" I get an error.

Remember that dotnet watch and other "tools" like it are not dependencies per se, but helpful sidecar apps. Tools can watch, squish css and js, precompile views, and do general administrivia that isn't appropriate at runtime.

It seems it's dotnet watch that wants something I don't have.

Now, I could go install the framework 1.1.0 that it's asking for, and the error would disappear, but would I know why? That would mean dotnet watch would use .NET Core 1.1.0 but my app (dotnet run) would use 1.0.1. That's likely fine, but is it intentional? Is it deterministic and what I wanted?

I'll open my generated project.lock.json. That's the calculated tree of what we ended up with after dotnet restore. It's a big calculated file but I can easily search it. I see two things. The internal details aren't interesting but version strings are.

First, I search for "dotnet.watcher" and I see this:

"projectFileToolGroups": {

".NETCoreApp,Version=v1.0": [
"Microsoft.AspNetCore.Razor.Tools >= 1.0.0-preview2-final",
"Microsoft.AspNetCore.Server.IISIntegration.Tools >= 1.0.0-preview2-final",
"Microsoft.DotNet.Watcher.Tools >= 1.0.0-*",
"Microsoft.EntityFrameworkCore.Tools >= 1.0.0-preview2-final",
"Microsoft.Extensions.SecretManager.Tools >= 1.0.0-preview2-final",
"Microsoft.VisualStudio.Web.CodeGeneration.Tools >= 1.0.0-preview2-final"
]

Ah, that's a reminder that I asked for 1.0.0-*. I asked for STAR for dotnet-watch but everything else was very clear. They were specific versions. I said "I don't care about the stuff after 1.0.0 for watch, gimme whatever's good."

It seems that a new version of dotnet-watch and other tools came out between the weekend and my demo.

Search more in project.lock.json and I can see what all it asked for...I can see my dotnet-watch's dependency tree.

"tools": {

".NETCoreApp,Version=v1.0": {
"Microsoft.DotNet.Watcher.Tools/1.0.0-preview3-final": {
"type": "package",
"dependencies": {
"Microsoft.DotNet.Cli.Utils": "1.0.0-preview2-003121",
"Microsoft.Extensions.CommandLineUtils": "1.1.0-preview1-final",
"Microsoft.Extensions.Logging": "1.1.0-preview1-final",
"Microsoft.Extensions.Logging.Console": "1.1.0-preview1-final",
"Microsoft.NETCore.App": "1.1.0-preview1-001100-00"
},

Hey now. I said "1.0.0-*" and I ended up with "1.0.0-preview3-final"

Looks like dotnet-watch is trying to bring in a whole new .NET Core. It wants 1.1.0. This new dotnet-watch is part of the wave of new preview stuff from 1.1.0.

But I want to stay on the released and supported "LTS" (long term support) stuff, not the new fancy builds.

I shouldn't have used 1.0.0-* as it was ambiguous. That might be great for my local versions or when I intend to chase the latest but not in this case.

I updated my version in my project.json to this and did a restore.

"Microsoft.DotNet.Watcher.Tools": "1.0.0-preview2-final",

Now I can reliably run dotnet restore and get what I want, and both dotnet watch and dotnet run use the same underlying runtime.


Sponsor: Big thanks to Telerik! They recently launched their UI toolset for ASP.NET Core so feel free to check it out or learn more about ASP.NET Core development in their recent whitepaper.


© 2016 Scott Hanselman. All rights reserved.
     
05 Nov 08:09

Your Digital Pinball Machine

by Jeff Atwood

I've had something of an obsession with digital pinball for years now. That recently culminated in me buying a Virtuapin Mini.

OK, yes, it's an extravagance. There's no question. But in my defense, it is a minor extravagance relative to a real pinball machine.

The mini is much smaller than a normal pinball machine, so it's easier to move around, takes up less space, and is less expensive. Plus you can emulate every pinball machine, ever! The Virtuapin Mini is a custom $3k build centered around three screens:

  • 27" main playfield (HDMI)
  • 23" backglass (DVI)
  • 8" digital matrix (USB LCD)

Most of the magic is in those screens, and whether the pinball sim in question allows you to arrange the three screens in its advanced settings, usually by enabling a "cabinet" mode.

Let me give you an internal tour. Open the front coin door and detach the two internal nuts for the front bolts, which are finger tight. Then remove the metal lockdown bar and slide the tempered glass out.

The most uniquely pinball item in the case is right at the front. This Digital Plunger Kit connects the 8 buttons (2 on each side, 3 on the front, 1 on the bottom) and includes an analog tilt sensor and analog plunger sensor. All of which shows up as a standard game controller in Windows.

On the left front side, the audio amplifier and left buttons.

On the right front side, the digital plunger and right buttons.

The 27" playfield monitor is mounted using a clever rod assembly to the standard VESA mount on the back, so we can easily rotate it up to work on the inside as needed.

To remove the playfield, disconnect the power cord and the HDMI connector. Then lift it up and out, and you now have complete access to the interior.

Notice the large down-firing subwoofer mounted in the middle of the body, as well as the ventilation holes. The PC "case" is just a back panel, and the power strip is the Smart Strip kind where it auto-powers everything based on the PC being powered on or off. The actual power switch is on the bottom front right of the case.

Powering it up and getting all three screens configured in the pinball sim of your choice results in … magic.

It is a thoroughly professional build, as you'd expect from a company that has been building these pinball rigs for the last decade. It uses real wood (not MDF), tempered glass, and authentic metal pinball parts throughout.

I was truly impressed by the build quality of this machine. Paul of Virtuapin said they're on roughly version four of the machine and it shows. It's over 100 pounds fully assembled and arrives on a shipping pallet. I can only imagine how heavy the full size version would be!

That said, I do have some tweaks I recommend:

  • Make absolutely sure you get an IPS panel as your 27" playfield monitor. As arrived, mine had a TN panel and while it was playable if you stood directly in front of the machine, playfield visibility was pretty dire outside that narrow range. I dropped in the BenQ GW2765HT to replace the GL2760H that was in there, and I was golden. If you plan to order, I would definitely talk to Paul at VirtuaPin and specify that you want this IPS display even if it costs a little more. The 23" backglass monitor is also TN but the viewing angles are reasonable-ish in that orientation and the backglass is mostly for decoration anyway.

  • The improved display has a 1440p resolution compared to the 1080p originally shipped, so you might want to upgrade from the GeForce 750 Ti video card to the just-released 1050 Ti. This is not strictly required, as I found the 750 Ti an excellent performer even at the higher resolution, but I plan to play only fully 3D pinball sims and the 1050 Ti gets excellent reviews for $140, so I went for it.

  • Internally everything is exceptionally well laid out, the only very minor improvement I'd recommend is connecting the rear exhaust fan to the motherboard header so its fan speed can be dynamically controlled by the computer rather than being at full power all the time.

  • On the Virtuapin website order form the PC they provide sounds quite outdated, but don't sweat it: I picked the lowest options thinking I would have to replace it all, and they shipped me a Haswell based quad-core PC with 8GB RAM and a 256GB SSD, even though those options weren't even on the order form.

I realize $3k (plus palletized shipping) is a lot of money, but I estimate it would cost you at least $1500 in parts to build this machine, plus a month of personal labor. Provided you get the IPS playfield monitor, this is a solidly constructed "real" pinball machine, and if you're into digital pinball like I am, it's an absolute joy to play and a good deal for what you actually get. As Ferris Bueller once said:

If you'd like to experiment with this and don't have three grand burning a hole in your pocket, 90% of digital pinball simulation is a widescreen display in portrait mode. Rotate one of your monitors, add another monitor if you're feeling extra fancy, and give it a go.

As for software, most people talk about Visual Pinball for these machines, and it works. But the combination of janky hacked-together 2D bitmap technology used in the gameplay, and the fact that all those designs are ripoffs that pay nothing in licensing back to the original pinball manufacturers really bothers me.

I prefer Pinball Arcade in DirectX 11 mode, which is downright beautiful, easily (and legally!) obtainable via Steam and offers a stable of 60+ incredible officially licensed classic pinball tables to choose from, all meticulously recreated in high resolution 3D with excellent physics.

As for getting pinball simulations running on your three monitor setup, if you're lucky the game will have a cabinet mode you can turn on. Unfortunately, this can be weird due to … licensing issues. Apparently building a pinball sim on the computer requires entirely different licensing than placing it inside a full-blown pinball cabinet.

Pinball Arcade has a nifty camera hack someone built that lets you position three cameras as needed to get the three displays. You will also need the excellent x360ce program to dynamically map joystick events and buttons to a simulated Xbox 360 controller.

Pinball FX2 added a cabinet mode about a year ago, but turning it on requires a special code and you have to send them a picture of your cabinet (!) to get that code. I did, and the cabinet mode works great; just enter your code, specify the coordinates of each screen in the settings and you are good to go. While these tables definitely have arcadey physics, I find them great fun and there are a ton to choose from.

Pro Pinball Timeshock Ultra is unique because it's originally from 1997 and was one of the first "simulation" level pinball games. The current rebooted version is still pre-rendered graphics rather than 3D, but the client downloads the necessary gigabytes of pre-rendered content at your exact screen resolution and it looks amazing.

Timeshock has explicit cabinet support in the settings and via command line tweaks. Also, in cabinet mode, when choosing table view, you want the bottom left one. Trust me on this! It supports maximum height for portrait cabinet mode.

Position each window as necessary, then enable fullscreen for each one and it'll snap to the monitor you placed it on. It's "only" one table, but arguably the most classic of all pinball sims. I sincerely hope they continue to reboot the rest of the Pro Pinball series, including Big Race USA which is my favorite.

I've always loved pinball machines, even though they struggled to keep up with digital arcade games. In some ways I view my current project, Discourse, as a similarly analog experience attempting to bridge the gap to the modern digital world:

The fantastic 60 minute documentary Tilt: The Battle to Save Pinball has so many parallels with what we're trying to do for forum software.

Pinball is threatened by Video Games, in the same way that Forums are threatened by Facebook and Twitter and Tumblr and Snapchat. They're considered old and archaic technology. They've stopped being sexy and interesting relative to what else is available.

Pinball was forced to reinvent itself several times throughout the years, from mechanical, to solid state, to computerized. And the defining characteristic of each "era" of pinball is that the new tables, once you played them, made all the previous pinball games seem immediately obsolete because of all the new technology.

The Pinball 2000 project was an attempt to invent the next generation of pinball machines:

It wasn't a new feature, a new hardware set, it was everything new. We have to get everything right. We thought that we had reinvented the wheel. And in many respects, we had.

This is exactly what we want to do with Discourse – build a forum experience so advanced that playing will make all previous forum software seem immediately obsolete.

Discourse aims to save forums and make them relevant and useful to a whole new generation.

So if I seem a little more nostalgic than most about pinball, perhaps a little too nostalgic at times, maybe that's why.

[advertisement] Building out your tech team? Stack Overflow Careers helps you hire from the largest community for programmers on the planet. We built our site with developers like you in mind.
20 Oct 06:31

Storing and using secrets in Azure

by Bertrand Le Roy

Most applications need access to secret information in order to function: it could be an API key, database credentials, or something else. In this post, we’ll create a simple service that will compare the temperatures in Seattle and Paris using the OpenWeatherMap API, for which we’ll need a secret API key. I’ll walk you through the usage of Azure’s Key Vault for storing the key, then I’ll show how to retrieve and use it in a simple Azure function.

Prerequisites

In order to be able to follow along, you’ll need an Azure subscription. It’s very easy to get a trial subscription, and get started for free.

Where to store secrets?

There are of course many different places where people store such secrets. From worst to best, one could think of the following: in your source code repository on GitHub (of course, nobody should ever do that), in configuration files (encrypted or not), in environment variables, or in specialized secret vaults.

Which one you choose depends on the level of security your application requires. Oftentimes, storing an API key in an environment variable will be adequate (what is never adequate is hard-coded values in code or configuration files checked into source control). If you require a higher level of security, however, you’ll need a specialized vault such as Azure Key Vault.

Azure Key Vault is a service that stores and retrieves secrets in a secure fashion. Once stored, your secrets can only be accessed by applications you authorize, and only on an encrypted channel. Each secret can be managed in a single secure place, while multiple applications can use it.

Setting up Key Vault

First, we’re going to set-up Key Vault. This requires a few steps, but only steps 4 and 5 have to be repeated for new secrets, the others being the one-time building of the vault.

  1. Open the Azure portal and click on Resource groups. Choose an existing group, or create a new one. This is a matter of preference, but is otherwise inconsequential from a security standpoint. For this tutorial, we’ll create a new one called “sample-weather-group”. After clicking the Create button, you may have to wait a few seconds and refresh the list of resource groups.

    Creating a new resource group

  2. Select the newly created group, then click the Add button over its property page. Enter “Key Vault” in the search box, select the Key Vault service, then click Create.

    Adding the Key Vault service to the resource group

  3. Enter “sample-weather-vault” as the name of the new vault. Select the right subscription and location, and leave the “sample-weather-group” resource group selected. Click Create.

    Creating a new vault

  4. If you refresh the resource group property page, you’ll see the new vault appear. We’re now ready to add a key to it. Get an API key from OpenWeatherMaps. Select the vault in the list of resources under the resource group, then select Secrets. You can now click Add to add a new secret. Under Upload options, select Manual. Enter “open-weather-map-key” as the name of the secret, and paste the API key from OpenWeatherMaps into the value field. Click Create.

    Storing the secret in Key Vault

  5. We will later need the URL for the secret we just created. This can be found under Secret Identifier on the property page of the current version of the secret, which can be reached by navigating to Secrets under the key vault, then clicking the secret, and then its latest version.

    Getting the URL for the secret

And this is it for now for Key Vault: we now have a vault, containing our secret. Next, we’ll need to setup access so that we can securely retrieve the key from our application.

Preparing Active Directory authentication

The application will need to securely connect to the vault, for which it will have to prove its identity, using some form of master secret. This is similar to the master password that a password vault uses, and makes sure the identity of our application can be managed independently from the secret, which may be shared with more than one application. We’ll use Active Directory for this.

In this post, we’ll authenticate using a secret key, but it’s important to note that it is possible to use a certificate instead for added security. Please refer to Authenticate with a Certificate instead of a Client Secret for more information.

  1. To access Active Directory, in the Azure portal, select More Services and choose Azure Active Directory (currently in preview). In the next menu that will appear, click App registrations. Click the Add button above the list of applications. You’ll be asked for a name for the application. We’ll choose “sample-weather-ad”. For the application type, leave the default Web app / API selected. We also have to provide a sign-on URL. For our purposes, this doesn’t need to actually exist, but only to be unique. Click the Create button.

    Naming the AD application

  2. Now that the application has been created, select it in the application list, so that you can see your application’s security principal ID, which can be found under Managed Application In Local Directory in the application’s property page. This principal ID will represent the authenticated identity of our application. You’ll need that and a key.

    Viewing the application's principal ID

    Currently, this principal ID does not get automatically generated until the Active Directory application is logged into for the first time. This is a temporary issue that is being looked into. In the meantime, it can be worked around by browsing to <oauth 2.0 authorization endpoint URL>?client_id=<application_id>, where <oauth 2.0 authorization endpoint URL> can be found by clicking the Endpoints button above the list of registered apps, and <application_id> can be found under the property page for the application. For example, if your authorization endpoint URL is https://login.windows.net/00000000-0000-0000-0000-000000000000/oauth2/authorize, and your application id is 11111111-1111-1111-1111-111111111111, the URL to browse to would be https://login.windows.net/00000000-0000-0000-0000-000000000000/oauth2/authorize?client_id=11111111-1111-1111-1111-111111111111.

    Getting the authorization URL

    Navigating to the URL we composed will require you to authenticate with the credentials of a user that has admin rights on the subscription, and then it will yield an error page that can be safely ignored. If all went well, you should now be able to see your application’s principal ID in the property page.

    Getting the AD application's principal ID

  3. We can now proceed to create credentials that our Azure Function will be able to use to authenticate to Active Directory as the application we created earlier. Click on All settings, then select Keys. We can add a new key by entering a description, selecting an expiration, and hitting the Save button. If you do choose to have the key expire, you should also take the time to create a reminder on the schedule of the team in charge of managing this application to renew it.

    Adding a new key

    Note that using a different key and id for each application that will use the secrets makes it possible to revoke access to the whole vault for a specific application in one operation.

  4. Once you’ve saved, the key can be viewed and copied to a safe place. Do it now, because this is the last time the Azure portal is going to show it.

    The generated key

  5. We’ll also need the URL of the Active Directory endpoint. This is not the URL that we manually entered earlier when we created the AD application. It can be obtained by clicking the Endpoints button above the list of applications.

    The "Endpoints" button

    The URL we want to copy for later use is the one under OAuth 2.0 Token Endpoint.

  6. We’re now ready to authorize the application to access the vault and get values out of it. Navigate back to the key vault’s property page, and select the vault we created earlier, then Access policies. Click Add new, then Select principal. Paste the principal ID into the text box. After a few seconds, the principal should appear checked. Click the Select button. Then click on Secret permissions and check Get, then click OK, then Save.

    Adding permissions for AD

We now have a set of Active Directory credentials that our Azure Function will be able to use, that will enable read access to the secrets in the key vault. Our last remaining step is to create the actual code.

Creating the Azure function

We’re going to use Azure Functions to implement the actual service, because it’s the easiest way to write code on Azure, but roughly the same steps would apply to any other kind of application.

  1. From the resource group’s property page, click Add, and type “Function App” in the filter box. Select Function App, then click Create. Name your new function app “sample-weather”. Select the relevant subscription, resource group, plan, and location.

    Setting up the new function app

  2. Refresh the resource group’s property page, then select the new “sample-weather” app service. Create a new function under the function app. Name it “ParisSeattleWeatherComparison” and choose the empty C# template:

    Creating the function

    Now we have an empty function. Let’s add some code.

  3. Click View Files under the code editor. This shows the list of files in the function’s directory. Click the + icon to add a new file, and name it “weather.csx”. Enter the following code in that file:

    Those classes will help deserialize the response from the weather server. The structure of the types WeatherList, Weather, and WeatherData reflects the schema of the JSON documents that the weather service will return. It does not, nor needs to reflect the entirety of the schema returned by the API: the Microsoft.AspNet.WebApi.Client library will figure out which parts to deserialize and how to map them onto the provided object model.

  4. Enter the following code as the body of the function:

    We also need to define the bindings for the function. Open function.json and enter the following as its content.

    This tells Azure Functions what objects to inject into the function.

  5. In the code above, you’ll notice that we’re reading the AD URL, client ID and key from configuration, because of course we haven’t done all this to end up storing secrets in code. For the code to function, we’ll have to enter that information into the function’s Azure configuration. This can be done by clicking Function app settings on the bottom-left of the function editing screen.

    The function app settings button

    In the screen this brings up, you’ll want to select the Configure app settings option. Once there, you’ll see a few general settings, but what we’re interested in is the table of custom App settings. We’ll add new key-value pairs in there with the names we used in the code:

    • “WeatherADURL” with the Active Directory OAuth 2.0 Token Endpoint URL.
    • “WeatherADClientID” with the Active Directory application ID we got in the previous section.
    • “WeatherADKey” with the Active Directory application key.
    • “WeatherKeyUrl” with the URL for the secret API key we stored in the vault earlier.

    Don’t forget to hit Save on top of the panel.

    Setting the Active Directory URL, master id and key in the app settings

  6. Go back to the function editor and add a project.json file to import the NuGet packages we need.

    Adding a project.json file

    Enter the following code into the file.

And this is it. Once you’ve saved all the files, you should see the trace of the package restoration and compilation of the function in the logs window. After that’s completed, you should be able to click the Run button and figure out where the weather is nicest, between Paris and Seattle.

Where is the weather the nicest?

Summary

We now have a function that connects to an Azure Key Vault using Azure Active Directory authentication, and then uses a secret stored in the vault to query a remote service.

Wait, what about .NET Core?

Before I conclude, and now that we’ve made this work in Azure Functions, how about re-using the knowledge we’ve gained in a different kind of application, such as a .NET Core console application? We actually wouldn’t have to change much. First, we’d need to check our dependencies to make sure that everything we need is available on .NET Core, then we’d have to modify the code so that it reads configuration using the new ConfigurationBuilder. Finally, we’d just change req.CreateResponse(HttpStatusCode.OK, $"..."); into simple Console.WriteLine calls.

Here’s what the code looks like once ported to a .NET Core console app.

And here’s the project.json that enables it to restore the right packages.

The weather data model remains unchanged from the Azure Functions version.

And that’s it, this console application will, like the Azure Function, authenticate to Active Directory, get the API key from Key Vault, and then query the API and tell you about the weather.

What’s next?

Azure services are evolving constantly, and so is .NET support for them. While I was writing this article, I was able to transfer some of the steps that previously required command-line operations, to using the portal, making them a lot easier, as well as more discoverable. One of the things to look forward to is a new ASP.NET configuration provider that will enable developers to get rid of much of the code I had to write to access the key vault.

Please let us know if tutorials like these are helpful. Happy programming!

References

  1. Manage Key Vault using CLI
  2. Azure Key Vault .NET Samples
  3. Azure Functions C# Reference
  4. Safe storage of app secrets during development
20 Oct 05:53

DocumentDB updates in the Azure Portal: New metrics blade and improved collections management

by Kirill Gavrylyuk

Recently we released several updates to the Azure DocumentDB portal experience. We added a new consolidated metrics experience with several new metrics available including new availability, throughput, consistency, and latency metrics which allows you to track how DocumentDB is meeting its SLA's. We also streamlined collection management by surfacing all collection operations and experiences on the left navigation bar, and eliminated excessive horizontal scrolling.

New metrics blade and SLA metrics

Azure DocumentDB is a globally distributed managed NoSQL database as a service that offers 99.99% guarantees for availability, throughput, <10ms read latency and <15ms write latency at 99th percentile, and guarantees 100% consistency.  We believe it is important for you to know how the service is performing against these guarantees. We also heard how important it is for you to have all metrics information readily available without browsing through multiple windows. In response to this feedback, we have added several new metrics to reflect service performance vs SLA, as well as introduced a new at-a-glance metrics experience for DocumentDB collections. If you click on the Metrics node under Collections, you will see all metrics DocumentDB provides for each of your collections on a single surface, including:

  • Actual collection availability vs. SLA
  • Requests rate grouped by status code
  • Consumed and provisioned throughput capacity measured in Request Units/second, as well as percentage of requests exceeding capacity
  • Observed latency in the regions where your collection is located
  • Percentage of requests that met consistency guarantees
  • Storage consumed vs. capacity

You can also zoom in and navigate to the standard metrics experience for the individual charts via zoom-in gesture.

portal-docdb-metrics

Streamlined collections management

DocumentDB stores data in collections, and majority of your time working with DocumentDB is spent working with collections. With this portal change, we bring collections front and center in the portal experience and eliminate the excessive horizontal scrolling you used to encounter when working in the portal.

  • The overview blade now provides one-click access to collections, as well as "Add Collection" command.
  • All collection management experiences and tools are now available on the left navigation menu.
  • With this update, we eliminated the need for horizontal scrolling when working with DocumentDB in Azure portal.

portal-docdb-collections-pivot

We hope these new features will make your time in the Azure portal more efficient and provide you with the monitoring information that’s most important to you. As always, let us know how we are doing and what improvements you'd like to see going forward through Uservoice,  StackOverflow #azure-documentdb, or Twitter @documentdb.

20 Oct 05:51

Azure DocumentDB SDK updates include Python 3 support

by Mimi Gentz

Azure DocumentDB, Microsoft’s globally replicated, low latency, NoSQL database, is pleased to announce updates to all four of its client-side SDKs. The biggest improvements were made to the Python SDK, including support for Python 3, connection pooling, consistency improvements, and Top/Order By support for partitioned collections.

This article describes the changes made to each of the new SDKs.

DocumentDB Python SDK 2.0.0

The Azure DocumentDB Python SDK now supports Python 3. The DocumentDB Python SDK previously supported Python 2.7, but it now supports Python 3.3, Python 3.4 and Python 3.5, in addition to Python 2.7. But that’s not all! Connection pooling is now built in, so instead of creating a new connection for each request, calls to the same host are now added to the same session, saving the cost of creating a new connection each time. We also added a few enhancements to consistency level support, and we added Top and Order By support for cross-partition queries, so you can retrieve the top results from multiple partitions and order those results based on the property you specify.

To get started, go to the DocumentDB Python SDK page to download the SDK, get the latest release notes, and browse to the API reference content.

DocumentDB .NET SDK 1.10.0

The new DocumentDB .NET SDK has a few specific improvements, the biggest of which is direct connectivity support for partitioned collections. If you're currently using a partitioned collection – this improvement is the go fast button!  In addition, the .NET SDK improves performance for the Bounded Staleness consistency level, and adds LINQ support for StringEnumConverter, IsoDateTimeConverter and UnixDateTimeConverter while translating predicates.

You can download the latest DocumentDB .NET SDK, get the latest release notes, and browse to the API reference content from the DocumentDB .NET SDK page.

DocumentDB Java SDK 1.9.0

In the new DocumentDB Java SDK, we’ve changed Top and Order By support to include queries across partitions within a collection. 

You can download the Java SDK, get the latest release notes, and browse to the API reference content from the DocumentDB Java SDK page.

DocumentDB Node.js SDK 1.10.0

In the new Node.js SDK, we also changed Top and Order By support to include queries across partitions within a collection.

You can download the Node.js SDK, get the latest release notes, and browse to the API reference content from the DocumentDB Node.js SDK page.

Please upgrade to the latest SDKs and take advantage of all these improvements. And as always, if you need any help or have questions or feedback regarding the new SDKs or anything related to Azure DocumentDB, please reach out to us on the developer forums on Stack Overflow. And stay up-to-date on the latest DocumentDB news and features by following us on Twitter @DocumentDB.