c-bass

Authored by — Madeline, Lauren, Boris, Tim, Parth, Eugene and Apurva

Introduction

At Netflix, the Content Platform Engineering and Global Product Creative teams know that imagery plays an incredibly important role in how viewers find new shows and movies to watch. We take pride in surfacing the unique elements of a story that connect our audiences to diverse characters and story lines. As our Original content slate continues to expand, our technical experts are tasked with finding new ways to scale our resources and alleviate our creatives from the tedious and ever-increasing demands of digital merchandising. One of the ways in which we do this is by harvesting static image frames directly from our source videos to provide a more flexible source of raw artwork.

The Business Case

Merchandising stills are static video frames taken directly from the source video content used to broaden the reach of a title on the Netflix service. Within a single one-hour episode of Stranger Things, there are nearly 86,000 static video frames.

Traditionally, these merchandising stills are selected by human curators or editors, and require an in-depth expertise of the source content that they’re intended to represent. We know through A/B testing that we can effectively drive increased viewing from expected and unexpected audience groups by exploring as many representations of a title as possible. When it comes to title key art, we like to test many artistic representations of a title in order to find the “right” artwork for the right audience. While this presents an exciting opportunity for innovation and testing, it simultaneously presents a very challenging expectation to scale this experience across every title in our growing global catalog.

AVA

AVA is a collection of tools and algorithms designed to surface high quality imagery from the videos on our service. A single season of average TV show (about 10 episodes) contains nearly 9 million total frames. Asking creative editors to efficiently sift through that many frames of video to identify one frame that will capture an audience’s attention is tedious and ineffective. We set out to build a tool that quickly and effectively identifies which frames are the best moments to represent a title on the Netflix service.

To achieve this goal, we first came up with objective signals that we can measure for each and every frame of the video using Frame Annotations. As result, we can collect an effective representation of each frame of the video. Subsequently, we created ranking algorithms that allows us to rank a subset of frames that meets aesthetic, creative and diversity objectives to represent content accurately for various canvases of our product.

Frame Annotation

As part of our automation pipeline, we process and annotate many different variables on each individual frame of video to best derive what the frame contains, and to understand why it is or isn’t important to the story. In order to scale horizontally and have predictable SLA for a growing catalog of content, we utilized the Archer framework to process our videos more efficiently. Archer allowed us to split the videos into smaller sized chunks that could each be processed in parallel. This has enabled us to scale by lending efficiency to our video processing pipelines, and allowing us to integrate more and more content intelligence algorithms into our tool sets.

Every frame of video in a piece of content is processed through a series of computer vision algorithms to gather objective frame metadata, latent representation of frame, as well as some of the contextual metadata that those frame(s) contain. The annotation properties that we process and apply to our video frames can be roughly grouped into 3 main categories:

Visual Metadata

Typically these properties are objective, measurable, and mostly contained at the pixel-level. Some examples of visual properties are brightness, color, contrast, and motion blur.

Contextual Metadata

Contextual metadata is comprised of a combination of elements that are aggregated to derive meaning from the actions or movement of the actors, objects and camera in the frame. Some examples include;

• Face detection with facial landmarks tracking, pose estimation, and sentiment analysis — This allows us to estimate posture and sentiment of the subjects in the frame.
• Motion estimation — This allows us to estimate the amount of motion (both camera movement and subject movement) contained within a particular shot. This allows us to control for elements such as motion blur, as well as to identify camera movement that makes for compelling still imagery.
• Camera shot identification — (e.g. close up shot vs. dolly shot) This provides insight into the intentions of the cinematographer, allowing us to quickly identify and surface stylistic camera choices that provide insight into the mood, tone and genre of the title.
• Object detection — The detection of props and animated object segmentation allow us to attribute importance to non-human subjects in the frame.

Composition Metadata

Composition metadata refers to a special set of heuristic characteristics that we’ve identified and defined based on some of the core principles in photography, cinematography and visual aesthetic design. Some examples of composition are rule-of-third, depth-of-field and symmetry.

Image Ranking

After we’ve processed and annotated every frame in a given video, the next step is to surface “the best” image candidates from those frames through an automated artwork pipeline. That way, when our creative teams are ready to begin work for a piece of content, they are automatically provided with a high quality image set to choose from. Below, we outline some of the key elements we use to surface the best images for a given title.

Actors

Actors play a very important role in artwork. One way we identify the key character for a given episode is by utilizing a combination of face clustering and actor recognition to prioritize main characters and de-prioritize secondary characters or extras. To accomplish this, we trained a deep-learning model to trace facial similarities from all qualifying candidate frames tagged with frame annotation to surface and rank the main actors of a given title without knowing anything about the cast members.

Beyond cast, we also take into account pose, facial landmarks, and the overall position of characters for a given cast member.

Frame Diversity

Creative and visual diversity is a highly subjective discipline, as there are many different ways to perceive and define diversity in imagery. In the context of this solution, image diversity more specifically refers to the algorithms ability to capture the heuristic variance that naturally occurs within a single movie or episode. In doing so, we hope to provide designers and creatives with a scalable mechanism to quickly understand which visual elements are most representative of the title, and which elements are misrepresentative of the title. Some of the visual heuristic variables that we’ve incorporated into AVA to surface a diverse image set for a title include elements such as camera shot types (long shot vs medium shot), visual similarity (rule of thirds, brightness, contrast), color (colors that are most prominent), and saliency maps (to identify negative space and complexity). By combining these heuristic variables, we can effectively cluster image frames based on a custom vector for diversity. Furthermore, by incorporating several vectors, we’re able to construct a diversity index against which all candidate imagery for a given episode or movie, can be scored.

Filters for Maturity

For content sensitivity and audience maturity reasons, we also needed to make sure we excluded frames containing harmful or offensive elements. Examples of editorial exclusion criteria are things like; sex/nudity, text, logos/unauthorized branding, and violence/gore. In order to de-prioritize frames containing these elements, we incorporated the probability of each of these variables as vectors, allowing us to quantify and ultimately attribute a lower score for these frames.

We additionally included elements such as title genre, content format, maturity rating, etc. as secondary elements or minor features and as feedback to the model for ranking prediction.

Conclusion

In this techblog, we’ve provided an overview of our unique approach to surfacing meaningful images from video and enabling our creative teams to design stunning artwork every single day. AVA is a collection of tools and algorithms encapsulating the key intersections of computer vision combined with the core principles of filmmaking and photo editing.

Stay tuned for a follow up blog in which we’ll dive into programmatic artwork composition, an exciting new solution that’s responsible for much of the artwork you see on the Netflix service today!

Thank you.

If you have great or innovative ideas come join us on the Content Platform Engineering team!

---

AVA: The Art and Science of Image Discovery at Netflix was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Das Krankenhaus in Friedrichshain hat Gewinne gemacht. Es soll auch in die Bauarbeiten am Neubau auf dem Gelände fließen. Das Krankenhaus in Friedrichshain, genauer im Friedrichshain, – das Vivantes – hat im letzten Jahr einen ordentlichen Gewinn erwirtschaftet. Tatsächlich finde ich es schon seltsam, dass ein Krankenhaus einen Profit erwirtschaftet – oder soll. Aber das […]

Your browser does not support iframes.

VIDEO: Steve Nash to miss entire 2014-15 season with nerve issue

This changes nothing, and this changes everything.

Steve Nash was locked in as a first-ballot Hall of Famer years ago, one of the stars of a generation and one of the standout point guards of any era. So, the agonizing slow leak into retirement — after Thursday’s announcement of Nash missing the entire 2014-15 season with a nerve issue — of what will become three consecutive seasons with serious injuries will not dent his legacy. He got old, not bad.

But what an insightful few years it was. We didn’t get to see Nash close to his best in L.A., what the Lakers hoped for when they sent a couple first-round picks, including the choice that is top-five protected in 2015, and a couple seconds to Phoenix in July 2012, but it was the best of Nash in some ways. The passion to play, the determination to work back instead of taking early retirement and a golden parachute — it was as telling in a strange way as any of the countless accomplishments on the court.

He was always faking people out like that. Nash didn’t have much of a future coming out of high school in the charming Vancouver suburb of Victoria, and then he turned one NCAA Division I scholarship offer, to Santa Clara, into being drafted in the first round and a career that would have reached Season 19 in 2014-15. He didn’t have the athleticism to hang with the speed point guards, and then he surgically steered the Phoenix jet offense of the Seven Seconds Or Less Days, running everyone else into the ground as it turned out. Now, at what by every indication is the end, although the Lakers have only said he is done for the season, Nash discovered a new way to impress.

Your browser does not support iframes.

VIDEO: Relive Steve Nash’s top 10 career assists

He had done it in most every other manner before: back-to-back MVPs, eight-time All-Star, the only player in NBA history to shoot at least 50 percent from the field, 40 percent on 3-pointers and 90 percent from the line four times. That’s two more than Larry Bird and three more than everybody else, third all-time in total assists, first all-time in free-throw percentage with at least 1,200 makes.

And if anything, Nash was underrated on offense — which is saying something considering the praise he earned. But to trigger one of the game’s lethal pick-and-roll games (particularly with superb finisher Amar’e Stoudemire) and also succeed in the high-octane offenses of coaches Mike D’Antoni and Alvin Gentry as the Suns reached the Western Conference finals is a note few point guards can reach. He was never a good defender who could get in the conversation with, say, John Stockton or Gary Payton as all-time great two-way point guards. But Nash with the ball was still a clinic.

That’s Nash’s direct impact. His final legacy, though, won’t be known for years, maybe even for a decade.

The wave of Canadian players into the Draft the last few seasons? That is partly on him, too. Probably not to the extent of the expansion Raptors taking root in Toronto and the expansion Grizzlies in Vancouver. Maybe not even equal to the impact of Vince Carter winning the slam-dunk crown at All-Star weekend 2000 as a Raptor, given the impact of that event on kids and the basketball explosion in Toronto in particular.

But the guy who hadn’t played for a team in Canada since high school became the Nash-ional hero.

There’s Andrew Wiggins. Anthony Bennett. Kelly Olynyk, from British Columbia. Tristan Thompson. Nik Stauskas.

Stauskas was 14 or 15 — he doesn’t remember exactly — and part of a new breed of Canadian kids, the ones who didn’t grow up automatically playing hockey. His AAU coach, Anthony Otto, had known Nash for years and arranged for Stauskas and another prospect, Kevin Zabo, to spend a couple days being tutored by Nash in Phoenix. Two star-struck teenagers, a future Hall of Famer and an empty gym.

“I got a chance to work out with him and see him up close and the fundamentals he had,” Stauskas said. “For me, it was just like, ‘He’s not quick, he’s not strong, he doesn’t have a crazy build or anything and here he is a two-time MVP.’ You’re like, ‘Man, this is possible. If you work hard and do what he does, this is really possible.’ “

There were times Zabo, now at San Diego State, and Stauskas, now a Kings rookie as a lottery pick, stopped their individual work and watched Nash — now also general manager of the Canadian national team —  in another part of the gym, for as long as 20 minutes. Just watching the Suns guard go through drills.

A technician like Nash had that kind of draw. It was hard not to stop and watch him at every opportunity, even when he played with Dirk Nowitzki in Dallas or Stoudemire and Shaquille O’Neal in Phoenix or Kobe Bryant and Pau Gasol in Los Angeles. The chance to watch is almost certainly over as age claims another victim, but the disappointment of the hobbling finish for someone who had earned the right to go out on his terms doesn’t matter to the legacy.

It changes nothing. And everything.

Wedding ist nicht Wedding. So steht es auf der Seite www.planet-wedding.de. Blog-Betreiberin Dominique Hensel schreibt dort seit 2008 über ihre Erlebnisse in Gesundbrunnen und im Wedding. Ab sofort lädt die Journalistin aus dem Brunnenviertel die Leser des Weddingweiser ein Mal im Monat dazu ein, einen Blick in die Welt einer Weddinger Familie zu werfen. Die Alltags-Kolumne über eine Familie auf dem Planeten Wedding – immer am ersten Mittwoch im Monat. Heute: Sonntagsfrühstück.

Für den Honig gibt es eine Warteliste. Ganz klar, dass man ihn nicht einfach so haben kann. Man stellt sich brav hinten an, wenn man etwas naschen will am Frühstückstisch und drängelt sich nicht vor. Das tut man nicht, schon gar nicht am Sonntagmorgen! Sonst kneift die kleine, aufmerksame Skeptikerin sofort aufgebracht die Augen zusammen, das Engelchen ergreift dann lautstark Partei und die diplomatische Lauryn strengt sich an, den entstehenden Streit zu schlichten.

Ein Frühstücksthema muss her, ganz klar! Oder ein Spiel. Also kommt auf den Tisch was die großen Mädchen soeben in der Schule gelernt haben. Und weil wir ausgelassen sind an diesem Morgen, steigern wir das Wort Frühstück und bis zu höchsten Form und freuen uns diebisch, dass das eigentlich gar nicht geht. Am frühstücksten. Der Honig aus dem Humboldthain geht rum, die Himbeermarmelade von Oma H., die Butter, dann wird Tee nachgegossen. Der Tisch ist fast zu klein für die erweiterte Frühstücksrunde, die Ananas muss sich anstrengen, um einen Platz zu bekommen. Was geschieht da? Es riecht so komisch plötzlich … Aber Lauryn hat ein gutes Näschen und rettet den Brotkorb vor dem übermütigen Teelicht. Wie gut, dass sie dabei ist am Familiensonntag!

Der große Skeptiker glänzt heute wie ein strahlender Stern und kann sich entspannt zurücklehnen. Er hat den Tisch gedeckt, Tee gekocht, Brötchen in der Brunnenstraße geholt und versüßt seinen Mädels damit den Tag. Dann soll er auch noch das letzte Croissant teilen! Acht Augen und Ohren warten auf das Donnerwetter. Aber er ist guter Stimmung und neckt den Besuch mit einem listigen Blick – aber nur zum Schein. Das Croissant wird fair halbiert. Und später, später besuchen wir noch Elvis, dichten und singen und das Engelchen probiert an die 100 Mal aus, wie lang man das „i“ mit Melodie in die Länge ziehen kann. Doch da ist schon alles verputzt und die Königin in ihrem langen Kleid an der Reihe. Sie hat dieses Mal leider nicht mitgefrühstückt. Aber nächste Woche, da laden wir sie ein an unseren Frühstückstisch! Aber nur wenn sie mitspielt und herausbekommt, was das bedeutet: SFIS.

Foto und Text: Dominique Hensel

Our dresser drawers look like a clothing bomb went off in them. That is to say, our folding skills are severely lacking. Now, instead of spending half an hour attempting to get everything folded nicely, we can be back on the couch with a beer in hand in no time. You can do the same, just watch the video and see how you can fold a shirt in under two seconds.