Spotify’s Music Recommendations Lambda Architecture

Понравилась презентация – покажи это...

Слайд 0

Spotify’s Music Recommendations Lambda Architecture Esh Kumar @eshvk Emily Samuels @emilymsa

Слайд 1

Overview Why Lambda? Use Case: Discover Recommendations Batch Architecture Real-time Architecture Challenges Future Work

Слайд 2

Why Lambda? 1 new user every 3 seconds. Contextual, time based recs more & more important

Слайд 3

Discover Recs

Слайд 4

The Discover Page Algorithmically generated fresh recs for users.

Слайд 5

The Discover Batch Pipeline

Слайд 6

Machine Learning Deep Dive

Слайд 7

Word2Vec Words with similar contexts have similar meaning

Слайд 8

Word2Vec King – Man + Woman = Queen

Слайд 9

Annoy Approximate Nearest Neighbors Oh Yeah! https://github.com/spotify/annoy

Слайд 10

Batch Architecture Strengths Recs based on complete user history Weakness User vector generation time increasing with no. users. Not reflective of current mood.

Слайд 11

Intro to Storm

Слайд 12

Storm Distributed real-time computation system

Слайд 13

Storm @ Spotify

Слайд 14

Real-time Architecture

Слайд 15

Workers die -> Cascading JVM Process death Memcache flakiness Cassandra JVM problems due to write/overwrite pattern Challenges

Слайд 16

Future/Ongoing Work Simplify the topology Keep listens for 24 hours Ongoing work on other real time personalization features.

Слайд 17

Questions Esh Kumar eshvk@spotify.com Emily Samuels esamuels@spotify.com