'

Developing Frameworks for Apache Mesos

Понравилась презентация – покажи это...





Слайд 0

Developing Frameworks for Apache Mesos


Слайд 1

Joe Stein CEO of Elodina http://www.elodina.net/ a big data as a service platform built on top open source software. The Elodina platform enables customers to analyze data streams and programmatically react to the results in real-time. We solve today’s data analytics needs by providing the tools and support necessary to utilize open source technologies. As users, contributors and committers, Elodina also provides support for frameworks that run on Mesos including Apache Kafka, Exhibitor (Zookeeper), Apache Storm, Apache Cassandra and a whole lot more! LinkedIn: http://linkedin.com/in/charmalloc Twitter : @allthingshadoop


Слайд 2

What goes on Mesos? Framework = (Scheduler + Executor) What does it look like without a scheduler? We can do better using a scheduler! Schedulers working together. Framework API & Examples. Overview


Слайд 3

What goes on Mesos?


Слайд 4


Слайд 5

Many, many things


Слайд 6

Anything can be run on Mesos with Marathon or Aurora https://mesosphere.github.io/marathon/ http://aurora.apache.org/


Слайд 7

Framework = (Scheduler + Executor)


Слайд 8


Слайд 9

Scheduler


Слайд 10

Executors


Слайд 11

mesos/kafka https://github.com/mesos/kafka


Слайд 12

Scheduler Provides the operational automation for a Kafka Cluster. Manages the changes to the broker's configuration. Exposes a REST API for the CLI to use or any other client. Runs on Marathon for high availability. Broker Failure Management “stickiness” Executor The executor interacts with the kafka broker as an intermediary to the scheduler Scheduler & Executor


Слайд 13

CLI & REST API scheduler - starts the scheduler. add - adds one more more brokers to the cluster. update - changes resources, constraints or broker properties one or more brokers. remove - take a broker out of the cluster. start - starts a broker up. stop - this can either a graceful shutdown or will force kill it (./kafka-mesos.sh help stop) rebalance - allows you to rebalance a cluster either by selecting the brokers or topics to rebalance. Manual assignment is still possible using the Apache Kafka project tools. Rebalance can also change the replication factor on a topic. help - ./kafka-mesos.sh help || ./kafka-mesos.sh help {command}


Слайд 14

Launch 20 brokers in seconds ./kafka-mesos.sh add 1000..1019 --cpus 0.01 --heap 128 --mem 256 --options num.io.threads=1 ./kafka-mesos.sh start 1000..1019


Слайд 15

What does it look like without a scheduler?


Слайд 16

without a scheduler


Слайд 17

without a scheduler


Слайд 18

without a scheduler


Слайд 19

without a scheduler


Слайд 20

without a scheduler


Слайд 21

We can do better using a scheduler


Слайд 22

with a scheduler


Слайд 23

with a scheduler


Слайд 24

with a scheduler


Слайд 25

with a scheduler


Слайд 26

with a scheduler


Слайд 27

Schedulers working together


Слайд 28


Слайд 29


Слайд 30


Слайд 31

Framework API & Examples


Слайд 32

Mesos Protos https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto Everything is good to understand but here is a good place to start FrameworkInfo TaskInfo TaskState MasterInfo SlaveInfo


Слайд 33

Mesos Framework Development Guide http://mesos.apache.org/documentation/latest/app-framework-development-guide/ Scheduler API registered, reregistered, disconnected resourceOffers, offerRescinded, statusUpdate, frameworkMessage slaveLost, executorLost, error Executor API registered, reregistered, disconnected launchTask, killTask, frameworkMessage


Слайд 34

Task Reconciliation http://mesos.apache.org/documentation/latest/reconciliation/ It is the responsibility of Mesos (scheduler driver / Master) to ensure that the framework is notified when a disconnection, and subsequent (re-)registration occurs. At this point, the scheduler should perform task state reconciliation.


Слайд 35

Language Bindings c++ > https://github.com/apache/mesos/tree/master/src/examples python > https://github.com/apache/mesos/tree/master/src/examples/python https://github.com/wickman/pesos java > https://github.com/apache/mesos/tree/master/src/examples/java http://mesos.apache.org/api/latest/java/ https://github.com/groupon/jesos go > https://github.com/mesos/mesos-go clojure > https://github.com/dgrnbrg/clj-mesos https://github.com/pyr/mesomatic scala > https://github.com/mesosphere/scala-sbt-mesos-framework.g8


Слайд 36

Rendler A rendering web crawler for Apache Mesos. https://github.com/mesosphere/RENDLER


Слайд 37

Questions? Joe Stein http://www.elodina.net


×

HTML:





Ссылка: