Airbnb Shares The Keys To Its Infrastructure

Date: September 10, 2015 Published by

  • Interestingly, even though the Mesos cluster controller was commercialized by techies Twitter and Airbnb, who started Mesosphere, Airbnb is not using Mesos very much today.
  • The infrastructure at Airbnb is not anywhere approaching that scale, but it is still large and growing fast.
  • To make embedding machine learning into its applications easier, Airbnb created a tool called Aerosolve, which is available open source on GitHub, that plugs into the Apache Spark in-memory processing engine, which Airbnb runs as a standalone platform for its early experiments in machine learning.
  • As far as machine learning training goes, we train on CPUs right now.
  • The two Hadoop clusters used by Airbnb today are kept in synch using Kafka for log collection and streaming data to both clusters at the same time.