Attending this event?
Virtual Event
August 17–August 20, 2020

The schedule is subject to change. As we adjust to a virtual experience, our plan is to keep the sessions the same, which is dependent on speaker availability.

Learn More and Register to Attend This Event
Back To Schedule
Thursday, August 20 • 15:30 - 16:05
Elephant on Wheels: Petabyte-scale AI @ LinkedIn - Cong Gu & Abin Shahab, LinkedIn

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Kubernetes has flourished at LinkedIn for AI workloads. It started as a proof of concept for Jupyter notebooks, and now it has become a key infrastructure for model training and model serving. LinkedIn AI has been traditionally Hadoop/YARN based, and its Hadoop data lake is one of the worlds largest. To allow AI and non-AI workloads to securely access HDFS, a scalable, secure, open-source integration with HDFS Kerberos called Kube2Hadoop was built. This enables AI modelers at LinkedIn to use data securely in their model exploration and training with KubeFlow components such as the mpi-operator. LinkedIn’s infra teams are also prototyping a multilevel scheduler on top of Kubernetes and YARN clusters on the cloud, which can intelligently route jobs to multiple clusters and can facilitate workflows across Kubernetes and YARN clusters.


Abin Shahab

Staff Software Engineer, LinkedIn
Abin Shahab is a Staff Engineer at Linkedin’s Big Data Platform (BDP) team. He joined Linkedin in 2017 and leads the Deep Learning infra team in BDP. He is a veteran KubeCon speaker.

Cong Gu

Software Engineer, LinkedIn
Cong is a Software Engineer at LinkedIn's Big Data Platform team. He helps AI engineers by building infrastructure to improve their productivity. He's been with LinkedIn for about 10 months. He has given technical deep-dive talks in company-wide settings as well as at KubeFlow su... Read More →

Thursday August 20, 2020 15:30 - 16:05
Feedback form isn't open yet.