Loading…
Timezone
Attending this event?
Virtual Event
August 17–August 20, 2020
Learn More and Register to Attend This Event

The Sched app allows you to build your schedule but is not a substitute for your event registration. You must be registered for KubeCon + CloudNativeCon Europe 2020 - Virtual to participate in the sessions. If you have not registered but would like to join us, please register here.

Please note: This schedule is automatically displayed in Central European Summer Time (CEST). To see the schedule in your preferred timezone, please select from the drop-down menu to the right, above "Filter by Date." The schedule is subject to change.
Back To Schedule
Thursday, August 20 • 18:05 - 18:40
Elephant on Wheels: Petabyte-scale AI @ LinkedIn - Cong Gu & Abin Shahab, LinkedIn

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Tweet Share
Kubernetes has flourished at LinkedIn for AI workloads. It started as a proof of concept for Jupyter notebooks, and now it has become a key infrastructure for model training and model serving. LinkedIn AI has been traditionally Hadoop/YARN based, and its Hadoop data lake is one of the worlds largest. To allow AI and non-AI workloads to securely access HDFS, a scalable, secure, open-source integration with HDFS Kerberos called Kube2Hadoop was built. This enables AI modelers at LinkedIn to use data securely in their model exploration and training with KubeFlow components such as the mpi-operator. LinkedIn’s infra teams are also prototyping a multilevel scheduler on top of Kubernetes and YARN clusters on the cloud, which can intelligently route jobs to multiple clusters and can facilitate workflows across Kubernetes and YARN clusters.
Speakers
AS

Abin Shahab

Staff Software Engineer, LinkedIn
Abin Shahab is a Staff Engineer at Linkedin’s Big Data Platform (BDP) team. He joined Linkedin in 2017 and leads the Deep Learning infra team in BDP. He is a veteran KubeCon speaker.
CG

Cong Gu

Software Engineer, LinkedIn
Cong is a Software Engineer at LinkedIn's Big Data Platform team. He helps AI engineers by building infrastructure to improve their productivity. He's been with LinkedIn for about 10 months. He has given technical deep-dive talks in company-wide settings as well as at KubeFlow su... Read More →

Thursday August 20, 2020 18:05 - 18:40
Virtual
  Customizing + Extending Kubernetes
Feedback form isn't open yet.