KubeCon + CloudNativeCon Europe 2020

Provisioning GPUs for ML workloads in data center can be very costly and more costly if they are not fully utilized. Thus, maximizing the GPU utilization is a must for ML workloads.

This session will show how a single GPU can be used to run multiple ML workloads, especially ML inference, in parallel and will deep dive into the understanding of how GPUs are provisioned and attached using K8s device plugins. It will show how the nvidia device plugin can be extended to schedule multiple ML workloads to a single GPU and collect desired GPU information with Prometheus.

This session will highlight and deep dive into native GPU sharing using K8s device plugin without additional technologies such as vGPUs from VMware.

Kubeflow is an open source machine learning platform built on Kubernetes. Every service in Kubeflow is implemented either as a Custom Resource Definition (CRD) (e.g., TensorFlow Job) or as a standalone service (e.g., Kubeflow Pipelines).

As enterprises start to adopt Kubeflow, the need for access control, authentication, and authorization is emerging. Kubernetes CRDs come with their own auth story, but what about Services with their own API and database, like Kubeflow Pipelines? In this talk, we explore how we enabled multi-user workflows for Kubeflow Pipelines, in a Kubernetes-native way.

We present how we combined open-source, cloud-native technologies to design and implement a flexible, Kubernetes-native solution for services with their own API and database. The talk will include a live demo.

The Kubeflow project brings incredibly powerful Machine Learning frameworks like TensorFlow and PyTorch to Kubernetes. The ability to parallelize training and the ability to scale workflows up and down is revolutionary. However, state and persistent storage are a much bigger challenge for machine learning workloads because of their training data, library files, and models. We will discuss what it took to create AI/ML environments running thousands of pods and that request petabytes of training data.

We will explore the various state and storage challenges that crop up when you are building Kubeflow applications. We will discuss where distributed persistent storage solutions fit in the picture. We will address various storage api's including: POSIX/CSI solutions, NFS, S3, and HDFS fit into solutions. Data security and privacy issues will be discussed.

The recent popularity of hybrid cloud architectures have left organizations with traditional data lakes behind. How do you build a data architecture that is cloud provider agnostic and is flexible enough to run in across public and private cloud data centers? How do you make this data available to data scientists and developers in a way that simplifies the creation of intelligent applications?

This talk will walk through a new way of building data lakes for the hybrid cloud using Rook and Ceph community projects running on Kubernetes. With a single data architecture deployment that can run in any cloud or across multiple clouds, IT and data engineers can use open source tools on Kubernetes such as Rook, Ceph, Hive metastore, Spark, and Presto to provide unified access to massive amounts of data across multiple data centers for data scientists and developers.

With the growing scale of DL and ML applications, distributed execution of jobs across multiple nodes becomes increasingly critical -- to solve bigger problems faster -- as illustrated by the recent MLperf results. However running such workloads in a production K8s cluster shared by multiple jobs/users has several challenges.

In this talk, we’ll give an overview of this area -- including distributed Tensorflow, Pytorch, Horovod, MPI -- and the use of GPU nodes with NCCL and RDMA for accelerated performance. We’ll describe our end-to-end flow for multi-node jobs in K8s including gang scheduling, quotas, fairness and backfilling implemented in our custom scheduler for GPUs. Our cluster includes high-speed networking through RoCE and SR-IOV / Multus CNI. We’ll share our design choices, learnings and operational experience including failure handling, performance and telemetry.

While machine learning is spreading like wildfire, very little attention has been paid to the ways that it can go wrong when moving from development to production. Even when models work perfectly, they can be attacked and/or degrade quickly if the data changes. Having a well understood MLOps process is necessary for ML security!

Using Kubeflow, we will demonstrate how to the common ways machine learning workflows go wrong, and how to mitigate them using MLOps pipelines to provide reproducibility, validation, versioning/tracking, and safe/compliant deployment. We will also talk about the direction for MLOps as an industry, and how we can use it to move faster, with less risk, than ever before.

In the search for unknown particles in the CERN Large Hadron Collider’s particle collisions, billions of events need to be analysed. Even though large parts of CERN’s computing infrastructure are deployed using Kubernetes, physics analysis jobs are still being run on classical high throughput computing batch systems. While developing a fully cloud native computing approach, one still needs to have access to the ten-thousands of cores available on the legacy batch system to have sufficient resources for the data processing.

In this presentation, Clemens will demonstrate how complex physics analysis workflows that are written and scheduled using Kubernetes can make use of classical batch systems. The audience will also learn what complexity a realistic physics analysis can reach, and the important role that software containers and Kubernetes play in the context of open science.

This session will provide a Kubeflow 1.0 Update by a Kubeflow Community Product Manager.  The presentation will include a review of the Kubeflow Community and feature development process, the Kubeflow user survey results, and Kubeflow 1.0 features.  The talk will highlight significant business benefits and review use cases from top deployments.  It will also include a live demonstration of a workflow to build, train and deploy a versioned Kubeflow Pipeline.

At Intuit, customer data sets are growing exponentially with the growth of the business and the capabilities offered. Processing this data and making it available for downstream applications such as ML, Analytics, Exploration etc. is crucial. Following the trend of running the services workload on Kubernetes, we built a data processing platform with Spark on Kubernetes as the backbone. This allowed us to reap all the benefits of well established processes for CI/CD, security and cluster management. With these we were able to reduce the cost footprint of our data processing jobs by 30%, while simultaneously increasing the speed to production.

Kubeflow is maturing as a cloud native Machine Learning (ML) platform that simplifies the journey of development, deployment, and management of ML on Kubernetes. As Kubeflow gets increasingly adopted, practitioners are looking beyond functions and starting to explore its performance and cost efficiency in the real world. MLPerf is a state-of-the-art benchmark suite that aims to set an industry standard for end-to-end performance evaluation of ML systems with real-world workloads covering both training and inference phases of ML lifecycle. This talk will provide a brief overview about MLPerf, followed by detailed discussions about how MLPerf can be adapted to evaluate ML performance on Kubeflow and Kubernetes, as well as how the performance results can be leveraged to guide the future design and optimization of cloud native ML platforms based on Kubeflow and Kubernetes.