Loading…
Virtual Event
August 17–August 20, 2020
Learn More and Register to Attend This Event

The Sched app allows you to build your schedule but is not a substitute for your event registration. You must be registered for KubeCon + CloudNativeCon Europe 2020 - Virtual to participate in the sessions. If you have not registered but would like to join us, please register here.

Please note: This schedule is automatically displayed in Central European Summer Time (CEST). To see the schedule in your preferred timezone, please select from the drop-down menu to the right, above "Filter by Date." The schedule is subject to change.
Back To Schedule
Wednesday, August 19 • 17:40 - 18:15
Monitoring GPUs at Scale for AI/ML and HPC Clusters - Bharti L Agrawal, NVIDIA

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Tweet Share
Feedback form is now closed.
At Nvidia we have several large GPU K8s clusters for running deep learning training (AI/ML) workloads. On these clusters we need monitoring to support a range of user personas . First we have the end users (AI/ML researchers) who want to get an insight into how well their workloads used the GPUs and the system. Then we have the operations team who would like to monitor the general health of the cluster and be alerted in real time to any issues. Finally we have the stakeholders who would like to see the GPU utilization and saturation over time for capacity planning. These requirements cannot be satisfied by a standard “out of the box” setup.

In this presentation we will show how we used a combination of open source tools to address our requirements. We will discuss various deployment, maintenance, security and scale challenges we hit and how we resolved them for monitoring GPU data.
Speakers
avatar for Bharti L Agrawal

Bharti L Agrawal

Senior Staff Engineer, NVIDIA
Bharti Agrawal been in the software industry for over 20 years. Her career has taken her from working on mainframes, to web 2.0 sites, to SAAS applications, to advertising platforms. She has worked in a wide range of companies from small startups to Google and Yahoo. She currently... Read More →

MonitoringGPUsAtScale pptx
CLOSE CAPTIONING TRANSCRIPT COCG 4067 txt
Wednesday August 19, 2020 17:40 - 18:15 CEST
InXpo https://onlinexperiences.com/Launch/Event.htm?ShowKey99259
  Observability