KubeCon + CloudNativeCon Europe 2020

Databases and stateful apps are gravitating rapidly to Kubernetes, so the sorrows of accidental data loss cannot be far behind. As long-time database engineers and authors of the ClickHouse Kubernetes operator, our team has seen lots of imaginative ways to lose data. We also learned how to prevent them.

The talk starts with standard high availability/disaster recovery models used in DBMS and demonstrates that classic ways to lose data are still alive and well on Kubernetes. We'll then pivot to Kubernetes-specific disasters-in-waiting, such as the PV that wasn't, affinity afflictions, and the dreaded fat fingers of fate. The talk will help instill a healthy sense of paranoia and give listeners tools to ensure their experiences with cloud-native data will be happy ones.

Vitess has a flexible sharding architecture and natively supports "cells" which correspond to infrastructure located in multiple locations. This allows for the creation of jurisdiction-aware database clusters that solve data locality without having to re-architect your application. Also, Vitess's built-in support for resharding workflows makes migrating from existing databases into databases resident in multiple locations easy.
In this talk the speakers will show how to build a custom sharding scheme in Vitess that respects data locality requirements. They will then demonstrate a database cluster built using this scheme that enables transfer of existing data belonging to people from 8 different countries from one jurisdiction to four(4) separate jurisdictions with zero downtime at the application level.

Containerized machine learning workloads running on Kubernetes receive benefits such as portability, declarative configuration, less administrative toil, all with marginal performance impact. The best published results for performance sensitive machine learning workloads, e.g. MLPerf v0.6, were obtained by reading the datasets from local SSDs. While the MLPerf datasets fit comfortably on a single SSD, it’s a luxury not afforded to folks training models against petabyte scale datasets. We’ll share our experience running MLPerf training jobs in Kubernetes, against datasets stored by Kubernetes stateful storage services orchestrated by Rook. Highlights include the performance and scalability tradeoffs associated with local and open source distributed storage, and how machine learning formats like RecordIO and TFRecord provide performance utility and model validation flexibility.

This session presents a summary of the existing Kubernetes features to use local storage devices (e.g. hostPath and local persistent volumes) in first, then introduces a novel CSI plugin named TopoLVM.

Applications that can replicate data themselves (e.g. Elasticsearch, MySQL, Ceph) are promising users of local storage because of their needs for lower prices and higher I/O performance storage. Although existing ways enable to use of local storage, the following desired features are not provided yet:

- Raw block volume
- Capacity-aware dynamic provision
- Online volume resizing

To satisfy these features, the authors created a novel CSI plugin named TopoLVM. It dynamically creates logical volumes of LVM and extends the standard Pod scheduler to consider volume group capacity of each node. Besides, it intends to support more features including online resizing.

Local Persistent Volumes are a powerful feature of Kubernetes, enabling users to build and operate high-performance stateful systems in their clusters and bridging the storage gap for on-premise users. However, their usage comes with subtle pitfalls that users may not be aware of, and understanding of these aspects is key to a seamless production experience.

In this talk, Matt will share best practices and how-to’s for local volumes based on experience running them in production since their alpha release. He will cover topics such as in-cluster volume provisioning, best practices not obvious from Kubernetes documentation, and cloud provider-specific local disk behavior.

The audience will walk away with a better understanding of the tradeoffs of local volumes and how to leverage them in their workloads, concrete tips for avoiding common mistakes, and day-two operational best practices.