KubeCon + CloudNativeCon Europe 2020

LinkedIn operates one of the world’s largest Hadoop environments, with ~450PB used data, 2 billion files/blocks, and over 400K jobs/day. However, testing cluster features in an isolated fashion has been traditionally fairly difficult. Infra teams such as HDFS, YARN, and Azkaban often step on top of one another for testing new features in our existing test Hadoop clusters. Setting up a new test cluster requires coordination between hardware, infra, and security teams, usually taking weeks to months. We have recently extended Kubernetes’ usage to test Hadoop(HDFS/YARN) clusters, by deploying production-like Hadoop cluster on Kubernetes. This has reduced infra setup time from weeks down to minutes with no network, hardware dependencies, and enables critical infra/workflow teams to test new features on the fly.