KubeCon + CloudNativeCon Europe 2020

If debugging distributed systems is like solving a murder mystery, then we need a way to gather clues without disturbing the scene of the crime. Some application errors (resource leaks and race conditions in particular) can take a long time to reproduce; other errors might occur unexpectedly in production. When this happens, we want to gather as much
information as possible without restarting the pod. Tools like detailed logging, debug sidecars, and Linkerd's "tap" are only usable if we had the foresight to include them before the error occurred. Ephemeral containers promise to be a useful tool, but are currently in alpha. We will compare these techniques and then see how we can use direct access to the node to enter a pod's network namespace and do detailed network debugging without needing to restart any running containers.