Author: Ananya Kumar (@AnanyaKumar)
This document presents the design of the Kubernetes DaemonSet, describes use cases, and gives an overview of the code.
Many users have requested for a way to run a daemon on every node in a Kubernetes cluster, or on a certain set of nodes in a cluster. This is essential for use cases such as building a sharded datastore, or running a logger on every node. In comes the DaemonSet, a way to conveniently create and manage daemon-like workloads in Kubernetes.
The DaemonSet can be used for user-specified system services, cluster-level applications with strong node ties, and Kubernetes node services. Below are example use cases in each category.
Logging: Some users want a way to collect statistics about nodes in a cluster and send those logs to an external database. For example, system administrators might want to know if their machines are performing as expected, if they need to add more machines to the cluster, or if they should switch cloud providers. The DaemonSet can be used to run a data collection service (for example fluentd) on every node and send the data to a service like ElasticSearch for analysis.
Datastore: Users might want to implement a sharded datastore in their cluster. A few nodes in the cluster, labeled ‘app=datastore’, might be responsible for storing data shards, and pods running on these nodes might serve data. This architecture requires a way to bind pods to specific nodes, so it cannot be achieved using a Replication Controller. A DaemonSet is a convenient way to implement such a datastore.
For other uses, see the related feature request
The DaemonSet supports standard API features: - create - The spec for DaemonSets has a pod template field. - Using the pod’s nodeSelector field, DaemonSets can be restricted to operate over nodes that have a certain label. For example, suppose that in a cluster some nodes are labeled ‘app=database’. You can use a DaemonSet to launch a datastore pod on exactly those nodes labeled ‘app=database’. - Using the pod’s nodeName field, DaemonSets can be restricted to operate on a specified node. - The PodTemplateSpec used by the DaemonSet is the same as the PodTemplateSpec used by the Replication Controller. - The initial implementation will not guarantee that DaemonSet pods are created on nodes before other pods. - The initial implementation of DaemonSet does not guarantee that DaemonSet pods show up on nodes (for example because of resource limitations of the node), but makes a best effort to launch DaemonSet pods (like Replication Controllers do with pods). Subsequent revisions might ensure that DaemonSet pods show up on nodes, preempting other pods if necessary. - The DaemonSet controller adds an annotation:
"kubernetes.io/created-by: \<json API object reference\>"
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: labels: app: datastore name: datastore spec: template: metadata: labels: app: datastore-shard spec: nodeSelector: app: datastore-node containers: - name: datastore-shard image: kubernetes/sharded ports: - containerPort: 9042 name: main
commands that get info:
In general, for all the supported features like get, describe, update, etc, the DaemonSet works in a similar way to the Replication Controller. However, note that the DaemonSet and the Replication Controller are different constructs.
We considered several alternatives, that were deemed inferior to the approach of creating a new DaemonSet abstraction.
One alternative is to include the daemon in the machine image. In this case it would run outside of Kubernetes proper, and thus not be monitored, health checked, usable as a service endpoint, easily upgradable, etc.
A related alternative is to package daemons as static pods. This would address most of the problems described above, but they would still not be easily upgradable, and more generally could not be managed through the API server interface.
A third alternative is to generalize the Replication Controller. We would do
something like: if you set the
replicas field of the ReplicationControllerSpec
to -1, then it means “run exactly one replica on every node matching the
nodeSelector in the pod template.” The ReplicationController would pretend
replicas had been set to some large number – larger than the largest number
of nodes ever expected in the cluster – and would use some anti-affinity
mechanism to ensure that no more than one Pod from the ReplicationController
runs on any given node. There are two downsides to this approach. First,
there would always be a large number of Pending pods in the scheduler (these
will be scheduled onto new machines when they are added to the cluster). The
second downside is more philosophical: DaemonSet and the Replication Controller
are very different concepts. We believe that having small, targeted controllers
for distinct purposes makes Kubernetes easier to understand and use, compared to
having larger multi-functional controllers (see
“Convert ReplicationController to a plugin” for
some discussion of this topic).