Please note: this proposal doesn’t reflect final implementation, it’s here for the purpose of capturing the original ideas. You should probably read
kubeadmdocs, to understand the end-result of this effor.
Luke Marsden & many others in SIG-cluster-lifecycle.
17th August 2016
This proposal aims to capture the latest consensus and plan of action of SIG-cluster-lifecycle. It should satisfy the first bullet point required by the feature description.
Kubernetes is hard to install, and there are many different ways to do it today. None of them are excellent. We believe this is hindering adoption.
Have one recommended, official, tested, “happy path” which will enable a majority of new and existing Kubernetes users to:
Kick the tires and easily turn up a new cluster on infrastructure of their choice
Get a reasonably secure, production-ready cluster, with reasonable defaults and a range of easily-installable add-ons
We plan to do so by improving and simplifying Kubernetes itself, rather than building lots of tooling which “wraps” Kubernetes by poking all the bits into the right place.
There are logically 3 steps to deploying a Kubernetes cluster:
Provisioning: Getting some servers - these may be VMs on a developer’s workstation, VMs in public clouds, or bare-metal servers in a user’s data center.
Install & Discovery: Installing the Kubernetes core components on those servers (kubelet, etc) - and bootstrapping the cluster to a state of basic liveness, including allowing each server in the cluster to discover other servers: for example teaching etcd servers about their peers, having TLS certificates provisioned, etc.
Add-ons: Now that basic cluster functionality is working, installing add-ons such as DNS or a pod network (should be possible using kubectl apply).
Notably, this project is only working on dramatically improving 2 and 3 from the perspective of users typing commands directly into root shells of servers. The reason for this is that there are a great many different ways of provisioning servers, and users will already have their own preferences.
What’s more, once we’ve radically improved the user experience of 2 and 3, it will make the job of tools that want to do all three much easier.
In time to be an alpha feature in Kubernetes 1.4.
Note: the current plan is to deliver
kubeadm which implements these stories as “alpha” packages built from master (after the 1.4 feature freeze), but which are capable of installing a Kubernetes 1.4 cluster.
Install: As a potential Kubernetes user, I can deploy a Kubernetes 1.4 cluster on a handful of computers running Linux and Docker by typing two commands on each of those computers. The process is so simple that it becomes obvious to me how to easily automate it if I so wish.
Pre-flight check: If any of the computers don’t have working dependencies installed (e.g. bad version of Docker, too-old Linux kernel), I am informed early on and given clear instructions on how to fix it so that I can keep trying until it works.
Control: Having provisioned a cluster, I can gain user credentials which allow me to remotely control it using kubectl.
Install-addons: I can select from a set of recommended add-ons to install directly after installing Kubernetes on my set of initial computers with kubectl apply.
Add-node: I can add another computer to the cluster.
Secure: As an attacker with (presumed) control of the network, I cannot add malicious nodes I control to the cluster created by the user. I also cannot remotely control the cluster.
In time for Kubernetes 1.5: Everything from Phase I as beta/stable feature, everything else below as beta feature in Kubernetes 1.5.
Upgrade: Later, when Kubernetes 1.4.1 or any newer release is published, I can upgrade to it by typing one other command on each computer.
HA: If one of the computers in the cluster fails, the cluster carries on working. I can find out how to replace the failed computer, including if the computer was one of the masters.
We will introduce a new binary, kubeadm, which ships with the Kubernetes OS packages (and binary tarballs, for OSes without package managers).
laptop$ kubeadm --help kubeadm: bootstrap a secure kubernetes cluster easily. /==========================================================\ | KUBEADM IS ALPHA, DO NOT USE IT FOR PRODUCTION CLUSTERS! | | | | But, please try it out! Give us feedback at: | | https://github.com/kubernetes/kubernetes/issues | | and at-mention @kubernetes/sig-cluster-lifecycle | \==========================================================/ Example usage: Create a two-machine cluster with one master (which controls the cluster), and one node (where workloads, like pods and containers run). On the first machine ==================== master# kubeadm init master Your token is: <token> On the second machine ===================== node# kubeadm join node --token=<token> <ip-of-master> Usage: kubeadm [command] Available Commands: init Run this on the first server you deploy onto. join Run this on other servers to join an existing cluster. user Get initial admin credentials for a cluster. manual Advanced, less-automated functionality, for power users. Use "kubeadm [command] --help" for more information about a command.
On first machine:
master# kubeadm init master Initializing kubernetes master... [done] Cluster token: 73R2SIPM739TNZOA Run the following command on machines you want to become nodes: kubeadm join node --token=73R2SIPM739TNZOA <master-ip> You can now run kubectl here.
On N “node” machines:
node# kubeadm join node --token=73R2SIPM739TNZOA <master-ip> Initializing kubernetes node... [done] Bootstrapping certificates... [done] Joined node to cluster, see 'kubectl get nodes' on master.
[done] would be colored green in all of the above.
The user (or their config management system) creates a token and passes the same one to both init and join.
master# kubeadm init master --token=73R2SIPM739TNZOA Initializing kubernetes master... [done] You can now run kubectl here.
master# kubeadm init master Error: socat not installed. Unable to proceed.
On master, after Install, kubectl is automatically able to talk to localhost:8080:
master# kubectl get pods [normal kubectl output]
To mint new user credentials on the master:
master# kubeadm user create -o kubeconfig-bob bob Waiting for cluster to become ready... [done] Creating user certificate for user... [done] Waiting for user certificate to be signed... [done] Your cluster configuration file has been saved in kubeconfig. laptop# scp <master-ip>:/root/kubeconfig-bob ~/.kubeconfig laptop# kubectl get pods [normal kubectl output]
Using CNI network as example:
master# kubectl apply --purge -f \ https://git.io/kubernetes-addons/<X>.yaml [normal kubectl apply output]
Same as Install – ”on node machines”.
node# kubeadm join --token=GARBAGE node <master-ip> Unable to join mesh network. Check your token.
TLS bootstrapping - so that kubeadm can mint credentials for kubelets and users
Fix for #30515 - so that kubeadm can install a kubeconfig which kubelet then picks up
kubeadm implementation - the kubeadm CLI itself, will get bundled into “alpha” kubeadm packages
Documentation - so that new users can see this in 1.4 (even if it’s caveated with alpha/experimental labels and flags all over it)
kubeadm alpha packages
Kubectl apply –purge - so that addons can be maintained using k8s infrastructure
Based on @philips’ comment here. The key point with this implementation plan is that it requires basically no changes to kubelet except #30515. It also doesn’t require kubelet to do TLS bootstrapping - kubeadm handles that.
/var/lib/kubelet/kubeconfigbut the kubelet is in a crash loop and is restarted by host init system
/var/lib/kubelet/kubeconfigand waits for kubelet to restart