REVIEWED in SIG-multicluster meeting on 10⁄24
This doc is a Markdown conversion of the original Cluster registry design and plan Google doc. That doc is deprecated, and this one is canonical; however, the old doc will be preserved so as not to lose comment and revision history that it contains.
SIG-multicluster has identified a cluster registry as being a key enabling component for multi-cluster use cases. The goal of the SIG in this project is that the API defined for the cluster registry become a standard for multi-cluster tools. The API design is being discussed in a separate doc. A working prototype of the cluster registry has been assembled in a new repository.
This document intends to lay out an initial plan for moving the cluster registry from the prototype state through alpha, beta and eventually to a stable release.
These requirements are derived mainly from the output of the August 9th multi-cluster SIG meeting. However, they also derive (at least indirectly) from the results of the SIG F2F meeting that took place earlier, as well as the use cases presented by various parties at that meeting.
kubectlintegration with the cluster registry API server is first-class and on par with kubectl integration with core Kubernetes APIs
The cluster registry will be implemented using the Kubernetes API machinery. The cluster registry API server will be a fork and rework of the existing Federation API server, scaled down and simplified to match the simpler set of requirements for the cluster registry. It will use the apiserver library, plus some code copied from the core Kubernetes repo that provides scaffolding for certain features in the API server. This is currently implemented in a prototype form in the cluster-registry repo.
The API will be implemented using the Kubernetes API machinery, as a new API
with two objects,
ClusterList. Other APIs may be added in the
future to support future use cases, but the intention is that the cluster
registry API server remain minimal and only provide the APIs that users of a
cluster registry would want for the cluster registry.
The cluster registry will not be responsible for storing secrets. It will contain pointers to other secret stores which will need to be implemented independently. The cluster registry API will not provide proxy access to clusters, and will not interact with clusters on a user’s behalf. Storing secret information in the cluster registry will be heavily discouraged by its documentation, and will be considered a misuse of the registry. This allows us to sidestep the complexity of implementing a secure credential storage.
The expectation is that Federation and other programs will use the cluster-registry as an aggregated API server rather than via direct code integration. Therefore, the cluster registry will explicitly support being deployed as an API server that can be aggregated by other API servers.
The cluster registry could be implemented as a CRD that is registered with a Kubernetes API server. This implementation is lighter weight than running a full API server. If desired, the administrator could then disable the majority of the Kubernetes APIs for non-admin users and so make it appear as if the API server only supports cluster objects. It should be possible for a user to migrate without much effort from a CRD-based to an API-server-based implementation of the cluster registry, but the cluster-registry project is not currently planning to spend time supporting this use case. CRDs do not support (and may never support) versioning, which is very desirable for the cluster registry API. Users who wish to use a CRD implementation will have to design and maintain it themselves.
In the alpha stage, the cluster registry will repurpose the kubefed tool from
Federation and use it to initialize a cluster registry. In early stages, this
will only create a deployment with one replica, running the API server and etcd
Pod. As the HA requirements for the cluster registry are fleshed out,
this tool may need to be updated or replaced to support deploying a cluster
registry in multiple clusters and with multiple etcd replicas.
Since the cluster registry is a Kubernetes API server that serves a custom
resource type, it will be usable by
kubectl. We expect that the kubectl
experience for custom APIs will soon be on par with that of core Kubernetes
APIs, since there has been a significant investment in making
very detailed output from its describe subcommand; and
kubectl 1.9 is expected
to introduce API server-originated columns. Therefore, we will not initially
implement any special tooling for interacting with the registry, and will tell
users to use
kubectl or generated client libraries.
The cluster registry is a top-level Kubernetes repository, and thus it requires some process to ensure stability for dependent projects and accountability. Since the Federation project wants to use the cluster registry instead of its current cluster API, there is a requirement to establish process even though the project is young and does not yet have a lot of contributors.
The standard Kubernetes Prow bots have been enabled on the repo. There is currently some functionality around managing PR approval and reviewer assignment and such that does not yet live in Prow, but given the limited number of contributors at this point it seems reasonable to wait for sig-testing to implement this functionality in Prow rather than enabling the deprecated tools. In most cases where a process is necessary, we will defer to the spirit of the Kubernetes process (if not the letter) though we will modify it as necessary for the scope and scale of the cluster registry project. There is not yet a merge queue, and until there is a clear need for one we do not intend to add one.
The code in the repository will use bazel as its build system. This is in-line with what the Kubernetes project is attempting to move towards, and since the cluster registry has a similar but more limited set of needs than Kubernetes, we expect that bazel will support our needs adequately. The structure that bazel uses is meant to be compatible with go tooling, so if necessary, we can migrate away from bazel in the future without having to entirely revamp the repository structure.
There is not currently a good process in Kubernetes for keeping vendored dependencies up-to-date. The strategy we expect to take with the cluster registry is to update only when necessary, and to use the same dependency versions that Kubernetes uses unless there is some particular incompatibility.
For early versions, the cluster registry release will consist of a container image and a client tool. The container image will contain the cluster registry API server, and the tool will be used to bootstrap this image plus an etcd image into a running cluster. The container will be published to GCR, and the client tool releases will be stored in a GCS bucket, following a pattern used by other k/ projects.
For now, the release process will be managed mostly manually by repository
maintainers. We will create a release branch that will be used for releases, and
use GitHub releases along with some additional per-release documentation
CHANGELOGs and release notes will be collected manually
until the volume of work becomes too great. We do not intend to create multiple
release branches until the project is more stable. Releases will undergo a
more-detailed set of tests that will ensure compatibility with recent released
kubectl (for the registry) and Kubernetes (for the cluster
registry as an aggregated API server). Having a well-defined release process
will be a stable release requirement, and by that point we expect to have gained
some practical experience that will make it easier to codify the requirements
around doing releases. The cluster registry will use semantic versioning, but
its versions will not map to Kubernetes versions. Cluster registry releases will
not follow the Kubernetes release cycle, though Kubernetes releases may trigger
cluster registry releases if there are compatibility issues that need to be
Projects that want to vendor the cluster registry will be able to do so. We expect that these projects will vendor from the release branch if they want a stable version, or from a desired SHA if they are comfortable using a version that has not necessarily been fully vetted.
As the project matures, we expect the tool to evolve (or be replaced) in order to support deployment against an existing etcd instance (potentially provided by an etcd operator), and to provide a HA story for hosting a cluster registry. This is considered future work and will not be addressed directly in this document.
Cross-compilation support in bazel is still a work in progress, so the cluster registry will not be able to easily provide binary releases for every platform until this is supported by bazel. If it becomes necessary to provide cross-platform binaries before bazel cross-compilation is available, the repository is setup to support common go tooling, so we should be able to devise a process for doing so.
There are several participants in the cluster registry ecosystem whose versions will be conceptually able to float relative to each other:
We will need to define the version skew restraints between these components and ensure that our testing validates key skews that we care about.
The cluster registry is a simple component, and so should be able to be tested extensively without too much difficulty. The API machinery code is already tested by its owning teams, and since the cluster registry is a straightforward API server, it should not require much testing of its own. The bulk of the tests written will be integration tests, to ensure that it runs correctly in aggregated and independent modes in a Kubernetes cluster; to verify that various versions of kubectl can interact with it; and to verify that it can be upgraded safely. A full testing strategy is a requirement for a GA launch; we expect development of a test suite to be an ongoing effort in the early stages of development.
The command-line tool will require testing. It should be E2E tested against recent versions of Kubernetes, to ensure that a simple cluster registry can be created in a Kubernetes cluster. The multiplicity of configuration options it provides cannot conveniently be tested in E2E tests, and so will be validated in unit tests.
| Date | Details | |–|–| | 10/9/17 | Initial draft | | 10/16/17 | Minor edits based on comments | | 10/19/17 | Added section specifically about version skew; change P0/P1/P2 to alpha/beta/stable; added some milestone requirements | | 11/2/17 | Resolved most comments and added minor tweaks to text in order to do so |