Code Organization and Release Process Improvement

**Identify the following before beginning your session. Do note move forward until these are decided / assigned: **

  • Session Topic: Code Organization and Release Process Improvement
  • Topic Facilitator(s): bgrant0607@
  • Note-taker(s) (Collaborating on this doc): jdumars@
  • Person responsible for converting to Markdown & uploading to Github: michellen@

Session Notes

Background from November dev summit:

Github orgs:

Github supports 2 levels of hierarchy, orgs and repos, and we need to use both effectively. We need to move away from monorepos, such as kubernetes/kubernetes and kubernetes/contrib, and mono-orgs, such as kubernetes and kubernetes-incubator. Kubernetes-client is the first focused org.

Ideally, code would be divided along lines of ownership, by SIG and subproject, with infrastructure for packaging the code (APIs and controllers) into the desired number of components (e.g., hyperkube as well as factored by layer) and releases (e.g., monthly as well as LTS).

Where we are on multiple repos:

  • Code organization issue
  • kubernetes multi-repo issue and contrib breakup issue
  • User docs were moved to kubernetes/kubernetes.github.io
  • Contributor docs were moved to kubernetes/community
  • Examples have been copied to kubernetes/examples, but haven’t yet been removed from the kubernetes repo
  • API machinery: In order to build virtually anything outside the kubernetes repo, the API machinery needs to be made available, for component configuration, for building APIs, for building controllers and other Go clients of the APIs, etc.
  • TODO: util
    • pkg/util moves thread
  • Kubectl:
    • In the process of breaking kubectl <-> kubernetes dependencies
    • Have kubernetes/kubectl repo, need to migrate issues there
    • Issue migration tool exists - creates lots of notifications

Next up: * kubeadm * Federation * cloudprovider and cluster * Scheduler * Kubelet

Build system and checking in generated code (or not):


What would be better than godeps? Dep? Glide? https://github.com/kubernetes/kubernetes/issues/44873


Multi-repo requirements

Release process

  • Components are combined from subrepos to build released bits
  • Components each have sufficient testing against their repo to determine if they are stable to release

Development process in multiple repos

  • PRs sent to external repos
  • PR and CI tests run against external repo. Sufficient to validate the code maybe released as a component.
    • Unit + Integration tests
    • e2e tests?
  • Automatically update vendored deps from consuming repos
  • e2e tests run against main repo against integrated components

Process to move code into its own repo


  • Does the code depend on kubernetes/kubernetes code?
    • Break these dependencies or move the deps to another repo
  • Does other code from kubernetes/kubernetes depend on the code?
    • Must update package names
  • Can code be fully tested independent of kubernetes/kubernetes code?
  • Will the code share the same release as kubernetes/kubernetes or be released independently?
    • Clients released independently need a way to perform version skew testing

Possible workflow

  • Maybe first move to “staging” directory to simulate multi-repo within single repo - this has its own costs
  • Remove cyclic dependencies between repositories
  • Update mungebot to close PRs that update moved packages
  • Copy code to new location
  • Vendor code from new location to main repo (if needed)
  • Change package names to the vendored code
  • Delete old code

Where do we need to get to in order to separate: [ diagram ] * Utility packages need to go somewhere * API packages need to move * Helper libraries (reusable server infrastructure) * Code generators

Consumer-visible * client go * kubectl

Need to get the API types into their own repo * currently have a staging area, and a copy/paste script Need to develop in those repos and then assemble a coherent release from them

re: kubectl moving, * 1. sits in the repo but build rules break bad dependencies, can start shipping it independently * 2. make sure everything still works

kubectl could get its own release

build files to add visibility

kubectl should never be imported

As these refactors happen, Brian Grant will be more lenient with privileges, establishing janitors

kubectl split is technical debt cleanup - testing should not rely on e2e, no compiled-in dependencies

breaking the dependencies yields more benefits

Q: Will kubectl be buildable with go get?
A: Probably, but definitely with bazel

Integration testing plan? Use last stable release in k/k e2e tests

We need to catch regressions when it’s being released, not when it’s vendored back in

Need better, smaller tests overall

kops has a blocking PR test - kops master + what is in your PR

cross-component problems at the main level

Need a central build pipe in order to untangle dependencies

Experience breaking apart has been negative as a contributor

Q: What is the contribx? Testing? A: Short term will be many separate PRs, need to solve dependencies in test management

Please do not add new binaries in the kubernetes repo

Need help * getting rid of generated code * build system * solve vendoring problem

Working group? Code health WG

Client tool SDK?

kubectl -> SIG CLI code -> API Machinery

Caution against validating undocumented functionality via CI, better: documentation and integration test as a validation ~ API testing

Too much testing is currently e2e

Don’t make a catchall util package


Key Takeaways / Analysis of Situation

Recommendations & Decisions Moving Forward (High-Level)

Should build and release be the same SIG? Where does build live? – SIG release / SIG testing

Q: Is it worthwhile to go through the project and specify what needs to be moved out? A: In the SIG assignment process there may be breakup

Action Item Owner(s)
Need a SIG owner on utilities Tim Hockin
Talk to Cloud Foundry and see how they do it Caleb at Community
Working group organization or a SIG that oversees this Machinery, CLI, Cluster Lifecycle
Regular community checkins Brian Grant
Describe the end state and then assign breakout work to SIGs Working group needs to be organized - Brian Grant


  1. Make a copy of the “Template” area of this document in the folder.
  2. Set the “Share” settings to be “anyone with the link can edit” and take your session notes there. Ensure to decide/assign (and document) the topic title, facilitator(s), note-taker(s), and Github uploader.
  3. Later, convert the notes to Markdown and upload to this GitHub directory:
  4. Save your session notes with the following format: a. HHMM-HHMM_SESSIONTITLE.md i. where HHMM is the time in 24-hour format in the PST zone.


  • Use the document outline (Tools > Document Outline) and headings (Format > Paragraph Styles) to organize your notes as you go

Link to original doc