Kubernetes, starting with the release 1.7, adds Alpha support ( via PRs 41939 and 46460) to encrypt secrets and resources in etcd3 via a configured Provider. This release supports three providers viz. aesgcm, aescbc, secretbox. These providers store the encryption key(s) locally in a server configuration file. The provider encrypts and decrypts secrets in-process. Building upon these, a KMS provider framework with an option to support different KMS providers like google cloud KMS is being added via PRs 48574 and 49350. The new KMS provider framework uses an envelope encryption scheme.
This proposal adopts the KMS provider framework and adds a new KMS provider that uses Hashicorp Vault with a transit backend, to encrypt and decrypt the DEK stored in encrypted form in etcd3 along with encrypted secrets.
Vault is widely used for Data encryption and securely storing secrets. Externalizing encryption/decryption of kubernetes secrets to vault provides various benefits
Note, that the Vault Provider in this proposal
This proposal assumes familiarity with Vault and the transit back-end.
As with existing providers, the Vault based provider will implement the
envelope.Service. Based on value of name in the KMS provider
EnvelopeTransformer module will use an instance of the
Vault provider for decryption and encryption of DEK before storing and after
reading from the storage.
The KEK will be stored and managed in Vault backend. The Vault based provider configured in KMS Transformer configuration will make REST requests to encrypt and decrypt DEKs over a secure channel (must enable TLS). KMS Transformer will store the DEKs in etcd in encrypted form along with encrypted secrets. As with existing providers, encrypted DEKs will be stored with metadata used to identify the provider and KEK to be used for decryption.
The provider will support following authentication back-ends
Deployers can choose an authentication mechanism best suited to their requirements. The provider will work with vault REST APIs and will not require Vault to be configured or deployed in any specific way other than requiring a Transit Backend.
Every encrypted secret will have the following metadata prefixed.
k8s:enc:kms:<api-version>:vault:len(<KEK-key-name>:<KEK-key-version>:<DEK encrypted with KEK>):<KEK-key-name>:<KEK-key-version>:<DEK encrypted with KEK>
<api-version>represents api version in the providers configuration file.
vaultrepresents the KMS service kind value. It is a fixed value for Vault based provider.
KEK-key-nameis determined from the vault service configuration in providers configuration file
KEK-key-versionis an internal identifier used by vault to identify specific key version used to encrypt and decrypt. Vault sends
kek-key-versionprefixed with encrypted data in the response to an encrypt request. The
kek-key-versionwill be stored as part of prefix and returned back to Vault during a decrypt request.
Of the above metadata,
k8s:enc:kms:<api-version>:vault:len(<KEK-key-name>:<KEK-key-version>:<DEK encrypted with KEK>)
<KEK-key-name>:<KEK-key-version>:<DEK encrypted with KEK>.
EnvelopeTransformer will write encrypted DEK along with encrypted secret in
Here’s the pseudocode for
vaultEnvelopeService.encrypt(), invoked on each
write of DEK.
KEY_NAME = <first key-name from vault provider config> PLAIN_DEK = <value of DEK> ENCRYPTED_DEK_WITH_KEY_VERSION = encrypt(base64(PLAIN_DEK), KEY_NAME) // output from vault will have an extra prefix "vault" (other than key version) which will be stripped. STORED_DEK = KEY_NAME:<ENCRYPTED_DEK_WITH_KEY_VERSION>
EnvelopeTransformer will read encrypted DEK along with encrypted secret from
Here’s the pseudocode
vaultEnvelopeService.decrypt() invoked on each read of
// parse the provider kind, key name and encrypted DEK prefixed with key version KEY_NAME = //key-name from the prefix ENCRYPTED_DEK_WITH_KEY_VERSION = //<key version>:<encrypted DEK> from the stored value // add "vault" prefix to ENCRYPTED_DEK_WITH_KEY_VERSION as required by vault decrypt API base64Encoded = decrypt(vault:ENCRYPTED_DEK_WITH_KEY_VERSION, KEY_NAME) PLAIN_DEK = base64.Decode(base64Encoded)
DEK = "the quick brown fox" provider kind = "vault" api version version = "v1" Key name = "kube-secret-enc-key" key version = v1 ciphertext returned from vault = vault:v1:aNOTZn0aUDMDbWAQL1E31tH/7zr7oslRjkSpRW0+BPdMfSJntyXZNCAwIbkTtn0= prefixed DEK used to tag secrets = vault:kube-secret-enc-key:v1:aNOTZn0aUDMDbWAQL1E31tH/7zr7oslRjkSpRW0+BPdMfSJntyXZNCAwIbkTtn0=
No new configuration file or startup parameter will be introduced.
The vault provider will be specified in the existing configuration file used to
configure any of the encryption providers. The location of this configuration
file is identified by the existing startup parameter:
Vault provider configuration will be identified by value “vault” for the
name attribute in
The actual configuration of the vault provider will be in a separate
configuration identified by the
configfile attribute in the KMS provider.
Here is a sample configuration file with the vault provider configured:
kind: EncryptionConfig apiVersion: v1 resources: - resources: - secrets providers: - kms: name: vault cachesize: 10 configfile: /home/myvault/vault-config.yaml
The Vault based Provider needs the following configuration elements, at a minimum:
addrVault service base endpoint eg. https://example.com:8200
key-nameslist of names of the keys in Vault to be used. eg: key-name: kube-secret-enc-key.
Note : key name does not need to be changed if the key is rotated in Vault, the rotated key is identified by key version which is prefix to ciphertext.
A new key can be added in the list. Encryption will be done using the first key in the list. Decryption can happen using any of the keys in the list based on the prefix to the encrypted DEK stored in etcd
For the Kubernetes cluster to authenticate the vault server, TLS must be enabled :
ca-cert location of x509 certificate to authenticate the vault server eg:
For client authentication, one of following must be used: (provider will reject the configuration if parameters for more than one authentication backends are specified )
client-cert: location of x509 certificate to authenticate kubernetes API server to vault server eg.
client-key: location of x509 private key to authenticate kubernetes API server to vault server eg.
Here’s a sample
vault-config.yaml configuration with
key-names: - kube-secret-enc-key addr: https://example.com:8200 ca-cert:/var/run/kubernetes/ssl/vault.crt client-cert:/var/run/kubernetes/ssl/vault-client-cert.pem client-key:/var/run/kubernetes/ssl/vault-client-key.pem
token: limited access vault token required by kubernetes API server to authenticate itself while making requests to vault eg: 8dad1053-4a4e-f359-2eab-d57968eb277f
Here’s a sample
vault-config.yaml configuration using a Vault Token for authentication.
the Kubernetes cluster as a client to Vault:
key-names: -kube-secret-enc-key addr: https://example.com:8200 ca-cert:/var/run/kubernetes/ssl/vault.crt token: 8dad1053-4a4e-f359-2eab-d57968eb277f
role-id: RoleID of the AppRole
secret-id: secret Id only if associated with the appRole.
Here’s a sample configuration file using a Vault AppRole for authentication.
key-names: - kube-secret-enc-key addr: https://localhost:8200 ca-cert: /var/run/kubernetes/ssl/vault.crt role-id: db02de05-fa39-4855-059b-67221c5c2f63
The KEK is generated in Vault and rotated using direct API call or CLI to Vault itself. The Key never leaves the vault.
Note that when a key is rotated, Vault does not allow choosing a different encryption algorithm or key size. If a key for different encryption algorithm or a different key size is desired, new key needs to be generated in Vault and the corresponding key name be added in the configuration. Subsequent encryption will be done using the first key in the list. Decryption can happen using any of the keys in the list based on the prefix to the encrypted DEK.