Vault
Vault on Red Hat Demo Platform (RHDP)
Red Hat's OpenShift is a distribution of the Kubernetes platform that provides a number of usability and security enhancements.
The Red Hat Demo Platform (RHDP) is a way for Red Hat employees and official Red Hat partners to deploy and demo Red Hat products for specific use cases alongside Red Hat partner technology. Visit the Red Hat website for demos and workshops.
In this tutorial, you will login to an OpenShift cluster and configure the authentication between Vault and the cluster. You will then deploy two web applications; One that authenticates and requests secrets directly from the Vault server. The other will utilize deployment annotations that enable it to remain Vault unaware.
Upon completion of this tutorial, you will gain a better understanding of how Vault is deployed on Red Hat OpenShift and how Vault integrates with applications running in OpenShift clusters.
Note
The RHDP system is only available for Red Hat/HashiCorp employees and official Red Hat partners. Please contact technologypartners@hashicorp.com for further information and access to the system.
Prerequisites
Please make sure your system and network can make external ssh connections.
Once the workshop admin spins up the cluster, you'd be provided with an access URL with username and password.
Upon login, users will receive credentials similar to what's shown below.
Example output:
HTPasswd Authentication is enabled on this cluster. Users user1 .. user30 are created with password openshift User opentlc-mgr with password <password_here> is cluster ad User opentlc-mgr with password <password_here> is clusterlearn-vault-kubernetes/openshift admin. You can access your bastion via SSH: ssh lab-user@bastion.r47nd.sandbox695.opentlc.com
Upon completion of the last step, you are connected to the bastion running RHEL. Confirm that OpenShift environment is running and the Vault namespace has been created and Vault pods are successfully running.
$ oc get ns vault NAME STATUS AGE vault Active 1h
Output:
$ kubectl get pods -n vault NAME READY STATUS RESTARTS AGE openshift-helm-charts-vault-0 1/1 Running 0 22m openshift-helm-charts-vault-agent-injector-777b86fbbd-cxrgb 1/1 Running 0 22m
Retrieve the web application and additional configuration by cloning the learn-vault-kubernetes repository from GitHub.
$ git clone https://github.com/hashicorp-education/learn-vault-kubernetes.git
Change the working directory to
learn-vault-kubernetes/openshift
.$ cd learn-vault-kubernetes/openshift
Note
This tutorial assumes that the remainder of commands are executed within this directory.
Configure the Vault Kubernetes authentication
Vault provides a Kubernetes authentication method that enables clients to authenticate with Vault within an OpenShift cluster. This authentication method configuration requires the location of the Kubernetes API, which is available in environment variables within the pod.
Set the current project to Vault
$ oc project vault Now using project "vault" on server "https://api.cluster-<id>.<id>.sandbox<id>.opentlc.com:6443".
Your system prompt is replaced with a new prompt
/ #
. Commands issued at this prompt are executed on theopenshift-helm-charts-vault-0
container.Start an interactive shell session on the
openshift-helm-charts-vault-0
pod.$ oc exec -it openshift-helm-charts-vault-0 -- /bin/sh / #
Your system prompt is replaced with a new prompt
/ #
. Commands issued at this prompt are executed on theopenshift-helm-charts-vault-0
container.Enable the Kubernetes authentication method.
$ vault auth enable kubernetes Success! Enabled kubernetes auth method at: kubernetes/
Configure the Kubernetes authentication method to use the location of the Kubernetes host. It will automatically use the pod's own identity to authenticate with Kubernetes when querying the token review API.
Note
For the best compatibility with recent Kubernetes versions, ensure you are using Vault v1.9.3 or greater.$ vault write auth/kubernetes/config \ kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
Output:
Success! Data written to: auth/kubernetes/config
The authentication method is now configured.
Exit the
openshift-helm-charts-vault-0
pod.$ exit
Deployment: Request secrets directly from Vault
Applications on pods can directly communicate with Vault to authenticate and request secrets. An application needs:
- a service account
- a Vault secret
- a Vault policy to read the secret
- a Kubernetes authentication role
Create the service account
Display the service account defined in
service-account-webapp.yml
.$ cat service-account-webapp.yml apiVersion: v1 kind: ServiceAccount metadata: name: webapp
This definition of the service account creates the account with the name
webapp
.Apply the service account.
$ oc apply --filename service-account-webapp.yml serviceaccount/webapp created
Get all the service accounts.
$ oc get serviceaccounts NAME SECRETS AGE builder 1 47m default 1 47m deployer 1 47m openshift-helm-charts-vault 1 31m openshift-helm-charts-vault-agent-injector 1 31m webapp 1 10s
The
webapp
service account is displayed.
Create the secret
Start an interactive shell session on the
openshift-helm-charts-vault-0
pod.$ oc exec -it openshift-helm-charts-vault-0 -- /bin/sh
Your system prompt is replaced with a new prompt
/ #
. Commands issued at this prompt are executed on theopenshift-helm-charts-vault-0
container.Create a secret at path
secret/webapp/config
with ausername
andpassword
.$ vault kv put secret/webapp/config username="static-user" \ password="static-password"
Output:
Key Value --- ----- created_time 2020-07-14T20:20:04.315677226Z deletion_time n/a destroyed false version 1
Read the secret at path
secret/webapp/config
.$ vault kv get secret/webapp/config ====== Metadata ====== Key Value --- ----- created_time 2020-07-14T20:20:04.315677226Z deletion_time n/a destroyed false version 1 ====== Data ====== Key Value --- ----- password static-password username static-user
The secret with the username and password is displayed.
Define the read policy
Write out the policy named
webapp
that enables theread
capability for secrets at pathsecret/data/webapp/config
.$ vault policy write webapp - <<EOF path "secret/data/webapp/config" { capabilities = ["read"] } EOF
Output:
Success! Uploaded policy: webapp
The policy
webapp
is used in the Kubernetes authentication role definition.
Create a Kubernetes authentication role
Create a Kubernetes authentication role, named
webapp
, that connects the Kubernetes service account name andwebapp
policy.$ vault write auth/kubernetes/role/webapp \ bound_service_account_names=webapp \ bound_service_account_namespaces=vault \ policies=webapp \ ttl=24h
Output:
Success! Data written to: auth/kubernetes/role/webapp
The role connects the Kubernetes service account,
webapp
, the namespace,vault
, with the Vault policy,webapp
. The tokens returned are valid for 24 hours.Exit the
openshift-helm-charts-vault-0
pod.$ exit
Deploy the application
Examine the webapp deployment defined in
deployment-webapp-rhdp.yml
.$ cat deployment-webapp-rhdp.yml
The deployment deploys a pod with a web application running under the
webapp
service account that talks directly to the Vault service created by the Vault Helm charthttp://openshift-helm-charts-vault:8200
.deployment-webapp-rhdp.yml
--- apiVersion: apps/v1 kind: Deployment metadata: name: webapp labels: app: webapp spec: replicas: 1 selector: matchLabels: app: webapp template: metadata: labels: app: webapp spec: serviceAccountName: webapp containers: - name: app image: hashieducation/simple-vault-client:latest imagePullPolicy: Always env: - name: VAULT_ADDR value: 'http://openshift-helm-charts-vault:8200' - name: JWT_PATH value: '/var/run/secrets/kubernetes.io/serviceaccount/token' - name: SERVICE_PORT value: '8080'
Apply the webapp deployment.
$ oc apply --filename deployment-webapp-rhdp.yml deployment.apps/webapp created
Display all the pods within the vault namespace.
$ oc get pods NAME READY STATUS RESTARTS AGE openshift-helm-charts-vault-0 1/1 Running 0 7m16s openshift-helm-charts-vault-agent-injector-597d5f5569-h4gbp 1/1 Running 0 7m16s webapp-55c464ff57-grjbp 1/1 Running 0 2m55s
Wait until the
webapp
pod is running and ready (1/1
).This web application runs an HTTP service that listens on port 8080.
Perform a
curl
request athttp://localhost:8080
on thewebapp
pod.$ oc exec \ $(oc get pod --selector='app=webapp' --output='jsonpath={.items[0].metadata.name}') \ --container app -- curl -s http://localhost:8080 ; echo
Output:
username:static-user password:static-password
The web application running on port 8080 in the webapp pod:
- authenticates with the Kubernetes service account token
- receives a Vault token with the read capability at the
secret/data/webapp/config
path - retrieves the secrets from
secret/data/webapp/config
path - displays the secrets as JSON
Deployment: Secrets through Annotations
Applications on pods can remain Vault unaware if they provide deployment annotations that the Vault Agent Injector detects. This injector service leverages the Kubernetes mutating admission webhook to intercept pods that define specific annotations and inject a Vault Agent container to manage these secrets. An application needs:
- a service account
- a Vault secret
- a Vault policy to read the secret
- a Kubernetes authentication role
- a deployment with Vault Agent Injector annotations
Create the service account
Display the service account defined in
service-account-issues.yml
.$ cat service-account-issues.yml apiVersion: v1 kind: ServiceAccount metadata: name: issues
This definition of the service account creates the account with the name
issues
.Apply the service account.
$ oc apply --filename service-account-issues.yml serviceaccount/issues created
Get all the service accounts.
$ oc get serviceaccounts NAME SECRETS AGE builder 1 47m default 1 47m deployer 1 47m openshift-helm-charts-vault 1 31m openshift-helm-charts-vault-agent-injector 1 31m webapp 1 10s
The
issues
service account is displayed.
Create the secret
Start an interactive shell session on the
openshift-helm-charts-vault-0
pod.$ oc exec -it openshift-helm-charts-vault-0 -- /bin/sh
Your system prompt is replaced with a new prompt
/ #
. Commands issued at this prompt are executed on theopenshift-helm-charts-vault-0
container.Create a secret at path
secret/issues/config
with ausername
andpassword
.$ vault kv put secret/issues/config username="annotation-user" \ password="annotation-password"
Output:
Key Value --- ----- created_time 2020-07-14T20:25:24.043709599Z deletion_time n/a destroyed false version 1
Get the secret at path
secret/issues/config
.$ vault kv get secret/issues/config ====== Metadata ====== Key Value --- ----- created_time 2020-07-14T20:25:24.043709599Z deletion_time n/a destroyed false version 1 ====== Data ====== Key Value --- ----- password annotation-password username annotation-user
The secret with the username and password is displayed.
Define the read policy
Write out the policy named
issues
that enables theread
capability for secrets at pathsecret/data/issues/config
.$ vault policy write issues - <<EOF path "secret/data/issues/config" { capabilities = ["read"] } EOF
Output:
Success! Uploaded policy: issues
The policy
issues
is used in the Kubernetes authentication role definition.
Create a Kubernetes authentication role
Create a Kubernetes authentication role, named
issues
, that connects the Kubernetes service account name andissues
policy.$ vault write auth/kubernetes/role/issues \ bound_service_account_names=issues \ bound_service_account_namespaces=vault \ policies=issues \ ttl=24h
Output:
Success! Data written to: auth/kubernetes/role/issue
The role connects the Kubernetes service account,
issues
, the namespace,vault
, with the Vault policy,issues
. The tokens returned are valid for 24 hours.Exit the
openshift-helm-charts-vault-0
pod.$ exit
Deploy the application
Examine the issues deployment defined in
deployment-issues.yml
.$ cat deployment-issues.yml
The Vault Agent Injector service reads the metadata annotations prefixed with
vault.hashicorp.com
.agent-inject
enables the Vault Agent injector servicerole
is the Vault Kubernetes authentication roleagent-inject-secret-FILEPATH
prefixes the path of the file,issues-config.txt
written to the/vault/secrets
directory. The value is the path to the Vault secret.agent-inject-template-FILEPATH
formats the secret with a provided template.
deployment-issues.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: issues labels: app: issues spec: selector: matchLabels: app: issues replicas: 1 template: metadata: annotations: vault.hashicorp.com/agent-inject: 'true' vault.hashicorp.com/role: 'issues' vault.hashicorp.com/agent-inject-secret-issues-config.txt: 'secret/data/issues/config' vault.hashicorp.com/agent-inject-template-issues-config.txt: | {{- with secret "secret/data/issues/config" -}} postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard {{- end -}} labels: app: issues spec: serviceAccountName: issues containers: - name: issues image: jweissig/app:0.0.1
Apply the issues deployment.
$ oc apply --filename deployment-issues.yml deployment.apps/issues created
Display all the pods within the vault namespace.
$ oc get pods NAME READY STATUS RESTARTS AGE issues-75d794c744-x9gnf 2/2 Running 0 17s openshift-helm-charts-vault-0 1/1 Running 0 9m53s openshift-helm-charts-vault-agent-injector-597d5f5569-h4gbp 1/1 Running 0 9m53s webapp-55c464ff57-grjbp 1/1 Running 0 5m32s Wait until the `issues` pod is running and ready (`2/2`). This new pod launches two containers. The application container, named `issues`, and the Vault Agent container, named `vault-agent`.
Display the logs of the
vault-agent
container in theissues
pod.$ oc logs \ $(oc get pod -l app=issues -o jsonpath="{.items[0].metadata.name}") \ --container vault-agent
Output:
==> Vault server started! Log data will stream in below: ==> Vault agent configuration: Cgo: disabled Log Level: info Version: Vault v1.4.2 [INFO] sink.file: creating file sink [INFO] sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r----- [INFO] auth.handler: starting auth handler [INFO] auth.handler: authenticating [INFO] template.server: starting template server [INFO] (runner) creating new runner (dry: false, once: false) [INFO] (runner) creating watcher [INFO] sink.server: starting sink server [INFO] auth.handler: authentication successful, sending token to sinks [INFO] auth.handler: starting renewal process [INFO] template.server: template server received new token [INFO] (runner) stopping [INFO] (runner) creating new runner (dry: false, once: false) [INFO] (runner) creating watcher [INFO] (runner) starting [INFO] sink.file: token written: path=/home/vault/.vault-token [INFO] auth.handler: renewed auth token
Display the secret written to the
issues
container.$ oc exec \ $(oc get pod -l app=issues -o jsonpath="{.items[0].metadata.name}") \ --container issues -- cat /vault/secrets/issues-config.txt ; echo
Output:
postgresql://annotation-user:annotation-password@postgres:5432/wizard
The secrets are rendered in a PostgreSQL connection string is present on the container.
Conclusion
Congratulations on finishing the tutorial!
In this lesson, you learned how to:
- Access the RHDP system and log into the OpenShift cluster
- Verify Vault is running on the OpenShift cluster
- Enable and configure Kubernetes auth method
- Deploy a web application that reads secrets directly from Vault
- Leverage Vault Agent Injector to manage secrets while keeping the application Vault unaware
Next steps
Learn more about the Vault Helm chart by reading the documentation or exploring the project source code.
Additional suggested reading:
- The blog post announcing the Injecting Vault Secrets into Kubernetes Pods via a Sidecar
- Documentation for Vault Agent Injector service.
- Repo for vault-hello-world