Resetting the Kubeadmin Password

When provisioning a new OpenShift Cluster there are 2 default cluster administrator accounts that are configured to allow for easy access to bootstrap other IdentityProviders and User accounts.

  • The kubeadmin credential consists of a user / password combination that is provided using the OpenShift Authentication module and can be accessed through both the WebConsole and CLI tool
  • The system:admin credential consists of a signed X509 certificate that can be used to communicate with the Kubernetes API and bypasses the OpenShift Authentication module. This account can only be used with the CLI tools and is not accessible via the WebConsole.

Both the kubeadmin and system:admin users are provided upon successful installation of the cluster and are associated with the cluster-admin ClusterRole, providing a high-tier of RBAC authorization.

The kubeadmin account is provided through the use of a hard-coded username provided in the Oauth-Apiserver Codebase with the associated password randomly-generated on install and stored in a Kubernetes Secret object in the kube-system Namespace.

As this account is granted authentication using the OAuth-APIServer, the kubeadmin account cannot be used for recovery of a cluster when the OpenShift Authentication ClusterOperator is not in a Ready state or a cluster is having other authentication issues.

The system:admin account can be used to communicate directly with the Kubernetes APIServer and bypasses the addition OpenShift authetication plugins/webhooks as the authentication is performed in-tree for the Kubernetes API using the certificate authority of the x509 signed-certificate.

It is a recommended practice to remove the kubeadmin account after an alternative IdentityProvider (IdP) is configured and a permanent User is associated with the cluster-admin ClusterRole.

If this has happened, or the kubeadmin password has been lost, is there a way to recreate/replace the password with a new known password?

Inspecting the old KubeAdmin password

Reviewing the kubeadmin definition on a newly provisioned cluster we can see the following:

auth|⇒ oc get secrets -n kube-system kubeadmin -o yaml 

apiVersion: v1
data:
  kubeadmin: JDJhJDEwJE85dkxPREtjYWQ4TlJiTEV6eHhKQk9OODBWaU1qbkJoeU5jQ1paOWJOa2gwS3duNzRtWkJh
...

auth|⇒ oc get secrets -n kube-system kubeadmin -o jsonpath="{.data.kubeadmin}" | base64 -d
$2a$10$O9vLODKcad8NRbLEzxxJBON80ViMjnBhyNcCZZ9bNkh0Kwn74mZBa

The provided output $2a$10$O9vLODKcad8NRbLEzxxJBON80ViMjnBhyNcCZZ9bNkh0Kwn74mZBa looks like the standard encrypted password similar to those found in the HTPasswd file-type.

Following the filetype the value can be split into sub-components representing the hashing algorithm and other relevant information as follows:

  • $2a - references the bcrypt hashing algorithm
  • $10 - references the cost of the algorithm
  • O9vLODKcad8NRbLEzxxJBO - the 22 character salt value
  • N80ViMjnBhyNcCZZ9bNkh0Kwn74mZBa - the 31 character output hash

Using htpasswd CLI tool with the extracted values, we can create a password hash string to replace this as following:

auth|⇒ htpasswd -bnBC 10 "" bxYJn-jvZBy-viWJi-HGxZA 
:$2y$10$y9MHpYM8LWGvN.K8n7mSbuOKTeid39XRn2orARXJDFa9DzEB4o/36

The htpasswd needs to be converted into base64 encoding and the first : character removed to create a Kubeadmin-compatible hash string as follows:

auth|⇒ htpasswd -bnBC 10 "" bxYJn-jvZBy-viWJi-HGxZA | cut -c 2- | base64
JDJ5JDEwJGdFbjRyTGI0ZTlmcXFGOTNLZlRubnVpR01CSWgxdDVjMzY0UXVqSTZzLzVRTmJFSDFxTHBLCgo=

We can now use the output from the htpasswd tool to set the secret value in OpenShift:

auth|⇒ oc patch -n kube-system secret/kubeadmin --patch '{"data": {"kubeadmin": "JDJ5JDEwJGdFbjRyTGI0ZTlmcXFGOTNLZlRubnVpR01CSWgxdDVjMzY0UXVqSTZzLzVRTmJFSDFxTHBLCgo="}}'
auth|⇒ oc login -u kubeadmin -p bxYJn-jvZBy-viWJi-HGxZA 
Login successful.
...

Password Requirements

An additional note about the password length must be addressed. The kubeadmin password has a requirement to be at least 23 characters long. The expected format is 5char-5char-5char-5char, however this is not enforced.

If the Secret has been removed?

If the Secret has been deleted from the OpenShift cluster, the following template can be used to re-create the Secret and regain access to the Kubeadmin account.

Replace the data.kubeadmin value if required, store this into a YAML file and apply the definitions as follows:

auth|⇒ cat kubeadmin.yaml
apiVersion: v1
data:
  kubeadmin: JDJ5JDEwJGdFbjRyTGI0ZTlmcXFGOTNLZlRubnVpR01CSWgxdDVjMzY0UXVqSTZzLzVRTmJFSDFxTHBLCgo=
kind: Secret
metadata:
  name: kubeadmin
  namespace: kube-system
type: Opaque

auth|⇒ oc apply -f ./kubeadmin.yaml
secret/kubeadmin configured

Troubleshooting techniques

If you run into issues deploying the changes to the cluster and are still unable to access the cluster as the kubeadmin User, the following examples can be used to help troubleshoot.

The relevant error logs for the authentication pipeline are available within Pods in openshift-authentication Namespace.

Some common examples of failures and resolutions are as below:

  1. Error log:
auth|⇒ oc logs -n openshift-authentication oauth-openshift-XXX | less
...
E0117 08:56:28.917755       1 basicauth.go:47] Error authenticating login "kubeadmin" with provider "kube:admin": kubeadmin password must be at least 23 characters long
E0117 08:56:28.917786       1 errorpage.go:26] AuthenticationError: kubeadmin password must be at least 23 characters long
...

Cause: The password does not meet the character requirement of 23 characters.
Resolution: Start from Step 1 with a longer password

  1. Error log:
auth|⇒ oc logs -n openshift-authentication oauth-openshift-XXX | less
...
E0117 06:13:20.531974       1 basicauth.go:47] Error authenticating login "kubeadmin" with provider "kube:admin": crypto/bcrypt: bcrypt hashes must start with '$', but hashedSecret started with ':'
E0117 06:13:20.532019       1 errorpage.go:26] AuthenticationError: crypto/bcrypt: bcrypt hashes must start with '$', but hashedSecret started with ':'
...

Cause: The : character is still present as the first character of the kubeadmin Secret.
Resolution: Ensure that the cut -c 2- command was not removed from the HASH generation step before storing the base64-encoded hash string into Kubernetes.