I recently was working with the open source project NICo using the setup script that provides automation for end-to-end installation of the solution in a Kubernetes cluster. One of those components is HashiCorp Vault. In the very first run of the setup script in my virtualized environment the automation error during the Vault installation. Rather than simply re-run the automation and move on, I wanted to understand what had actually happened and confirm the environment was sound by stepping through the Vault initialization and unseal process manually. This post documents that troubleshooting walk-through.
Before getting into the steps, a brief word on what Vault is and why the init and unseal process matters. HashiCorp Vault is a secrets management platform that provides encrypted storage and fine-grained access control for sensitive data like credentials, certificates, and API keys. When Vault is deployed, it starts in a sealed state which means it cannot decrypt its storage backend or serve any requests until it has been initialized and unsealed. Initialization generates the encryption keys that protect Vault's data, and unsealing provides enough of those key shares to reconstruct the master key. Until that process completes, the pods will run but the vault container itself will not reach a ready state.
The Automation Failure
In my virtualized environment the automation output showed that the Helm deployment of vault-0.25.0 (Vault 1.14.0) completed successfully, but the subsequent unseal script failed immediately after confirming all three pods were Running.
Release "vault" does not exist. Installing it now.
NAME: vault
LAST DEPLOYED: Thu Jul 2 15:04:29 2026
NAMESPACE: vault
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!
Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:
https://www.vaultproject.io/docs/
Your release is named vault. To learn more about the release, try:
$ helm status vault
$ helm get manifest vault
Listing releases matching ^vault$
vault vault 1 2026-07-02 15:04:29.03472951 +0000 UTC deployed vault-0.25.0 1.14.0
========== Updated Releases ==========
NAME NAMESPACE CHART VERSION DURATION
vault vault hashicorp/vault 0.25.0 3s
=== [4/6] unseal vault ===
Waiting for all 3 Vault pods to be Running...
pod/vault-0 condition met
pod/vault-1 condition met
pod/vault-2 condition met
All Vault pods are Running
Checking Vault status on vault-0...
ERROR: Unable to retrieve Vault status from vault-0.
Make sure the Vault pods are running and try again.
=========================================================================
SETUP FAILED
Phase : [4/6] vault init + unseal
Command : ./unseal_vault.sh
Code : 1
=========================================================================
The pods were Running but the script could not retrieve a Vault status. The first thing to check was whether the pods were actually healthy.
Checking Pod Readiness
Checking pod status reveals the issue immediately in that all three pods show 1/2 rather than 2/2, meaning only one of the two containers in each pod is ready.
$ kubectl get pods -n vault
NAME READY STATUS RESTARTS AGE
vault-0 1/2 Running 0 11m
vault-1 1/2 Running 0 11m
vault-2 1/2 Running 0 11m
Each Vault pod runs two containers: the vault container itself and a vault-cert-reload sidecar. The cert-reload sidecar depends on Vault being operational, so when the vault container is sealed and unresponsive, the sidecar never reaches a ready state. The 1/2 readiness is actually the expected symptom of a sealed, uninitialized Vault.
Checking Vault Status Directly
We can verify this by exec-ing into vault-0 and running vault status directly with TLS verification skipped.
$ kubectl exec -it -n vault vault-0 -- /bin/sh -c "vault status -tls-skip-verify"
Defaulted container "vault" out of: vault, vault-cert-reload
Key Value
--- -----
Seal Type shamir
Initialized false
Sealed true
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
HA Enabled true
command terminated with exit code 2
This confirms it: Initialized: false. The automation script checked for Vault status before the cluster had ever been initialized, which is why it could not retrieve a meaningful response. The pods were Running, just not yet in a state where Vault could respond to status queries the way the script expected.
Reading the Logs
Looking through the Vault logs at the time of failure shows what the three pods were doing while the automation was waiting.
2026-07-02T16:25:47.971Z [ERROR] core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"
2026-07-02T16:25:48.878Z [INFO] core: security barrier not initialized
2026-07-02T16:25:49.091Z [INFO] core: security barrier not initialized
2026-07-02T16:25:49.971Z [INFO] core: security barrier not initialized
2026-07-02T16:25:49.977Z [INFO] core: attempting to join possible raft leader node: leader_addr=https://vault-0.vault-internal:8200
2026-07-02T16:25:49.977Z [INFO] core: attempting to join possible raft leader node: leader_addr=https://vault-1.vault-internal:8200
2026-07-02T16:25:49.977Z [INFO] core: attempting to join possible raft leader node: leader_addr=https://vault-2.vault-internal:8200
2026-07-02T16:25:49.983Z [ERROR] core: failed to get raft challenge: leader_addr=https://vault-0.vault-internal:8200
error=
| error during raft bootstrap init call: Error making API request.
|
| URL: PUT https://vault-0.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge
| Code: 503. Errors:
|
| * Vault is sealed
2026-07-02T16:25:49.984Z [ERROR] core: failed to get raft challenge: leader_addr=https://vault-1.vault-internal:8200
error=
| error during raft bootstrap init call: Error making API request.
|
| URL: PUT https://vault-1.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge
| Code: 503. Errors:
|
| * Vault is sealed
2026-07-02T16:25:49.987Z [ERROR] core: failed to get raft challenge: leader_addr=https://vault-2.vault-internal:8200
error=
| error during raft bootstrap init call: Error making API request.
|
| URL: PUT https://vault-2.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge
| Code: 503. Errors:
|
| * Vault is sealed
All three pods were stuck in a loop trying to join the Raft cluster and failing because every node they tried to reach returned a 503 because Vault is sealed. This is the classic bootstrap chicken-and-egg: no node can join the Raft cluster until at least one node has been initialized and unsealed, but all three are waiting on each other. The solution is to manually initialize vault-0 first, which establishes the Raft leader, and then unseal each node in sequence.
Initializing Vault
Vault initialization is a one-time operation that generates the encryption keys and produces the unseal key shares. We initialize with five total key shares and a threshold of three, meaning any three of the five shares are sufficient to unseal. The output is saved to a local JSON file for key extraction.
bschmaus@asus2-vm1:~$ kubectl exec -n vault vault-0 -- vault operator init -tls-skip-verify -key-shares=5 -key-threshold=3 -format=json > ~/vault-init.json
Defaulted container "vault" out of: vault, vault-cert-reload
Treat the vault-init.json file as highly sensitive. In a production environment these key shares and the root token should be distributed securely across separate custodians and never stored in plain text on a shared system.
Checking Vault status on vault-0 again confirms initialization succeeded.
$ kubectl exec -it -n vault vault-0 -- /bin/sh -c "vault status -tls-skip-verify"
Defaulted container "vault" out of: vault, vault-cert-reload
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 0/0
Unseal Nonce n/a
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
HA Enabled true
command terminated with exit code 2
Initialized: true confirms we are on the right track. Vault is initialized but still sealed. Now we need to extract three of the five key shares from the init output and apply them.
Extracting Unseal Keys
The vault-init.json file holds all five unseal keys and the root token. We use jq to pull three key shares and the root token out of the file. Note these keys shared here are okay because this environment no longer exists.
$ jq -r '.unseal_keys_b64[0]' ~/vault-init.json
VoPGJXy60WL1xdJDle2t1TG/fDKmszTJ5dRF0uQlDsjf
$ jq -r '.unseal_keys_b64[1]' ~/vault-init.json
DY5YVcAB4Kxr4leDnQzO5btj3BlyWSweR1iaiXIDxKV9
$ jq -r '.unseal_keys_b64[2]' ~/vault-init.json
9jorS66mfteDlUE48yIWVd4HwMqHOZ2Gy03tZzg9VWbL
$ jq -r '.root_token' ~/vault-init.json
hvs.Vin9sc6jjm4GRvwATjODvnsY
With the three key shares in hand, let's move on to unsealing each pod.
Unsealing vault-0
The unseal process is interactive and each call to vault operator unseal prompts for one key share. We need to provide three shares to reach the threshold and bring vault-0 out of the sealed state. The status output after each call shows the running Unseal Progress counter.
$ kubectl exec -it -n vault vault-0 -- vault operator unseal -tls-skip-verify
Defaulted container "vault" out of: vault, vault-cert-reload
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce ab666c33-1bac-c845-c178-9b93494fdc96
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
HA Enabled true
Now the second unseal call.
$ kubectl exec -it -n vault vault-0 -- vault operator unseal -tls-skip-verify
Defaulted container "vault" out of: vault, vault-cert-reload
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce ab666c33-1bac-c845-c178-9b93494fdc96
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
HA Enabled true
Then the third unseal call.
$ kubectl exec -it -n vault vault-0 -- vault operator unseal -tls-skip-verify
Defaulted container "vault" out of: vault, vault-cert-reload
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
Cluster Name vault-cluster-690f97b0
Cluster ID e97c222b-f44c-d31a-8476-54587966d724
HA Enabled true
HA Cluster https://vault-0.vault-internal:8201
HA Mode active
Active Since 2026-07-02T16:30:11.957556681Z
Raft Committed Index 36
Raft Applied Index 36
After the third key share vault-0 comes fully unsealed: Sealed: false, HA Mode: active. vault-0 has elected itself as the Raft leader and the cluster is now bootstrapped. With that in place we can move onto unsealing the standby nodes.
Unsealing vault-1 and vault-2
Unsealing vault-1 and vault-2 follows the same three-key pattern, but with one notable behavior worth calling out. After applying the first two keys to vault-1, the unseal progress resets to 0/3 with a new nonce (number used once) before finally unsealing on the next key entry. This happens because vault-1 was in the middle of its unseal sequence when vault-0 became the active Raft leader and vault-1 joined the cluster as a follower. The Raft join disrupts the in-progress unseal, invalidating the nonce, and vault-1 has to start its unseal counter over. The same thing happens with vault-2. It is a bit surprising the first time it happens, but it is expected behavior in an HA Raft setup.
$ kubectl exec -it -n vault vault-1 -- vault operator unseal -tls-skip-verify
Defaulted container "vault" out of: vault, vault-cert-reload
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce 4906d625-32f3-8c48-7dff-231d4233d33e
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
HA Enabled true
Second unseal call for vault-1.
$ kubectl exec -it -n vault vault-1 -- vault operator unseal -tls-skip-verify
Defaulted container "vault" out of: vault, vault-cert-reload
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce 4906d625-32f3-8c48-7dff-231d4233d33e
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
HA Enabled true
Third unseal call for vault-1.
$ kubectl exec -it -n vault vault-1 -- vault operator unseal -tls-skip-verify
Defaulted container "vault" out of: vault, vault-cert-reload
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 0/3
Unseal Nonce n/a
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
HA Enabled true
Fourth and final unseal call for vault-1.
$ kubectl exec -it -n vault vault-1 -- vault operator unseal -tls-skip-verify
Defaulted container "vault" out of: vault, vault-cert-reload
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
Cluster Name vault-cluster-690f97b0
Cluster ID e97c222b-f44c-d31a-8476-54587966d724
HA Enabled true
HA Cluster https://vault-0.vault-internal:8201
HA Mode standby
Active Node Address https://10.233.117.89:8200
Raft Committed Index 38
Raft Applied Index 38
vault-1 comes up in HA Mode: standby, correctly pointing at vault-0 as the active node. Now for vault-2.
$ kubectl exec -it -n vault vault-2 -- vault operator unseal -tls-skip-verify
Defaulted container "vault" out of: vault, vault-cert-reload
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce 913558eb-4ff3-63d3-169a-e96a73bf3675
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
HA Enabled true
Second unseal call for vault-2.
$ kubectl exec -it -n vault vault-2 -- vault operator unseal -tls-skip-verify
Defaulted container "vault" out of: vault, vault-cert-reload
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce 913558eb-4ff3-63d3-169a-e96a73bf3675
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
HA Enabled true
Third unseal call for vault-2.
$ kubectl exec -it -n vault vault-2 -- vault operator unseal -tls-skip-verify
Defaulted container "vault" out of: vault, vault-cert-reload
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 0/3
Unseal Nonce n/a
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
HA Enabled true
Fourth and final unseal call for vault-2.
$ kubectl exec -it -n vault vault-2 -- vault operator unseal -tls-skip-verify
Defaulted container "vault" out of: vault, vault-cert-reload
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
Cluster Name vault-cluster-690f97b0
Cluster ID e97c222b-f44c-d31a-8476-54587966d724
HA Enabled true
HA Cluster https://vault-0.vault-internal:8201
HA Mode standby
Active Node Address https://10.233.117.89:8200
Raft Committed Index 39
Raft Applied Index 39
vault-2 comes up in standby as well. All three nodes are now part of the same Raft cluster — vault-0 as active, vault-1 and vault-2 as standbys.
Verifying the Cluster
With all three nodes unsealed, we can do a final check on pod readiness and Vault status.
$ kubectl get pods -n vault
NAME READY STATUS RESTARTS AGE
vault-0 2/2 Running 0 88m
vault-1 2/2 Running 0 88m
vault-2 2/2 Running 0 88m
All three pods are now showing 2/2. The vault-cert-reload sidecar in each pod has come up now that the vault container is operational. Checking the full status on vault-0 confirms everything is healthy.
$ kubectl exec -it -n vault vault-0 -- /bin/sh -c "vault status -tls-skip-verify"
Defaulted container "vault" out of: vault, vault-cert-reload
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
Cluster Name vault-cluster-690f97b0
Cluster ID e97c222b-f44c-d31a-8476-54587966d724
HA Enabled true
HA Cluster https://vault-0.vault-internal:8201
HA Mode active
Active Since 2026-07-02T16:30:11.957556681Z
Raft Committed Index 40
Raft Applied Index 40
At this point I feel pretty confident the environment is healthy. A final pod check a little while later confirms everything stayed stable.
$ kubectl get pods -n vault
NAME READY STATUS RESTARTS AGE
vault-0 2/2 Running 0 137m
vault-1 2/2 Running 0 137m
vault-2 2/2 Running 0 137m
No restarts, all 2/2, steady at 137 minutes. The automation failure turned out to be a timing issue in that the unseal script was checking Vault status before initialization had been triggered, which is a sequencing problem rather than an environment problem. The environment itself was perfectly sound. With the manual steps confirmed I now have a clear picture of what the automation needs to do and in what order, which makes fixing the script straightforward.
Hopefully this walk-through provides a useful reference for anyone who runs into a similar Vault init and unseal failure on Kubernetes and wants to understand what is actually happening before reaching for a re-run.
