Improve Vault traffic resiliency with Vault Proxy

Use static secret caching with Vault Proxy to cache KVv1 and KVv2 secrets to minimize requests made to Vault and provide resilient connections for clients.

Vault Proxy utilizes the Enterprise only Vault event notification system feature for cache freshness. As a result, static secret caching can only be used with Vault Enterprise installations.

When using a Vault cluster with performance standbys, Proxy may receive secret update events before the secret update has been fully replicated. To make sure that Proxy can get updated secret values after receiving an event notification, Proxy must be configured to point to the address of the active node in its Vault stanza, or allow_forwarding_via_header must be set to true on the cluster. When allow_forwarding_via_header is configured, Proxy will only forward requests to update a secret in its cache after receiving an event indicating that secret got updated. This approach would be recommended if access to Vault was behind, for example, a load balancer.

Step 1: Subscribe Vault Proxy to KV events

Vault Proxy uses Vault events and auto-auth to monitor secret status and make appropriate cache updates.

1. Enable auto-auth.

2. Create an auto-auth token with permission to subscribe to KV event updates with the Vault event notification system. For example, to create a policy that grants access to static secret (KVv1 and KVv2) events, you need permission to subscribe to the events endpoint, as well as the list and subscribe permissions on KV secrets you want to get secrets from:

    path "sys/events/subscribe/kv*" {
      capabilities = ["read"]
    }
   
    path "*" {
      capabilities = ["list", "subscribe"]
      subscribe_event_types = ["kv*"]
    }
   

Subscribing to KV events means that Proxy receives updates as soon as a secret changes, which reduces staleness in the cache. Vault Proxy only checks for a secret update if an event notification indicates that the related secret was updated.

Step 2: Ensure tokens have capabilities-self access

Tokens require update access to the sys/capabilies-self endpoint to request cached secrets. Vault tokens receive update permissions by default. If you have modified or removed the default policy, you must explicitly create a policy with the appropriate permissions. For example:

  path "sys/capabilities-self" {
    capabilities = ["update"]
  }

Step 3: Configure an appropriate refresh interval

By default, Vault Proxy refreshes tokens every five minutes. You can change the default behavior and configure Proxy to verify and update cached token capabilities with the static_secret_token_capability_refresh_interval parameter in the cache configuration stanza. For example, to set a refresh interval of one minute:

cache {
  cache_static_secrets = true
  static_secret_token_capability_refresh_interval = "1m"
}

Functionality

With static secret caching, Vault Proxy caches GET requests for KVv1 and KVv2 endpoints.

When a client sends a GET request for a new KV secret, Proxy forwards the request to Vault but caches the response before forwarding it to the client. If that client makes subsequent GET requests for the same secret, Vault Proxy serves the cached response rather than forwarding the request to Vault.

Similarly, when a token requests access to a KV secret, it must complete a success GET request. If the request is successful, Proxy caches the fact that the token was successful in addition to the result. Subsequent requests by the same token can then access this secret from the cache instead of Vault.

Vault Proxy uses the event notification system to keep the cache up to date. It monitors the KV event feed for events related to any secret currently stored in the cache, including modification events like updates and deletes. When Proxy detects a change in a cached secret, it will update or evict the cache entry as appropriate.

Vault Proxy also checks and refreshes the access permissions of known tokens according to the window set with static_secret_token_capability_refresh_interval. By default, the refresh interval is five minutes.

Every interval, Proxy calls sys/capabilies-self on behalf of every token in the cache to confirm the token still has permission to access the cached secret. If the result from Vault indicates that permission (or the token itself) was revoked, Proxy updates the cache entry so that the affected token can no longer access the relevant paths from the cache. The refresh interval is essentially the maximum period after which permission to read a KV secret is fully revoked for the relevant token.

If the capabilities have been removed, or Proxy receives a 403 response, the capability is removed from the token, and that token cannot be used to access the cache. For other kinds of errors, such as Vault being unreachable or sealed, the static_secret_token_capability_refresh_behavior config is consulted. If set to optimistic (the default), the capability will not be removed unless we receive a 403 or valid response without the capability. If set to pessimistic, the capability will be removed for any error, such as would occur if Vault is sealed.

For token refresh to work, any token that will access the cache also needs update permission for sys/capabilies-self. Having update permission for the token lets Proxy test capabilities for the token against multiple paths with a single request instead of testing for a 403 response for each path explicitly.

When static secret caching is enabled, Proxy returns HIT or MISS in the X-Cache response header for requests so client can tell if the response was served from the cache or forwarded from Vault. In the event of a hit, Proxy also sets the Age header to indicate, in seconds, how old the cache entry is.

Configuration

The top level cache block has the following configuration entries relating to static secret caching:

• cache_static_secrets (bool: false) - Enables static secret caching when set to true. When cache_static_secrets and auto_auth are both enabled, Vault Proxy serves KV secrets directly from the cache to clients with sufficient permission.

• static_secret_token_capability_refresh_interval (duration: "5m", optional) - Sets the interval as a duration format string at which Vault Proxy rechecks the permissions of tokens used to access cached secrets. The refresh interval is the maximum period after which permission to read a cached KV secret is fully revoked. Ignored when cache_static_secrets is false.

• static_secret_token_capability_refresh_behavior (string: "optimistic", optional) - Sets the capability refresh behavior in the case of an error when attempting to refresh capabilities. In the case of a 403, capabilities will be removed for the token with either option. In case of other errors, such as Vault being sealed or Vault being unavailable, this setting controls the behavior. If set to optimistic (the default), capabilities will be removed for only 403 errors. If set to pessimistic, capabilities will be removed for any error. This essentially allows configuring a preference between favoring availability (optimistic) or access fidelity (pessimistic) of cached static secrets. Ignored when cache_static_secrets is false.

Example configuration

The following example Vault Proxy configuration:

• Defines a TCP listener (listener) with TLS disabled.
• Forces clients using API proxy (api_proxy) to identify with an auto-auth token.
• Configures auto-authentication (auto-auth) for approle.
• Enables static secret caching with cache_static_secrets.
• Sets an explicit token capability refresh window of 1 hour with static_secret_token_capability_refresh_interval.

# Other Vault Proxy configuration blocks
# ...

cache {
  cache_static_secrets = true
  static_secret_token_capability_refresh_interval = "1h"
}

api_proxy {
  use_auto_auth_token = "force"
}

listener "tcp" {
    address = "127.0.0.1:8100"
    tls_disable = true
}

auto_auth {
  method {
    type = "approle"
    config = {
      role_id_file_path = "roleid"
      secret_id_file_path = "secretid"
      remove_secret_id_file_after_reading = false
    }
  }