r/kubernetes
External Secrets Operator on OKE leaks memory above 50 Workload Identity SecretStores: root cause is in oci-go-sdk (fix inside)
- upvotes
- 10
- comments
- 2
Post
Highlighted: the lines this signal was extracted from
TL;DR: If you run External Secrets Operator (ESO) on OKE with principalType: Workload and more than 50 SecretStores that set serviceAccountRef, the controller leaks memory, about 20–25 MiB/h, until it is OOM-killed. The root cause is in the OCI Go SDK: an OKE Workload Identity provider that is thrown away keeps its proxymux connection open forever. The SDK fix is one line, and ESO also needs a configurable cache size. What we saw ESO v2.8.0 on OKE. We have one SecretStore per namespace, each with its own service account, so that IAM only lets a namespace read its own secrets. At 50 stores, memory was flat. The day the 51st store was created, the controller started growing 20–25 MiB/h. It was then SIGKILLed at its 1 GiB limit every ~36 h. ExternalSecrets with a 1m refresh were 4–5 minutes behind. Why ESO caches one Workload Identity configuration provider per store, in an LRU fixed at 50 entries. With 51+ stores, the reconciles cycle through more keys than the cache holds, so every lookup misses and ESO builds a new provider. In oci-go-sdk, the provider's token client talks to the OKE proxymux over an http.Transport with keep-alives on and no IdleConnTimeout, and nothing ever closes it. The idle connection's goroutines keep the transport, the federation client and its RSA session key reachable. So a provider that is dropped is never...
Keep reading with a free account
The rest of this post, and every signal for Oracle, is in your free account.