If your Team, Organization or Company is serious about where to keep credentials used by services (or to interact with them), they are likely in Kubernetes Secrets, Hashicorp Vault, one of the various cloud-based Secrets Managers, 1Password, etc. Then, when running locally, they might be in your Environment Variables, in a .env, in a file or just in plaintext in a script (naughty!).

This might lead to services and tools written to deal with those different “secret provision scenarios”. Or tricky/brittle local setups.

Introducing Spelunk

Ready to dig-up your secrets 😜

Ready to dig-up your secrets 😜

Spelunk is a Golang library for extracting secrets from various sources (Kubernetes, Vault, env vars, files) using a unified URI-based string: Secret Coordinates. For example:

1
2
3
4
5
6
7
8
9

# Get the value of `the-key` out of `my-team-secret` secret,
# stored in the namespace `my-namespace`
k8s://my-namespace/my-team-secret/the-key

# Secret provided in Base64 encoded form
base64://bXktYmlnLXNlY3JldAo=

# Field `$.kafka.password` extracted from the local `kafka-credentials.json`
file://kafka-credentials.json?jp=$.kafka.password

Spelunk simplifies the access to secrets by requiring only the coordinates for “digging it up”. I deal for cloud-native environments, and suitable for command-line tools, as well as deployed services.

Use cases?

  • Cloud-Native CLI Tools & Applications: simplifies secret management for CLI tools via “Secret Coordinates” instead of hardcoding secrets or managing integrations
  • Adaptable Secret Fetching: if your Security Team decides to move from a secret storage to another, all there is to do is changing coordinates (i.e. k8s:// → vault://)
  • Secret Modification/Formatting: Using Modifiers, users can extract exactly what they need from a secret (e.g. via JSONPath)
  • Idiomatic Parsing: It integrates with CLI configuration libraries (like flag, Kong, urfave/cli, Viper, …) to treat secrets just like any other parsed argument (code examples)

Let’s see some code!

You have a Kubernetes Secret:

1
2
3
4
5
6
7

apiVersion: v1
kind: Secret
metadata:
  namespace: my-namespace
  name: my-secret
data:
  my-data-key: Rm9yemEgTmFwb2xpIFNlbXByZSEgSnV2ZSBNZXJkYSEK

Spelunk can dig it up for you:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

package main

import (
	"github.com/detro/spelunk"
	"github.com/detro/spelunk/plugin/source/kubernetes"
	v1 "k8s.io/client-go/kubernetes/typed/core/v1"
)

// Initialize the Kubernetes client...
k8sClient, _ := v1.NewForConfig(restConfig)

// Create a Spelunker
spelunker := spelunk.NewSpelunker(
	kubernetes.WithKubernetes(k8sClient),
)

// Dig-up secrets!
coord, _ := types.NewSecretCoord("k8s://my-namespace/my-secret/my-data-key")
secret, _ := spelunker.DigUp(ctx, coord)

And it can dig-up secrets out of?

Currently Spelunk supports fetching secrets from these sources:

And more are in the works.

“Modifiers” you said?

Modifiers are optional behaviour applied to a secret after it has been dug-up by Spelunk. It can be seen as a function in the mathematical sense:

$$ Modifier(Val,Input) = ModifiedVal $$

Each modifier is applied in the same order provided to the secret value found at the coordinates:

1

<type>://<location>?mod1=A&mod2=B&mod1=C

Currently we have a JSONPath extractor (?jp=<JSONPath>) and Base64 encoder (?b64). And, like for secret sources, more modifiers are in the works.

Only what you need

Spelunk follows a “only import what you need” principle. Aside from the most simple SecretSources and SecretModifiers, all others are provided as plug-in that you import in your project when needed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

package main

import (
	"github.com/detro/spelunk"
	"github.com/detro/spelunk/plugin/source/vault"
)

func main() {
	// ... initialize a `vaultClient` and plug it in...
	spelunker = spelunk.NewSpelunker(
		vault.WithVault(vaultClient),
	)
}

Alternatives?

When the need for something like Spelunk rose, I looked for what options were out there. The closest thing I found was Go CDK Secrets. But, to my surprise, it does not support Kubernetes!

So, I decided to roll my own.

Next steps

Right now the latest release of Spelunk is v1.1.0, that introduced support for HashiCorp Vault. In the works I got: