secret adoption conflicts with other operators

[steven-rand]

Hi,

We're having an issue on our kubernetes clusters where the spicedb-operator is trying to adopt secrets which are owned by other operators, and this is causing an endless back and forth in which each operator mutates the secret's labels and annotations in its reconciliation loop. We have two different secrets, each configured for use by SpiceDB as follows:

apiVersion: authzed.com/v1alpha1
kind: SpiceDBCluster
spec:
  credentials:
    datastoreURI:
      secretName: {{ .Values.secrets.dbCredentialsSecret }}
      key: connection-string
    presharedKey:
      secretName: {{ .Values.secrets.presharedKeySecret }}
      key: preshared-key

The secret being used for the datastoreURI is managed by an internal operator which creates postgres databases and writes k8s secrets containing connection info for those databases. The secret being used for the presharedKey comes from the external secrets operator (https://github.com/external-secrets/external-secrets).

The loop for each secret is:

1. SpiceDB operator adopts the secret, adding its label and annotation
2. The operator which owns the secret sees that it has changed, and updates it, thereby removing the SpiceDB label and annotation
3. Back to step 1

We are working around this problem by configuring external-secrets-operator to apply the SpiceDB operator's label and annotation to the secret that it creates, and by telling our internal database operator to ignore label/annotation changes. However, hardcoding the spicedb operator's label and annotation onto the secret owned by external secrets operator in an attempt to trick spicedb-operator into believing that it owns the secret feels prety hacky, so I wanted to open an issue to see if we can find a better solution.

Is there a way for the spicedb-operator to not rely on the label and annotation that result secret adoption, and instead read the values from the configured secrets? I would expect that having the secrets used by the operator come from other kubernetes operators is not an uncommon use case.

Thanks,
Steven