Bug: No way to pass custom credentials/session vars to AWS Secrets Manager provider

[mrharpo]

Summary

When the AWS Secrets Manager secrets provider needs to authenticate against a different AWS account/credentials than the default S3 datastore, there is no configuration to supply custom session variables or client parameters.

The S3 datatools client supports this via METAFLOW_DATATOOLS_SESSION_VARS / METAFLOW_DATATOOLS_CLIENT_PARAMS, but there is no equivalent METAFLOW_AWS_SECRETS_MANAGER_SESSION_VARS / METAFLOW_AWS_SECRETS_MANAGER_CLIENT_PARAMS.

Details

get_aws_client already accepts session_vars and client_params, and the S3 datatools path wires config values straight through:

metaflow/metaflow/plugins/aws/aws_client.py

Lines 94 to 96 in 9088ec4

	def get_aws_client(
	module, with_error=False, role_arn=None, session_vars=None, client_params=None
	):

metaflow/metaflow/plugins/aws/aws_client.py

Lines 110 to 116 in 9088ec4

	return cached_provider_class.get_client(
	module,
	with_error,
	role_arn=role_arn,
	session_vars=session_vars,
	client_params=client_params,
	)

The AWS Secrets Manager provider, however, only passes region_name and role_arn, leaving session_vars / client_params unset:

metaflow/metaflow/plugins/aws/secrets_manager/aws_secrets_manager_secrets_provider.py

Lines 97 to 101 in 9088ec4

	secrets_manager_client = get_aws_client(
	"secretsmanager",
	client_params={"region_name": effective_aws_region},
	role_arn=role,
	)

As a result there is no way to customize the botocore session/client for Secrets Manager (e.g. supplying a different profile, credential provider, or endpoint) when the secret lives in a different account/credential context than the S3 store.

The only available override is AWS_SECRETS_MANAGER_DEFAULT_ROLE (role assumption), which doesn't cover all credential scenarios, including ours.

Expected behavior

Provide config options comparable to the datatools ones, e.g. METAFLOW_AWS_SECRETS_MANAGER_SESSION_VARS and METAFLOW_AWS_SECRETS_MANAGER_CLIENT_PARAMS passed through to get_aws_client when the Secrets Manager provider builds its client, so custom credentials can be supplied via the environment/config.

Relevant files

• metaflow/plugins/aws/secrets_manager/aws_secrets_manager_secrets_provider.py
• metaflow/metaflow_config.py
• metaflow/plugins/aws/aws_client.py

Environment

Metaflow version: 2.19.33

[PreethamNoelP]

Hi! I'd like to contribute a fix for this.
The pattern is clear, it mirrors how METAFLOW_DATATOOLS_SESSION_VARS / METAFLOW_DATATOOLS_CLIENT_PARAMS are wired through for S3, just applied to the Secrets Manager provider.
My plan: add METAFLOW_AWS_SECRETS_MANAGER_SESSION_VARS and METAFLOW_AWS_SECRETS_MANAGER_CLIENT_PARAMS to metaflow_config.py and pass them through to get_aws_client in aws_secrets_manager_secrets_provider.py.
Could a maintainer assign me and confirm this approach is acceptable before I open a PR?
Thanks!

[mrharpo]

@PreethamNoelP Thanks for offering to help!

I have a working implementation already in our fork: WGBH-MLA#1

Runs are working locally and --with kubernetes. It still needs tests and documentation, and I'm not sure if there's other places in the code where the pattern also needs to be extended. But feel free to use that as a starting point!

[PreethamNoelP]

Hi @mrharpo! Thanks for sharing your fork implementation.
I'll use it as a starting point.