[Announcement] New Managed Policies Available for the EBS CSI Driver

[mdzraf]

Summary

We are announcing the release of two new AWS managed IAM policies for the EBS CSI Driver that provide more restrictive permission scoping compared to the existing AmazonEBSCSIDriverPolicy:

• AmazonEBSCSIDriverPolicyV2 - restricts actions to resources tagged as managed by the EBS CSI Driver.
• AmazonEBSCSIDriverEKSClusterScopedPolicy - restricts actions to resources belonging to a specific EKS cluster, preventing cross-cluster access.

IMPORTANT NOTE

If your workflow relies on importing an existing EBS volume or snapshot for use by the EBS CSI Driver by manually creating a Kubernetes object that reference them (static provisioning), please read the full migration guidance below before switching policies, as this is a breaking change for that use case.

Why are we publishing the two new policies?

As part of our ongoing effort to enhance the security posture of the EBS CSI Driver, we are aligning the driver's managed policies with least-privilege best practices. The two new policies give you the ability to choose a permission model that better fits your workload requirements.

Background

The existing AmazonEBSCSIDriverPolicy grants the driver permissions to perform actions on any EBS volume and snapshot in the account, regardless of whether those resources are managed by the driver. This scope is intentional, as it enables workflows such as static provisioning and snapshot restore that require access to EBS resources not created by the driver. However, for the most common use case of dynamic provisioning, the driver's IAM identity has access to resources beyond what is strictly needed.

• AmazonEBSCSIDriverPolicyV2 addresses this by scoping permissions to only resources tagged with ebs.csi.aws.com/cluster: true. It also permits actions on resources tagged with kubernetes.io/created-for/pvc/name: <pvc name> to support volumes migrated from the legacy in-tree AWS EBS plugin to CSI.
• AmazonEBSCSIDriverEKSClusterScopedPolicy goes further by scoping permissions to resources tagged with ebs.csi.aws.com/cluster-name matching the eks-cluster-name tag on the driver's IAM principal. This ensures each cluster's driver instance can only manage EBS resources belonging to its own cluster.

Below is a table outlining the differences in how you can take actions on each resource with the different policies:

Resource	AmazonEBSCSIDriverPolicy	AmazonEBSCSIDriverPolicyV2	AmazonEBSCSIDriverEKSClusterScopedPolicy
Volumes/Snapshots created by the EBS CSI Driver	No additional tagging needed.	Supported via ebs.csi.aws.com/cluster: true tag being present on resource. This is already done by the driver.	Requires manually tagging the resource with ebs.csi.aws.com/cluster-name: <your-cluster-name>. On version v1.58.0 and later of the driver, this tag will be added on resource creation by the driver when configured (see migration guidance).
Volumes created by the legacy in-tree EBS plugin	No additional tagging needed.	Supported via kubernetes.io/created-for/pvc/name tag being present on resource.	Requires manually tagging the resource with ebs.csi.aws.com/cluster-name: <your-cluster-name>.
Volumes/Snapshots imported for management by the EBS CSI Driver (static provisioning)	No additional tagging needed.	Requires manually tagging the volume or snapshot with ebs.csi.aws.com/cluster: true.	Requires manually tagging the resource with ebs.csi.aws.com/cluster-name: <your-cluster-name>.
EC2 Instances (Kubernetes Nodes)	No additional tagging needed.	No additional tagging needed.	Requires resource to have eks:cluster-name: <your-cluster-name> tag (Added by default on nodes created with EKS). If using instances not created by EKS, you will need to manually tag these instances with the ebs.csi.aws.com/cluster-name: <your-cluster-name> tag.

Which policy should I choose?

• AmazonEBSCSIDriverPolicy is suitable if your security posture allows the driver permission to not be restricted to only tagged resources.
• AmazonEBSCSIDriverPolicyV2 is suitable if you want tighter scoping than the existing policy while keeping a single IAM role across multiple clusters, or if you run the driver on self-managed Kubernetes clusters and do not need to prevent cross-cluster access.
• AmazonEBSCSIDriverEKSClusterScopedPolicy is suitable if your workflows are composed of multiple EKS clusters in one single AWS account and you want to prevent cross-cluster access. If you are not using EKS Pod Identity and instead using something like IAM Roles For Service Account (IRSA), you will need to manually tag the IAM role with the tag eks-cluster-name: <your-cluster-name>, and maintain a separate role per cluster. Additionally, if your cluster nodes are not managed by EKS, you will need to also manually tag your instances with ebs.csi.aws.com/cluster-name: <your-cluster-name>.
• If none of the policy fits your use case, you can create a custom IAM policy tailored to your requirements. [1]

Migration guidance

If you do not deploy the EBS CSI Driver using the AWS managed IAM policy AmazonEBSCSIDriverPolicy you may skip this section.

Migrating to AmazonEBSCSIDriverPolicyV2:

• If you use static provisioning, tag any externally created volumes and snapshots that you want the driver to manage with ebs.csi.aws.com/cluster: true. This can be done through the AWS Management Console, the AWS CLI (aws ec2 create-tags), or any other method you use to manage resource tags.
• Attach AmazonEBSCSIDriverPolicyV2 to the IAM role used by the EBS CSI Driver. Follow the instructions for adding and removing IAM policies [1].
• Detach AmazonEBSCSIDriverPolicy from the IAM role. It is important to detach the old policy, as keeping both attached will result in the broader permissions of AmazonEBSCSIDriverPolicy remaining in effect.
• No driver upgrade is required. The driver already tags dynamically provisioned resources with ebs.csi.aws.com/cluster: true.

Migrating to AmazonEBSCSIDriverEKSClusterScopedPolicy:

• If using EKS Pod Identity:
  • Upgrade the EBS CSI Driver to version v1.58.0 or later and configure --k8s-tag-cluster-id to be your cluster name, if deploying as an EKS addon there is no need to manually configure as it will be set automatically. After upgrading, the driver will automatically tag all newly provisioned resources with ebs.csi.aws.com/cluster-name: <your-cluster-name>.
  • Tag any existing EBS volumes and snapshots created by the driver with ebs.csi.aws.com/cluster-name: <your-cluster-name>. This can be done through the AWS Management Console, the AWS CLI, or any other method you use to manage resource tags.
  • If you use static provisioning, also tag any externally-created volumes and snapshots that you want the driver to manage with ebs.csi.aws.com/cluster-name: <your-cluster-name>.
  • Ensure the EBS CSI Driver is configured to use EKS Pod Identity. EKS Pod Identity automatically sets the eks-cluster-name session tag on the driver's IAM principal, which the policy uses to scope permissions to the correct cluster.
  • Attach AmazonEBSCSIDriverEKSClusterScopedPolicy to the IAM role used by the EBS CSI Driver. Follow the instructions for adding and removing IAM policies [1].
  • Detach AmazonEBSCSIDriverPolicy from the IAM role. It is important to detach the old policy, as keeping both attached will result in the broader permissions of AmazonEBSCSIDriverPolicy remaining in effect.
• If using EKS without Pod Identity (i.e. with IAM Roles For Service Accounts (IRSA)):
  • Upgrade the EBS CSI Driver to version v1.58.0 or later and configure --k8s-tag-cluster-id to be your cluster name, if deploying as an EKS addon there is no need to manually configure as it will be set automatically. After upgrading, the driver will automatically tag all newly provisioned resources with ebs.csi.aws.com/cluster-name: <your-cluster-name>.
  • Tag any existing EBS volumes and snapshots created by the driver with ebs.csi.aws.com/cluster-name: <your-cluster-name>. This can be done through the AWS Management Console, the AWS CLI, or any other method you use to manage resource tags.
  • If you use static provisioning, also tag any externally-created volumes and snapshots that you want the driver to manage with ebs.csi.aws.com/cluster-name: <your-cluster-name>.
  • If you have any nodes in the cluster that are not managed by EKS, you must manually tag the corresponding EC2 instances with ebs.csi.aws.com/cluster-name: <your-cluster-name>.
  • Attach AmazonEBSCSIDriverEKSClusterScopedPolicy to the IAM role used by the EBS CSI Driver. Follow the instructions for adding and removing IAM policies [1].
  • DetachAmazonEBSCSIDriverPolicy from the IAM role. It is important to detach the old policy, as keeping both attached will result in the broader permissions of AmazonEBSCSIDriverPolicy remaining in effect.
  • Since EKS Pod Identity is not being used, you must manually tag the IAM role with the tag eks-cluster-name: <your-cluster-name> so that the policy variable ${aws:PrincipalTag/eks-cluster-name} resolves correctly.
  • Note that you will need to maintain a separate IAM role per cluster, as each role must be tagged with its corresponding cluster name.

FAQ

How can I know which IAM roles have the AmazonEBSCSIDriverPolicy attached?

If you would like to get a list of all the IAM roles which have the AmazonEBSCSIDriverPolicy attached, you can run the following CLI command:

aws iam list-entities-for-policy \
    --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
    --entity-filter Role \
    --query 'PolicyRoles[].RoleName' --output text

How can I know if my workflow relies on being able to static provision volumes and/or snapshots?

Your workflow relies on static provisioning if you create EBS volumes or snapshots outside of the EBS CSI Driver (for example, through the AWS Console, or AWS CLI) and then reference them in Kubernetes PersistentVolume (PV) objects for use in your cluster. A common indicator is PersistentVolume manifests that specify an existing volumeHandle in their CSI configuration rather than using a StorageClass to dynamically create new volumes. Similarly, if you create EBS snapshots outside of the driver and then reference them as the source for new volumes through a VolumeSnapshotContent object, your workflow depends on access to snapshots not created by the driver.

Additionally, if you would like to get a list of all the volume and snapshot IDs in a specific region that do not have the tag key value pair of ebs.csi.aws.com/cluster: true, and have a corresponding PersistentVolume or VolumeSnapshotContent object, you can run the following script: (i.e. ./script-name.sh us-west-2)

#!/bin/bash 
set -euo pipefail 
  
REGION="${1:?Usage: $0 <region>}" 
  
# Untagged EBS volumes (missing ebs.csi.aws.com/cluster=true) 
untagged_volumes=$(aws ec2 describe-volumes --region "$REGION" --output json --no-paginate \ 
  | jq -r '.Volumes[] | select((.Tags // []) | map(select(.Key == "ebs.csi.aws.com/cluster" and .Value == "true")) | length == 0) | .VolumeId') 
  
# Untagged EBS snapshots 
untagged_snapshots=$(aws ec2 describe-snapshots --region "$REGION" --owner-ids self --output json --no-paginate \ 
  | jq -r '.Snapshots[] | select((.Tags // []) | map(select(.Key == "ebs.csi.aws.com/cluster" and .Value == "true")) | length == 0) | .SnapshotId') 
  
# Volume IDs referenced by PVs using the EBS CSI driver 
pv_volume_ids=$(kubectl get pv -o json | jq -r '.items[] | select(.spec.csi.driver == "ebs.csi.aws.com") | .spec.csi.volumeHandle') 
  
# Snapshot IDs referenced by VolumeSnapshotContent objects (skip if CRD not installed) 
vsc_snapshot_ids=$(kubectl get volumesnapshotcontent -o json 2>/dev/null \ 
  | jq -r '.items[] | select(.spec.driver == "ebs.csi.aws.com") | .status.snapshotHandle // .spec.source.snapshotHandle // empty') || true 
  
echo "=== Untagged volumes with a corresponding PV ===" 
results=$(comm -12 <(echo "$untagged_volumes" | sort) <(echo "$pv_volume_ids" | sort)) 
[ -n "$results" ] && echo "$results" || echo "(none)" 
  
echo "" 
echo "=== Untagged snapshots with a corresponding VolumeSnapshotContent ===" 
results=$(comm -12 <(echo "$untagged_snapshots" | sort) <(echo "$vsc_snapshot_ids" | sort)) 
[ -n "$results" ] && echo "$results" || echo "(none)" 

What do I do if my workflow relies on being able to statically provision volumes and snapshots?

The new policies do not grant access to all volumes or snapshots in an account by default. If your workflow relies on associating externally-created EBS volumes and snapshots with their corresponding Kubernetes objects, you must manually tag those resources with the appropriate tag for your chosen policy (ebs.csi.aws.com/cluster: true for AmazonEBSCSIDriverPolicyV2, or ebs.csi.aws.com/cluster-name: <your-cluster-name> for AmazonEBSCSIDriverEKSClusterScopedPolicy) for the driver to manage them.

Alternatively, you can stay on the current AmazonEBSCSIDriverPolicy or create a custom IAM policy tailored to your requirements that will not require your resources being tagged. [1]

What will happen if I try to use a statically provisioned volume or snapshot not properly tagged with the new policy attached?

You will receive an UnauthorizedOperation error from the AWS API.

What will happen if I try to use a self-managed node/instance not properly tagged with the new policy attached?

You will receive an UnauthorizedOperation error from the AWS API.

What happens if I attach one of the new policies but also keep the AmazonEBSCSIDriverPolicy attached on the same identity?

Actions allowed by the existing AmazonEBSCSIDriverPolicy policy will continue to be authorized until it is detached.

Support

Please reach out directly on this issue if you have any questions about these new policies.

[1] https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html
[2] https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html

[amos-ddir]

This might be a "me" problem, but from the above documentation:

Requires resource to have eks:cluster-name: tag (Added by default on nodes created with EKS).

However, looking at my ec2 nodes (managed by EKS auto-mode), the tags that provide are aws:eks:cluster-name and eks:eks-cluster-name. Manually adding the tag seems incorrect because it won't be automatically maintained when nodes are swapped out.

[ConnorJC3]

@amos-ddir the above doesn't apply to EKS Auto Mode instances, as the OSS EBS CSI Driver isn't supported on those nodes.

EKS Auto's Storage Controller uses the managed policy AmazonEKSBlockStoragePolicy which references the eks:eks-cluster-name tag.