Add ARM multiarchitecture support for the Pipeline managed cluster components

[pregnor]

Is your feature request related to a problem? Please describe.

I would like to start a pure ARM nodes backed Pipeline client cluster and run every Pipeline components in it.
I need ARM architecture support for all available Pipeline components.

Describe the solution you'd like to see

Make Pipeline components able to run on both non-ARM and ARM nodes.

Describe alternatives you've considered

Additional context

ARM support current state:

Note: deprioritized, ignored or removed components are yet to be added to the table, I focused on the "important" stuff for the initial state.

Service	Pod	ARM support	Comment
Cluster autoscaler	(aws-)cluster-autoscaler	❌	CA 1.20 (supporting K8s 1.20) has multi-arch ARM64 support. Being backported to 1.19, waiting for tag.
Horizontal pod autoscaling		❌	See components below.
	hpa-operator	❌	To be removed
	kube-metrics-adapter	❌	To be removed with hpa-operator. We build our own image from this, requires update for ARM multiarch support.
	metrics-server	❌	To be transitioned to self-serve with hpa-operator removal. Minor update available with ARM multiarch support.
Instance termination handler	ith-instance-termination-handler	✅
IS Backup	velero	❌	Manifest based multi-arch images are available for current velero container version v1.5.1, single arch ARM and ARM64 images are available for current velero-plugin-for-aws container version 1.1.0, multiarch image is available for latest/master state.
IS DNS	external-dns	✅	Since PR #3393
IS Expiry	-	✅
IS Ingress	traefik	✅
IS Logging		❌	See components below.
	fluentbit	✅
	fluentd	❌	Officially only single arch ARM images are available, but the banzaicloud image supports manifest based multiarch.
	logging-operator	❌	Available from 3.8.4 (major version change).
	loki	✅
IS Monitoring		❌	See components below.
	alertmanager	✅	Since PR #3394
	grafana	✅
	kube-state-metrics	❌	Latest chart version 2.11.0 supports image version 1.9.7 and no ARM support, ARM multiarch support is available since 2.0.0-alpha.3, latest image version 2.0.0-beta doesn't have official chart yet and also introduces breaking changes to CLI arguments and HPA metrics among others. Decided to postpone this until official chart is available.
	prometheus	✅	Since PR #3394
	prometheus-node-exporter	✅
	prometheus-operator	✅	Since PR #3394
	prometheus-pushgateway	✅
IS Security Scan	anchore-policy-validator	✅
IS Vault	vault-secrets-webhook	✅	Since PR #3385
Node pool label sets	nodepool-labels-operator	✅

[sagikazarmark]

Logging and monitoring integrated services are not that important: they are going to be replaced by one eye.