diff --git a/OCI/README.md b/OCI/README.md new file mode 100644 index 0000000..161409f --- /dev/null +++ b/OCI/README.md @@ -0,0 +1,159 @@ +# OCI Cloud Resource Estimator + +This guide explains how to run the OCI CSPM benchmark script to count billable resources in your Oracle Cloud Infrastructure tenancy. + +## What it counts + +| Resource | OCI Service | +|---|---| +| Running VMs | OCI Compute — `RUNNING` instances | +| Stopped VMs | OCI Compute — `STOPPED` instances | +| OKE Clusters | Container Engine for Kubernetes — active clusters | +| OKE Managed Nodes | OKE managed node pool sizes | +| OKE Virtual Nodes | OKE virtual (serverless) node pool sizes | +| Container Instances | OCI Container Instances — active containers | +| Functions | OCI Functions — active functions across all applications | +| Autonomous Databases | Oracle Autonomous Database — all workload types | +| MySQL DB Systems | MySQL HeatWave Database Service | +| Object Storage Buckets | OCI Object Storage — per compartment | + +## Prerequisites + +- Python 3 +- pip +- [OCI CLI](https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm) — `oci` command must be in PATH + +### Install the OCI CLI + +```shell +bash -c "$(curl -L https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh)" +``` + +### Configure credentials + +```shell +oci setup config +``` + +This creates `~/.oci/config` and generates an API key pair. Upload the public key to your OCI user's API keys in the Console. + +Verify authentication: + +```shell +oci iam region list +``` + +## IAM Policy (least-privilege) + +Create an IAM group (e.g. `CrowdStrikeEstimatorGroup`) and attach the following policy at the tenancy level: + +``` +Allow group CrowdStrikeEstimatorGroup to inspect compartments in tenancy +Allow group CrowdStrikeEstimatorGroup to inspect instances in tenancy +Allow group CrowdStrikeEstimatorGroup to inspect cluster-family in tenancy +Allow group CrowdStrikeEstimatorGroup to inspect container-instances in tenancy +Allow group CrowdStrikeEstimatorGroup to inspect fn-app in tenancy +Allow group CrowdStrikeEstimatorGroup to inspect fn-function in tenancy +Allow group CrowdStrikeEstimatorGroup to inspect autonomous-databases in tenancy +Allow group CrowdStrikeEstimatorGroup to inspect mysql-instances in tenancy +Allow group CrowdStrikeEstimatorGroup to inspect buckets in tenancy +Allow group CrowdStrikeEstimatorGroup to read region-subscriptions in tenancy +``` + +### Using Instance Principal (running on OCI Compute) + +Create a Dynamic Group matching your scanner instance: + +``` +All {instance.compartment.id = 'ocid1.compartment.oc1..'} +``` + +Then apply the same `inspect` / `read` policies to that Dynamic Group instead of a user group. + +## Running in OCI Cloud Shell + +OCI Cloud Shell has the OCI CLI pre-installed and is automatically authenticated via Instance Principal. + +```shell +RELEASE_VERSION="v1.0.0" +curl -sLO "https://github.com/CrowdStrike/cloud-resource-estimator/releases/download/${RELEASE_VERSION}/benchmark.sh" +curl -sL "https://github.com/CrowdStrike/cloud-resource-estimator/releases/download/${RELEASE_VERSION}/checksum.txt" \ + | grep benchmark.sh | sha256sum -c +chmod +x benchmark.sh +./benchmark.sh oci +``` + +## Running locally + +```shell +./benchmark.sh oci +``` + +Or scan multiple providers at once: + +```shell +./benchmark.sh aws oci +``` + +## Configuration + +All options can be set via environment variables before running the script. + +| Variable | Default | Description | +|---|---|---| +| `OCI_PROFILE` | `DEFAULT` | OCI config file profile name | +| `OCI_TENANCY_OCID` | from config | Override tenancy OCID | +| `OCI_REGIONS` | all subscribed | Comma-separated regions to scan | +| `OCI_SKIP_COMPARTMENTS` | none | Comma-separated compartment OCIDs to exclude | +| `OCI_INCLUDE_COMPARTMENTS` | all | Comma-separated compartment OCIDs to scan exclusively | +| `OCI_THREADS` | `5` | Parallel scan workers | +| `OCI_API_DELAY` | `0.05` | Seconds between API calls | +| `OCI_MAX_RETRIES` | `5` | Retry attempts per failed operation | +| `OCI_RESUME_FILE` | `oci_benchmark_progress.json` | Progress tracking file for resume support | +| `OCI_DRY_RUN` | `false` | Set to `true` to simulate without API calls | + +**Important:** `OCI_INCLUDE_COMPARTMENTS` takes full precedence — if set, `OCI_SKIP_COMPARTMENTS` is ignored. + +### Resuming an interrupted scan + +If the scan is interrupted (Ctrl+C or SIGTERM), partial results are written to the CSV and progress is saved to `OCI_RESUME_FILE`. Re-running the same command resumes from where it left off — already-completed `(compartment, region)` pairs are skipped automatically. + +The progress file is deleted automatically on a fully successful run. To force a fresh scan, delete it manually or set `OCI_RESUME_FILE` to a new path. + +### Example: scan only specific compartments + +```shell +export OCI_INCLUDE_COMPARTMENTS="ocid1.compartment.oc1..aaaa...,ocid1.compartment.oc1..bbbb..." +./benchmark.sh oci +``` + +### Example: scan specific regions only + +```shell +export OCI_REGIONS="us-ashburn-1,us-phoenix-1" +./benchmark.sh oci +``` + +### Example: large tenancies (many compartments and regions) + +```shell +export OCI_THREADS=10 +export OCI_API_DELAY=0.1 +./benchmark.sh oci +``` + +## Output + +The script writes a timestamped CSV file: + +``` +./cloud-benchmark/oci-benchmark-YYYYMMDD_HHMMSS.csv +``` + +Results are also printed to the terminal as a grid table. + +To view the output: + +```shell +cat ./cloud-benchmark/oci-benchmark-*.csv +``` diff --git a/OCI/oci_cspm_benchmark.py b/OCI/oci_cspm_benchmark.py new file mode 100644 index 0000000..fbb8f13 --- /dev/null +++ b/OCI/oci_cspm_benchmark.py @@ -0,0 +1,919 @@ +# pylint: disable=C0301,C0302,E0401,W1203,W0718 +# flake8: noqa: E501 +""" +oci_cspm_benchmark.py + +Assists with provisioning calculations by retrieving a count of +all billable resources attached to an OCI tenancy. +""" + +import argparse +import csv +import concurrent.futures +import threading +import time +import json +import os +import random +import signal +import logging +from datetime import datetime, timezone +from typing import Dict, List, Optional, Any, Union, Tuple + +import oci +import oci.config +import oci.exceptions +import oci.pagination +import oci.auth.signers +import oci.identity +import oci.core +import oci.container_engine +import oci.container_instances +import oci.functions +import oci.database +import oci.mysql +import oci.object_storage +from tabulate import tabulate + + +# Global data structures +data: List[Dict[str, Any]] = [] +headers = { + "tenancy_id": "OCI Tenancy ID", + "compartment_id": "Compartment OCID", + "compartment_name": "Compartment Name", + "region": "Region", + "vms_running": "Running VMs", + "vms_stopped": "Stopped VMs", + "oke_clusters": "OKE Clusters", + "oke_nodes": "OKE Managed Nodes", + "oke_virtual_nodes": "OKE Virtual Nodes", + "container_instances": "Container Instances", + "functions": "Functions", + "autonomous_dbs": "Autonomous Databases", + "mysql_dbs": "MySQL DB Systems", + "buckets": "Object Storage Buckets", +} +totals: Dict[str, Union[str, int]] = { + "tenancy_id": "totals", + "compartment_id": "totals", + "compartment_name": "totals", + "region": "totals", + "vms_running": 0, + "vms_stopped": 0, + "oke_clusters": 0, + "oke_nodes": 0, + "oke_virtual_nodes": 0, + "container_instances": 0, + "functions": 0, + "autonomous_dbs": 0, + "mysql_dbs": 0, + "buckets": 0, +} + +# Thread-safe locks +data_lock = threading.Lock() +totals_lock = threading.Lock() +console_lock = threading.Lock() + +# Global configuration — initialized in main() +args: Optional[argparse.Namespace] = None +log: Optional[logging.Logger] = None + + +# --------------------------------------------------------------------------- +# Logging +# --------------------------------------------------------------------------- + +def setup_logging() -> logging.Logger: + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s - %(name)s - %(levelname)s - %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", + ) + return logging.getLogger("oci_cspm") + + +# --------------------------------------------------------------------------- +# Signal handling +# --------------------------------------------------------------------------- + +def setup_signal_handlers() -> None: + """Map SIGINT and SIGTERM to KeyboardInterrupt so main() can do cleanup.""" + def _handler(signum, _frame): + if log: + log.info(f"Received signal {signum}, initiating graceful shutdown...") + raise KeyboardInterrupt("Shutdown signal received") + + signal.signal(signal.SIGINT, _handler) + signal.signal(signal.SIGTERM, _handler) + + +# --------------------------------------------------------------------------- +# Progress tracker +# --------------------------------------------------------------------------- + +class ProgressTracker: + """ + Persists completed/failed (compartment_id, region) pairs to a JSON file so + an interrupted scan can be resumed without re-scanning already-finished pairs. + + File format: + { + "completed_pairs": [["ocid...", "us-ashburn-1"], ...], + "failed_pairs": [["ocid...", "eu-frankfurt-1"], ...], + "start_time": "2024-01-01T00:00:00+00:00", + "total_pairs": 500, + "last_updated": "2024-01-01T01:23:45+00:00" + } + """ + + def __init__(self, progress_file: str): + self.progress_file = progress_file + self._lock = threading.Lock() + self.completed: set = set() + self.failed: set = set() + self.start_time: Optional[str] = None + self.total_pairs: int = 0 + self._load() + + def _load(self) -> None: + if not os.path.exists(self.progress_file): + return + try: + with open(self.progress_file, "r", encoding="utf-8") as f: + saved = json.load(f) + self.completed = {tuple(p) for p in saved.get("completed_pairs", [])} + self.failed = {tuple(p) for p in saved.get("failed_pairs", [])} + self.start_time = saved.get("start_time") + self.total_pairs = saved.get("total_pairs", 0) + msg = f"Resumed from progress file: {len(self.completed)} pairs already complete" + if log: + log.info(msg) + else: + print(msg) + except (json.JSONDecodeError, KeyError, TypeError) as exc: + msg = f"Could not parse progress file {self.progress_file}: {exc} — starting fresh" + if log: + log.warning(msg) + else: + print(msg) + + def _save(self) -> None: + try: + payload = { + "completed_pairs": [list(p) for p in self.completed], + "failed_pairs": [list(p) for p in self.failed], + "start_time": self.start_time, + "total_pairs": self.total_pairs, + "last_updated": datetime.now(timezone.utc).isoformat(), + } + with open(self.progress_file, "w", encoding="utf-8") as f: + json.dump(payload, f, indent=2) + except OSError as exc: + if log: + log.error(f"Could not save progress to {self.progress_file}: {exc}") + + def should_skip(self, compartment_id: str, region: str) -> bool: + return (compartment_id, region) in self.completed + + def mark_completed(self, compartment_id: str, region: str) -> None: + with self._lock: + self.completed.add((compartment_id, region)) + self.failed.discard((compartment_id, region)) + self._save() + + def mark_failed(self, compartment_id: str, region: str) -> None: + with self._lock: + self.failed.add((compartment_id, region)) + self._save() + + def remove(self) -> None: + try: + os.remove(self.progress_file) + except OSError: + pass + + def print_resume_guidance(self) -> None: + completed_count = len(self.completed) + failed_count = len(self.failed) + remaining = self.total_pairs - completed_count + + print("\n" + "=" * 70) + print("PROGRESS SAVED - SCAN CAN BE RESUMED") + print("=" * 70) + print(f" Completed pairs : {completed_count}") + print(f" Failed pairs : {failed_count}") + print(f" Remaining pairs : {remaining}") + if self.total_pairs > 0: + pct = completed_count / self.total_pairs * 100 + print(f" Progress : {pct:.1f}%") + print(f"\n Progress file : {self.progress_file}") + print("\nTo resume, run the same command again.") + print("The script will skip already-completed pairs automatically.") + print("=" * 70) + + +# --------------------------------------------------------------------------- +# Retry handler +# --------------------------------------------------------------------------- + +class RetryHandler: + """Exponential backoff retry for OCI SDK calls.""" + + RETRYABLE_HTTP = {429, 500, 503, 504} + RETRYABLE_CODES = { + "TooManyRequests", + "InternalError", + "ServiceUnavailable", + "RequestTimeout", + "ConnectionError", + } + + @staticmethod + def exponential_backoff(attempt: int, base: float = 1.0, max_delay: float = 120.0) -> float: + delay = min(base * (2 ** attempt), max_delay) + delay *= 0.5 + random.random() * 0.5 # nosec B311 + return delay + + @classmethod + def should_retry(cls, exc: Exception, attempt: int, max_retries: int) -> bool: + if attempt >= max_retries: + return False + if isinstance(exc, oci.exceptions.ServiceError): + return exc.status in cls.RETRYABLE_HTTP or exc.code in cls.RETRYABLE_CODES + if isinstance(exc, oci.exceptions.RequestException): + return True + return False + + def retry_with_backoff(self, func, max_retries: int = 5, operation_name: str = "operation"): + for attempt in range(max_retries + 1): + try: + return func() + except Exception as exc: + if not self.should_retry(exc, attempt, max_retries): + raise + delay = self.exponential_backoff(attempt) + if log: + log.debug(f"Retry {attempt + 1}/{max_retries} for {operation_name} in {delay:.2f}s: {exc}") + time.sleep(delay) + raise RuntimeError(f"Max retries exceeded for {operation_name}") + + +# --------------------------------------------------------------------------- +# Authentication +# --------------------------------------------------------------------------- + +def build_config_and_signer() -> Tuple[dict, Optional[Any]]: + """Return (config, signer). Prefers instance principal, falls back to config file.""" + # Try instance principal first (works on OCI Compute instances without a config file) + try: + signer = oci.auth.signers.InstancePrincipalsSecurityTokenSigner() + tenancy_id = signer.get_tenancy_id() + region = signer.initialize_and_return_region() + config = {"tenancy": tenancy_id, "region": region} + if log: + log.info("Authenticated via Instance Principal") + return config, signer + except oci.exceptions.RequestException as exc: + if log: + log.debug(f"Instance principal not available (network/metadata unreachable): {exc}") + except Exception as exc: # pylint: disable=broad-except + # Instance principal fails fast on non-OCI hosts; swallow and try config file + if log: + log.debug(f"Instance principal not available: {type(exc).__name__}: {exc}") + + # Fall back to config file + profile = args.profile if args else "DEFAULT" + config = oci.config.from_file(profile_name=profile) + oci.config.validate_config(config) + if log: + log.info(f"Authenticated via config file profile '{profile}'") + return config, None + + +# --------------------------------------------------------------------------- +# Pagination helper +# --------------------------------------------------------------------------- + +def paginated_list(list_fn, **kwargs) -> List[Any]: + """Exhaust all pages from any OCI list_* call.""" + results = [] + page = None + while True: + if page: + kwargs["page"] = page + response = list_fn(**kwargs) + results.extend(response.data) + page = response.headers.get("opc-next-page") + if not page: + break + return results + + +# --------------------------------------------------------------------------- +# OCI client handle +# --------------------------------------------------------------------------- + +class OCIHandle: + """Wraps OCI config/signer and provides lazy, region-aware client creation.""" + + def __init__(self, config: dict, signer=None): + self.config = config + self.signer = signer + self.tenancy_id = config["tenancy"] + self._identity: Optional[oci.identity.IdentityClient] = None + self._namespace: Optional[str] = None + self._compartment_ids: Optional[List[str]] = None + self._compartment_names: Optional[Dict[str, str]] = None + self._regions: Optional[List[str]] = None + self._retry = RetryHandler() + # Locks protecting lazy-initialized attributes accessed from multiple threads + self._identity_lock = threading.Lock() + self._namespace_lock = threading.Lock() + self._compartment_lock = threading.Lock() + self._regions_lock = threading.Lock() + + # -- identity client (region-agnostic) ----------------------------------- + + @property + def identity(self) -> oci.identity.IdentityClient: + if self._identity is None: + with self._identity_lock: + if self._identity is None: # double-checked locking + if self.signer: + self._identity = oci.identity.IdentityClient(config={}, signer=self.signer) + else: + self._identity = oci.identity.IdentityClient(self.config) + return self._identity + + # -- compartment enumeration --------------------------------------------- + + def get_all_compartment_ids(self) -> List[str]: + """Return OCIDs of all ACTIVE compartments accessible in this tenancy, including the root.""" + if self._compartment_ids is not None: + return self._compartment_ids + + with self._compartment_lock: + if self._compartment_ids is not None: # another thread may have populated it + return self._compartment_ids + + def fetch(): + return paginated_list( + self.identity.list_compartments, + compartment_id=self.tenancy_id, + compartment_id_in_subtree=True, + lifecycle_state="ACTIVE", + access_level="ACCESSIBLE", + ) + + compartments = self._retry.retry_with_backoff(fetch, args.max_retries, "list_compartments") + names = {c.id: c.name for c in compartments} + names[self.tenancy_id] = "root" + self._compartment_names = names + # Root tenancy is itself a valid compartment for resource scanning + self._compartment_ids = [self.tenancy_id] + [c.id for c in compartments] + + return self._compartment_ids + + def compartment_name(self, compartment_id: str) -> str: + if self._compartment_names is None: + self.get_all_compartment_ids() # populates _compartment_names as a side effect + return (self._compartment_names or {}).get(compartment_id, compartment_id) + + # -- region enumeration -------------------------------------------------- + + def get_subscribed_regions(self) -> List[str]: + """Return region name strings the tenancy is subscribed to.""" + if self._regions is not None: + return self._regions + + with self._regions_lock: + if self._regions is not None: + return self._regions + + def fetch(): + response = self.identity.list_region_subscriptions(self.tenancy_id) + return [r.region_name for r in response.data if r.status == "READY"] + + self._regions = self._retry.retry_with_backoff(fetch, args.max_retries, "list_region_subscriptions") + + return self._regions + + # -- per-region client factory ------------------------------------------- + + def _make_client(self, client_class, region: str): + """Instantiate a region-specific OCI client.""" + if self.signer: + client = client_class(config={}, signer=self.signer) + client.base_client.set_region(region) + return client + region_config = dict(self.config) + region_config["region"] = region + return client_class(region_config) + + # -- object storage namespace (tenancy-scoped, fetched once) ------------- + + def object_storage_namespace(self, region: str) -> str: + if self._namespace is None: + with self._namespace_lock: + if self._namespace is None: # double-checked locking + client = self._make_client(oci.object_storage.ObjectStorageClient, region) + self._namespace = client.get_namespace().data + return self._namespace + + # -- resource counting --------------------------------------------------- + + def count_resources(self, compartment_id: str, region: str) -> Dict[str, int]: + """Count all tracked resource types in a single (compartment, region) pair.""" + row: Dict[str, int] = { + "vms_running": 0, + "vms_stopped": 0, + "oke_clusters": 0, + "oke_nodes": 0, + "oke_virtual_nodes": 0, + "container_instances": 0, + "functions": 0, + "autonomous_dbs": 0, + "mysql_dbs": 0, + "buckets": 0, + } + + retry = self._retry + + # -- Compute instances ----------------------------------------------- + compute = self._make_client(oci.core.ComputeClient, region) + + def _list_instances(): + return paginated_list(compute.list_instances, compartment_id=compartment_id) + + try: + instances = retry.retry_with_backoff( + _list_instances, args.max_retries, f"list_instances/{compartment_id}/{region}" + ) + for inst in instances: + if inst.lifecycle_state in ("TERMINATING", "TERMINATED"): + continue + if inst.lifecycle_state == "RUNNING": + row["vms_running"] += 1 + else: + row["vms_stopped"] += 1 + except oci.exceptions.ServiceError as exc: + if exc.status not in (401, 404): + if log: + log.warning(f"list_instances {compartment_id}/{region}: {exc.status} {exc.code}") + + # -- OKE clusters and node pools ------------------------------------ + ce = self._make_client(oci.container_engine.ContainerEngineClient, region) + + def _list_clusters(): + return paginated_list(ce.list_clusters, compartment_id=compartment_id) + + def _list_node_pools(): + return paginated_list(ce.list_node_pools, compartment_id=compartment_id) + + try: + clusters = retry.retry_with_backoff( + _list_clusters, args.max_retries, f"list_clusters/{compartment_id}/{region}" + ) + row["oke_clusters"] = sum(1 for c in clusters if c.lifecycle_state == "ACTIVE") + except oci.exceptions.ServiceError as exc: + if exc.status not in (401, 404): + if log: + log.warning(f"list_clusters {compartment_id}/{region}: {exc.status} {exc.code}") + + try: + node_pools = retry.retry_with_backoff( + _list_node_pools, args.max_retries, f"list_node_pools/{compartment_id}/{region}" + ) + for pool in node_pools: + if pool.lifecycle_state != "ACTIVE": + continue + # Virtual node pools (serverless) have a separate node_pool_type attribute. + # OCI SDK uses "VIRTUAL" vs "MANAGED". getattr guards against SDK version gaps. + pool_type = getattr(pool, "node_pool_type", None) + if pool_type is None: + # Attribute absent in older SDK versions — treat as managed + pool_type = "MANAGED" + if log: + log.debug(f"node_pool_type absent on pool {pool.id}, defaulting to MANAGED") + size = 0 + if pool.node_config_details: + size = pool.node_config_details.size or 0 + if pool_type == "VIRTUAL": + row["oke_virtual_nodes"] += size + else: + row["oke_nodes"] += size + except oci.exceptions.ServiceError as exc: + if exc.status not in (401, 404): + if log: + log.warning(f"list_node_pools {compartment_id}/{region}: {exc.status} {exc.code}") + + # -- Container Instances -------------------------------------------- + try: + ci_client = self._make_client(oci.container_instances.ContainerInstanceClient, region) + + def _list_cis(): + return paginated_list(ci_client.list_container_instances, compartment_id=compartment_id) + + cis = retry.retry_with_backoff( + _list_cis, args.max_retries, f"list_container_instances/{compartment_id}/{region}" + ) + # The list response returns ContainerInstanceSummary objects. + # container_count is a documented field on ContainerInstanceSummary. + # Each ContainerInstance can hold multiple containers (like a pod). + row["container_instances"] = sum( + getattr(ci, "container_count", 1) or 1 + for ci in cis + if ci.lifecycle_state == "ACTIVE" + ) + except oci.exceptions.ServiceError as exc: + if exc.status not in (401, 404): + if log: + log.warning(f"list_container_instances {compartment_id}/{region}: {exc.status} {exc.code}") + + # -- Functions ------------------------------------------------------- + try: + fn_client = self._make_client(oci.functions.FunctionsManagementClient, region) + + def _list_apps(): + return paginated_list(fn_client.list_applications, compartment_id=compartment_id) + + apps = retry.retry_with_backoff( + _list_apps, args.max_retries, f"list_applications/{compartment_id}/{region}" + ) + for app in apps: + if app.lifecycle_state != "ACTIVE": + continue + + def _list_fns(app_id=app.id): + return paginated_list(fn_client.list_functions, application_id=app_id) + + try: + fns = retry.retry_with_backoff( + _list_fns, args.max_retries, f"list_functions/{app.id}" + ) + row["functions"] += sum(1 for f in fns if f.lifecycle_state == "ACTIVE") + except oci.exceptions.ServiceError as exc: + if exc.status not in (401, 404): + if log: + log.warning(f"list_functions {app.id}: {exc.status} {exc.code}") + except oci.exceptions.ServiceError as exc: + if exc.status not in (401, 404): + if log: + log.warning(f"list_applications {compartment_id}/{region}: {exc.status} {exc.code}") + + # -- Autonomous Databases -------------------------------------------- + try: + db_client = self._make_client(oci.database.DatabaseClient, region) + + def _list_adbs(): + return paginated_list(db_client.list_autonomous_databases, compartment_id=compartment_id) + + adbs = retry.retry_with_backoff( + _list_adbs, args.max_retries, f"list_autonomous_databases/{compartment_id}/{region}" + ) + row["autonomous_dbs"] = sum( + 1 for db in adbs + if db.lifecycle_state not in ("TERMINATING", "TERMINATED") + ) + except oci.exceptions.ServiceError as exc: + if exc.status not in (401, 404): + if log: + log.warning(f"list_autonomous_databases {compartment_id}/{region}: {exc.status} {exc.code}") + + # -- MySQL DB Systems ------------------------------------------------ + # OCI SDK uses MysqlaasClient (the service was formerly called "MySQL as a Service"). + # Older SDK versions may expose DbSystemClient instead — try both. + try: + _mysql_cls = getattr(oci.mysql, "MysqlaasClient", None) or getattr(oci.mysql, "DbSystemClient", None) + if _mysql_cls is None: + raise AttributeError("Neither MysqlaasClient nor DbSystemClient found in oci.mysql") + mysql_client = self._make_client(_mysql_cls, region) + + def _list_mysql(): + return paginated_list(mysql_client.list_db_systems, compartment_id=compartment_id) + + dbs = retry.retry_with_backoff( + _list_mysql, args.max_retries, f"list_db_systems/{compartment_id}/{region}" + ) + row["mysql_dbs"] = sum( + 1 for db in dbs + if db.lifecycle_state not in ("DELETING", "DELETED") + ) + except oci.exceptions.ServiceError as exc: + if exc.status not in (401, 404): + if log: + log.warning(f"list_db_systems {compartment_id}/{region}: {exc.status} {exc.code}") + + # -- Object Storage Buckets ----------------------------------------- + try: + os_client = self._make_client(oci.object_storage.ObjectStorageClient, region) + namespace = self.object_storage_namespace(region) + + def _list_buckets(): + return paginated_list( + os_client.list_buckets, + namespace_name=namespace, + compartment_id=compartment_id, + ) + + buckets = retry.retry_with_backoff( + _list_buckets, args.max_retries, f"list_buckets/{compartment_id}/{region}" + ) + row["buckets"] = len(buckets) + except oci.exceptions.ServiceError as exc: + if exc.status not in (401, 404): + if log: + log.warning(f"list_buckets {compartment_id}/{region}: {exc.status} {exc.code}") + + return row + + +# --------------------------------------------------------------------------- +# Per-(compartment, region) scan worker +# --------------------------------------------------------------------------- + +def scan_pair( + handle: OCIHandle, + compartment_id: str, + region: str, + progress: ProgressTracker, +) -> Optional[Dict[str, Any]]: + """Scan one (compartment, region) pair, update progress, and return a data row.""" + if args.dry_run: + with console_lock: + print(f" [dry-run] Would scan compartment={handle.compartment_name(compartment_id)} region={region}") + return None + + if args.api_delay > 0: + time.sleep(args.api_delay) + + try: + counts = handle.count_resources(compartment_id, region) + except Exception as exc: + if log: + log.error(f"Failed scanning {compartment_id}/{region}: {exc}") + progress.mark_failed(compartment_id, region) + return None + + progress.mark_completed(compartment_id, region) + + row = { + "tenancy_id": handle.tenancy_id, + "compartment_id": compartment_id, + "compartment_name": handle.compartment_name(compartment_id), + "region": region, + **counts, + } + return row + + +# --------------------------------------------------------------------------- +# Output helper +# --------------------------------------------------------------------------- + +def write_output(csv_filename: str) -> None: + """Write the current data (plus totals) to CSV and print the grid table.""" + output_rows = data + [totals] + print(tabulate(output_rows, headers=headers, tablefmt="grid")) + with open(csv_filename, "w", newline="", encoding="utf-8") as csv_file: + csv_writer = csv.DictWriter(csv_file, fieldnames=headers.keys()) + csv_writer.writeheader() + csv_writer.writerows(output_rows) + print(f"\nCSV file stored in: ./cloud-benchmark/{csv_filename}") + + +# --------------------------------------------------------------------------- +# Argument parsing +# --------------------------------------------------------------------------- + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="OCI tenancy resource estimator") + + parser.add_argument( + "--profile", + default=os.environ.get("OCI_PROFILE", "DEFAULT"), + help="OCI config file profile name (default: DEFAULT, env: OCI_PROFILE).", + ) + parser.add_argument( + "--tenancy-id", + default=os.environ.get("OCI_TENANCY_OCID"), + help="Override tenancy OCID (default: from config, env: OCI_TENANCY_OCID).", + ) + parser.add_argument( + "--regions", + default=os.environ.get("OCI_REGIONS"), + help="Comma-separated list of regions to scan (default: all subscribed, env: OCI_REGIONS).", + ) + parser.add_argument( + "--skip-compartments", + default=os.environ.get("OCI_SKIP_COMPARTMENTS"), + help="Comma-separated compartment OCIDs to exclude (env: OCI_SKIP_COMPARTMENTS).", + ) + parser.add_argument( + "--include-compartments", + default=os.environ.get("OCI_INCLUDE_COMPARTMENTS"), + help="Comma-separated compartment OCIDs to include exclusively (env: OCI_INCLUDE_COMPARTMENTS).", + ) + parser.add_argument( + "--threads", + type=int, + default=int(os.environ.get("OCI_THREADS", "5")), + help="Parallel (compartment, region) workers (default: 5, env: OCI_THREADS).", + ) + parser.add_argument( + "--api-delay", + type=float, + default=float(os.environ.get("OCI_API_DELAY", "0.05")), + help="Seconds to wait between API calls (default: 0.05, env: OCI_API_DELAY).", + ) + parser.add_argument( + "--max-retries", + type=int, + default=int(os.environ.get("OCI_MAX_RETRIES", "5")), + help="Maximum retry attempts per operation (default: 5, env: OCI_MAX_RETRIES).", + ) + parser.add_argument( + "--resume-file", + default=os.environ.get("OCI_RESUME_FILE", "oci_benchmark_progress.json"), + help="File to store/resume scan progress (default: oci_benchmark_progress.json, env: OCI_RESUME_FILE).", + ) + parser.add_argument( + "--dry-run", + action="store_true", + default=os.environ.get("OCI_DRY_RUN", "").lower() == "true", + help="Print what would be scanned without making API calls (env: OCI_DRY_RUN).", + ) + + parsed = parser.parse_args() + + if parsed.threads < 1 or parsed.threads > 50: + parser.error("--threads must be between 1 and 50") + if parsed.api_delay < 0 or parsed.api_delay > 10: + parser.error("--api-delay must be between 0 and 10") + if parsed.max_retries < 0 or parsed.max_retries > 20: + parser.error("--max-retries must be between 0 and 20") + + return parsed + + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + +def main() -> None: + global args, log # pylint: disable=W0603 + + args = parse_args() + log = setup_logging() + setup_signal_handlers() + + log.info("Starting OCI CSPM benchmark") + log.info(f" Profile: {args.profile}") + log.info(f" Threads: {args.threads}") + log.info(f" API delay: {args.api_delay}s") + log.info(f" Max retries: {args.max_retries}") + log.info(f" Resume file: {args.resume_file}") + log.info(f" Dry run: {args.dry_run}") + + # --- Authentication --- + try: + config, signer = build_config_and_signer() + except (oci.exceptions.ConfigFileNotFound, oci.exceptions.ProfileNotFound, oci.exceptions.InvalidConfig) as exc: + print(f"OCI authentication failed: {exc}") + print("Run 'oci setup config' to configure credentials, or set OCI_PROFILE.") + raise SystemExit(1) + + # Allow CLI override of tenancy OCID + if args.tenancy_id: + config["tenancy"] = args.tenancy_id + + handle = OCIHandle(config, signer) + + # --- Enumerate compartments --- + print("Enumerating compartments...") + try: + all_compartment_ids = handle.get_all_compartment_ids() + except Exception as exc: + print(f"Failed to enumerate compartments: {exc}") + raise SystemExit(1) + + # Apply compartment filters + if args.include_compartments: + include_set = {c.strip() for c in args.include_compartments.split(",")} + compartment_ids = [c for c in all_compartment_ids if c in include_set] + if not compartment_ids: + print("No compartments matched --include-compartments filter.") + raise SystemExit(1) + elif args.skip_compartments: + skip_set = {c.strip() for c in args.skip_compartments.split(",")} + compartment_ids = [c for c in all_compartment_ids if c not in skip_set] + else: + compartment_ids = all_compartment_ids + + log.info(f"Compartments to scan: {len(compartment_ids)}") + + # --- Enumerate regions --- + print("Enumerating subscribed regions...") + try: + if args.regions: + regions = [r.strip() for r in args.regions.split(",")] + else: + regions = handle.get_subscribed_regions() + except Exception as exc: + print(f"Failed to enumerate regions: {exc}") + raise SystemExit(1) + + log.info(f"Regions to scan: {len(regions)} — {', '.join(regions)}") + + # Build the full list of (compartment_id, region) pairs + all_pairs = [(c, r) for r in regions for c in compartment_ids] + total_pairs = len(all_pairs) + print(f"Scanning {len(compartment_ids)} compartments × {len(regions)} regions = {total_pairs} scan units") + + if args.dry_run: + for compartment_id, region in all_pairs: + print(f" [dry-run] {handle.compartment_name(compartment_id)} / {region}") + return + + # --- Progress tracking --- + progress = ProgressTracker(args.resume_file) + progress.total_pairs = total_pairs + if progress.start_time is None: + progress.start_time = datetime.now(timezone.utc).isoformat() + + # Skip pairs already completed in a previous run + pending_pairs = [(c, r) for c, r in all_pairs if not progress.should_skip(c, r)] + skipped = total_pairs - len(pending_pairs) + if skipped: + print(f" Skipping {skipped} already-completed pairs, {len(pending_pairs)} remaining") + + # Prepare the timestamped output filename once so partial + final writes use the same file + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + csv_filename = f"oci-benchmark-{timestamp}.csv" + + # --- Parallel scan --- + completed = 0 + interrupted = False + + executor = concurrent.futures.ThreadPoolExecutor(max_workers=args.threads) + try: + future_to_pair = { + executor.submit(scan_pair, handle, c, r, progress): (c, r) + for c, r in pending_pairs + } + + for future in concurrent.futures.as_completed(future_to_pair): + compartment_id, region = future_to_pair[future] + completed += 1 + try: + row = future.result() + if row is None: + continue + + with data_lock: + data.append(row) + + with totals_lock: + for key in totals: + if key not in ("tenancy_id", "compartment_id", "compartment_name", "region"): + totals[key] += row.get(key, 0) + + if completed % 50 == 0 or completed == len(pending_pairs): + with console_lock: + print(f" Progress: {completed}/{len(pending_pairs)} scan units complete") + + except Exception as exc: + if log: + log.error(f"Error scanning {compartment_id}/{region}: {exc}") + + except KeyboardInterrupt: + interrupted = True + print("\nScan interrupted. Cancelling queued work...") + finally: + # cancel_futures drops queued-but-not-started work; in-flight threads finish normally. + # This runs before __exit__ would have, so partial results are available immediately. + executor.shutdown(wait=True, cancel_futures=True) + + if interrupted: + if data: + print(f"Writing partial results ({len(data)} rows collected)...") + write_output(csv_filename) + progress.print_resume_guidance() + return + + # --- Clean completion --- + write_output(csv_filename) + + # Remove progress file only when there are no failures — failed pairs can be retried + if not progress.failed: + progress.remove() + log.info("Progress file cleaned up (all pairs completed successfully)") + else: + log.warning(f"{len(progress.failed)} pair(s) failed — progress file retained for resume") + progress.print_resume_guidance() + + +if __name__ == "__main__": + main() diff --git a/OCI/requirements.txt b/OCI/requirements.txt new file mode 100644 index 0000000..f5c7ea5 --- /dev/null +++ b/OCI/requirements.txt @@ -0,0 +1,2 @@ +oci>=2.126.0 +tabulate diff --git a/OCI/setup.cfg b/OCI/setup.cfg new file mode 100644 index 0000000..0d6c186 --- /dev/null +++ b/OCI/setup.cfg @@ -0,0 +1,6 @@ +[flake8] +max-line-length = 120 +max-complexity = 10 + +[pylint.MASTER] +disable=C0301,C0116,C0115,C0103 diff --git a/README.md b/README.md index 3ec13b0..928be1f 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ # CrowdStrike Cloud Resource Estimator -This multi-cloud resource auditing utility helps organizations calculate the size of their cloud deployments across AWS, Azure, and Google Cloud Platform. It's designed for **CrowdStrike CWP/Horizon licensing calculations** and cloud security posture management (CSPM) benchmarking. +This multi-cloud resource auditing utility helps organizations calculate the size of their cloud deployments across AWS, Azure, Google Cloud Platform, and Oracle Cloud Infrastructure. It's designed for **CrowdStrike CWP/Horizon licensing calculations** and cloud security posture management (CSPM) benchmarking. ## Security @@ -24,7 +24,7 @@ The Cloud Resource Estimator performs **read-only** scanning of your cloud infra ## Running an audit -The `benchmark.sh` entrypoint script helps you to perform sizing calculations for your cloud resources. It detects the cloud provider (AWS, Azure, or GCP) and downloads the necessary scripts to perform the calculation. You can also pass one or more cloud providers as arguments. +The `benchmark.sh` entrypoint script helps you to perform sizing calculations for your cloud resources. It detects the cloud provider (AWS, Azure, GCP, or OCI) and downloads the necessary scripts to perform the calculation. You can also pass one or more cloud providers as arguments. ## Configuration @@ -81,16 +81,46 @@ export AZURE_INCLUDE_SUBSCRIPTIONS="sub-id-3,sub-id-4" GCP supports performance tuning and filtering options including project filtering (with automatic sys-* project exclusion) and threading options. +### OCI Configuration + +| Variable | Default | Description | +| :--- | :--- | :--- | +| `OCI_PROFILE` | `DEFAULT` | OCI config file profile name | +| `OCI_TENANCY_OCID` | from config | Override tenancy OCID | +| `OCI_REGIONS` | all subscribed | Comma-separated list of regions to scan | +| `OCI_SKIP_COMPARTMENTS` | None | Comma-separated compartment OCIDs to exclude | +| `OCI_INCLUDE_COMPARTMENTS` | None | Comma-separated compartment OCIDs to scan (exclusive filter, takes full precedence) | +| `OCI_THREADS` | `5` | Number of parallel scan workers | +| `OCI_API_DELAY` | `0.05` | Seconds to wait between API calls | +| `OCI_MAX_RETRIES` | `5` | Maximum retry attempts for failed operations | +| `OCI_RESUME_FILE` | `oci_benchmark_progress.json` | Progress tracking file for resume support | +| `OCI_DRY_RUN` | `false` | Set to `true` to simulate without API calls | + +**Important**: `OCI_INCLUDE_COMPARTMENTS` takes **full precedence** — if set, `OCI_SKIP_COMPARTMENTS` is completely ignored. + +Example usage: + +```shell +# Skip specific compartments +export OCI_SKIP_COMPARTMENTS="ocid1.compartment.oc1..example1,ocid1.compartment.oc1..example2" + +# OR (use one or the other, not both) + +# Include only specific compartments +export OCI_INCLUDE_COMPARTMENTS="ocid1.compartment.oc1..example3" +``` + For complete configuration details, see the provider-specific README files: - **AWS**: See [AWS README](AWS/README.md) for detailed configuration options - **Azure**: See [Azure README](Azure/README.md) for subscription filtering and performance settings - **GCP**: See [GCP README](GCP/README.md) for project filtering (including sys-* project handling) and threading options +- **OCI**: See [OCI README](OCI/README.md) for compartment filtering, IAM policy setup, and threading options ## Usage ```shell -./benchmark.sh [aws|azure|gcp]... +./benchmark.sh [aws|azure|gcp|oci]... ``` Below are two different ways to execute the script. @@ -102,6 +132,7 @@ To execute the script in your environment using Cloud Shell, follow the appropri - [AWS](AWS/README.md) - [Azure](Azure/README.md) - [GCP](GCP/README.md) +- [OCI](OCI/README.md) ### In your Local Environment @@ -112,7 +143,7 @@ For those who prefer to run the script locally, or would like to run the script - Python 3 - pip - curl -- Appropriate cloud provider CLI ([AWS](https://aws.amazon.com/cli/), [Azure](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli), [GCP](https://cloud.google.com/sdk/docs/install)) +- Appropriate cloud provider CLI ([AWS](https://aws.amazon.com/cli/), [Azure](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli), [GCP](https://cloud.google.com/sdk/docs/install), [OCI](https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm)) #### Steps diff --git a/benchmark.sh b/benchmark.sh index 579e678..524bb1a 100755 --- a/benchmark.sh +++ b/benchmark.sh @@ -9,7 +9,7 @@ base_url="https://raw.githubusercontent.com/CrowdStrike/cloud-resource-estimator # Usage message usage() { echo """ - Usage: $0 [OPTIONS] [aws|azure|gcp]... + Usage: $0 [OPTIONS] [aws|azure|gcp|oci]... The script recognizes the following environment variables: @@ -42,6 +42,25 @@ usage() { export AZURE_SKIP_SUBSCRIPTIONS="sub-id-1,sub-id-2" OR export AZURE_INCLUDE_SUBSCRIPTIONS="sub-id-3,sub-id-4" + + OCI: + - OCI_PROFILE: OCI config file profile name (optional, default: DEFAULT) + - OCI_TENANCY_OCID: Override tenancy OCID (optional, default: from config file) + - OCI_REGIONS: Comma-separated list of regions to scan (optional, default: all subscribed) + - OCI_SKIP_COMPARTMENTS: Comma-separated compartment OCIDs to exclude from scanning (optional) + - OCI_INCLUDE_COMPARTMENTS: Comma-separated compartment OCIDs to scan exclusively (optional) + - OCI_THREADS: Number of parallel scan workers (optional, default: 5) + - OCI_API_DELAY: Seconds to wait between API calls (optional, default: 0.05) + - OCI_MAX_RETRIES: Maximum retry attempts for failed operations (optional, default: 5) + - OCI_RESUME_FILE: File to store/resume progress (optional, default: oci_benchmark_progress.json) + - OCI_DRY_RUN: Set to 'true' to simulate without API calls (optional) + + Note: OCI_INCLUDE_COMPARTMENTS takes full precedence. If set, OCI_SKIP_COMPARTMENTS is ignored. + + Example (use one or the other, not both): + export OCI_SKIP_COMPARTMENTS="ocid1.compartment.oc1..example1,ocid1.compartment.oc1..example2" + OR + export OCI_INCLUDE_COMPARTMENTS="ocid1.compartment.oc1..example3" """ } @@ -76,6 +95,10 @@ is_valid_cloud() { echo "GCP" return 0 ;; + oci) + echo "OCI" + return 0 + ;; *) return 1 ;; @@ -109,6 +132,18 @@ call_benchmark_script() { ;; GCP) ;; + OCI) + [[ -n $OCI_PROFILE ]] && args+=("--profile" "$OCI_PROFILE") + [[ -n $OCI_TENANCY_OCID ]] && args+=("--tenancy-id" "$OCI_TENANCY_OCID") + [[ -n $OCI_REGIONS ]] && args+=("--regions" "$OCI_REGIONS") + [[ -n $OCI_SKIP_COMPARTMENTS ]] && args+=("--skip-compartments" "$OCI_SKIP_COMPARTMENTS") + [[ -n $OCI_INCLUDE_COMPARTMENTS ]] && args+=("--include-compartments" "$OCI_INCLUDE_COMPARTMENTS") + [[ -n $OCI_THREADS ]] && args+=("--threads" "$OCI_THREADS") + [[ -n $OCI_API_DELAY ]] && args+=("--api-delay" "$OCI_API_DELAY") + [[ -n $OCI_MAX_RETRIES ]] && args+=("--max-retries" "$OCI_MAX_RETRIES") + [[ -n $OCI_RESUME_FILE ]] && args+=("--resume-file" "$OCI_RESUME_FILE") + [[ $OCI_DRY_RUN == "true" ]] && args+=("--dry-run") + ;; *) echo "Invalid cloud provider specified: $cloud" usage @@ -156,6 +191,26 @@ audit() { file="${cloud}_cspm_benchmark.py" curl -s -o "${file}" "${base_url}/${CLOUD}/${file}" ;; + OCI) + # Use local OCI script if available + if [ -f "../OCI/oci_cspm_benchmark.py" ]; then + echo "Using local OCI CSPM benchmark script..." + file="../OCI/oci_cspm_benchmark.py" + if [ -f "../OCI/requirements.txt" ]; then + python3 -m pip install --disable-pip-version-check -qq -r "../OCI/requirements.txt" + else + curl -s -o requirements.txt "${base_url}/OCI/requirements.txt" + python3 -m pip install --disable-pip-version-check -qq -r requirements.txt + fi + else + echo "Local OCI script not found, downloading from remote" + curl -s -o requirements.txt "${base_url}/OCI/requirements.txt" + echo "Installing python dependencies for communicating with OCI into (~/cloud-benchmark)" + python3 -m pip install --disable-pip-version-check -qq -r requirements.txt + file="oci_cspm_benchmark.py" + curl -s -o "${file}" "${base_url}/OCI/${file}" + fi + ;; *) echo "Unsupported cloud provider: $CLOUD" exit 1 @@ -210,6 +265,11 @@ if [ $# -eq 0 ]; then audit "GCP" found_provider=true fi + + if type oci >/dev/null 2>&1; then + audit "OCI" + found_provider=true + fi fi if [ "$found_provider" = false ]; then