Terraform template for infra setup for starting a migration#3867
Terraform template for infra setup for starting a migration#3867shreyakhajanchi wants to merge 7 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3867 +/- ##
============================================
+ Coverage 53.73% 63.31% +9.58%
+ Complexity 6743 2315 -4428
============================================
Files 1087 514 -573
Lines 66794 29929 -36865
Branches 7478 3298 -4180
============================================
- Hits 35890 18950 -16940
+ Misses 28477 10017 -18460
+ Partials 2427 962 -1465
🚀 New features to boost your workflow:
|
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request provides a robust Terraform-based infrastructure automation solution for setting up complex sharded database environments. It streamlines the provisioning of Cloud SQL and Spanner resources, handles schema distribution, and generates the necessary configuration artifacts for migration pipelines. The solution is designed to be resilient against common cloud API limitations and ensures a clean lifecycle management of resources. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a Terraform-based infrastructure setup for source database (Cloud SQL) and target Spanner database migration testing, including automated schema importing and cleanup scripts. The review feedback highlights critical Terraform evaluation issues where resources with conditional counts (such as the private network and database password) are indexed directly, which will cause errors when they are not created. Additionally, the feedback suggests improving configuration flexibility and preventing deployment failures by dynamically defaulting the database version and Spanner region when they are not explicitly provided.
| fi | ||
|
|
||
| # Add a random jitter of 1-5 seconds to prevent thundering herds across parallel shards | ||
| jitter=$(( RANDOM % 5 + 1 )) |
There was a problem hiding this comment.
This will run sequentially, what are the scenarios where you would need this jitter ?
There was a problem hiding this comment.
While imports within a single physical instance are sequential, Terraform runs multiple physical instances in parallel (e.g., terraform apply -parallelism=100). If those 100 parallel bash scripts hit a 429 Rate Limit simultaneously and sleep for a fixed duration, they will all wake up at the exact same millisecond and hammer the API again, causing a 'thundering herd'. The random jitter staggers the sleep times across the parallel physical instances.
| # SOURCE DATABASE (Cloud SQL) | ||
| # ------------------------------------------------------------------------------ | ||
| database_provider = "MYSQL" # MYSQL or POSTGRES | ||
| database_version = "8_0" # MySQL: 8_0, 5_7 | Postgres: 14, 15, 16 |
There was a problem hiding this comment.
Are these the default values ?
| # Generate the Shard Config json file matching the Shard.java model properties | ||
| locals { | ||
| shards = [ | ||
| for idx in range(var.physical_shards_count * var.logical_shards_count) : { |
There was a problem hiding this comment.
There should be some logic for this in the sharded migration terraform setup a couple years back. We should look for opportunities to re-use some of that stuff to generate these files.
There was a problem hiding this comment.
we take it as input in the migration pipelines , hence there is no code for creating this file - checked live and reverse samples
| one([for ip in google_sql_database_instance.instances[tostring(floor(idx / var.logical_shards_count))].ip_address : ip.ip_address if ip.type == "PRIVATE"]), | ||
| google_sql_database_instance.instances[tostring(floor(idx / var.logical_shards_count))].ip_address[0].ip_address | ||
| ), | ||
| "127.0.0.1" |
There was a problem hiding this comment.
i'm not sure if this would ever be localhost. Can you check in what case this would be required ?
There was a problem hiding this comment.
During terraform plan, the Cloud SQL instances don't exist yet, so their ip_address lists are empty. Terraform's strict type evaluator crashes if it tries to index into an empty list. Providing a fallback string safely satisfies the type-checker during the planning phase.
| ## Step-by-Step Guide to Deploying | ||
|
|
||
| ### Step 1: Prepare Your Local Database Structure | ||
| Create a local SQL file named `schema.sql` in this folder. Define the tables and columns you want to load into your source databases. For example: |
There was a problem hiding this comment.
not immediately relevant, but it will be interesting to see how this extends to schema less databases.
There was a problem hiding this comment.
yeah it would be interesting in case of data generation , but for infra setup we could just skip this step in those cases
2 participants
Description
This PR introduces a complete Terraform module that automates the end-to-end provisioning of sharded Cloud SQL databases, target Spanner instances, and the required dynamic configurations necessary for running the CDC Data Generator.
This has been tested against a Ck scale data set of - 128* 8 shards
✨ Key Features
logical_shards_countvariable.import_schema.sh):schema.sqlto GCS and securely imports it into every logical database.429 RESOURCE_EXHAUSTED) or IAM eventual consistency errors.shard_config.jsonandbulk_shard_config.jsonartifacts containing resolved IP addresses, connection strings, and Secret Manager references required by the Dataflow pipeline templates.teardown_vpc_peering.shanddelete_spanner_backups.shhooks that run duringterraform destroyto bypass Google Cloud's native deletion locks and ensure a clean environment teardown.📝 Documentation & Templates
README.mddetailing architecture, prerequisites, and execution commands.terraform_simple.tfvarsandterraform_advanced.tfvarstemplates to provide standardized "Getting Started" configurations.