feat(Spanner): integrate SourceConfigParser to centralize shard configuration loading for SourceDbToSpanner pipelines. by pratickchokhani · Pull Request #3854 · GoogleCloudPlatform/DataflowTemplates

pratickchokhani · 2026-05-25T07:06:55Z

Implemented centralize shard configuration loading for SourceDbToSpanner
Added integration tests for Astra DB
Updated integration tests across the template.

codecov · 2026-05-25T07:19:03Z

Codecov Report

❌ Patch coverage is 85.41667% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.09%. Comparing base (46df425) to head (7007b5f).
⚠️ Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
...loud/teleport/v2/templates/PipelineController.java	0.00%	7 Missing ⚠️
...ource/jdbc/AbstractJdbcSrcToSpSourceConnector.java	0.00%	4 Missing ⚠️
...cloud/teleport/v2/templates/SourceDbToSpanner.java	83.33%	1 Missing and 1 partial ⚠️
...rce/cassandra/CassandraSrcToSpSourceConnector.java	50.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3854      +/-   ##
============================================
- Coverage     56.10%   56.09%   -0.01%     
+ Complexity     7319     7310       -9     
============================================
  Files          1126     1126              
  Lines         68550    68555       +5     
  Branches       7753     7754       +1     
============================================
+ Hits          38457    38459       +2     
+ Misses        27610    27609       -1     
- Partials       2483     2487       +4

Components	Coverage Δ
spanner-templates	`87.63% <83.72%> (+0.07%)`	⬆️
spanner-import-export	`68.86% <ø> (-0.17%)`	⬇️
spanner-live-forward-migration	`89.47% <ø> (+0.20%)`	⬆️
spanner-live-reverse-replication	`83.60% <ø> (+0.16%)`	⬆️
spanner-bulk-migration	`92.33% <83.72%> (+0.10%)`	⬆️
gcs-spanner-dv	`88.78% <ø> (+0.23%)`	⬆️

Files with missing lines	Coverage Δ
...eam/it/gcp/cloudsql/CloudSqlShardOrchestrator.java	`85.83% <100.00%> (-0.52%)`	⬇️
...ud/teleport/v2/options/OptionsToConfigBuilder.java	`92.94% <100.00%> (-0.40%)`	⬇️
...ud/teleport/v2/source/ISrcToSpSourceConnector.java	`100.00% <ø> (ø)`
...cassandra/iowrapper/CassandraIOWrapperFactory.java	`100.00% <100.00%> (ø)`
.../cassandra/iowrapper/CassandraIOWrapperHelper.java	`100.00% <100.00%> (ø)`
...der/io/cassandra/iowrapper/CassandraIoWrapper.java	`100.00% <ø> (ø)`
...t/v2/source/jdbc/ShardedJdbcDbConfigContainer.java	`100.00% <100.00%> (ø)`
...urce/jdbc/SingleInstanceJdbcDbConfigContainer.java	`100.00% <100.00%> (ø)`
...t/v2/source/mysql/MySqlSrcToSpSourceConnector.java	`98.85% <100.00%> (-0.02%)`	⬇️
...ource/postgres/PostgresSrcToSpSourceConnector.java	`98.92% <100.00%> (-0.02%)`	⬇️
... and 4 more

... and 12 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

bharadwaj-aditya · 2026-08-03T06:30:01Z

+   * @throws IllegalArgumentException if the provided options are invalid for the pipeline.
+   */
+  @VisibleForTesting
+  static void validateOptions(


couple of points.

This looks very very similar to the previous method, why is this created as a different method here ? It would be better to just tweak the existing method for this extra parameter to keep the amount of change less ?

As this is PG specific code, it should ideally go into the source directory and be exposed via the src connector. Please check if that is feasible here.

Have added the TO-DO and moved the method.

bharadwaj-aditya · 2026-08-03T06:31:43Z

            () ->
                CassandraIOWrapperHelper.buildDataSource(
-                    TEST_BUCKET_CASSANDRA_CONFIG_CONF,
+                    null,


It doesn't seem like there is a test with this TEST_BUCKET_CASSANDRA_CONFIG_CONF config ? Can you double check ?

bharadwaj-aditya · 2026-08-03T06:32:26Z

-                  .oss()
-                  .numPartitions())
-          .isEqualTo(42);
-      assertThrows(


Is there an alternate test for invalid config url ?

bharadwaj-aditya · 2026-08-03T06:33:31Z

    ResourceManagerUtils.cleanResources(spannerResourceManager, mySQLResourceManager);
  }

-  /**


is this comment not relevant anymore ?

Yes, SHARD Id is populated. See expected data at line 106

bharadwaj-aditya · 2026-08-03T06:34:16Z

+    shard.setLogicalShardId("Shard1");
+    shard.setUser(jdbcResourceManager.getUsername());
+    shard.setPassword(jdbcResourceManager.getPassword());
+    if (jdbcResourceManager instanceof PostgresResourceManager pgRm) {


please check if the test cleanup scripts will handle these directories

We are not creating any new folder here. This uses the run-id folder used by all other files.
protected String getGcsPath(String artifactId) {
return ArtifactUtils.getFullGcsPath(
artifactBucketName, getClass().getSimpleName(), gcsClient.runId(), artifactId);
}

bharadwaj-aditya · 2026-08-03T06:35:38Z

+
+    JdbcShardConfig jdbcConfig = new JdbcShardConfig();
+    Shard shard1 = new Shard();
+    shard1.setNamespace("public");


do we have a test for non-public namespaces as well ?

This current test is testing non-public name space (line:155),

At line:166 only public name space is getting tested.

…guration loading for SourceDbToSpanner pipelines

…nectionConfig instead of pipeline options for improved configuration handling.

…straDbToSpanner

…es and update JDBC migration logic to utilize shard configurations

…object directly to builder methods

bharadwaj-aditya

LGTM

…e standardized shard config GCS file

pull-request-size Bot added the size/XXL label May 25, 2026

pratickchokhani added the addition New feature or request label May 25, 2026

pratickchokhani force-pushed the shard-config-bulk branch from 453a9f9 to b7b20a3 Compare June 22, 2026 05:50

pull-request-size Bot added size/L and removed size/XXL labels Jun 22, 2026

pratickchokhani force-pushed the shard-config-bulk branch 6 times, most recently from a19cf9e to 581ffd3 Compare June 29, 2026 07:25

pratickchokhani force-pushed the shard-config-bulk branch 4 times, most recently from 0da7475 to e13bdd2 Compare June 30, 2026 14:13

pull-request-size Bot added size/XL and removed size/L labels Jul 1, 2026

pratickchokhani force-pushed the shard-config-bulk branch from c90217f to c58f5cb Compare July 1, 2026 09:19

pratickchokhani commented Jul 1, 2026

View reviewed changes

Comment thread ...ogle/cloud/teleport/v2/source/cassandra/reader/io/cassandra/iowrapper/AstraDbDataSource.java Outdated

pull-request-size Bot added size/XXL and removed size/XL labels Jul 2, 2026

pratickchokhani force-pushed the shard-config-bulk branch 3 times, most recently from 41f2355 to 461938b Compare July 2, 2026 10:46

pratickchokhani marked this pull request as ready for review July 2, 2026 10:55

pratickchokhani requested a review from a team as a code owner July 2, 2026 10:55

pratickchokhani requested review from jsuhani-2026 and sm745052 and removed request for sm745052 July 2, 2026 10:55

pratickchokhani force-pushed the shard-config-bulk branch 6 times, most recently from c7f9560 to 2649e0a Compare July 28, 2026 08:38

pratickchokhani requested a review from bharadwaj-aditya July 28, 2026 09:21

bharadwaj-aditya reviewed Aug 3, 2026

View reviewed changes

pratickchokhani and others added 14 commits August 4, 2026 15:29

feat(Spanner): integrate SourceConfigParser to centralize shard confi…

3f3f46f

…guration loading for SourceDbToSpanner pipelines

test(Spanner): Integration test fix.

2e6aea6

feat(Spanner): refactor CassandraIOWrapperFactory to accept SourceCon…

f386644

…nectionConfig instead of pipeline options for improved configuration handling.

fix: resolve Astra token parsing issue and add integration test for A…

8fc86c1

…straDbToSpanner

Nit fixes.

d66cc81

Cassandra to use SourceConnectionConfig

8f7b482

Extend unit test coverage

c87e9a7

Astra DB Integration test update

77224bb

feat: updated the documentation to reflect the config changes.

3746052

refactor: propagate SourceConnectionConfig through connector interfac…

d5ae55b

…es and update JDBC migration logic to utilize shard configurations

refactor: simplify Shard configuration handling by passing the Shard …

441f1b7

…object directly to builder methods

fix(Spanner): removed stale validations.

458e675

PR fixes

5e045dc

Readme fix.

157262e

pratickchokhani force-pushed the shard-config-bulk branch from 2649e0a to 157262e Compare August 4, 2026 10:00

pratickchokhani requested a review from bharadwaj-aditya August 4, 2026 11:04

minor refactoring.

2ddbd87

pratickchokhani force-pushed the shard-config-bulk branch from f2d0565 to 2ddbd87 Compare August 4, 2026 11:07

bharadwaj-aditya previously approved these changes Aug 5, 2026

View reviewed changes

pratickchokhani dismissed bharadwaj-aditya’s stale review via 23b9871 August 6, 2026 08:13

refactor: unify shard configuration format and update E2E tests to us…

7007b5f

…e standardized shard config GCS file

pratickchokhani force-pushed the shard-config-bulk branch from 23b9871 to 7007b5f Compare August 6, 2026 08:17

Navigation Menu

Uh oh!

feat(Spanner): integrate SourceConfigParser to centralize shard configuration loading for SourceDbToSpanner pipelines. - #3854

feat(Spanner): integrate SourceConfigParser to centralize shard configuration loading for SourceDbToSpanner pipelines.#3854
pratickchokhani wants to merge 16 commits into
GoogleCloudPlatform:mainfrom
pratickchokhani:shard-config-bulk

pratickchokhani commented May 25, 2026 •

edited

Loading

Uh oh!

codecov Bot commented May 25, 2026 •

edited

Loading

Uh oh!

Uh oh!

bharadwaj-aditya Aug 3, 2026

Uh oh!

pratickchokhani Aug 4, 2026

Uh oh!

bharadwaj-aditya Aug 3, 2026

Uh oh!

bharadwaj-aditya Aug 3, 2026

Uh oh!

bharadwaj-aditya Aug 3, 2026

Uh oh!

pratickchokhani Aug 3, 2026

Uh oh!

bharadwaj-aditya Aug 3, 2026

Uh oh!

pratickchokhani Aug 3, 2026

Uh oh!

bharadwaj-aditya Aug 3, 2026

Uh oh!

pratickchokhani Aug 3, 2026

Uh oh!

bharadwaj-aditya left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

pratickchokhani commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bharadwaj-aditya left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pratickchokhani commented May 25, 2026 •

edited

Loading

codecov Bot commented May 25, 2026 •

edited

Loading