Skip to content

Call WalDataToDDLEvent once per event in the processor chain#970

Open
kvch wants to merge 3 commits into
mainfrom
perf/ddl-event-memoisation
Open

Call WalDataToDDLEvent once per event in the processor chain#970
kvch wants to merge 3 commits into
mainfrom
perf/ddl-event-memoisation

Conversation

@kvch

@kvch kvch commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Description

Every processor in the chain (filter, injector, transformer, and the postgres DDL adapter) independently called WalDataToDDLEvent on the same event, and each call ran json.Unmarshal([]byte(d.Content)).
From now on parsing result is cached. So there is no need for the expensive json.Unmarshals.

Cachingthe parse on wal.Data reduces 5 json.Unmarshal passes per event down to 1 (+4 cached hits), cutting time and allocations ~74%. The gain scales with how many processors in the chain call WalDataToDDLEvent on the same event.

Metric main (baseline) this pr Δ vs main
Time 6.669µ 1.747µ −73.80%
Bytes/op 5.104Ki 1.321Ki −74.12%
Allocs/op 25.000 6.000 −76.00%

Type of Change

Please select the relevant option(s):

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • 📚 Documentation update
  • 🔧 Refactoring (no functional changes)
  • ⚡ Performance improvement
  • 🧪 Test coverage improvement
  • 🔨 Build/CI changes
  • 🧹 Code cleanup

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • All existing tests pass

Checklist

  • Code follows project style guidelines
  • Self-review completed
  • Code is well-commented
  • Documentation updated where necessary

@kvch kvch force-pushed the perf/ddl-event-memoisation branch from 9948ebe to b902ac2 Compare July 7, 2026 15:13
@kvch kvch requested a review from exekias July 7, 2026 16:05
@kvch kvch marked this pull request as draft July 7, 2026 16:12
@kvch kvch removed the request for review from exekias July 7, 2026 16:12
@kvch kvch marked this pull request as ready for review July 7, 2026 16:51
@kvch kvch requested a review from exekias July 7, 2026 16:51
@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown

Merging this branch will increase overall coverage

Impacted Packages Coverage Δ 🤖
github.com/xataio/pgstream/pkg/snapshot/generator/postgres/data 78.75% (ø)
github.com/xataio/pgstream/pkg/wal 72.31% (+0.65%) 👍
github.com/xataio/pgstream/pkg/wal/listener/kafka 69.57% (ø)
github.com/xataio/pgstream/pkg/wal/processor/filter 94.00% (ø)

Coverage by file

Changed files (no unit tests)

Changed File Coverage Δ Total Covered Missed 🤖
github.com/xataio/pgstream/pkg/wal/wal_data.go 0.00% (ø) 11 0 11
github.com/xataio/pgstream/pkg/wal/wal_ddl.go 71.15% (+1.77%) 52 (+3) 37 (+3) 15 👍

Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code.

Changed unit test files

  • github.com/xataio/pgstream/pkg/snapshot/generator/postgres/data/pg_snapshot_generator_test.go
  • github.com/xataio/pgstream/pkg/wal/listener/kafka/wal_kafka_reader_test.go
  • github.com/xataio/pgstream/pkg/wal/processor/filter/wal_filter_test.go
  • github.com/xataio/pgstream/pkg/wal/wal_ddl_test.go

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant