Add pipeline mode support for PostgreSQL 14+ (RFC)#182
Open
jjn1056 wants to merge 19 commits into
Open
Conversation
Add DBDPG_HAS_PIPELINE compile-time define for PostgreSQL 14+, trace macros for pipeline libpq functions, pipeline tracking field in imp_dbh_st, and function declarations for all pipeline methods. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Create t/11pipeline.t with PG14+ version detection. Implement pg_pipeline_status wrapping PQpipelineStatus — returns 0 (off), 1 (on), or 2 (aborted). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wrap PQenterPipelineMode and PQexitPipelineMode as $dbh methods. Both are idempotent, croak if compiled without PG14+ support. Tests cover enter, exit, idempotency, and post-pipeline query sanity. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
pg_pipeline_sync wraps PQpipelineSync — sends a sync point that delimits an implicit transaction boundary. pg_getresult wraps PQgetResult — returns a hashref with status, error, ntuples, nfields, cmdtuples, and rows (for SELECT results), or undef when no more results are available. This enables manual pipeline result collection. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wrap PQsendQueryParams for pipeline query submission with $1/$2 parameter syntax. Tests cover INSERT pipeline with data verification, SELECT with row data extraction, and PIPELINE_ABORTED error flow with recovery after sync point. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…g_flush pg_send_prepare wraps PQsendPrepare for pipelining PREPARE commands. pg_send_query_prepared wraps PQsendQueryPrepared for executing prepared statements in pipeline mode. pg_send_flush_request wraps PQsendFlushRequest to retrieve results without a sync point. pg_flush wraps PQflush for manual output buffer management. Tests cover prepared statement pipeline (PREPARE + 2 EXECUTEs), and flush request to retrieve results without a sync point. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reject pg_putcopydata/pg_putcopyend in pipeline mode with clear croak. Test exit_pipeline_mode failure with unconsumed results. Add multiple pipeline sync cycles without exiting. Test 100-INSERT large pipeline with result draining. Add dbd_pg_test_pipeline to cleanup list. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document all pipeline methods in Pg.pm POD with examples and return values. Fix spellcheck (nfields, ntuples, PQpipelineStatus), add missing TRACE macros in pg_db_getresult, update MANIFEST and README.dev for t/11pipeline.t. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Initialize imp_dbh->pipeline = 0 in dbd_db_login6 alongside other fields. Remove unused D_imp_dbh from pg_getresult XS binding. Move variable declarations inside #ifdef DBDPG_HAS_PIPELINE in send functions to avoid unused variable warnings on PG < 14 builds. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add standalone pg_flush POD section. Add pipeline mode to Changes for v3.21.0. Document blocking mode deadlock risk for large pipelines. Add pg_getcopydata pipeline guard matching putcopydata/putcopyend. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Test SELECT returning zero rows (no rows key in result), NULL/undef parameter values, pg_pipeline_status == 2 during aborted state, pg_flush return value, pg_send_query_params without params argument, and two consecutive pipeline syncs without draining between them. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use single-line entry format that the Changes file parser in t/00_release.t can handle. Keep entry under current version. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add Perl-side ? to $N conversion in pg_send_query_params so that DBI-standard ? placeholders work alongside libpq-native $1/$2 syntax. This enables compatibility with SQL generators like DBIx::Class that produce ? placeholders exclusively. The conversion is a simple regex that detects ? when no $N placeholders are present. The XS functions are renamed to _pg_send_query_params and _pg_send_query_prepared with Perl wrappers handling the conversion. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add pg_convert_placeholders() static function in dbdimp.c that converts DBI ? placeholders to libpq $N style while properly skipping single-quoted strings, double-quoted identifiers, dollar-quoted strings, and SQL comments. Follows the same skip patterns as pg_st_split_statement for correctness. Remove the Perl-side regex wrapper and revert XS function names back to pg_send_query_params/pg_send_query_prepared. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Test that ? inside single-quoted strings, double-quoted identifiers, dollar-quoted strings, line comments, and block comments is NOT converted. Test multiple ? sequence ($1,$2,$3) and $1-present bypass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pass SV *dbh to pg_convert_placeholders so TRACE_PQPARAMETERSTATUS works correctly. Rename seg_start to segment_start to avoid spellcheck hit on 'seg'. Add edge case tests for ? inside quoted strings, comments, dollar-quoting, multiple ? sequencing, and $1 bypass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
esabol
reviewed
Apr 8, 2026
esabol
reviewed
Apr 8, 2026
The 3.20.0 version has already been released, so the pipeline mode entry doesn't belong there. The entry will be added under the next version when a release is prepared. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Explain that pg_getresult returns undef as a delimiter between each query's results, and that callers must consume each separator. This addresses reviewer feedback that the undef calls seemed unclear. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Working reference implementation of the pipeline mode API proposed in #181. See the issue for API rationale, open questions, and design alternatives.
Summary
$dbhmethods wrapping libpq's pipeline API?and$1/$2placeholder syntax supported via C-level converter#if PGLIBVERSION >= 140000) — builds cleanly against older libpq, methods croak if calledAUTHOR_TESTING=1 RELEASE_TESTING=1)This is an RFC to get a stake in the ground and start a conversation. The pipeline mode TODO item has been in the project for a while, and the public API here is a proposal. The code works and is well-tested, but we'd like maintainer input on the API design and approach before considering it final. See #181 for the full discussion.
Test plan
?placeholder conversion: basic, multiple, inside quotes/comments/dollar-quoting$1passthrough when already present🤖 Generated with Claude Code