-
Notifications
You must be signed in to change notification settings - Fork 43
Add non-blocking async COPY FROM support (pg_putcopydata_async, pg_putcopyend_async, pg_flush) #176
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from 5 commits
940df53
d6506ff
f0b0a3d
a20aa96
2a1308c
8a03679
cc9027f
311682b
c7691b8
633e496
379e0a5
bc0b83d
34e82a1
6dfeeb5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -158,8 +158,11 @@ use 5.008001; | |
| DBD::Pg::db->install_method('pg_getcopydata'); | ||
| DBD::Pg::db->install_method('pg_getcopydata_async'); | ||
| DBD::Pg::db->install_method('pg_notifies'); | ||
| DBD::Pg::db->install_method('pg_flush'); | ||
| DBD::Pg::db->install_method('pg_putcopydata'); | ||
| DBD::Pg::db->install_method('pg_putcopydata_async'); | ||
| DBD::Pg::db->install_method('pg_putcopyend'); | ||
| DBD::Pg::db->install_method('pg_putcopyend_async'); | ||
| DBD::Pg::db->install_method('pg_ping'); | ||
| DBD::Pg::db->install_method('pg_putline'); | ||
| DBD::Pg::db->install_method('pg_ready'); | ||
|
|
@@ -4424,12 +4427,106 @@ the COPY statement. Returns a 1 on successful input. Examples: | |
| $dbh->pg_putcopydata("Anchovies~6\n"); | ||
| $dbh->pg_putcopyend(); | ||
|
|
||
| =head3 B<pg_putcopydata_async> | ||
|
|
||
| Non-blocking version of pg_putcopydata for use by async libraries. When called, the | ||
| connection is switched into non-blocking mode (via C<PQsetnonblocking>), which is safe | ||
| because no other operations are permitted on this connection during a COPY. The | ||
| non-blocking mode is automatically restored to blocking when L</pg_putcopyend_async> | ||
| completes. | ||
|
|
||
| Note: the connection performing the COPY is restricted to COPY operations until | ||
| the COPY ends. However, the non-blocking methods allow the event loop to service | ||
| other connections and tasks between calls, which is the primary benefit over the | ||
| blocking variants. | ||
|
|
||
| Return values match C<PQputCopyData>: | ||
|
|
||
| 1 = data queued successfully (caller should call L</pg_flush> to send) | ||
| 0 = output buffer full; caller should poll the socket for | ||
| write-readiness and retry the same pg_putcopydata_async call | ||
| -1 = error | ||
|
|
||
| After a successful return of 1, call L</pg_flush> to push the data to the | ||
| server. If C<pg_flush> returns 1 (data pending), poll the socket for | ||
| write-readiness and call C<pg_flush> again. | ||
|
|
||
| Example usage: | ||
|
|
||
| ## Simple usage (flush after each row): | ||
| use IO::Select; | ||
| $dbh->do("COPY mytable(id, flavor, slices) FROM STDIN"); | ||
| for my $row ("123\tPepperoni\t3\n", "314\tMushroom\t8\n") { | ||
| $dbh->pg_putcopydata_async($row); | ||
| while ($dbh->pg_flush()) { | ||
|
turnstep marked this conversation as resolved.
Outdated
|
||
| IO::Select->new($dbh->{pg_socket})->can_write(); | ||
|
turnstep marked this conversation as resolved.
Outdated
|
||
| } | ||
| } | ||
| $dbh->pg_putcopyend(); | ||
|
|
||
| ## Robust usage (handles buffer-full and async end): | ||
| use IO::Select; | ||
| my $sel = IO::Select->new($dbh->{pg_socket}); | ||
|
|
||
| ## Column list is optional but recommended. Default format is | ||
| ## tab-delimited text with newline row terminators, matching | ||
| ## PostgreSQL's COPY text format. Use COPY ... WITH (FORMAT csv) | ||
| ## for CSV data, or WITH (DELIMITER '|') for custom delimiters. | ||
| $dbh->do("COPY mytable(id, flavor, slices) FROM STDIN"); | ||
| my @data = ("123\tPepperoni\t3\n", "314\tMushroom\t8\n", | ||
| "6\tAnchovies\t100\n"); | ||
| for my $row (@data) { | ||
| my $status = $dbh->pg_putcopydata_async($row); | ||
| while ($status == 0) { # buffer full | ||
| $sel->can_write(); | ||
| $status = $dbh->pg_putcopydata_async($row); | ||
| } | ||
| die "COPY error" if $status == -1; | ||
| while ($dbh->pg_flush()) { # push to server | ||
| $sel->can_write(); | ||
| } | ||
| } | ||
|
|
||
| ## Non-blocking end: poll until server confirms | ||
| while ((my $end = $dbh->pg_putcopyend_async()) == 0) { | ||
| $sel->can_read(); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Er...can_read or can_write? Can buffer be full here?
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You're right — |
||
| } | ||
| die "COPY end error" if $end == -1; | ||
|
|
||
| =head3 B<pg_putcopyend> | ||
|
|
||
| When you are finished with pg_putcopydata, call pg_putcopyend to let the server know | ||
| that you are done, and it will return to a normal, non-COPY state. Returns a 1 on | ||
| When you are finished with pg_putcopydata, call pg_putcopyend to let the server know | ||
| that you are done, and it will return to a normal, non-COPY state. Returns a 1 on | ||
| success. This method will fail if called when not in COPY IN mode. | ||
|
|
||
| =head3 B<pg_putcopyend_async> | ||
|
|
||
| Non-blocking version of pg_putcopyend for use by async libraries. Sends the COPY | ||
| end marker and attempts to collect the server result without blocking. Designed to | ||
| be called in a poll loop. | ||
|
|
||
| Return values: | ||
|
|
||
| 1 = COPY completed successfully, connection is back in normal blocking mode | ||
| 0 = not ready yet; caller should poll the socket for readiness, then call | ||
| pg_putcopyend_async again | ||
| -1 = error | ||
|
|
||
| After pg_putcopyend_async returns 1, the connection is back in blocking mode and | ||
| normal queries can be issued. | ||
|
|
||
| =head3 B<pg_flush> | ||
|
|
||
| Flushes the libpq output buffer. Wraps C<PQflush> directly. Used after | ||
| L</pg_putcopydata_async> returns 1 to push queued data to the server. | ||
|
|
||
| Return values match C<PQflush>: | ||
|
|
||
| 0 = all data flushed successfully | ||
|
esabol marked this conversation as resolved.
|
||
| 1 = data still pending; caller should poll the socket for write-readiness | ||
| and call pg_flush again | ||
| -1 = error | ||
|
|
||
| =head2 Postgres limits | ||
|
|
||
| For convenience, DBD::Pg can export certain constants representing the limits of | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.