You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "lidavidm (via GitHub)" <gi...@apache.org> on 2023/12/21 18:47:29 UTC
[PR] fix(c/driver/postgresql): fix ingest with multiple batches [arrow-adbc]
lidavidm opened a new pull request, #1393:
URL: https://github.com/apache/arrow-adbc/pull/1393
Fixes #1310.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/postgresql): fix ingest with multiple batches [arrow-adbc]
Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on code in PR #1393:
URL: https://github.com/apache/arrow-adbc/pull/1393#discussion_r1434591341
##########
c/driver/postgresql/statement.cc:
##########
@@ -611,25 +605,26 @@ struct BindStream {
return ADBC_STATUS_IO;
}
- if (PQputCopyEnd(conn, NULL) <= 0) {
- SetError(error, "Error message returned by PQputCopyEnd: %s",
- PQerrorMessage(conn));
- return ADBC_STATUS_IO;
- }
+ if (rows_affected) *rows_affected += array->length;
Review Comment:
Ah perfect. Ignore what I said then - I just missed it because I didn't see it in the diff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/postgresql): fix ingest with multiple batches [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #1393:
URL: https://github.com/apache/arrow-adbc/pull/1393#discussion_r1434590685
##########
c/driver/postgresql/statement.cc:
##########
@@ -611,25 +605,26 @@ struct BindStream {
return ADBC_STATUS_IO;
}
- if (PQputCopyEnd(conn, NULL) <= 0) {
- SetError(error, "Error message returned by PQputCopyEnd: %s",
- PQerrorMessage(conn));
- return ADBC_STATUS_IO;
- }
+ if (rows_affected) *rows_affected += array->length;
Review Comment:
Sorry, what do you mean? We're still calling PQputCopyData for each batch, we're just not calling PQputCopyEnd until we've written *all* the batches
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/postgresql): fix ingest with multiple batches [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #1393:
URL: https://github.com/apache/arrow-adbc/pull/1393#issuecomment-1866798330
CC @WillAyd do the COPY changes here look reasonable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/postgresql): fix ingest with multiple batches [arrow-adbc]
Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on code in PR #1393:
URL: https://github.com/apache/arrow-adbc/pull/1393#discussion_r1434574632
##########
c/driver/postgresql/statement.cc:
##########
@@ -611,25 +605,26 @@ struct BindStream {
return ADBC_STATUS_IO;
}
- if (PQputCopyEnd(conn, NULL) <= 0) {
- SetError(error, "Error message returned by PQputCopyEnd: %s",
- PQerrorMessage(conn));
- return ADBC_STATUS_IO;
- }
+ if (rows_affected) *rows_affected += array->length;
Review Comment:
What we might want to do (certainly fine as a follow up) is send this data to the server at the end of each loop iteration using `PQputCopyData`:
https://www.postgresql.org/docs/current/libpq-copy.html#LIBPQ-COPY-SEND
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/postgresql): fix ingest with multiple batches [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #1393:
URL: https://github.com/apache/arrow-adbc/pull/1393#discussion_r1434590959
##########
c/driver/postgresql/statement.cc:
##########
@@ -611,25 +605,26 @@ struct BindStream {
return ADBC_STATUS_IO;
}
- if (PQputCopyEnd(conn, NULL) <= 0) {
- SetError(error, "Error message returned by PQputCopyEnd: %s",
- PQerrorMessage(conn));
- return ADBC_STATUS_IO;
- }
+ if (rows_affected) *rows_affected += array->length;
Review Comment:
(We could also be smarter here and flush to the server at a certain amount of data instead of each batch to smooth things out between large/small batches)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/postgresql): fix ingest with multiple batches [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm merged PR #1393:
URL: https://github.com/apache/arrow-adbc/pull/1393
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org