You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "WillAyd (via GitHub)" <gi...@apache.org> on 2023/10/06 20:09:37 UTC
[PR] feat(c/driver/postgresql): Binary COPY Writer [arrow-adbc]
WillAyd opened a new pull request, #1181:
URL: https://github.com/apache/arrow-adbc/pull/1181
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/driver/postgresql): Binary COPY Writer [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #1181:
URL: https://github.com/apache/arrow-adbc/pull/1181#discussion_r1354872987
##########
c/validation/adbc_validation.cc:
##########
@@ -1507,7 +1507,7 @@ void StatementTest::TestSqlIngestLargeString() {
void StatementTest::TestSqlIngestBinary() {
ASSERT_NO_FATAL_FAILURE(TestSqlIngestType<std::string>(
- NANOARROW_TYPE_BINARY, {std::nullopt, "", "\x00\x01\x02\x04", "\xFE\xFF"}));
+ NANOARROW_TYPE_BINARY, {std::nullopt, "", "\x01\x02\x03\x04", "\xFE\xFF"}));
Review Comment:
It was not, but I think we should still test that bytestrings containing nulls work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/driver/postgresql): Binary COPY Writer [arrow-adbc]
Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on code in PR #1181:
URL: https://github.com/apache/arrow-adbc/pull/1181#discussion_r1349282752
##########
c/validation/adbc_validation.cc:
##########
@@ -1507,7 +1507,7 @@ void StatementTest::TestSqlIngestLargeString() {
void StatementTest::TestSqlIngestBinary() {
ASSERT_NO_FATAL_FAILURE(TestSqlIngestType<std::string>(
- NANOARROW_TYPE_BINARY, {std::nullopt, "", "\x00\x01\x02\x04", "\xFE\xFF"}));
+ NANOARROW_TYPE_BINARY, {std::nullopt, "", "\x01\x02\x03\x04", "\xFE\xFF"}));
Review Comment:
The leading \x00 was causing this to construct as an empty string, which I don't think was the intention of the test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/driver/postgresql): Binary COPY Writer [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #1181:
URL: https://github.com/apache/arrow-adbc/pull/1181#discussion_r1355505334
##########
c/driver/postgresql/postgres_copy_reader_test.cc:
##########
@@ -774,6 +774,87 @@ TEST(PostgresCopyUtilsTest, PostgresCopyWriteLargeString) {
}
}
+// COPY (SELECT CAST("col" AS BYTEA) AS "col" FROM ( VALUES ((""), ("\x00\x01"),
+// ("\x01\x02\x03\x04"), ("\xFE\xFF"), (NULL)) AS drvd("col")) TO STDOUT
+// WITH (FORMAT binary);
+static uint8_t kTestPgCopyBinary[] = {
+ 0x50, 0x47, 0x43, 0x4f, 0x50, 0x59, 0x0a, 0xff, 0x0d, 0x0a, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x04,
+ 0x01, 0x02, 0x03, 0x04, 0x00, 0x01, 0x00, 0x00, 0x00, 0x02, 0xfe, 0xff,
+ 0x00, 0x01, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff};
+
+TEST(PostgresCopyUtilsTest, PostgresCopyReadBinary) {
+ ArrowBufferView data;
+ data.data.as_uint8 = kTestPgCopyBinary;
+ data.size_bytes = sizeof(kTestPgCopyBinary);
+
+ auto col_type = PostgresType(PostgresTypeId::kBytea);
+ PostgresType input_type(PostgresTypeId::kRecord);
+ input_type.AppendChild("col", col_type);
+
+ PostgresCopyStreamTester tester;
+ ASSERT_EQ(tester.Init(input_type), NANOARROW_OK);
+ ASSERT_EQ(tester.ReadAll(&data), ENODATA);
+ ASSERT_EQ(data.data.as_uint8 - kTestPgCopyBinary, sizeof(kTestPgCopyBinary));
+ ASSERT_EQ(data.size_bytes, 0);
+
+ nanoarrow::UniqueArray array;
+ ASSERT_EQ(tester.GetArray(array.get()), NANOARROW_OK);
+ ASSERT_EQ(array->length, 5);
+ ASSERT_EQ(array->n_children, 1);
+
+ auto validity = reinterpret_cast<const uint8_t*>(array->children[0]->buffers[0]);
+ auto offsets = reinterpret_cast<const int32_t*>(array->children[0]->buffers[1]);
+ auto data_buffer = reinterpret_cast<const char*>(array->children[0]->buffers[2]);
+ ASSERT_NE(validity, nullptr);
+ ASSERT_NE(data_buffer, nullptr);
+
+ ASSERT_TRUE(ArrowBitGet(validity, 0));
+ ASSERT_TRUE(ArrowBitGet(validity, 1));
+ ASSERT_TRUE(ArrowBitGet(validity, 2));
+ ASSERT_TRUE(ArrowBitGet(validity, 3));
+ ASSERT_FALSE(ArrowBitGet(validity, 4));
+
+ ASSERT_EQ(offsets[0], 0);
+ ASSERT_EQ(offsets[1], 0);
+ ASSERT_EQ(offsets[2], 0);
+ ASSERT_EQ(offsets[3], 4);
+ ASSERT_EQ(offsets[4], 6);
+ ASSERT_EQ(offsets[5], 6);
+
+ ASSERT_EQ(std::string(data_buffer + 0, 0), "");
+ ASSERT_EQ(std::string(data_buffer + 0, 0), "");
Review Comment:
Hmm, so Postgres also doesn't handle an embedded null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/driver/postgresql): Binary COPY Writer [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm merged PR #1181:
URL: https://github.com/apache/arrow-adbc/pull/1181
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/driver/postgresql): Binary COPY Writer [arrow-adbc]
Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on code in PR #1181:
URL: https://github.com/apache/arrow-adbc/pull/1181#discussion_r1355625995
##########
c/driver/postgresql/postgres_copy_reader_test.cc:
##########
@@ -774,6 +774,87 @@ TEST(PostgresCopyUtilsTest, PostgresCopyWriteLargeString) {
}
}
+// COPY (SELECT CAST("col" AS BYTEA) AS "col" FROM ( VALUES ((""), ("\x00\x01"),
+// ("\x01\x02\x03\x04"), ("\xFE\xFF"), (NULL)) AS drvd("col")) TO STDOUT
+// WITH (FORMAT binary);
+static uint8_t kTestPgCopyBinary[] = {
+ 0x50, 0x47, 0x43, 0x4f, 0x50, 0x59, 0x0a, 0xff, 0x0d, 0x0a, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x04,
+ 0x01, 0x02, 0x03, 0x04, 0x00, 0x01, 0x00, 0x00, 0x00, 0x02, 0xfe, 0xff,
+ 0x00, 0x01, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff};
+
+TEST(PostgresCopyUtilsTest, PostgresCopyReadBinary) {
+ ArrowBufferView data;
+ data.data.as_uint8 = kTestPgCopyBinary;
+ data.size_bytes = sizeof(kTestPgCopyBinary);
+
+ auto col_type = PostgresType(PostgresTypeId::kBytea);
+ PostgresType input_type(PostgresTypeId::kRecord);
+ input_type.AppendChild("col", col_type);
+
+ PostgresCopyStreamTester tester;
+ ASSERT_EQ(tester.Init(input_type), NANOARROW_OK);
+ ASSERT_EQ(tester.ReadAll(&data), ENODATA);
+ ASSERT_EQ(data.data.as_uint8 - kTestPgCopyBinary, sizeof(kTestPgCopyBinary));
+ ASSERT_EQ(data.size_bytes, 0);
+
+ nanoarrow::UniqueArray array;
+ ASSERT_EQ(tester.GetArray(array.get()), NANOARROW_OK);
+ ASSERT_EQ(array->length, 5);
+ ASSERT_EQ(array->n_children, 1);
+
+ auto validity = reinterpret_cast<const uint8_t*>(array->children[0]->buffers[0]);
+ auto offsets = reinterpret_cast<const int32_t*>(array->children[0]->buffers[1]);
+ auto data_buffer = reinterpret_cast<const char*>(array->children[0]->buffers[2]);
+ ASSERT_NE(validity, nullptr);
+ ASSERT_NE(data_buffer, nullptr);
+
+ ASSERT_TRUE(ArrowBitGet(validity, 0));
+ ASSERT_TRUE(ArrowBitGet(validity, 1));
+ ASSERT_TRUE(ArrowBitGet(validity, 2));
+ ASSERT_TRUE(ArrowBitGet(validity, 3));
+ ASSERT_FALSE(ArrowBitGet(validity, 4));
+
+ ASSERT_EQ(offsets[0], 0);
+ ASSERT_EQ(offsets[1], 0);
+ ASSERT_EQ(offsets[2], 0);
+ ASSERT_EQ(offsets[3], 4);
+ ASSERT_EQ(offsets[4], 6);
+ ASSERT_EQ(offsets[5], 6);
+
+ ASSERT_EQ(std::string(data_buffer + 0, 0), "");
+ ASSERT_EQ(std::string(data_buffer + 0, 0), "");
Review Comment:
Ah nice catch. I need to change the tests around to better account for this, but yea it looks like postgres should still display those bytes (confirmed in binary out file as well):
````
SELECT CAST("col" AS BYTEA) AS "col" FROM ( VALUES (''), ('\x0001'),
('\x01020304'), ('\xFEFF'), (NULL)) AS drvd("col");
col
------------
\x
\x0001
\x01020304
\xfeff
```
I think I just need to be careful about constructing a C++ string from that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org