You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "efegokdemir (via GitHub)" <gi...@apache.org> on 2026/10/10 08:48:28 UTC
[PR] fix(c/driver/sqlite): preserve REAL precision on string inference [arrow-adbc]
efegokdemir opened a new pull request, #4878:
URL: https://github.com/apache/arrow-adbc/pull/4878
## What
Preserve SQLite's text representation when REAL values are converted to binary strings during mixed-type column inference.
## Why
The current conversion used `%e` with its default precision, truncating REAL values when a column containing them is inferred as string.
## How
Use SQLite's formatter with 17 significant digits for the binary64 value. The regression test covers REAL values both before and after a string value in the result set, comparing with SQLite's own `CAST(... AS TEXT)` output.
## Testing
- Confirmed the new regression test fails before the fix (`123456789.123` became `1.234568e+08`).
- Targeted SQLite reader tests passed (2/2).
- Full SQLite test binary passed (144 run, 16 skipped, 0 failed).
- CTest SQLite target passed (1/1).
- Relevant clang-format, cpplint, trailing-whitespace, and `git diff --check` checks passed.
Closes #4862
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/sqlite): preserve REAL precision on string inference [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #4878:
URL: https://github.com/apache/arrow-adbc/pull/4878#discussion_r4266745458
##########
c/driver/sqlite/sqlite_test.cc:
##########
@@ -1013,9 +1015,116 @@ TEST_F(SqliteReaderTest, IntsFloatsStrs) {
ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[0].type);
ASSERT_NO_FATAL_FAILURE(reader.Next());
+ ASSERT_NO_FATAL_FAILURE(CompareArray<std::string>(reader.array_view->children[0],
+ {"1.0", "1.0", "", "Inf", "-Inf"}));
+}
+
+TEST_F(SqliteReaderTest, InferFloatToStringPreservesRoundTripPrecision) {
+ ASSERT_NO_FATAL_FAILURE(Exec("CREATE TABLE foo (upcast_col, append_col)"));
+ ASSERT_NO_FATAL_FAILURE(
+ Exec("INSERT INTO foo VALUES "
+ "(123456789.123, 'start'), "
+ "(3.141592653589793, 123456789.123), "
+ "(0.1, 3.141592653589793), "
+ "('end', 0.1)"));
+
+ struct ExpectedValue {
+ std::optional<double> number;
+ std::string text;
+ };
+ std::vector<ExpectedValue> expected_upcast;
+ std::vector<ExpectedValue> expected_append;
+ sqlite3_stmt* expected_stmt = nullptr;
+ ASSERT_EQ(SQLITE_OK, sqlite3_prepare_v2(db, "SELECT upcast_col, append_col FROM foo",
+ -1, &expected_stmt, nullptr));
+ int rc = SQLITE_OK;
+ while ((rc = sqlite3_step(expected_stmt)) == SQLITE_ROW) {
+ auto read_expected = [expected_stmt](int column) {
+ ExpectedValue value;
+ const int type = sqlite3_column_type(expected_stmt, column);
+ if (type == SQLITE_INTEGER || type == SQLITE_FLOAT) {
+ value.number = sqlite3_column_double(expected_stmt, column);
+ } else {
+ value.text = std::string(
+ reinterpret_cast<const char*>(sqlite3_column_text(expected_stmt, column)),
+ sqlite3_column_bytes(expected_stmt, column));
+ }
+ return value;
+ };
+ expected_upcast.emplace_back(read_expected(0));
+ expected_append.emplace_back(read_expected(1));
+ }
+ EXPECT_EQ(SQLITE_DONE, rc);
+ sqlite3_finalize(expected_stmt);
Review Comment:
Why do we need to do this vs being able to put the expected values directly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/sqlite): preserve REAL precision on string inference [arrow-adbc]
Posted by "efegokdemir (via GitHub)" <gi...@apache.org>.
efegokdemir commented on PR #4878:
URL: https://github.com/apache/arrow-adbc/pull/4878#issuecomment-6140084056
I used AI assistance. I’ve switched to SQLite’s %!.17g formatter, which keeps locale-independent, round-trippable REAL text and preserves .0 for integral values. The SQLite test binary passed: 129 passed, 16 skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/sqlite): preserve REAL precision on string inference [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #4878:
URL: https://github.com/apache/arrow-adbc/pull/4878#discussion_r4337068854
##########
c/driver/sqlite/sqlite_test.cc:
##########
@@ -1013,9 +1016,98 @@ TEST_F(SqliteReaderTest, IntsFloatsStrs) {
ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[0].type);
ASSERT_NO_FATAL_FAILURE(reader.Next());
+ ASSERT_NO_FATAL_FAILURE(CompareArray<std::string>(reader.array_view->children[0],
+ {"1.0", "1.0", "", "Inf", "-Inf"}));
+}
+
+TEST_F(SqliteReaderTest, InferFloatToStringPreservesRoundTripPrecision) {
+ ASSERT_NO_FATAL_FAILURE(Exec("CREATE TABLE foo (upcast_col, append_col)"));
ASSERT_NO_FATAL_FAILURE(
- CompareArray<std::string>(reader.array_view->children[0],
- {"1.000000e+00", "1.000000e+00", "", "inf", "-inf"}));
+ Exec("INSERT INTO foo VALUES "
+ "(123456789.123, 'start'), "
+ "(3.141592653589793, 123456789.123), "
+ "(0.1, 3.141592653589793), "
+ "('end', 0.1)"));
+
+ const std::vector<std::optional<double>> expected_upcast = {
+ 123456789.123, 3.141592653589793, 0.1, std::nullopt};
+ const std::vector<std::optional<double>> expected_append = {std::nullopt, 123456789.123,
+ 3.141592653589793, 0.1};
+ const std::vector<std::string> expected_text_upcast = {"", "", "", "end"};
+ const std::vector<std::string> expected_text_append = {"start", "", "", ""};
+
+ adbc_validation::StreamReader reader;
+ ASSERT_NO_FATAL_FAILURE(Exec("SELECT * FROM foo", kInferRows, &reader));
+ ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[0].type);
+ ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[1].type);
+ ASSERT_NO_FATAL_FAILURE(reader.Next());
+ auto expect_round_trip = [](ArrowArrayView* actual,
+ const std::vector<std::optional<double>>& expected_values,
+ const std::vector<std::string>& expected_text) {
+ ASSERT_EQ(actual->length, static_cast<int64_t>(expected_values.size()));
+ ASSERT_EQ(expected_values.size(), expected_text.size());
+ for (int64_t i = 0; i < actual->length; i++) {
+ const ArrowStringView actual_view = ArrowArrayViewGetStringUnsafe(actual, i);
+ const std::string actual_value(actual_view.data, actual_view.size_bytes);
+ if (expected_values[i].has_value()) {
+ double parsed = 0.0;
+ const char* begin = actual_value.data();
+ const char* end = begin + actual_value.size();
+ const auto result =
+ std::from_chars(begin, end, parsed, std::chars_format::general);
+ ASSERT_EQ(result.ec, std::errc{});
+ ASSERT_EQ(result.ptr, end);
+ EXPECT_EQ(parsed, expected_values[i].value());
+ } else {
+ EXPECT_EQ(actual_value, expected_text[i]);
+ }
+ }
Review Comment:
Another point: the PR description doesn't need to reflect this comment. "Rather than parsing the output into doubles" is useless information once the change has been made.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/sqlite): preserve REAL precision on string inference [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #4878:
URL: https://github.com/apache/arrow-adbc/pull/4878#issuecomment-6139203556
Also, why does preserving the `.0` matter so much?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/sqlite): preserve REAL precision on string inference [arrow-adbc]
Posted by "efegokdemir (via GitHub)" <gi...@apache.org>.
efegokdemir commented on code in PR #4878:
URL: https://github.com/apache/arrow-adbc/pull/4878#discussion_r4266532284
##########
c/driver/sqlite/sqlite_test.cc:
##########
@@ -1013,9 +1015,116 @@ TEST_F(SqliteReaderTest, IntsFloatsStrs) {
ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[0].type);
ASSERT_NO_FATAL_FAILURE(reader.Next());
+ ASSERT_NO_FATAL_FAILURE(CompareArray<std::string>(reader.array_view->children[0],
+ {"1.0", "1.0", "", "Inf", "-Inf"}));
+}
+
+TEST_F(SqliteReaderTest, InferFloatToStringPreservesRoundTripPrecision) {
+ ASSERT_NO_FATAL_FAILURE(Exec("CREATE TABLE foo (upcast_col, append_col)"));
+ ASSERT_NO_FATAL_FAILURE(
+ Exec("INSERT INTO foo VALUES "
+ "(123456789.123, 'start'), "
+ "(3.141592653589793, 123456789.123), "
+ "(0.1, 3.141592653589793), "
+ "('end', 0.1)"));
+
+ struct ExpectedValue {
+ std::optional<double> number;
+ std::string text;
+ };
+ std::vector<ExpectedValue> expected_upcast;
+ std::vector<ExpectedValue> expected_append;
+ sqlite3_stmt* expected_stmt = nullptr;
+ ASSERT_EQ(SQLITE_OK, sqlite3_prepare_v2(db, "SELECT upcast_col, append_col FROM foo",
+ -1, &expected_stmt, nullptr));
+ int rc = SQLITE_OK;
+ while ((rc = sqlite3_step(expected_stmt)) == SQLITE_ROW) {
+ auto read_expected = [expected_stmt](int column) {
+ ExpectedValue value;
+ const int type = sqlite3_column_type(expected_stmt, column);
+ if (type == SQLITE_INTEGER || type == SQLITE_FLOAT) {
+ value.number = sqlite3_column_double(expected_stmt, column);
+ } else {
+ value.text = std::string(
+ reinterpret_cast<const char*>(sqlite3_column_text(expected_stmt, column)),
+ sqlite3_column_bytes(expected_stmt, column));
+ }
+ return value;
+ };
+ expected_upcast.emplace_back(read_expected(0));
+ expected_append.emplace_back(read_expected(1));
+ }
+ EXPECT_EQ(SQLITE_DONE, rc);
+ sqlite3_finalize(expected_stmt);
Review Comment:
Changed the test to compare Arrow’s output with SQLite’s sqlite3_column_text result directly, rather than parsing both values as doubles. The focused precision and decimal-comma tests pass.
##########
c/driver/sqlite/sqlite_test.cc:
##########
@@ -1013,9 +1015,116 @@ TEST_F(SqliteReaderTest, IntsFloatsStrs) {
ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[0].type);
ASSERT_NO_FATAL_FAILURE(reader.Next());
+ ASSERT_NO_FATAL_FAILURE(CompareArray<std::string>(reader.array_view->children[0],
+ {"1.0", "1.0", "", "Inf", "-Inf"}));
+}
+
+TEST_F(SqliteReaderTest, InferFloatToStringPreservesRoundTripPrecision) {
+ ASSERT_NO_FATAL_FAILURE(Exec("CREATE TABLE foo (upcast_col, append_col)"));
+ ASSERT_NO_FATAL_FAILURE(
+ Exec("INSERT INTO foo VALUES "
+ "(123456789.123, 'start'), "
+ "(3.141592653589793, 123456789.123), "
+ "(0.1, 3.141592653589793), "
+ "('end', 0.1)"));
+
+ struct ExpectedValue {
+ std::optional<double> number;
+ std::string text;
+ };
+ std::vector<ExpectedValue> expected_upcast;
+ std::vector<ExpectedValue> expected_append;
+ sqlite3_stmt* expected_stmt = nullptr;
+ ASSERT_EQ(SQLITE_OK, sqlite3_prepare_v2(db, "SELECT upcast_col, append_col FROM foo",
+ -1, &expected_stmt, nullptr));
+ int rc = SQLITE_OK;
+ while ((rc = sqlite3_step(expected_stmt)) == SQLITE_ROW) {
+ auto read_expected = [expected_stmt](int column) {
+ ExpectedValue value;
+ const int type = sqlite3_column_type(expected_stmt, column);
+ if (type == SQLITE_INTEGER || type == SQLITE_FLOAT) {
+ value.number = sqlite3_column_double(expected_stmt, column);
+ } else {
+ value.text = std::string(
+ reinterpret_cast<const char*>(sqlite3_column_text(expected_stmt, column)),
+ sqlite3_column_bytes(expected_stmt, column));
+ }
+ return value;
+ };
+ expected_upcast.emplace_back(read_expected(0));
+ expected_append.emplace_back(read_expected(1));
+ }
+ EXPECT_EQ(SQLITE_DONE, rc);
+ sqlite3_finalize(expected_stmt);
+
+ adbc_validation::StreamReader reader;
+ ASSERT_NO_FATAL_FAILURE(Exec("SELECT * FROM foo", kInferRows, &reader));
+ ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[0].type);
+ ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[1].type);
+ ASSERT_NO_FATAL_FAILURE(reader.Next());
+ auto expect_round_trip = [](ArrowArrayView* actual,
+ const std::vector<ExpectedValue>& expected) {
+ ASSERT_EQ(actual->length, static_cast<int64_t>(expected.size()));
+ for (int64_t i = 0; i < actual->length; i++) {
+ const ArrowStringView actual_view = ArrowArrayViewGetStringUnsafe(actual, i);
+ const std::string actual_value(actual_view.data, actual_view.size_bytes);
+
+ if (expected[i].number.has_value()) {
+ char* actual_end = nullptr;
+ const double actual_number = std::strtod(actual_value.c_str(), &actual_end);
Review Comment:
Changed the test to compare Arrow’s output with SQLite’s sqlite3_column_text result directly, rather than parsing both values as doubles. The focused precision and decimal-comma tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/sqlite): preserve REAL precision on string inference [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #4878:
URL: https://github.com/apache/arrow-adbc/pull/4878#discussion_r4245784922
##########
c/driver/sqlite/sqlite_test.cc:
##########
@@ -1087,6 +1086,40 @@ TEST_F(SqliteReaderTest, InferFloatToStringPreservesRoundTripPrecision) {
expect_round_trip(reader.array_view->children[1], expected_append));
}
+TEST_F(SqliteReaderTest, InferFloatToStringUsesDotDecimalInCommaLocale) {
+ const char* current_locale = std::setlocale(LC_NUMERIC, nullptr);
+ ASSERT_NE(nullptr, current_locale);
+ const std::string original_locale(current_locale);
+
+ const char* comma_locales[] = {"de_DE.UTF-8", "de_DE.utf8", "de_DE",
+ "German_Germany.1252"};
+ bool comma_locale_available = false;
+ for (const char* locale : comma_locales) {
+ if (std::setlocale(LC_NUMERIC, locale) != nullptr &&
+ std::strcmp(std::localeconv()->decimal_point, ",") == 0) {
+ comma_locale_available = true;
+ break;
+ }
+ }
+ if (!comma_locale_available) {
+ std::setlocale(LC_NUMERIC, original_locale.c_str());
+ GTEST_SKIP() << "No decimal-comma locale is installed";
+ }
+
+ struct RestoreLocale {
+ std::string name;
+ ~RestoreLocale() { std::setlocale(LC_NUMERIC, name.c_str()); }
+ } restore{original_locale};
Review Comment:
No, I meant that RestoreLocale should be set up before we set locale above, then you don't need to explicitly restore it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/sqlite): preserve REAL precision on string inference [arrow-adbc]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #4878:
URL: https://github.com/apache/arrow-adbc/pull/4878#discussion_r4244224458
##########
c/driver/sqlite/statement_reader.c:
##########
@@ -953,27 +953,30 @@ AdbcStatusCode InternalSqliteStatementReaderAppendDoubleToBinary(
struct ArrowBuffer* offsets, struct ArrowBuffer* binary, double value,
int32_t* offset, struct AdbcError* error) {
static const size_t kReserve = 64;
- size_t buffer_size = kReserve;
- CHECK_NA(INTERNAL, ArrowBufferReserve(binary, buffer_size), error);
+ CHECK_NA(INTERNAL, ArrowBufferReserve(binary, kReserve), error);
char* output = (char*)(binary->data + binary->size_bytes);
- int written = 0;
- while (1) {
- written = snprintf(output, buffer_size, "%e", value);
- if (written < 0) {
- InternalAdbcSetError(error, "Encoding error when upcasting double to string");
- return ADBC_STATUS_INTERNAL;
- } else if (((size_t)written) >= buffer_size) {
- // Truncated, resize and try again
- // Check for overflow - presumably this can never happen...?
- if (UINT_MAX - buffer_size < buffer_size) {
- InternalAdbcSetError(error, "Overflow when upcasting double to string");
- return ADBC_STATUS_INTERNAL;
- }
- CHECK_NA(INTERNAL, ArrowBufferReserve(binary, buffer_size), error);
- buffer_size += buffer_size;
- continue;
+ // Keep enough significant digits to round-trip the binary64 value. SQLite's
+ // printf precision can be capped by the linked SQLite build, so use libc for
+ // finite values and SQLite's formatter for its non-finite spellings.
+ int written;
+ if (isfinite(value)) {
+ written = snprintf(output, (int)kReserve, "%.17g", value);
+ if (written >= 0 && written < (int)kReserve - 2 && strchr(output, '.') == NULL) {
Review Comment:
`snprintf("%.17g")` honors the process's `LC_NUMERIC`. With a decimal-comma locale, a value such as `1.5` is written as `1,5`; the following `strchr(output, '.')` then inserts `.0`, producing the invalid `1,5.0`. This makes query output locale-dependent and breaks round-tripping. Please use a locale-independent conversion (or safely normalize the locale decimal separator before the `.0` logic) and cover a decimal-comma locale in the regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/sqlite): preserve REAL precision on string inference [arrow-adbc]
Posted by "efegokdemir (via GitHub)" <gi...@apache.org>.
efegokdemir commented on code in PR #4878:
URL: https://github.com/apache/arrow-adbc/pull/4878#discussion_r4245198483
##########
c/driver/sqlite/statement_reader.c:
##########
@@ -953,27 +953,30 @@ AdbcStatusCode InternalSqliteStatementReaderAppendDoubleToBinary(
struct ArrowBuffer* offsets, struct ArrowBuffer* binary, double value,
int32_t* offset, struct AdbcError* error) {
static const size_t kReserve = 64;
- size_t buffer_size = kReserve;
- CHECK_NA(INTERNAL, ArrowBufferReserve(binary, buffer_size), error);
+ CHECK_NA(INTERNAL, ArrowBufferReserve(binary, kReserve), error);
char* output = (char*)(binary->data + binary->size_bytes);
- int written = 0;
- while (1) {
- written = snprintf(output, buffer_size, "%e", value);
- if (written < 0) {
- InternalAdbcSetError(error, "Encoding error when upcasting double to string");
- return ADBC_STATUS_INTERNAL;
- } else if (((size_t)written) >= buffer_size) {
- // Truncated, resize and try again
- // Check for overflow - presumably this can never happen...?
- if (UINT_MAX - buffer_size < buffer_size) {
- InternalAdbcSetError(error, "Overflow when upcasting double to string");
- return ADBC_STATUS_INTERNAL;
- }
- CHECK_NA(INTERNAL, ArrowBufferReserve(binary, buffer_size), error);
- buffer_size += buffer_size;
- continue;
+ // Keep enough significant digits to round-trip the binary64 value. SQLite's
+ // printf precision can be capped by the linked SQLite build, so use libc for
+ // finite values and SQLite's formatter for its non-finite spellings.
+ int written;
+ if (isfinite(value)) {
+ written = snprintf(output, (int)kReserve, "%.17g", value);
+ if (written >= 0 && written < (int)kReserve - 2 && strchr(output, '.') == NULL) {
Review Comment:
Normalised the decimal separator before the integral-value handling and added a decimal-comma locale regression test. The SQLite tests, clang-format and diff check pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/sqlite): preserve REAL precision on string inference [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #4878:
URL: https://github.com/apache/arrow-adbc/pull/4878#discussion_r4266054139
##########
c/driver/sqlite/sqlite_test.cc:
##########
@@ -1013,9 +1015,116 @@ TEST_F(SqliteReaderTest, IntsFloatsStrs) {
ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[0].type);
ASSERT_NO_FATAL_FAILURE(reader.Next());
+ ASSERT_NO_FATAL_FAILURE(CompareArray<std::string>(reader.array_view->children[0],
+ {"1.0", "1.0", "", "Inf", "-Inf"}));
+}
+
+TEST_F(SqliteReaderTest, InferFloatToStringPreservesRoundTripPrecision) {
+ ASSERT_NO_FATAL_FAILURE(Exec("CREATE TABLE foo (upcast_col, append_col)"));
+ ASSERT_NO_FATAL_FAILURE(
+ Exec("INSERT INTO foo VALUES "
+ "(123456789.123, 'start'), "
+ "(3.141592653589793, 123456789.123), "
+ "(0.1, 3.141592653589793), "
+ "('end', 0.1)"));
+
+ struct ExpectedValue {
+ std::optional<double> number;
+ std::string text;
+ };
+ std::vector<ExpectedValue> expected_upcast;
+ std::vector<ExpectedValue> expected_append;
+ sqlite3_stmt* expected_stmt = nullptr;
+ ASSERT_EQ(SQLITE_OK, sqlite3_prepare_v2(db, "SELECT upcast_col, append_col FROM foo",
+ -1, &expected_stmt, nullptr));
+ int rc = SQLITE_OK;
+ while ((rc = sqlite3_step(expected_stmt)) == SQLITE_ROW) {
+ auto read_expected = [expected_stmt](int column) {
+ ExpectedValue value;
+ const int type = sqlite3_column_type(expected_stmt, column);
+ if (type == SQLITE_INTEGER || type == SQLITE_FLOAT) {
+ value.number = sqlite3_column_double(expected_stmt, column);
+ } else {
+ value.text = std::string(
+ reinterpret_cast<const char*>(sqlite3_column_text(expected_stmt, column)),
+ sqlite3_column_bytes(expected_stmt, column));
+ }
+ return value;
+ };
+ expected_upcast.emplace_back(read_expected(0));
+ expected_append.emplace_back(read_expected(1));
+ }
+ EXPECT_EQ(SQLITE_DONE, rc);
+ sqlite3_finalize(expected_stmt);
+
+ adbc_validation::StreamReader reader;
+ ASSERT_NO_FATAL_FAILURE(Exec("SELECT * FROM foo", kInferRows, &reader));
+ ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[0].type);
+ ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[1].type);
+ ASSERT_NO_FATAL_FAILURE(reader.Next());
+ auto expect_round_trip = [](ArrowArrayView* actual,
+ const std::vector<ExpectedValue>& expected) {
+ ASSERT_EQ(actual->length, static_cast<int64_t>(expected.size()));
+ for (int64_t i = 0; i < actual->length; i++) {
+ const ArrowStringView actual_view = ArrowArrayViewGetStringUnsafe(actual, i);
+ const std::string actual_value(actual_view.data, actual_view.size_bytes);
+
+ if (expected[i].number.has_value()) {
+ char* actual_end = nullptr;
+ const double actual_number = std::strtod(actual_value.c_str(), &actual_end);
Review Comment:
Why compare the doubles? If the string representations match then it should be fine?
##########
c/driver/sqlite/sqlite_test.cc:
##########
@@ -1013,9 +1015,116 @@ TEST_F(SqliteReaderTest, IntsFloatsStrs) {
ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[0].type);
ASSERT_NO_FATAL_FAILURE(reader.Next());
+ ASSERT_NO_FATAL_FAILURE(CompareArray<std::string>(reader.array_view->children[0],
+ {"1.0", "1.0", "", "Inf", "-Inf"}));
+}
+
+TEST_F(SqliteReaderTest, InferFloatToStringPreservesRoundTripPrecision) {
+ ASSERT_NO_FATAL_FAILURE(Exec("CREATE TABLE foo (upcast_col, append_col)"));
+ ASSERT_NO_FATAL_FAILURE(
+ Exec("INSERT INTO foo VALUES "
+ "(123456789.123, 'start'), "
+ "(3.141592653589793, 123456789.123), "
+ "(0.1, 3.141592653589793), "
+ "('end', 0.1)"));
+
+ struct ExpectedValue {
+ std::optional<double> number;
+ std::string text;
+ };
+ std::vector<ExpectedValue> expected_upcast;
+ std::vector<ExpectedValue> expected_append;
+ sqlite3_stmt* expected_stmt = nullptr;
+ ASSERT_EQ(SQLITE_OK, sqlite3_prepare_v2(db, "SELECT upcast_col, append_col FROM foo",
+ -1, &expected_stmt, nullptr));
+ int rc = SQLITE_OK;
+ while ((rc = sqlite3_step(expected_stmt)) == SQLITE_ROW) {
+ auto read_expected = [expected_stmt](int column) {
+ ExpectedValue value;
+ const int type = sqlite3_column_type(expected_stmt, column);
+ if (type == SQLITE_INTEGER || type == SQLITE_FLOAT) {
+ value.number = sqlite3_column_double(expected_stmt, column);
+ } else {
+ value.text = std::string(
+ reinterpret_cast<const char*>(sqlite3_column_text(expected_stmt, column)),
+ sqlite3_column_bytes(expected_stmt, column));
+ }
+ return value;
+ };
+ expected_upcast.emplace_back(read_expected(0));
+ expected_append.emplace_back(read_expected(1));
+ }
+ EXPECT_EQ(SQLITE_DONE, rc);
+ sqlite3_finalize(expected_stmt);
Review Comment:
Why are we doing all this?
##########
c/driver/sqlite/statement_reader.c:
##########
@@ -953,27 +952,16 @@ AdbcStatusCode InternalSqliteStatementReaderAppendDoubleToBinary(
struct ArrowBuffer* offsets, struct ArrowBuffer* binary, double value,
int32_t* offset, struct AdbcError* error) {
static const size_t kReserve = 64;
- size_t buffer_size = kReserve;
- CHECK_NA(INTERNAL, ArrowBufferReserve(binary, buffer_size), error);
+ CHECK_NA(INTERNAL, ArrowBufferReserve(binary, kReserve), error);
char* output = (char*)(binary->data + binary->size_bytes);
- int written = 0;
- while (1) {
- written = snprintf(output, buffer_size, "%e", value);
- if (written < 0) {
- InternalAdbcSetError(error, "Encoding error when upcasting double to string");
- return ADBC_STATUS_INTERNAL;
- } else if (((size_t)written) >= buffer_size) {
- // Truncated, resize and try again
- // Check for overflow - presumably this can never happen...?
- if (UINT_MAX - buffer_size < buffer_size) {
- InternalAdbcSetError(error, "Overflow when upcasting double to string");
- return ADBC_STATUS_INTERNAL;
- }
- CHECK_NA(INTERNAL, ArrowBufferReserve(binary, buffer_size), error);
- buffer_size += buffer_size;
- continue;
- }
- break;
+ // SQLite's alternate-form-2 formatter is locale-independent, keeps enough
+ // significant digits to round-trip binary64, and preserves REAL values as
+ // SQL-compatible text (including a decimal point for integral values).
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/sqlite): preserve REAL precision on string inference [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #4878:
URL: https://github.com/apache/arrow-adbc/pull/4878#discussion_r4331389060
##########
c/driver/sqlite/sqlite_test.cc:
##########
@@ -1013,9 +1016,98 @@ TEST_F(SqliteReaderTest, IntsFloatsStrs) {
ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[0].type);
ASSERT_NO_FATAL_FAILURE(reader.Next());
+ ASSERT_NO_FATAL_FAILURE(CompareArray<std::string>(reader.array_view->children[0],
+ {"1.0", "1.0", "", "Inf", "-Inf"}));
+}
+
+TEST_F(SqliteReaderTest, InferFloatToStringPreservesRoundTripPrecision) {
+ ASSERT_NO_FATAL_FAILURE(Exec("CREATE TABLE foo (upcast_col, append_col)"));
ASSERT_NO_FATAL_FAILURE(
- CompareArray<std::string>(reader.array_view->children[0],
- {"1.000000e+00", "1.000000e+00", "", "inf", "-inf"}));
+ Exec("INSERT INTO foo VALUES "
+ "(123456789.123, 'start'), "
+ "(3.141592653589793, 123456789.123), "
+ "(0.1, 3.141592653589793), "
+ "('end', 0.1)"));
+
+ const std::vector<std::optional<double>> expected_upcast = {
+ 123456789.123, 3.141592653589793, 0.1, std::nullopt};
+ const std::vector<std::optional<double>> expected_append = {std::nullopt, 123456789.123,
+ 3.141592653589793, 0.1};
+ const std::vector<std::string> expected_text_upcast = {"", "", "", "end"};
+ const std::vector<std::string> expected_text_append = {"start", "", "", ""};
+
+ adbc_validation::StreamReader reader;
+ ASSERT_NO_FATAL_FAILURE(Exec("SELECT * FROM foo", kInferRows, &reader));
+ ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[0].type);
+ ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[1].type);
+ ASSERT_NO_FATAL_FAILURE(reader.Next());
+ auto expect_round_trip = [](ArrowArrayView* actual,
+ const std::vector<std::optional<double>>& expected_values,
+ const std::vector<std::string>& expected_text) {
+ ASSERT_EQ(actual->length, static_cast<int64_t>(expected_values.size()));
+ ASSERT_EQ(expected_values.size(), expected_text.size());
+ for (int64_t i = 0; i < actual->length; i++) {
+ const ArrowStringView actual_view = ArrowArrayViewGetStringUnsafe(actual, i);
+ const std::string actual_value(actual_view.data, actual_view.size_bytes);
+ if (expected_values[i].has_value()) {
+ double parsed = 0.0;
+ const char* begin = actual_value.data();
+ const char* end = begin + actual_value.size();
+ const auto result =
+ std::from_chars(begin, end, parsed, std::chars_format::general);
+ ASSERT_EQ(result.ec, std::errc{});
+ ASSERT_EQ(result.ptr, end);
+ EXPECT_EQ(parsed, expected_values[i].value());
+ } else {
+ EXPECT_EQ(actual_value, expected_text[i]);
+ }
+ }
Review Comment:
Again:
> Why compare the doubles? If the string representations match then it should be fine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/sqlite): preserve REAL precision on string inference [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #4878:
URL: https://github.com/apache/arrow-adbc/pull/4878#discussion_r4331395797
##########
c/driver/sqlite/sqlite_test.cc:
##########
@@ -1013,9 +1016,98 @@ TEST_F(SqliteReaderTest, IntsFloatsStrs) {
ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[0].type);
ASSERT_NO_FATAL_FAILURE(reader.Next());
+ ASSERT_NO_FATAL_FAILURE(CompareArray<std::string>(reader.array_view->children[0],
+ {"1.0", "1.0", "", "Inf", "-Inf"}));
+}
+
+TEST_F(SqliteReaderTest, InferFloatToStringPreservesRoundTripPrecision) {
+ ASSERT_NO_FATAL_FAILURE(Exec("CREATE TABLE foo (upcast_col, append_col)"));
ASSERT_NO_FATAL_FAILURE(
- CompareArray<std::string>(reader.array_view->children[0],
- {"1.000000e+00", "1.000000e+00", "", "inf", "-inf"}));
+ Exec("INSERT INTO foo VALUES "
+ "(123456789.123, 'start'), "
+ "(3.141592653589793, 123456789.123), "
+ "(0.1, 3.141592653589793), "
+ "('end', 0.1)"));
+
+ const std::vector<std::optional<double>> expected_upcast = {
+ 123456789.123, 3.141592653589793, 0.1, std::nullopt};
+ const std::vector<std::optional<double>> expected_append = {std::nullopt, 123456789.123,
+ 3.141592653589793, 0.1};
+ const std::vector<std::string> expected_text_upcast = {"", "", "", "end"};
+ const std::vector<std::string> expected_text_append = {"start", "", "", ""};
+
+ adbc_validation::StreamReader reader;
+ ASSERT_NO_FATAL_FAILURE(Exec("SELECT * FROM foo", kInferRows, &reader));
+ ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[0].type);
+ ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[1].type);
+ ASSERT_NO_FATAL_FAILURE(reader.Next());
+ auto expect_round_trip = [](ArrowArrayView* actual,
+ const std::vector<std::optional<double>>& expected_values,
+ const std::vector<std::string>& expected_text) {
+ ASSERT_EQ(actual->length, static_cast<int64_t>(expected_values.size()));
+ ASSERT_EQ(expected_values.size(), expected_text.size());
+ for (int64_t i = 0; i < actual->length; i++) {
+ const ArrowStringView actual_view = ArrowArrayViewGetStringUnsafe(actual, i);
+ const std::string actual_value(actual_view.data, actual_view.size_bytes);
+ if (expected_values[i].has_value()) {
+ double parsed = 0.0;
+ const char* begin = actual_value.data();
+ const char* end = begin + actual_value.size();
+ const auto result =
+ std::from_chars(begin, end, parsed, std::chars_format::general);
+ ASSERT_EQ(result.ec, std::errc{});
+ ASSERT_EQ(result.ptr, end);
+ EXPECT_EQ(parsed, expected_values[i].value());
+ } else {
+ EXPECT_EQ(actual_value, expected_text[i]);
+ }
+ }
Review Comment:
@efegokdemir please review https://arrow.apache.org/docs/developers/overview.html#ai-generated-code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/sqlite): preserve REAL precision on string inference [arrow-adbc]
Posted by "efegokdemir (via GitHub)" <gi...@apache.org>.
efegokdemir commented on code in PR #4878:
URL: https://github.com/apache/arrow-adbc/pull/4878#discussion_r4245777241
##########
c/driver/sqlite/sqlite_test.cc:
##########
@@ -1087,6 +1086,40 @@ TEST_F(SqliteReaderTest, InferFloatToStringPreservesRoundTripPrecision) {
expect_round_trip(reader.array_view->children[1], expected_append));
}
+TEST_F(SqliteReaderTest, InferFloatToStringUsesDotDecimalInCommaLocale) {
+ const char* current_locale = std::setlocale(LC_NUMERIC, nullptr);
+ ASSERT_NE(nullptr, current_locale);
+ const std::string original_locale(current_locale);
+
+ const char* comma_locales[] = {"de_DE.UTF-8", "de_DE.utf8", "de_DE",
+ "German_Germany.1252"};
+ bool comma_locale_available = false;
+ for (const char* locale : comma_locales) {
+ if (std::setlocale(LC_NUMERIC, locale) != nullptr &&
+ std::strcmp(std::localeconv()->decimal_point, ",") == 0) {
+ comma_locale_available = true;
+ break;
+ }
+ }
+ if (!comma_locale_available) {
+ std::setlocale(LC_NUMERIC, original_locale.c_str());
+ GTEST_SKIP() << "No decimal-comma locale is installed";
+ }
+
+ struct RestoreLocale {
+ std::string name;
+ ~RestoreLocale() { std::setlocale(LC_NUMERIC, name.c_str()); }
+ } restore{original_locale};
Review Comment:
Moved the table setup and statement preparation before changing `LC_NUMERIC`; reader inference still runs under the decimal-comma locale. The focused regression and SQLite CTest target pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/sqlite): preserve REAL precision on string inference [arrow-adbc]
Posted by "efegokdemir (via GitHub)" <gi...@apache.org>.
efegokdemir commented on code in PR #4878:
URL: https://github.com/apache/arrow-adbc/pull/4878#discussion_r4266953655
##########
c/driver/sqlite/sqlite_test.cc:
##########
@@ -1013,9 +1015,116 @@ TEST_F(SqliteReaderTest, IntsFloatsStrs) {
ASSERT_EQ(NANOARROW_TYPE_STRING, reader.fields[0].type);
ASSERT_NO_FATAL_FAILURE(reader.Next());
+ ASSERT_NO_FATAL_FAILURE(CompareArray<std::string>(reader.array_view->children[0],
+ {"1.0", "1.0", "", "Inf", "-Inf"}));
+}
+
+TEST_F(SqliteReaderTest, InferFloatToStringPreservesRoundTripPrecision) {
+ ASSERT_NO_FATAL_FAILURE(Exec("CREATE TABLE foo (upcast_col, append_col)"));
+ ASSERT_NO_FATAL_FAILURE(
+ Exec("INSERT INTO foo VALUES "
+ "(123456789.123, 'start'), "
+ "(3.141592653589793, 123456789.123), "
+ "(0.1, 3.141592653589793), "
+ "('end', 0.1)"));
+
+ struct ExpectedValue {
+ std::optional<double> number;
+ std::string text;
+ };
+ std::vector<ExpectedValue> expected_upcast;
+ std::vector<ExpectedValue> expected_append;
+ sqlite3_stmt* expected_stmt = nullptr;
+ ASSERT_EQ(SQLITE_OK, sqlite3_prepare_v2(db, "SELECT upcast_col, append_col FROM foo",
+ -1, &expected_stmt, nullptr));
+ int rc = SQLITE_OK;
+ while ((rc = sqlite3_step(expected_stmt)) == SQLITE_ROW) {
+ auto read_expected = [expected_stmt](int column) {
+ ExpectedValue value;
+ const int type = sqlite3_column_type(expected_stmt, column);
+ if (type == SQLITE_INTEGER || type == SQLITE_FLOAT) {
+ value.number = sqlite3_column_double(expected_stmt, column);
+ } else {
+ value.text = std::string(
+ reinterpret_cast<const char*>(sqlite3_column_text(expected_stmt, column)),
+ sqlite3_column_bytes(expected_stmt, column));
+ }
+ return value;
+ };
+ expected_upcast.emplace_back(read_expected(0));
+ expected_append.emplace_back(read_expected(1));
+ }
+ EXPECT_EQ(SQLITE_DONE, rc);
+ sqlite3_finalize(expected_stmt);
Review Comment:
Updated the test to use explicit expected values rather than querying SQLite for its own string output. The focused regression passed, and the SQLite test binary passed (129 passed, 16 skipped).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(c/driver/sqlite): preserve REAL precision on string inference [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #4878:
URL: https://github.com/apache/arrow-adbc/pull/4878#discussion_r4245385245
##########
c/driver/sqlite/sqlite_test.cc:
##########
@@ -1087,6 +1086,40 @@ TEST_F(SqliteReaderTest, InferFloatToStringPreservesRoundTripPrecision) {
expect_round_trip(reader.array_view->children[1], expected_append));
}
+TEST_F(SqliteReaderTest, InferFloatToStringUsesDotDecimalInCommaLocale) {
+ const char* current_locale = std::setlocale(LC_NUMERIC, nullptr);
+ ASSERT_NE(nullptr, current_locale);
+ const std::string original_locale(current_locale);
+
+ const char* comma_locales[] = {"de_DE.UTF-8", "de_DE.utf8", "de_DE",
+ "German_Germany.1252"};
+ bool comma_locale_available = false;
+ for (const char* locale : comma_locales) {
+ if (std::setlocale(LC_NUMERIC, locale) != nullptr &&
+ std::strcmp(std::localeconv()->decimal_point, ",") == 0) {
+ comma_locale_available = true;
+ break;
+ }
+ }
+ if (!comma_locale_available) {
+ std::setlocale(LC_NUMERIC, original_locale.c_str());
+ GTEST_SKIP() << "No decimal-comma locale is installed";
+ }
+
+ struct RestoreLocale {
+ std::string name;
+ ~RestoreLocale() { std::setlocale(LC_NUMERIC, name.c_str()); }
+ } restore{original_locale};
Review Comment:
Shouldn't you set this up before you set the locale above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org