[GitHub] [arrow] lidavidm commented on a change in pull request #10344: ARROW-12603: [C++][Dataset] Backport fix for specifying CSV column types

[GitBox <gi...@apache.org>]

lidavidm commented on a change in pull request #10344:
URL: https://github.com/apache/arrow/pull/10344#discussion_r633695279



##########
File path: cpp/src/arrow/dataset/file_csv_test.cc
##########
@@ -112,6 +112,28 @@ N/A
   ASSERT_EQ(row_count, 3);
 }
 
+TEST_P(TestCsvFileFormat, ScanRecordBatchReaderSchema) {
+  // Regression test for ARROW-12603
+  auto source = GetFileSource(R"(f64,str
+1.0,2.0
+N/A,3.0
+2,foo)");
+  SetSchema({field("f64", float64()), field("str", utf8())});
+  ASSERT_OK_AND_ASSIGN(auto fragment, format_->MakeFragment(*source));
+  auto fragment_scan_options = std::make_shared<CsvFragmentScanOptions>();
+  // Force a small buffer size so type inference gets it wrong
+  fragment_scan_options->read_options.block_size = 20;
+  opts_->fragment_scan_options = fragment_scan_options;
+  ASSERT_OK(SetProjection(opts_.get(), {"f64"}));
+
+  int64_t row_count = 0;
+  for (auto maybe_batch : Batches(fragment.get())) {
+    ASSERT_OK_AND_ASSIGN(auto batch, maybe_batch);
+    row_count += batch->num_rows();

Review comment:
       Ah yes, without the patch, this will fail before it gets here because it will use the auto-inferred type and try to read the string column as a float64 column.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org