[GitHub] [arrow-datafusion] alamb commented on a change in pull request #1697: [arrow2] Merge arrow2 and datafusion latest

[GitBox <gi...@apache.org>]

alamb commented on a change in pull request #1697:
URL: https://github.com/apache/arrow-datafusion/pull/1697#discussion_r794756250



##########
File path: datafusion/src/field_util.rs
##########
@@ -109,3 +111,321 @@ pub fn struct_array_from(pairs: Vec<(Field, ArrayRef)>) -> StructArray {
     let values = pairs.iter().map(|v| v.1.clone()).collect();
     StructArray::from_data(DataType::Struct(fields), values, None)
 }
+
+/// Imitate arrow-rs Schema behavior by extending arrow2 Schema
+pub trait SchemaExt {

Review comment:
       this is cool

##########
File path: datafusion/src/datasource/file_format/csv.rs
##########
@@ -102,25 +103,18 @@ impl FileFormat for CsvFormat {
                 .has_headers(self.has_header)
                 .from_reader(obj_reader?.sync_reader()?);
 
-            let schema = csv::read::infer_schema(
+            let (fields, records_read) = csv::read::infer_schema(
                 &mut reader,
                 Some(records_to_read),
                 self.has_header,
                 &csv::read::infer,
             )?;
 
-            // if records_read == 0 {
-            //     continue;
-            // }
-            // schemas.push(schema.clone());
-            // records_to_read -= records_read;
-            // if records_to_read == 0 {
-            //     break;
-            // }
-            //
-            // FIXME: return recods_read from infer_schema
-            schemas.push(schema.clone());
-            records_to_read -= records_to_read;
+            if records_read == 0 {

Review comment:
       👍 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org