You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2023/01/04 10:08:31 UTC

[GitHub] [arrow-datafusion] tustvold opened a new pull request, #4818: Update to arrow 30

tustvold opened a new pull request, #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818

   # Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   Closes #.
   
   # Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   # What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   # Are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   # Are there any user-facing changes?
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4818: Update to arrow `30.0.1`

Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#discussion_r1070626048


##########
datafusion/core/src/physical_plan/file_format/csv.rs:
##########
@@ -444,24 +444,11 @@ mod tests {
         assert_eq!(14, csv.schema().fields().len());
 
         let mut it = csv.execute(0, task_ctx)?;
-        let batch = it.next().await.unwrap()?;
-        assert_eq!(14, batch.num_columns());
-        assert_eq!(5, batch.num_rows());
-
-        let expected = vec![
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c1 | c2 | c3  | c4     | c5         | c6                   | c7  | c8    | c9         | c10                  | c11         | c12                 | c13                            | missing_col |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c  | 2  | 1   | 18109  | 2033001162 | -6513304855495910254 | 25  | 43062 | 1491205016 | 5863949479783605708  | 0.110830784 | 0.9294097332465232  | 6WfVFBVGJSQb7FhA7E0lBwdvjfZnSW |             |",
-            "| d  | 5  | -40 | 22614  | 706441268  | -7542719935673075327 | 155 | 14337 | 3373581039 | 11720144131976083864 | 0.69632107  | 0.3114712539863804  | C2GT5KVyOPZpgKVl110TyZO0NcJ434 |             |",
-            "| b  | 1  | 29  | -18218 | 994303988  | 5983957848665088916  | 204 | 9489  | 3275293996 | 14857091259186476033 | 0.53840446  | 0.17909035118828576 | AyYVExXK6AR2qUTxNZ7qRHQOVGMLcz |             |",
-            "| a  | 1  | -85 | -15154 | 1171968280 | 1919439543497968449  | 77  | 52286 | 774637006  | 12101411955859039553 | 0.12285209  | 0.6864391962767343  | 0keZ5G8BffGwgF2RwQD59TFzMStxCB |             |",
-            "| b  | 5  | -82 | 22080  | 1824882165 | 7373730676428214987  | 208 | 34331 | 3342719438 | 3330177516592499461  | 0.82634634  | 0.40975383525297016 | Ig1QcuKsjHXkproePdERo2w0mYzIqd |             |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-        ];
-
-        crate::assert_batches_eq!(expected, &[batch]);
-
+        let err = it.next().await.unwrap().unwrap_err().to_string();

Review Comment:
   Ticket link: https://github.com/apache/arrow-datafusion/pull/4919



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold commented on pull request #4818: Update to arrow 30

Posted by GitBox <gi...@apache.org>.
tustvold commented on PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#issuecomment-1371396498

   https://github.com/apache/arrow-datafusion/pull/4821 contains the necessary changes to make the benchmarks pass


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold commented on a diff in pull request #4818: Update to arrow `30.0.1`

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#discussion_r1070610760


##########
datafusion/core/src/physical_plan/file_format/csv.rs:
##########
@@ -444,24 +444,11 @@ mod tests {
         assert_eq!(14, csv.schema().fields().len());
 
         let mut it = csv.execute(0, task_ctx)?;
-        let batch = it.next().await.unwrap()?;
-        assert_eq!(14, batch.num_columns());
-        assert_eq!(5, batch.num_rows());
-
-        let expected = vec![
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c1 | c2 | c3  | c4     | c5         | c6                   | c7  | c8    | c9         | c10                  | c11         | c12                 | c13                            | missing_col |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c  | 2  | 1   | 18109  | 2033001162 | -6513304855495910254 | 25  | 43062 | 1491205016 | 5863949479783605708  | 0.110830784 | 0.9294097332465232  | 6WfVFBVGJSQb7FhA7E0lBwdvjfZnSW |             |",
-            "| d  | 5  | -40 | 22614  | 706441268  | -7542719935673075327 | 155 | 14337 | 3373581039 | 11720144131976083864 | 0.69632107  | 0.3114712539863804  | C2GT5KVyOPZpgKVl110TyZO0NcJ434 |             |",
-            "| b  | 1  | 29  | -18218 | 994303988  | 5983957848665088916  | 204 | 9489  | 3275293996 | 14857091259186476033 | 0.53840446  | 0.17909035118828576 | AyYVExXK6AR2qUTxNZ7qRHQOVGMLcz |             |",
-            "| a  | 1  | -85 | -15154 | 1171968280 | 1919439543497968449  | 77  | 52286 | 774637006  | 12101411955859039553 | 0.12285209  | 0.6864391962767343  | 0keZ5G8BffGwgF2RwQD59TFzMStxCB |             |",
-            "| b  | 5  | -82 | 22080  | 1824882165 | 7373730676428214987  | 208 | 34331 | 3342719438 | 3330177516592499461  | 0.82634634  | 0.40975383525297016 | Ig1QcuKsjHXkproePdERo2w0mYzIqd |             |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-        ];
-
-        crate::assert_batches_eq!(expected, &[batch]);
-
+        let err = it.next().await.unwrap().unwrap_err().to_string();

Review Comment:
   I've filed a ticket showing this being broken on master - https://github.com/apache/arrow-datafusion/issues/4918



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold commented on pull request #4818: Update to arrow 30

Posted by GitBox <gi...@apache.org>.
tustvold commented on PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#issuecomment-1380055994

   I'm working on it :smile: There be shenanigans


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold commented on a diff in pull request #4818: Update to arrow 30

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#discussion_r1067944592


##########
datafusion/physical-expr/src/expressions/nullif.rs:
##########
@@ -17,7 +17,7 @@
 
 use arrow::array::Array;
 use arrow::compute::eq_dyn;
-use arrow::compute::nullif::nullif;
+use arrow::compute::kernels::nullif::nullif;

Review Comment:
   https://github.com/apache/arrow-rs/pull/3515



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4818: Update to arrow `30.1.0`

Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#discussion_r1070567335


##########
datafusion/core/src/physical_plan/file_format/csv.rs:
##########
@@ -444,24 +444,11 @@ mod tests {
         assert_eq!(14, csv.schema().fields().len());
 
         let mut it = csv.execute(0, task_ctx)?;
-        let batch = it.next().await.unwrap()?;
-        assert_eq!(14, batch.num_columns());
-        assert_eq!(5, batch.num_rows());
-
-        let expected = vec![
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c1 | c2 | c3  | c4     | c5         | c6                   | c7  | c8    | c9         | c10                  | c11         | c12                 | c13                            | missing_col |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c  | 2  | 1   | 18109  | 2033001162 | -6513304855495910254 | 25  | 43062 | 1491205016 | 5863949479783605708  | 0.110830784 | 0.9294097332465232  | 6WfVFBVGJSQb7FhA7E0lBwdvjfZnSW |             |",
-            "| d  | 5  | -40 | 22614  | 706441268  | -7542719935673075327 | 155 | 14337 | 3373581039 | 11720144131976083864 | 0.69632107  | 0.3114712539863804  | C2GT5KVyOPZpgKVl110TyZO0NcJ434 |             |",
-            "| b  | 1  | 29  | -18218 | 994303988  | 5983957848665088916  | 204 | 9489  | 3275293996 | 14857091259186476033 | 0.53840446  | 0.17909035118828576 | AyYVExXK6AR2qUTxNZ7qRHQOVGMLcz |             |",
-            "| a  | 1  | -85 | -15154 | 1171968280 | 1919439543497968449  | 77  | 52286 | 774637006  | 12101411955859039553 | 0.12285209  | 0.6864391962767343  | 0keZ5G8BffGwgF2RwQD59TFzMStxCB |             |",
-            "| b  | 5  | -82 | 22080  | 1824882165 | 7373730676428214987  | 208 | 34331 | 3342719438 | 3330177516592499461  | 0.82634634  | 0.40975383525297016 | Ig1QcuKsjHXkproePdERo2w0mYzIqd |             |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-        ];
-
-        crate::assert_batches_eq!(expected, &[batch]);
-
+        let err = it.next().await.unwrap().unwrap_err().to_string();

Review Comment:
   So I looked at this test more carefully -- I think it is supposed to be demonstrating that we can read from a CSV file where the schema in the file is a subset of the schema in the plan and the columns are supposed to get padded with nulls
   
   It appears to have been added by @thinkharderdev  in https://github.com/apache/arrow-datafusion/commit/7bec762d1f1ebef4801af2eefd7a5033c474fe77 last year. 
   
   I think we should update the code so that it continues to pass. 
   
   I can try and look at it later this week if no one else has a chance



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on pull request #4818: Update to arrow `30.0.1`

Posted by GitBox <gi...@apache.org>.
alamb commented on PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#issuecomment-1383182948

   Thank you @tustvold 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on pull request #4818: Update to arrow 30

Posted by GitBox <gi...@apache.org>.
alamb commented on PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#issuecomment-1380047381

   Now that arrow 30.0.1 is released, this PR can be unblocked, right?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb merged pull request #4818: Update to arrow `30.0.1`

Posted by GitBox <gi...@apache.org>.
alamb merged PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold commented on a diff in pull request #4818: Update to arrow `30.1.0`

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#discussion_r1068802571


##########
datafusion/core/src/physical_plan/file_format/csv.rs:
##########
@@ -444,24 +444,11 @@ mod tests {
         assert_eq!(14, csv.schema().fields().len());
 
         let mut it = csv.execute(0, task_ctx)?;
-        let batch = it.next().await.unwrap()?;
-        assert_eq!(14, batch.num_columns());
-        assert_eq!(5, batch.num_rows());
-
-        let expected = vec![
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c1 | c2 | c3  | c4     | c5         | c6                   | c7  | c8    | c9         | c10                  | c11         | c12                 | c13                            | missing_col |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c  | 2  | 1   | 18109  | 2033001162 | -6513304855495910254 | 25  | 43062 | 1491205016 | 5863949479783605708  | 0.110830784 | 0.9294097332465232  | 6WfVFBVGJSQb7FhA7E0lBwdvjfZnSW |             |",
-            "| d  | 5  | -40 | 22614  | 706441268  | -7542719935673075327 | 155 | 14337 | 3373581039 | 11720144131976083864 | 0.69632107  | 0.3114712539863804  | C2GT5KVyOPZpgKVl110TyZO0NcJ434 |             |",
-            "| b  | 1  | 29  | -18218 | 994303988  | 5983957848665088916  | 204 | 9489  | 3275293996 | 14857091259186476033 | 0.53840446  | 0.17909035118828576 | AyYVExXK6AR2qUTxNZ7qRHQOVGMLcz |             |",
-            "| a  | 1  | -85 | -15154 | 1171968280 | 1919439543497968449  | 77  | 52286 | 774637006  | 12101411955859039553 | 0.12285209  | 0.6864391962767343  | 0keZ5G8BffGwgF2RwQD59TFzMStxCB |             |",
-            "| b  | 5  | -82 | 22080  | 1824882165 | 7373730676428214987  | 208 | 34331 | 3342719438 | 3330177516592499461  | 0.82634634  | 0.40975383525297016 | Ig1QcuKsjHXkproePdERo2w0mYzIqd |             |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-        ];
-
-        crate::assert_batches_eq!(expected, &[batch]);
-
+        let err = it.next().await.unwrap().unwrap_err().to_string();

Review Comment:
   The CSV reader got more picky



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold commented on a diff in pull request #4818: Update to arrow `30.1.0`

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#discussion_r1070588199


##########
datafusion/core/src/physical_plan/file_format/csv.rs:
##########
@@ -444,24 +444,11 @@ mod tests {
         assert_eq!(14, csv.schema().fields().len());
 
         let mut it = csv.execute(0, task_ctx)?;
-        let batch = it.next().await.unwrap()?;
-        assert_eq!(14, batch.num_columns());
-        assert_eq!(5, batch.num_rows());
-
-        let expected = vec![
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c1 | c2 | c3  | c4     | c5         | c6                   | c7  | c8    | c9         | c10                  | c11         | c12                 | c13                            | missing_col |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c  | 2  | 1   | 18109  | 2033001162 | -6513304855495910254 | 25  | 43062 | 1491205016 | 5863949479783605708  | 0.110830784 | 0.9294097332465232  | 6WfVFBVGJSQb7FhA7E0lBwdvjfZnSW |             |",
-            "| d  | 5  | -40 | 22614  | 706441268  | -7542719935673075327 | 155 | 14337 | 3373581039 | 11720144131976083864 | 0.69632107  | 0.3114712539863804  | C2GT5KVyOPZpgKVl110TyZO0NcJ434 |             |",
-            "| b  | 1  | 29  | -18218 | 994303988  | 5983957848665088916  | 204 | 9489  | 3275293996 | 14857091259186476033 | 0.53840446  | 0.17909035118828576 | AyYVExXK6AR2qUTxNZ7qRHQOVGMLcz |             |",
-            "| a  | 1  | -85 | -15154 | 1171968280 | 1919439543497968449  | 77  | 52286 | 774637006  | 12101411955859039553 | 0.12285209  | 0.6864391962767343  | 0keZ5G8BffGwgF2RwQD59TFzMStxCB |             |",
-            "| b  | 5  | -82 | 22080  | 1824882165 | 7373730676428214987  | 208 | 34331 | 3342719438 | 3330177516592499461  | 0.82634634  | 0.40975383525297016 | Ig1QcuKsjHXkproePdERo2w0mYzIqd |             |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-        ];
-
-        crate::assert_batches_eq!(expected, &[batch]);
-
+        let err = it.next().await.unwrap().unwrap_err().to_string();

Review Comment:
   Yeah, that logic has never been correct, it incorrectly assumes that the CSV reader will pad nulls which it hasn't ever except for in the case of a prefix. Unlike parquet or JSON columns aren't named, and so there isn't a way to perform this.
   
   Consider a schema containing columns a, b, and c, if the file actually had schema a,c previously this would interpret column c as b 😱. Now it errors as it should.
   
   I would advise we merge this as is, and potentially file a follow up ticket to investigate this further. I think the mistake is in this test, but I'm not 100% sure



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4818: Update to arrow 30

Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#discussion_r1068695423


##########
datafusion/core/src/physical_plan/file_format/csv.rs:
##########
@@ -444,24 +444,11 @@ mod tests {
         assert_eq!(14, csv.schema().fields().len());
 
         let mut it = csv.execute(0, task_ctx)?;
-        let batch = it.next().await.unwrap()?;
-        assert_eq!(14, batch.num_columns());
-        assert_eq!(5, batch.num_rows());
-
-        let expected = vec![
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c1 | c2 | c3  | c4     | c5         | c6                   | c7  | c8    | c9         | c10                  | c11         | c12                 | c13                            | missing_col |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c  | 2  | 1   | 18109  | 2033001162 | -6513304855495910254 | 25  | 43062 | 1491205016 | 5863949479783605708  | 0.110830784 | 0.9294097332465232  | 6WfVFBVGJSQb7FhA7E0lBwdvjfZnSW |             |",
-            "| d  | 5  | -40 | 22614  | 706441268  | -7542719935673075327 | 155 | 14337 | 3373581039 | 11720144131976083864 | 0.69632107  | 0.3114712539863804  | C2GT5KVyOPZpgKVl110TyZO0NcJ434 |             |",
-            "| b  | 1  | 29  | -18218 | 994303988  | 5983957848665088916  | 204 | 9489  | 3275293996 | 14857091259186476033 | 0.53840446  | 0.17909035118828576 | AyYVExXK6AR2qUTxNZ7qRHQOVGMLcz |             |",
-            "| a  | 1  | -85 | -15154 | 1171968280 | 1919439543497968449  | 77  | 52286 | 774637006  | 12101411955859039553 | 0.12285209  | 0.6864391962767343  | 0keZ5G8BffGwgF2RwQD59TFzMStxCB |             |",
-            "| b  | 5  | -82 | 22080  | 1824882165 | 7373730676428214987  | 208 | 34331 | 3342719438 | 3330177516592499461  | 0.82634634  | 0.40975383525297016 | Ig1QcuKsjHXkproePdERo2w0mYzIqd |             |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-        ];
-
-        crate::assert_batches_eq!(expected, &[batch]);
-
+        let err = it.next().await.unwrap().unwrap_err().to_string();

Review Comment:
   What happened here? Why is this erroring now?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold commented on a diff in pull request #4818: Update to arrow `30.1.0`

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#discussion_r1070588199


##########
datafusion/core/src/physical_plan/file_format/csv.rs:
##########
@@ -444,24 +444,11 @@ mod tests {
         assert_eq!(14, csv.schema().fields().len());
 
         let mut it = csv.execute(0, task_ctx)?;
-        let batch = it.next().await.unwrap()?;
-        assert_eq!(14, batch.num_columns());
-        assert_eq!(5, batch.num_rows());
-
-        let expected = vec![
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c1 | c2 | c3  | c4     | c5         | c6                   | c7  | c8    | c9         | c10                  | c11         | c12                 | c13                            | missing_col |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c  | 2  | 1   | 18109  | 2033001162 | -6513304855495910254 | 25  | 43062 | 1491205016 | 5863949479783605708  | 0.110830784 | 0.9294097332465232  | 6WfVFBVGJSQb7FhA7E0lBwdvjfZnSW |             |",
-            "| d  | 5  | -40 | 22614  | 706441268  | -7542719935673075327 | 155 | 14337 | 3373581039 | 11720144131976083864 | 0.69632107  | 0.3114712539863804  | C2GT5KVyOPZpgKVl110TyZO0NcJ434 |             |",
-            "| b  | 1  | 29  | -18218 | 994303988  | 5983957848665088916  | 204 | 9489  | 3275293996 | 14857091259186476033 | 0.53840446  | 0.17909035118828576 | AyYVExXK6AR2qUTxNZ7qRHQOVGMLcz |             |",
-            "| a  | 1  | -85 | -15154 | 1171968280 | 1919439543497968449  | 77  | 52286 | 774637006  | 12101411955859039553 | 0.12285209  | 0.6864391962767343  | 0keZ5G8BffGwgF2RwQD59TFzMStxCB |             |",
-            "| b  | 5  | -82 | 22080  | 1824882165 | 7373730676428214987  | 208 | 34331 | 3342719438 | 3330177516592499461  | 0.82634634  | 0.40975383525297016 | Ig1QcuKsjHXkproePdERo2w0mYzIqd |             |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-        ];
-
-        crate::assert_batches_eq!(expected, &[batch]);
-
+        let err = it.next().await.unwrap().unwrap_err().to_string();

Review Comment:
   Yeah, that logic has never been correct, it incorrectly assumes that the CSV reader will pad nulls which it hasn't ever except for in the case of a prefix. Unlike parquet or JSON columns aren't named, and so there isn't a way to perform this.
   
   I would advise we merge this as is, and potentially file a follow up ticket to investigate further



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4818: Update to arrow `30.1.0`

Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#discussion_r1069112547


##########
datafusion/core/src/physical_plan/file_format/csv.rs:
##########
@@ -444,24 +444,11 @@ mod tests {
         assert_eq!(14, csv.schema().fields().len());
 
         let mut it = csv.execute(0, task_ctx)?;
-        let batch = it.next().await.unwrap()?;
-        assert_eq!(14, batch.num_columns());
-        assert_eq!(5, batch.num_rows());
-
-        let expected = vec![
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c1 | c2 | c3  | c4     | c5         | c6                   | c7  | c8    | c9         | c10                  | c11         | c12                 | c13                            | missing_col |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c  | 2  | 1   | 18109  | 2033001162 | -6513304855495910254 | 25  | 43062 | 1491205016 | 5863949479783605708  | 0.110830784 | 0.9294097332465232  | 6WfVFBVGJSQb7FhA7E0lBwdvjfZnSW |             |",
-            "| d  | 5  | -40 | 22614  | 706441268  | -7542719935673075327 | 155 | 14337 | 3373581039 | 11720144131976083864 | 0.69632107  | 0.3114712539863804  | C2GT5KVyOPZpgKVl110TyZO0NcJ434 |             |",
-            "| b  | 1  | 29  | -18218 | 994303988  | 5983957848665088916  | 204 | 9489  | 3275293996 | 14857091259186476033 | 0.53840446  | 0.17909035118828576 | AyYVExXK6AR2qUTxNZ7qRHQOVGMLcz |             |",
-            "| a  | 1  | -85 | -15154 | 1171968280 | 1919439543497968449  | 77  | 52286 | 774637006  | 12101411955859039553 | 0.12285209  | 0.6864391962767343  | 0keZ5G8BffGwgF2RwQD59TFzMStxCB |             |",
-            "| b  | 5  | -82 | 22080  | 1824882165 | 7373730676428214987  | 208 | 34331 | 3342719438 | 3330177516592499461  | 0.82634634  | 0.40975383525297016 | Ig1QcuKsjHXkproePdERo2w0mYzIqd |             |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-        ];
-
-        crate::assert_batches_eq!(expected, &[batch]);
-
+        let err = it.next().await.unwrap().unwrap_err().to_string();

Review Comment:
   I guess I am wondering if this test should then be updated:
   1. Removed entirely (testing the csv reader error doesn't sound useful)
   2. Updated to use a non broken CSV reader
   
   I can try and look at it more carefully later today 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold commented on a diff in pull request #4818: Update to arrow `30.1.0`

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#discussion_r1068802571


##########
datafusion/core/src/physical_plan/file_format/csv.rs:
##########
@@ -444,24 +444,11 @@ mod tests {
         assert_eq!(14, csv.schema().fields().len());
 
         let mut it = csv.execute(0, task_ctx)?;
-        let batch = it.next().await.unwrap()?;
-        assert_eq!(14, batch.num_columns());
-        assert_eq!(5, batch.num_rows());
-
-        let expected = vec![
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c1 | c2 | c3  | c4     | c5         | c6                   | c7  | c8    | c9         | c10                  | c11         | c12                 | c13                            | missing_col |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c  | 2  | 1   | 18109  | 2033001162 | -6513304855495910254 | 25  | 43062 | 1491205016 | 5863949479783605708  | 0.110830784 | 0.9294097332465232  | 6WfVFBVGJSQb7FhA7E0lBwdvjfZnSW |             |",
-            "| d  | 5  | -40 | 22614  | 706441268  | -7542719935673075327 | 155 | 14337 | 3373581039 | 11720144131976083864 | 0.69632107  | 0.3114712539863804  | C2GT5KVyOPZpgKVl110TyZO0NcJ434 |             |",
-            "| b  | 1  | 29  | -18218 | 994303988  | 5983957848665088916  | 204 | 9489  | 3275293996 | 14857091259186476033 | 0.53840446  | 0.17909035118828576 | AyYVExXK6AR2qUTxNZ7qRHQOVGMLcz |             |",
-            "| a  | 1  | -85 | -15154 | 1171968280 | 1919439543497968449  | 77  | 52286 | 774637006  | 12101411955859039553 | 0.12285209  | 0.6864391962767343  | 0keZ5G8BffGwgF2RwQD59TFzMStxCB |             |",
-            "| b  | 5  | -82 | 22080  | 1824882165 | 7373730676428214987  | 208 | 34331 | 3342719438 | 3330177516592499461  | 0.82634634  | 0.40975383525297016 | Ig1QcuKsjHXkproePdERo2w0mYzIqd |             |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-        ];
-
-        crate::assert_batches_eq!(expected, &[batch]);
-
+        let err = it.next().await.unwrap().unwrap_err().to_string();

Review Comment:
   The CSV reader got more picky, it now consistently errors on schema mismatch



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold commented on a diff in pull request #4818: Update to arrow `30.1.0`

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#discussion_r1069118831


##########
datafusion/core/src/physical_plan/file_format/csv.rs:
##########
@@ -444,24 +444,11 @@ mod tests {
         assert_eq!(14, csv.schema().fields().len());
 
         let mut it = csv.execute(0, task_ctx)?;
-        let batch = it.next().await.unwrap()?;
-        assert_eq!(14, batch.num_columns());
-        assert_eq!(5, batch.num_rows());
-
-        let expected = vec![
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c1 | c2 | c3  | c4     | c5         | c6                   | c7  | c8    | c9         | c10                  | c11         | c12                 | c13                            | missing_col |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c  | 2  | 1   | 18109  | 2033001162 | -6513304855495910254 | 25  | 43062 | 1491205016 | 5863949479783605708  | 0.110830784 | 0.9294097332465232  | 6WfVFBVGJSQb7FhA7E0lBwdvjfZnSW |             |",
-            "| d  | 5  | -40 | 22614  | 706441268  | -7542719935673075327 | 155 | 14337 | 3373581039 | 11720144131976083864 | 0.69632107  | 0.3114712539863804  | C2GT5KVyOPZpgKVl110TyZO0NcJ434 |             |",
-            "| b  | 1  | 29  | -18218 | 994303988  | 5983957848665088916  | 204 | 9489  | 3275293996 | 14857091259186476033 | 0.53840446  | 0.17909035118828576 | AyYVExXK6AR2qUTxNZ7qRHQOVGMLcz |             |",
-            "| a  | 1  | -85 | -15154 | 1171968280 | 1919439543497968449  | 77  | 52286 | 774637006  | 12101411955859039553 | 0.12285209  | 0.6864391962767343  | 0keZ5G8BffGwgF2RwQD59TFzMStxCB |             |",
-            "| b  | 5  | -82 | 22080  | 1824882165 | 7373730676428214987  | 208 | 34331 | 3342719438 | 3330177516592499461  | 0.82634634  | 0.40975383525297016 | Ig1QcuKsjHXkproePdERo2w0mYzIqd |             |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-        ];
-
-        crate::assert_batches_eq!(expected, &[batch]);
-
+        let err = it.next().await.unwrap().unwrap_err().to_string();

Review Comment:
   > Updated to use a non broken CSV reader
   
   The CSV reader behaviour is correct and expected, whether the test is still providing value, I'm not sure - I thought removing it would be more controversial :sweat_smile: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] ursabot commented on pull request #4818: Update to arrow `30.0.1`

Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#issuecomment-1383185020

   Benchmark runs are scheduled for baseline = f3762700d5d7433667c8248d4d19ffeb1c6861ff and contender = 2801c8c01c09febe14a414fb303ad69808bff920. 2801c8c01c09febe14a414fb303ad69808bff920 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/7b4dbe33eaef415e8381fced86c94099...dc9ceb3798564cadb23d5ff5c8214b5e/)
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/8936e6d01e3e4db3b93b2939b4a42af6...0aa7247f08a749c0a45c8aead5fb3149/)
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/9f1562b6cc66458fba935475ab5a2397...c03ce0d2c00e48e38439412c040f76d3/)
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/e072ba661f794043ad217fa330db563a...1d2d7ab77ba346488cb0ceab2eca9e65/)
   Buildkite builds:
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold commented on a diff in pull request #4818: Update to arrow 30

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#discussion_r1061323132


##########
datafusion/physical-expr/src/expressions/nullif.rs:
##########
@@ -17,7 +17,7 @@
 
 use arrow::array::Array;
 use arrow::compute::eq_dyn;
-use arrow::compute::nullif::nullif;
+use arrow_select::nullif::nullif;

Review Comment:
   Upstream fix - https://github.com/apache/arrow-rs/pull/3451
   
   Longer term I hope to move datafusion away from the `arrow` dependency, and so this isn't really the end of the world



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold commented on a diff in pull request #4818: Update to arrow 30

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#discussion_r1061398337


##########
datafusion/core/src/physical_plan/file_format/csv.rs:
##########
@@ -444,24 +444,8 @@ mod tests {
         assert_eq!(14, csv.schema().fields().len());
 
         let mut it = csv.execute(0, task_ctx)?;
-        let batch = it.next().await.unwrap()?;
-        assert_eq!(14, batch.num_columns());
-        assert_eq!(5, batch.num_rows());
-
-        let expected = vec![
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c1 | c2 | c3  | c4     | c5         | c6                   | c7  | c8    | c9         | c10                  | c11         | c12                 | c13                            | missing_col |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-            "| c  | 2  | 1   | 18109  | 2033001162 | -6513304855495910254 | 25  | 43062 | 1491205016 | 5863949479783605708  | 0.110830784 | 0.9294097332465232  | 6WfVFBVGJSQb7FhA7E0lBwdvjfZnSW |             |",
-            "| d  | 5  | -40 | 22614  | 706441268  | -7542719935673075327 | 155 | 14337 | 3373581039 | 11720144131976083864 | 0.69632107  | 0.3114712539863804  | C2GT5KVyOPZpgKVl110TyZO0NcJ434 |             |",
-            "| b  | 1  | 29  | -18218 | 994303988  | 5983957848665088916  | 204 | 9489  | 3275293996 | 14857091259186476033 | 0.53840446  | 0.17909035118828576 | AyYVExXK6AR2qUTxNZ7qRHQOVGMLcz |             |",
-            "| a  | 1  | -85 | -15154 | 1171968280 | 1919439543497968449  | 77  | 52286 | 774637006  | 12101411955859039553 | 0.12285209  | 0.6864391962767343  | 0keZ5G8BffGwgF2RwQD59TFzMStxCB |             |",
-            "| b  | 5  | -82 | 22080  | 1824882165 | 7373730676428214987  | 208 | 34331 | 3342719438 | 3330177516592499461  | 0.82634634  | 0.40975383525297016 | Ig1QcuKsjHXkproePdERo2w0mYzIqd |             |",
-            "+----+----+-----+--------+------------+----------------------+-----+-------+------------+----------------------+-------------+---------------------+--------------------------------+-------------+",
-        ];
-
-        crate::assert_batches_eq!(expected, &[batch]);
-
+        let err = it.next().await.unwrap().unwrap_err().to_string();
+        assert_eq!(err, "Csv error: incorrect number of fields, expected 14 got 13");

Review Comment:
   The new arrow CSV reader made this an error - https://github.com/apache/arrow-rs/pull/3365#discussion_r1051686401



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold commented on pull request #4818: Update to arrow 30

Posted by GitBox <gi...@apache.org>.
tustvold commented on PR #4818:
URL: https://github.com/apache/arrow-datafusion/pull/4818#issuecomment-1370914447

   Practically speaking this is blocked on https://github.com/calebzulawski/target-features/issues/1 it makes working on this crate completely unworkable


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org