You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/11/26 20:37:20 UTC

[GitHub] [arrow] Dandandan opened a new pull request #8781: ARROW-10747: [DataFusion]: WIP CSV reader optimization

Dandandan opened a new pull request #8781:
URL: https://github.com/apache/arrow/pull/8781


   Still WIP, but makes CSV reading (quite a bit) faster by reusing a bit more allocations, doing things a bit more manually.
   
   The nytaxi benchmark speeds up from ~4500ms to ~3500ms.
   
   I think a further optimization would be to stop using the `StringRecords` altogether (e.g. by using the underlying https://docs.rs/csv-core/0.1.10/csv_core/ library instead) but this could be a next step.
   
   FYI @alamb @nevi-me @jorgecarleitao 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] Dandandan commented on a change in pull request #8781: ARROW-10747: [Rust]: CSV reader optimization

Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#discussion_r531689316



##########
File path: rust/benchmarks/src/bin/nyctaxi.rs
##########
@@ -92,7 +92,7 @@ async fn datafusion_sql_benchmarks(
     debug: bool,
 ) -> Result<()> {
     let mut queries = HashMap::new();
-    queries.insert("fare_amt_by_passenger", "SELECT passenger_count, MIN(fare_amount), MIN(fare_amount), SUM(fare_amount) FROM tripdata GROUP BY passenger_count");
+    queries.insert("fare_amt_by_passenger", "SELECT passenger_count, MIN(fare_amount), SUM(fare_amount) FROM tripdata GROUP BY passenger_count");

Review comment:
       Great, addressed!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] andygrove commented on pull request #8781: ARROW-10747: [Rust]: CSV reader optimization

Posted by GitBox <gi...@apache.org>.
andygrove commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734956532


   @Dandandan I'm not sure why the Windows build failed and I assume it is unrelated but the logs are not available. Could you push an empty commit to trigger CI again?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] Dandandan commented on pull request #8781: ARROW-10747: [Rust]: CSV reader optimization

Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-735222176


   I also removed the now unused buffered iterator as it is unused by now, and I think will not lead to efficient code in general.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nevi-me commented on a change in pull request #8781: ARROW-10747: [Rust]: CSV reader optimization

Posted by GitBox <gi...@apache.org>.
nevi-me commented on a change in pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#discussion_r531452276



##########
File path: rust/benchmarks/src/bin/nyctaxi.rs
##########
@@ -92,7 +92,7 @@ async fn datafusion_sql_benchmarks(
     debug: bool,
 ) -> Result<()> {
     let mut queries = HashMap::new();
-    queries.insert("fare_amt_by_passenger", "SELECT passenger_count, MIN(fare_amount), MIN(fare_amount), SUM(fare_amount) FROM tripdata GROUP BY passenger_count");
+    queries.insert("fare_amt_by_passenger", "SELECT passenger_count, MIN(fare_amount), SUM(fare_amount) FROM tripdata GROUP BY passenger_count");

Review comment:
       Thanks for fixing this, I saw it a few weeks ago, but forgot to open a JIRA. I think it should be `MAX(fare_amount)` though.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nevi-me commented on pull request #8781: ARROW-10747: [Rust]: CSV reader optimization

Posted by GitBox <gi...@apache.org>.
nevi-me commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734958652


   > Could you push an empty commit to trigger CI again?
   
   We can rerun failed CI jobs from the UI, which is often better as it doesn't trigger AppVeyor and Travis CI


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] Dandandan commented on pull request #8781: ARROW-10747: [Rust]: CSV reader optimization

Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734958248


   > @Dandandan I'm not sure why the Windows build failed and I assume it is unrelated but the logs are not available. Could you push an empty commit to trigger CI again?
   
   Just did. Let's see what happens!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] Dandandan commented on pull request #8781: ARROW-10747: [Rust]: CSV reader optimization

Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734490178


   Is ready for review now.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #8781: ARROW-10747: [DataFusion]: WIP CSV reader optimization

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734472724


   <!--
     Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
     regarding copyright ownership.  The ASF licenses this file
     to you under the Apache License, Version 2.0 (the
     "License"); you may not use this file except in compliance
     with the License.  You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
     Unless required by applicable law or agreed to in writing,
     software distributed under the License is distributed on an
     "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
     KIND, either express or implied.  See the License for the
     specific language governing permissions and limitations
     under the License.
   -->
   
   Thanks for opening a pull request!
   
   Could you open an issue for this pull request on JIRA?
   https://issues.apache.org/jira/browse/ARROW
   
   Then could you also rename pull request title in the following format?
   
       ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
   
   See also:
   
     * [Other pull requests](https://github.com/apache/arrow/pulls/)
     * [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nevi-me commented on pull request #8781: ARROW-10747: [Rust]: CSV reader optimization

Posted by GitBox <gi...@apache.org>.
nevi-me commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734965964


   > Now some other jobs failed. Maybe we can rerun those?
   
   CI seems to be misbehaving, it's not letting me cancel the workflow, even though the tests have failed. I'll leave this tab open, and retry the Rust jobs in about an hour. I'll merge this after CI passes


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] Dandandan commented on pull request #8781: ARROW-10747: [Rust]: CSV reader optimization

Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734488236


   I found some further opportunities for optimizing by also reusing the stringrecord items, for another speed up.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nevi-me closed pull request #8781: ARROW-10747: [Rust]: CSV reader optimization

Posted by GitBox <gi...@apache.org>.
nevi-me closed pull request #8781:
URL: https://github.com/apache/arrow/pull/8781


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] Dandandan commented on pull request #8781: ARROW-10747: [Rust]: CSV reader optimization

Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734963747


   Now some other jobs failed. Maybe we can rerun those? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] alamb commented on a change in pull request #8781: ARROW-10747: [Rust]: CSV reader optimization

Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#discussion_r532949648



##########
File path: rust/benchmarks/src/bin/nyctaxi.rs
##########
@@ -92,7 +92,7 @@ async fn datafusion_sql_benchmarks(
     debug: bool,
 ) -> Result<()> {
     let mut queries = HashMap::new();
-    queries.insert("fare_amt_by_passenger", "SELECT passenger_count, MIN(fare_amount), MIN(fare_amount), SUM(fare_amount) FROM tripdata GROUP BY passenger_count");
+    queries.insert("fare_amt_by_passenger", "SELECT passenger_count, MIN(fare_amount), MAX(fare_amount), SUM(fare_amount) FROM tripdata GROUP BY passenger_count");

Review comment:
       👍 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #8781: ARROW-10747: [Rust]: WIP CSV reader optimization

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734481779


   https://issues.apache.org/jira/browse/ARROW-10747


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org