You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/11/26 20:37:20 UTC
[GitHub] [arrow] Dandandan opened a new pull request #8781: ARROW-10747: [DataFusion]: WIP CSV reader optimization
Dandandan opened a new pull request #8781:
URL: https://github.com/apache/arrow/pull/8781
Still WIP, but makes CSV reading (quite a bit) faster by reusing a bit more allocations, doing things a bit more manually.
The nytaxi benchmark speeds up from ~4500ms to ~3500ms.
I think a further optimization would be to stop using the `StringRecords` altogether (e.g. by using the underlying https://docs.rs/csv-core/0.1.10/csv_core/ library instead) but this could be a next step.
FYI @alamb @nevi-me @jorgecarleitao
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Dandandan commented on a change in pull request #8781: ARROW-10747: [Rust]: CSV reader optimization
Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#discussion_r531689316
##########
File path: rust/benchmarks/src/bin/nyctaxi.rs
##########
@@ -92,7 +92,7 @@ async fn datafusion_sql_benchmarks(
debug: bool,
) -> Result<()> {
let mut queries = HashMap::new();
- queries.insert("fare_amt_by_passenger", "SELECT passenger_count, MIN(fare_amount), MIN(fare_amount), SUM(fare_amount) FROM tripdata GROUP BY passenger_count");
+ queries.insert("fare_amt_by_passenger", "SELECT passenger_count, MIN(fare_amount), SUM(fare_amount) FROM tripdata GROUP BY passenger_count");
Review comment:
Great, addressed!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] andygrove commented on pull request #8781: ARROW-10747: [Rust]: CSV reader optimization
Posted by GitBox <gi...@apache.org>.
andygrove commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734956532
@Dandandan I'm not sure why the Windows build failed and I assume it is unrelated but the logs are not available. Could you push an empty commit to trigger CI again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Dandandan commented on pull request #8781: ARROW-10747: [Rust]: CSV reader optimization
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-735222176
I also removed the now unused buffered iterator as it is unused by now, and I think will not lead to efficient code in general.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nevi-me commented on a change in pull request #8781: ARROW-10747: [Rust]: CSV reader optimization
Posted by GitBox <gi...@apache.org>.
nevi-me commented on a change in pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#discussion_r531452276
##########
File path: rust/benchmarks/src/bin/nyctaxi.rs
##########
@@ -92,7 +92,7 @@ async fn datafusion_sql_benchmarks(
debug: bool,
) -> Result<()> {
let mut queries = HashMap::new();
- queries.insert("fare_amt_by_passenger", "SELECT passenger_count, MIN(fare_amount), MIN(fare_amount), SUM(fare_amount) FROM tripdata GROUP BY passenger_count");
+ queries.insert("fare_amt_by_passenger", "SELECT passenger_count, MIN(fare_amount), SUM(fare_amount) FROM tripdata GROUP BY passenger_count");
Review comment:
Thanks for fixing this, I saw it a few weeks ago, but forgot to open a JIRA. I think it should be `MAX(fare_amount)` though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nevi-me commented on pull request #8781: ARROW-10747: [Rust]: CSV reader optimization
Posted by GitBox <gi...@apache.org>.
nevi-me commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734958652
> Could you push an empty commit to trigger CI again?
We can rerun failed CI jobs from the UI, which is often better as it doesn't trigger AppVeyor and Travis CI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Dandandan commented on pull request #8781: ARROW-10747: [Rust]: CSV reader optimization
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734958248
> @Dandandan I'm not sure why the Windows build failed and I assume it is unrelated but the logs are not available. Could you push an empty commit to trigger CI again?
Just did. Let's see what happens!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Dandandan commented on pull request #8781: ARROW-10747: [Rust]: CSV reader optimization
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734490178
Is ready for review now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #8781: ARROW-10747: [DataFusion]: WIP CSV reader optimization
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734472724
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nevi-me commented on pull request #8781: ARROW-10747: [Rust]: CSV reader optimization
Posted by GitBox <gi...@apache.org>.
nevi-me commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734965964
> Now some other jobs failed. Maybe we can rerun those?
CI seems to be misbehaving, it's not letting me cancel the workflow, even though the tests have failed. I'll leave this tab open, and retry the Rust jobs in about an hour. I'll merge this after CI passes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Dandandan commented on pull request #8781: ARROW-10747: [Rust]: CSV reader optimization
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734488236
I found some further opportunities for optimizing by also reusing the stringrecord items, for another speed up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nevi-me closed pull request #8781: ARROW-10747: [Rust]: CSV reader optimization
Posted by GitBox <gi...@apache.org>.
nevi-me closed pull request #8781:
URL: https://github.com/apache/arrow/pull/8781
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Dandandan commented on pull request #8781: ARROW-10747: [Rust]: CSV reader optimization
Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734963747
Now some other jobs failed. Maybe we can rerun those?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #8781: ARROW-10747: [Rust]: CSV reader optimization
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#discussion_r532949648
##########
File path: rust/benchmarks/src/bin/nyctaxi.rs
##########
@@ -92,7 +92,7 @@ async fn datafusion_sql_benchmarks(
debug: bool,
) -> Result<()> {
let mut queries = HashMap::new();
- queries.insert("fare_amt_by_passenger", "SELECT passenger_count, MIN(fare_amount), MIN(fare_amount), SUM(fare_amount) FROM tripdata GROUP BY passenger_count");
+ queries.insert("fare_amt_by_passenger", "SELECT passenger_count, MIN(fare_amount), MAX(fare_amount), SUM(fare_amount) FROM tripdata GROUP BY passenger_count");
Review comment:
👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #8781: ARROW-10747: [Rust]: WIP CSV reader optimization
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #8781:
URL: https://github.com/apache/arrow/pull/8781#issuecomment-734481779
https://issues.apache.org/jira/browse/ARROW-10747
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org