You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/09/28 08:13:51 UTC
[GitHub] [arrow] cyb70289 opened a new pull request #11258: ARROW-14150: [C++] Don't check delimiter in CSV chunker if no quoting
cyb70289 opened a new pull request #11258:
URL: https://github.com/apache/arrow/pull/11258
Improves performance significantly for no quoting case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11258: ARROW-14150: [C++] Don't check delimiter in CSV chunker if no quoting
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11258:
URL: https://github.com/apache/arrow/pull/11258#issuecomment-928961409
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #11258: ARROW-14150: [C++] Don't check delimiter in CSV chunker if no quoting
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #11258:
URL: https://github.com/apache/arrow/pull/11258#discussion_r717536610
##########
File path: cpp/src/arrow/csv/chunker.cc
##########
@@ -116,7 +120,8 @@ class Lexer {
if (ARROW_PREDICT_FALSE(c == '\n')) {
goto LineEnd;
}
- if (ARROW_PREDICT_FALSE(c == options_.delimiter)) {
+ // treat delimiter as a normal token in quoting is disabled
Review comment:
Typo: "if quoting is disabled"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #11258: ARROW-14150: [C++] Don't check delimiter in CSV chunker if no quoting
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #11258:
URL: https://github.com/apache/arrow/pull/11258#discussion_r717536610
##########
File path: cpp/src/arrow/csv/chunker.cc
##########
@@ -116,7 +120,8 @@ class Lexer {
if (ARROW_PREDICT_FALSE(c == '\n')) {
goto LineEnd;
}
- if (ARROW_PREDICT_FALSE(c == options_.delimiter)) {
+ // treat delimiter as a normal token in quoting is disabled
Review comment:
Typo: "if quoting is disabled"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 commented on pull request #11258: ARROW-14150: [C++] Don't check delimiter in CSV chunker if no quoting
Posted by GitBox <gi...@apache.org>.
cyb70289 commented on pull request #11258:
URL: https://github.com/apache/arrow/pull/11258#issuecomment-928961387
$ archery benchmark diff --suite-filter="arrow-csv-parser-benchmark" --cc=clang-10 --cxx=clang++-10
```
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Non-regressions: (8)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
benchmark baseline contender change % counters
ChunkCSVEscapedBlock 746.435 MiB/sec 1.325 GiB/sec 81.754 {'run_name': 'ChunkCSVEscapedBlock', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 3420}
ParseCSVFlightsExample 394.605 MiB/sec 397.147 MiB/sec 0.644 {'run_name': 'ParseCSVFlightsExample', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 284}
ParseCSVQuotedBlock 482.529 MiB/sec 484.527 MiB/sec 0.414 {'run_name': 'ParseCSVQuotedBlock', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 2088}
ParseCSVEscapedBlock 515.472 MiB/sec 515.476 MiB/sec 0.001 {'run_name': 'ParseCSVEscapedBlock', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 2364}
ChunkCSVQuotedBlock 836.720 MiB/sec 836.350 MiB/sec -0.044 {'run_name': 'ChunkCSVQuotedBlock', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 3613}
ParseCSVVehiclesExample 933.527 MiB/sec 932.065 MiB/sec -0.157 {'run_name': 'ParseCSVVehiclesExample', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 54}
ParseCSVStocksExample 776.589 MiB/sec 774.898 MiB/sec -0.218 {'run_name': 'ParseCSVStocksExample', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 258}
ChunkCSVNoNewlinesBlock 213.676275 211.736137 -0.908 {'run_name': 'ChunkCSVNoNewlinesBlock', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 3269815}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a change in pull request #11258: ARROW-14150: [C++] Don't check delimiter in CSV chunker if no quoting
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #11258:
URL: https://github.com/apache/arrow/pull/11258#discussion_r717548905
##########
File path: cpp/src/arrow/csv/chunker.cc
##########
@@ -116,7 +120,8 @@ class Lexer {
if (ARROW_PREDICT_FALSE(c == '\n')) {
goto LineEnd;
}
- if (ARROW_PREDICT_FALSE(c == options_.delimiter)) {
+ // treat delimiter as a normal token in quoting is disabled
Review comment:
```suggestion
// treat delimiter as a normal token if quoting is disabled
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a change in pull request #11258: ARROW-14150: [C++] Don't check delimiter in CSV chunker if no quoting
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #11258:
URL: https://github.com/apache/arrow/pull/11258#discussion_r717548905
##########
File path: cpp/src/arrow/csv/chunker.cc
##########
@@ -116,7 +120,8 @@ class Lexer {
if (ARROW_PREDICT_FALSE(c == '\n')) {
goto LineEnd;
}
- if (ARROW_PREDICT_FALSE(c == options_.delimiter)) {
+ // treat delimiter as a normal token in quoting is disabled
Review comment:
```suggestion
// treat delimiter as a normal token if quoting is disabled
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson closed pull request #11258: ARROW-14150: [C++] Don't check delimiter in CSV chunker if no quoting
Posted by GitBox <gi...@apache.org>.
nealrichardson closed pull request #11258:
URL: https://github.com/apache/arrow/pull/11258
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 commented on pull request #11258: ARROW-14150: [C++] Don't check delimiter in CSV chunker if no quoting
Posted by GitBox <gi...@apache.org>.
cyb70289 commented on pull request #11258:
URL: https://github.com/apache/arrow/pull/11258#issuecomment-928961387
$ archery benchmark diff --suite-filter="arrow-csv-parser-benchmark" --cc=clang-10 --cxx=clang++-10
```
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Non-regressions: (8)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
benchmark baseline contender change % counters
ChunkCSVEscapedBlock 746.435 MiB/sec 1.325 GiB/sec 81.754 {'run_name': 'ChunkCSVEscapedBlock', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 3420}
ParseCSVFlightsExample 394.605 MiB/sec 397.147 MiB/sec 0.644 {'run_name': 'ParseCSVFlightsExample', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 284}
ParseCSVQuotedBlock 482.529 MiB/sec 484.527 MiB/sec 0.414 {'run_name': 'ParseCSVQuotedBlock', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 2088}
ParseCSVEscapedBlock 515.472 MiB/sec 515.476 MiB/sec 0.001 {'run_name': 'ParseCSVEscapedBlock', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 2364}
ChunkCSVQuotedBlock 836.720 MiB/sec 836.350 MiB/sec -0.044 {'run_name': 'ChunkCSVQuotedBlock', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 3613}
ParseCSVVehiclesExample 933.527 MiB/sec 932.065 MiB/sec -0.157 {'run_name': 'ParseCSVVehiclesExample', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 54}
ParseCSVStocksExample 776.589 MiB/sec 774.898 MiB/sec -0.218 {'run_name': 'ParseCSVStocksExample', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 258}
ChunkCSVNoNewlinesBlock 213.676275 211.736137 -0.908 {'run_name': 'ChunkCSVNoNewlinesBlock', 'repetitions': 0, 'repetition_index': 0, 'threads': 1, 'iterations': 3269815}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson closed pull request #11258: ARROW-14150: [C++] Don't check delimiter in CSV chunker if no quoting
Posted by GitBox <gi...@apache.org>.
nealrichardson closed pull request #11258:
URL: https://github.com/apache/arrow/pull/11258
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11258: ARROW-14150: [C++] Don't check delimiter in CSV chunker if no quoting
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11258:
URL: https://github.com/apache/arrow/pull/11258#issuecomment-928961409
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org