You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kazantsev-maksim (via GitHub)" <gi...@apache.org> on 2026/01/06 15:36:31 UTC
[PR] Experimental: Native CSV files read [datafusion-comet]
kazantsev-maksim opened a new pull request, #3044:
URL: https://github.com/apache/datafusion-comet/pull/3044
## Which issue does this PR close?
- N/A
## Rationale for this change
Added an experimental implementation of native CSV file reading (currently only for DataSourceV2 version)
Required improvements:
1. Conduct more benchmark tests
2. Try to implement the idea from - https://github.com/apache/datafusion-comet/issues/882
3. Test reading files from S3/HDFS (currently only tested on local files)
Results of simple benchmark test (1 iteration)
Running benchmark: Native csv read: orders table
Running case: Spark
Stopped after 1 iterations, 238272 ms
Running case: Comet
Stopped after 1 iterations, 55326 ms
OpenJDK 64-Bit Server VM 11.0.26+4-LTS on Mac OS X 26.2
Apple M1 Pro
Native csv read: orders table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 238272 238272 0 0,0 113617,0 1,0X
Comet 55327 55327 0 0,0 26381,9 4,3X
Running benchmark: Native csv read: region table
Running case: Spark
Stopped after 1 iterations, 36 ms
Running case: Comet
Stopped after 1 iterations, 38 ms
OpenJDK 64-Bit Server VM 11.0.26+4-LTS on Mac OS X 26.2
Apple M1 Pro
Native csv read: region table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 36 36 0 57,7 17,3 1,0X
Comet 39 39 0 54,0 18,5 0,9X
Running benchmark: Native csv read: nation table
Running case: Spark
Stopped after 1 iterations, 35 ms
Running case: Comet
Stopped after 1 iterations, 39 ms
OpenJDK 64-Bit Server VM 11.0.26+4-LTS on Mac OS X 26.2
Apple M1 Pro
Native csv read: nation table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 36 36 0 59,0 16,9 1,0X
Comet 39 39 0 53,4 18,7 0,9X
Running benchmark: Native csv read: part table
Running case: Spark
Stopped after 1 iterations, 28883 ms
Running case: Comet
Stopped after 1 iterations, 7870 ms
OpenJDK 64-Bit Server VM 11.0.26+4-LTS on Mac OS X 26.2
Apple M1 Pro
Native csv read: part table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 28884 28884 0 0,1 13773,0 1,0X
Comet 7871 7871 0 0,3 3753,2 3,7X
Running benchmark: Native csv read: supplier table
Running case: Spark
Stopped after 1 iterations, 1380 ms
Running case: Comet
Stopped after 1 iterations, 413 ms
OpenJDK 64-Bit Server VM 11.0.26+4-LTS on Mac OS X 26.2
Apple M1 Pro
Native csv read: supplier table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 1381 1381 0 1,5 658,3 1,0X
Comet 414 414 0 5,1 197,2 3,3X
Running benchmark: Native csv read: partsupp table
Running case: Spark
Stopped after 1 iterations, 107904 ms
Running case: Comet
Stopped after 1 iterations, 26308 ms
OpenJDK 64-Bit Server VM 11.0.26+4-LTS on Mac OS X 26.2
Apple M1 Pro
Native csv read: partsupp table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 107905 107905 0 0,0 51452,9 1,0X
Comet 26308 26308 0 0,1 12544,8 4,1X
Running benchmark: Native csv read: customer table
Running case: Spark
Stopped after 1 iterations, 22288 ms
Running case: Comet
Stopped after 1 iterations, 6089 ms
OpenJDK 64-Bit Server VM 11.0.26+4-LTS on Mac OS X 26.2
Apple M1 Pro
Native csv read: customer table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 22289 22289 0 0,1 10628,0 1,0X
Comet 6090 6090 0 0,3 2903,8 3,7X
Running benchmark: Native csv read: lineitem table
Running case: Spark
Stopped after 1 iterations, 1331928 ms
Running case: Comet
Stopped after 1 iterations, 342292 ms
OpenJDK 64-Bit Server VM 11.0.26+4-LTS on Mac OS X 26.2
Apple M1 Pro
Native csv read: lineitem table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 1331928 1331928 0 0,0 635112,8 1,0X
Comet 342293 342293 0 0,0 163218,0 3,9X
## How are these changes tested?
1. Added new unit test
2. Added simple benchmark test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Experimental: Native CSV files read [datafusion-comet]
Posted by "kazantsev-maksim (via GitHub)" <gi...@apache.org>.
kazantsev-maksim commented on code in PR #3044:
URL: https://github.com/apache/datafusion-comet/pull/3044#discussion_r2691437000
##########
dev/benchmarks/tpcbench.py:
##########
@@ -146,15 +146,16 @@ def main(benchmark: str, data_path: str, query_path: str, iterations: int, outpu
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="DataFusion benchmark derived from TPC-H / TPC-DS")
- parser.add_argument("--benchmark", required=True, help="Benchmark to run (tpch or tpcds)")
+ parser.add_argument("--benchmark", required=True, default="tpch", help="Benchmark to run (tpch or tpcds)")
parser.add_argument("--data", required=True, help="Path to data files")
parser.add_argument("--queries", required=True, help="Path to query files")
parser.add_argument("--iterations", required=False, default="1", help="How many iterations to run")
parser.add_argument("--output", required=True, help="Path to write output")
parser.add_argument("--name", required=True, help="Prefix for result file e.g. spark/comet/gluten")
parser.add_argument("--query", required=False, type=int, help="Specific query number to run (1-based). If not specified, all queries will be run.")
parser.add_argument("--write", required=False, help="Path to save query results to, in Parquet format.")
+ parser.add_argument("--format", required=True, default="parquet", help="Input file format (parquet, csv, json)")
Review Comment:
It is necessary to add the ability to pass CSV reading options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Experimental: Native CSV files read [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #3044:
URL: https://github.com/apache/datafusion-comet/pull/3044#issuecomment-3715614581
nice, would love to see benches )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Experimental: Native CSV files read [datafusion-comet]
Posted by "kazantsev-maksim (via GitHub)" <gi...@apache.org>.
kazantsev-maksim commented on PR #3044:
URL: https://github.com/apache/datafusion-comet/pull/3044#issuecomment-3764137083
@comphead @parthchandra @andygrove @mbutrovich I would like to get your feedback — does this PR make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Experimental: Native CSV files read [datafusion-comet]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3044:
URL: https://github.com/apache/datafusion-comet/pull/3044#issuecomment-3731712513
## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/3044?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `74.60317%` with `32 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 16.62%. Comparing base ([`f09f8af`](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`65251eb`](https://app.codecov.io/gh/apache/datafusion-comet/commit/65251eb40b003101d2307cb1ef094e2df4adae34?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 834 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion-comet/pull/3044?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...pache/spark/sql/comet/CometCsvNativeScanExec.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/3044?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fcomet%2FCometCsvNativeScanExec.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9zcGFyay9zcWwvY29tZXQvQ29tZXRDc3ZOYXRpdmVTY2FuRXhlYy5zY2FsYQ==) | 84.21% | [4 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3044?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...cala/org/apache/comet/serde/operator/package.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/3044?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Fserde%2Foperator%2Fpackage.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9zZXJkZS9vcGVyYXRvci9wYWNrYWdlLnNjYWxh) | 71.42% | [7 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3044?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...n/scala/org/apache/comet/rules/CometScanRule.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/3044?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Frules%2FCometScanRule.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9ydWxlcy9Db21ldFNjYW5SdWxlLnNjYWxh) | 76.92% | [2 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3044?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...n/scala/org/apache/spark/sql/comet/operators.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/3044?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fcomet%2Foperators.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9zcGFyay9zcWwvY29tZXQvb3BlcmF0b3JzLnNjYWxh) | 0.00% | [0 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3044?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../apache/comet/serde/operator/CometNativeScan.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/3044?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Fserde%2Foperator%2FCometNativeScan.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9zZXJkZS9vcGVyYXRvci9Db21ldE5hdGl2ZVNjYW4uc2NhbGE=) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3044?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...n/scala/org/apache/comet/rules/CometExecRule.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/3044?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Frules%2FCometExecRule.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9ydWxlcy9Db21ldEV4ZWNSdWxlLnNjYWxh) | 50.00% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3044?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #3044 +/- ##
=============================================
- Coverage 56.12% 16.62% -39.50%
+ Complexity 976 398 -578
=============================================
Files 119 169 +50
Lines 11743 15576 +3833
Branches 2251 2560 +309
=============================================
- Hits 6591 2590 -4001
- Misses 4012 12583 +8571
+ Partials 1140 403 -737
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/3044?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Experimental: Native CSV files read [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #3044:
URL: https://github.com/apache/datafusion-comet/pull/3044#issuecomment-3731600787
Shouldn't CSV be a file format and part of `ScanExec`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Experimental: Native CSV files read [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #3044:
URL: https://github.com/apache/datafusion-comet/pull/3044#issuecomment-3775096098
> I ran a micro-benchmark test to measure the read speed for each TPC-H table.
>
> Results: [native_csv_read.txt](https://github.com/user-attachments/files/24622885/native_csv_read.txt)
That's a nice speedup so I think that it is worth it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Experimental: Native CSV files read [datafusion-comet]
Posted by "kazantsev-maksim (via GitHub)" <gi...@apache.org>.
kazantsev-maksim commented on PR #3044:
URL: https://github.com/apache/datafusion-comet/pull/3044#issuecomment-3750764590
Run TPCH-H test on my local machine
<img width="1000" height="600" alt="tpch_allqueries" src="https://github.com/user-attachments/assets/008c9082-ae43-488b-a968-c99b538a5e34" />
<img width="1500" height="600" alt="tpch_queries_compare" src="https://github.com/user-attachments/assets/3a30b09f-a110-440e-98a3-e73503e1cce8" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Experimental: Native CSV files read [datafusion-comet]
Posted by "kazantsev-maksim (via GitHub)" <gi...@apache.org>.
kazantsev-maksim commented on code in PR #3044:
URL: https://github.com/apache/datafusion-comet/pull/3044#discussion_r2694860679
##########
dev/benchmarks/tpcbench.py:
##########
@@ -146,15 +146,16 @@ def main(benchmark: str, data_path: str, query_path: str, iterations: int, outpu
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="DataFusion benchmark derived from TPC-H / TPC-DS")
- parser.add_argument("--benchmark", required=True, help="Benchmark to run (tpch or tpcds)")
+ parser.add_argument("--benchmark", required=True, default="tpch", help="Benchmark to run (tpch or tpcds)")
parser.add_argument("--data", required=True, help="Path to data files")
parser.add_argument("--queries", required=True, help="Path to query files")
parser.add_argument("--iterations", required=False, default="1", help="How many iterations to run")
parser.add_argument("--output", required=True, help="Path to write output")
parser.add_argument("--name", required=True, help="Prefix for result file e.g. spark/comet/gluten")
parser.add_argument("--query", required=False, type=int, help="Specific query number to run (1-based). If not specified, all queries will be run.")
parser.add_argument("--write", required=False, help="Path to save query results to, in Parquet format.")
+ parser.add_argument("--format", required=True, default="parquet", help="Input file format (parquet, csv, json)")
Review Comment:
For CSV:
tpcbench.py \
--name spark \
--benchmark tpch \
--data $TPCH_DATA \
--queries $TPCH_QUERIES \
--output . \
--iterations 1 \
--format csv
--options '{"header": "true", "delimiter": ","}'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Experimental: Native CSV files read [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #3044:
URL: https://github.com/apache/datafusion-comet/pull/3044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Experimental: Native CSV files read [datafusion-comet]
Posted by "kazantsev-maksim (via GitHub)" <gi...@apache.org>.
kazantsev-maksim commented on PR #3044:
URL: https://github.com/apache/datafusion-comet/pull/3044#issuecomment-3751145205
I ran a micro-benchmark test to measure the read speed for each TPC-H table.
Running benchmark: Native csv read: orders table
Running case: Spark
Stopped after 1 iterations, 238272 ms
Running case: Comet
Stopped after 1 iterations, 55326 ms
OpenJDK 64-Bit Server VM 11.0.26+4-LTS on Mac OS X 26.2
Apple M1 Pro
Native csv read: orders table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 238272 238272 0 0,0 113617,0 1,0X
Comet 55327 55327 0 0,0 26381,9 4,3X
Running benchmark: Native csv read: region table
Running case: Spark
Stopped after 1 iterations, 36 ms
Running case: Comet
Stopped after 1 iterations, 38 ms
OpenJDK 64-Bit Server VM 11.0.26+4-LTS on Mac OS X 26.2
Apple M1 Pro
Native csv read: region table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 36 36 0 57,7 17,3 1,0X
Comet 39 39 0 54,0 18,5 0,9X
Running benchmark: Native csv read: nation table
Running case: Spark
Stopped after 1 iterations, 35 ms
Running case: Comet
Stopped after 1 iterations, 39 ms
OpenJDK 64-Bit Server VM 11.0.26+4-LTS on Mac OS X 26.2
Apple M1 Pro
Native csv read: nation table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 36 36 0 59,0 16,9 1,0X
Comet 39 39 0 53,4 18,7 0,9X
Running benchmark: Native csv read: part table
Running case: Spark
Stopped after 1 iterations, 28883 ms
Running case: Comet
Stopped after 1 iterations, 7870 ms
OpenJDK 64-Bit Server VM 11.0.26+4-LTS on Mac OS X 26.2
Apple M1 Pro
Native csv read: part table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 28884 28884 0 0,1 13773,0 1,0X
Comet 7871 7871 0 0,3 3753,2 3,7X
Running benchmark: Native csv read: supplier table
Running case: Spark
Stopped after 1 iterations, 1380 ms
Running case: Comet
Stopped after 1 iterations, 413 ms
OpenJDK 64-Bit Server VM 11.0.26+4-LTS on Mac OS X 26.2
Apple M1 Pro
Native csv read: supplier table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 1381 1381 0 1,5 658,3 1,0X
Comet 414 414 0 5,1 197,2 3,3X
Running benchmark: Native csv read: partsupp table
Running case: Spark
Stopped after 1 iterations, 107904 ms
Running case: Comet
Stopped after 1 iterations, 26308 ms
OpenJDK 64-Bit Server VM 11.0.26+4-LTS on Mac OS X 26.2
Apple M1 Pro
Native csv read: partsupp table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 107905 107905 0 0,0 51452,9 1,0X
Comet 26308 26308 0 0,1 12544,8 4,1X
Running benchmark: Native csv read: customer table
Running case: Spark
Stopped after 1 iterations, 22288 ms
Running case: Comet
Stopped after 1 iterations, 6089 ms
OpenJDK 64-Bit Server VM 11.0.26+4-LTS on Mac OS X 26.2
Apple M1 Pro
Native csv read: customer table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 22289 22289 0 0,1 10628,0 1,0X
Comet 6090 6090 0 0,3 2903,8 3,7X
Running benchmark: Native csv read: lineitem table
Running case: Spark
Stopped after 1 iterations, 1331928 ms
Running case: Comet
Stopped after 1 iterations, 342292 ms
OpenJDK 64-Bit Server VM 11.0.26+4-LTS on Mac OS X 26.2
Apple M1 Pro
Native csv read: lineitem table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 1331928 1331928 0 0,0 635112,8 1,0X
Comet 342293 342293 0 0,0 163218,0 3,9X
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Experimental: Native CSV files read [datafusion-comet]
Posted by "kazantsev-maksim (via GitHub)" <gi...@apache.org>.
kazantsev-maksim commented on PR #3044:
URL: https://github.com/apache/datafusion-comet/pull/3044#issuecomment-3734909247
Thanks @parthchandra, you are absolutely right. In the first phase, I wanted to implement it only for DataSourceV2 to check the performance improvement. I hope to finish the benchmark tests in the coming days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org