You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "parthchandra (via GitHub)" <gi...@apache.org> on 2024/09/25 18:26:32 UTC
[PR] chore: Make parquet reader options Comet options instead of Hadoop options [datafusion-comet]
parthchandra opened a new pull request, #968:
URL: https://github.com/apache/datafusion-comet/pull/968
## Which issue does this PR close?
Closes #876 .
## Rationale for this change
Moves configuration options of the parallel reader into CometConf and adds documentation so that end users can easily understand the configuration of the reader.
## How are these changes tested?
Existing unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Make parquet reader options Comet options instead of Hadoop options [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #968:
URL: https://github.com/apache/datafusion-comet/pull/968#discussion_r1776037634
##########
common/src/test/java/org/apache/comet/parquet/TestFileReader.java:
##########
@@ -615,12 +617,12 @@ public void testColumnIndexReadWrite() throws Exception {
@Test
public void testWriteReadMergeScanRange() throws Throwable {
Configuration conf = new Configuration();
- conf.set(ReadOptions.COMET_IO_MERGE_RANGES, Boolean.toString(true));
+ conf.set(CometConf.COMET_IO_MERGE_RANGES().key(), Boolean.toString(true));
Review Comment:
This test is setting the config values in a Hadoop Configuration still, and not setting them in the Spark config. Would it make sense to update the test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Make parquet reader options Comet options instead of Hadoop options [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #968:
URL: https://github.com/apache/datafusion-comet/pull/968#discussion_r1776038555
##########
common/src/main/java/org/apache/comet/parquet/ReadOptions.java:
##########
@@ -173,14 +147,24 @@ public Builder(Configuration conf) {
this.conf = conf;
this.parallelIOEnabled =
conf.getBoolean(
Review Comment:
This is reading from the Hadoop conf. If I set the new configs on my Spark context, how would they get propagated to the Hadoop conf?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org