You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "beyond1920 (via GitHub)" <gi...@apache.org> on 2023/11/17 11:16:51 UTC
[PR] [HUDI-7118] set conf 'spark.sql.parquet.enableVectorizedReader' to true automatically only if the value is not explicitly set [hudi]
beyond1920 opened a new pull request, #10134:
URL: https://github.com/apache/hudi/pull/10134
### Change Logs
Currently, conf 'spark.sql.parquet.enableVectorizedReader' does not work properly. See complete discussion in [issue-9129](https://github.com/apache/hudi/issues/9129).
The pr aims to fix it.
### Impact
none
### Risk level (write none, low medium or high below)
none
### Documentation Update
NA
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7118] set conf 'spark.sql.parquet.enableVectorizedReader' to true automatically only if the value is not explicitly set [hudi]
Posted by "xiarixiaoyao (via GitHub)" <gi...@apache.org>.
xiarixiaoyao commented on code in PR #10134:
URL: https://github.com/apache/hudi/pull/10134#discussion_r1398156052
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/BaseFileOnlyRelation.scala:
##########
@@ -81,7 +81,9 @@ case class BaseFileOnlyRelation(override val sqlContext: SQLContext,
super.imbueConfigs(sqlContext)
// TODO Issue with setting this to true in spark 332
if (HoodieSparkUtils.gteqSpark3_4 || !HoodieSparkUtils.gteqSpark3_3_2) {
- sqlContext.sparkSession.sessionState.conf.setConfString("spark.sql.parquet.enableVectorizedReader", "true")
Review Comment:
should not use contains.
spark always inject this config.
The reason why this place is forcibly set to true is because the parent class of BaseFileOnlyRelationship previously forced set spark.sql.parquet.enableVectorizedReader=false. users unable to turn on vectorization, resulting in a significant decrease in query performance. So at that time, it was forcibly set to true in this location.
however https://issues.apache.org/jira/browse/HUDI-3639 This PR removes the behavior of the parent class of BaseFileOnlyRelationship.
Therefore, we no longer need to force set spark.sql.parquet.enableVectorizedReader=true in BaseFileOnlyRelationship; Whether to enable vectorization is left to the user for decision
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7118] set conf 'spark.sql.parquet.enableVectorizedReader' to true automatically only if the value is not explicitly set [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10134:
URL: https://github.com/apache/hudi/pull/10134#issuecomment-1816271730
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20977",
"triggerID" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59879faa666b5e819d00c4a8a6870a019c299bae Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20977)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7118] set conf 'spark.sql.parquet.enableVectorizedReader' to true automatically only if the value is not explicitly set [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10134:
URL: https://github.com/apache/hudi/pull/10134#issuecomment-1817475762
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20977",
"triggerID" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"triggerType" : "PUSH"
}, {
"hash" : "ed3c70df047f15d896385eebbbc21ec820ecf152",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20995",
"triggerID" : "ed3c70df047f15d896385eebbbc21ec820ecf152",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed3c70df047f15d896385eebbbc21ec820ecf152 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20995)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7118] set conf 'spark.sql.parquet.enableVectorizedReader' to true automatically only if the value is not explicitly set [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10134:
URL: https://github.com/apache/hudi/pull/10134#issuecomment-1817451558
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20977",
"triggerID" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"triggerType" : "PUSH"
}, {
"hash" : "ed3c70df047f15d896385eebbbc21ec820ecf152",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed3c70df047f15d896385eebbbc21ec820ecf152",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59879faa666b5e819d00c4a8a6870a019c299bae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20977)
* ed3c70df047f15d896385eebbbc21ec820ecf152 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7118] set conf 'spark.sql.parquet.enableVectorizedReader' to true automatically only if the value is not explicitly set [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10134:
URL: https://github.com/apache/hudi/pull/10134#issuecomment-1817518043
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20977",
"triggerID" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"triggerType" : "PUSH"
}, {
"hash" : "ed3c70df047f15d896385eebbbc21ec820ecf152",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20995",
"triggerID" : "ed3c70df047f15d896385eebbbc21ec820ecf152",
"triggerType" : "PUSH"
}, {
"hash" : "557ed312562e69900233d07a983fbc1d3db271ba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "557ed312562e69900233d07a983fbc1d3db271ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed3c70df047f15d896385eebbbc21ec820ecf152 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20995)
* 557ed312562e69900233d07a983fbc1d3db271ba UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7118] set conf 'spark.sql.parquet.enableVectorizedReader' to true automatically only if the value is not explicitly set [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10134:
URL: https://github.com/apache/hudi/pull/10134#issuecomment-1816260119
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59879faa666b5e819d00c4a8a6870a019c299bae UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7118] set conf 'spark.sql.parquet.enableVectorizedReader' to true automatically only if the value is not explicitly set [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10134:
URL: https://github.com/apache/hudi/pull/10134#issuecomment-1817519641
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20977",
"triggerID" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"triggerType" : "PUSH"
}, {
"hash" : "ed3c70df047f15d896385eebbbc21ec820ecf152",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20995",
"triggerID" : "ed3c70df047f15d896385eebbbc21ec820ecf152",
"triggerType" : "PUSH"
}, {
"hash" : "557ed312562e69900233d07a983fbc1d3db271ba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20997",
"triggerID" : "557ed312562e69900233d07a983fbc1d3db271ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed3c70df047f15d896385eebbbc21ec820ecf152 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20995)
* 557ed312562e69900233d07a983fbc1d3db271ba Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20997)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7118] set conf 'spark.sql.parquet.enableVectorizedReader' to true automatically only if the value is not explicitly set [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10134:
URL: https://github.com/apache/hudi/pull/10134#issuecomment-1817544813
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20977",
"triggerID" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"triggerType" : "PUSH"
}, {
"hash" : "ed3c70df047f15d896385eebbbc21ec820ecf152",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20995",
"triggerID" : "ed3c70df047f15d896385eebbbc21ec820ecf152",
"triggerType" : "PUSH"
}, {
"hash" : "557ed312562e69900233d07a983fbc1d3db271ba",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20997",
"triggerID" : "557ed312562e69900233d07a983fbc1d3db271ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 557ed312562e69900233d07a983fbc1d3db271ba Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20997)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7118] set conf 'spark.sql.parquet.enableVectorizedReader' to true automatically only if the value is not explicitly set [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #10134:
URL: https://github.com/apache/hudi/pull/10134
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7118] set conf 'spark.sql.parquet.enableVectorizedReader' to true automatically only if the value is not explicitly set [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10134:
URL: https://github.com/apache/hudi/pull/10134#issuecomment-1816574877
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20977",
"triggerID" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59879faa666b5e819d00c4a8a6870a019c299bae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20977)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7118] set conf 'spark.sql.parquet.enableVectorizedReader' to true automatically only if the value is not explicitly set [hudi]
Posted by "xiarixiaoyao (via GitHub)" <gi...@apache.org>.
xiarixiaoyao commented on code in PR #10134:
URL: https://github.com/apache/hudi/pull/10134#discussion_r1398157539
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/BaseFileOnlyRelation.scala:
##########
@@ -81,7 +81,9 @@ case class BaseFileOnlyRelation(override val sqlContext: SQLContext,
super.imbueConfigs(sqlContext)
// TODO Issue with setting this to true in spark 332
if (HoodieSparkUtils.gteqSpark3_4 || !HoodieSparkUtils.gteqSpark3_3_2) {
- sqlContext.sparkSession.sessionState.conf.setConfString("spark.sql.parquet.enableVectorizedReader", "true")
Review Comment:
just remove line 83. to line 85
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7118] set conf 'spark.sql.parquet.enableVectorizedReader' to true automatically only if the value is not explicitly set [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10134:
URL: https://github.com/apache/hudi/pull/10134#issuecomment-1817453518
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20977",
"triggerID" : "59879faa666b5e819d00c4a8a6870a019c299bae",
"triggerType" : "PUSH"
}, {
"hash" : "ed3c70df047f15d896385eebbbc21ec820ecf152",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20995",
"triggerID" : "ed3c70df047f15d896385eebbbc21ec820ecf152",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59879faa666b5e819d00c4a8a6870a019c299bae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20977)
* ed3c70df047f15d896385eebbbc21ec820ecf152 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20995)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7118] set conf 'spark.sql.parquet.enableVectorizedReader' to true automatically only if the value is not explicitly set [hudi]
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 commented on code in PR #10134:
URL: https://github.com/apache/hudi/pull/10134#discussion_r1398208223
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/BaseFileOnlyRelation.scala:
##########
@@ -81,7 +81,9 @@ case class BaseFileOnlyRelation(override val sqlContext: SQLContext,
super.imbueConfigs(sqlContext)
// TODO Issue with setting this to true in spark 332
if (HoodieSparkUtils.gteqSpark3_4 || !HoodieSparkUtils.gteqSpark3_3_2) {
- sqlContext.sparkSession.sessionState.conf.setConfString("spark.sql.parquet.enableVectorizedReader", "true")
Review Comment:
Thanks for reminder. Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org