You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lvhu-goodluck (via GitHub)" <gi...@apache.org> on 2023/02/16 02:38:20 UTC
[GitHub] [hudi] lvhu-goodluck opened a new pull request, #7975: Hash partition in spark data source
lvhu-goodluck opened a new pull request, #7975:
URL: https://github.com/apache/hudi/pull/7975
### Change Logs
Add hash partition in spark data source.
How to use hash partition in spark data source can refer to hudi/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala#testHashPartition
### Impact
No public API or user-facing feature change if hash partition parameter is not specified.
When hash.partition.fields is specified and partition.fields contains _hoodie_hash_partition, a column named _hoodie_hash_partition will be added in this table as one of the partition key.
If predicates of hash.partition.fields appear in the query statement, the _hoodie_hash_partition = X predicate will be automatically added to the query statement for partition pruning.
### Risk level (write none, low medium or high below)
Low medium.
### Documentation Update
To be continued.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] lvhu-goodluck closed pull request #7975: Hash partition in spark data source
Posted by "lvhu-goodluck (via GitHub)" <gi...@apache.org>.
lvhu-goodluck closed pull request #7975: Hash partition in spark data source
URL: https://github.com/apache/hudi/pull/7975
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7975: Hash partition in spark data source
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7975:
URL: https://github.com/apache/hudi/pull/7975#issuecomment-1432474868
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b175c2cd6dedb484a60284bc347ee67655c3621e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15226",
"triggerID" : "b175c2cd6dedb484a60284bc347ee67655c3621e",
"triggerType" : "PUSH"
}, {
"hash" : "8f4fa79f5ebd485c07517df9fc4dcac5154f5035",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8f4fa79f5ebd485c07517df9fc4dcac5154f5035",
"triggerType" : "PUSH"
}, {
"hash" : "21d11c1ab3d55bcdb8b1aea0b95d7d8dbcbfd26d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15231",
"triggerID" : "21d11c1ab3d55bcdb8b1aea0b95d7d8dbcbfd26d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b175c2cd6dedb484a60284bc347ee67655c3621e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15226)
* 8f4fa79f5ebd485c07517df9fc4dcac5154f5035 UNKNOWN
* 21d11c1ab3d55bcdb8b1aea0b95d7d8dbcbfd26d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15231)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7975: Hash partition in spark data source
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7975:
URL: https://github.com/apache/hudi/pull/7975#issuecomment-1432468827
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b175c2cd6dedb484a60284bc347ee67655c3621e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15226",
"triggerID" : "b175c2cd6dedb484a60284bc347ee67655c3621e",
"triggerType" : "PUSH"
}, {
"hash" : "8f4fa79f5ebd485c07517df9fc4dcac5154f5035",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8f4fa79f5ebd485c07517df9fc4dcac5154f5035",
"triggerType" : "PUSH"
}, {
"hash" : "21d11c1ab3d55bcdb8b1aea0b95d7d8dbcbfd26d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "21d11c1ab3d55bcdb8b1aea0b95d7d8dbcbfd26d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b175c2cd6dedb484a60284bc347ee67655c3621e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15226)
* 8f4fa79f5ebd485c07517df9fc4dcac5154f5035 UNKNOWN
* 21d11c1ab3d55bcdb8b1aea0b95d7d8dbcbfd26d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7975: Hash partition in spark data source
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7975:
URL: https://github.com/apache/hudi/pull/7975#issuecomment-1432460657
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b175c2cd6dedb484a60284bc347ee67655c3621e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15226",
"triggerID" : "b175c2cd6dedb484a60284bc347ee67655c3621e",
"triggerType" : "PUSH"
}, {
"hash" : "8f4fa79f5ebd485c07517df9fc4dcac5154f5035",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8f4fa79f5ebd485c07517df9fc4dcac5154f5035",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b175c2cd6dedb484a60284bc347ee67655c3621e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15226)
* 8f4fa79f5ebd485c07517df9fc4dcac5154f5035 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7975: Hash partition in spark data source
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7975:
URL: https://github.com/apache/hudi/pull/7975#issuecomment-1432406888
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b175c2cd6dedb484a60284bc347ee67655c3621e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b175c2cd6dedb484a60284bc347ee67655c3621e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b175c2cd6dedb484a60284bc347ee67655c3621e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7975: Hash partition in spark data source
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7975:
URL: https://github.com/apache/hudi/pull/7975#issuecomment-1432413913
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b175c2cd6dedb484a60284bc347ee67655c3621e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15226",
"triggerID" : "b175c2cd6dedb484a60284bc347ee67655c3621e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b175c2cd6dedb484a60284bc347ee67655c3621e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15226)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org