You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/07/12 09:20:42 UTC
[GitHub] [hudi] TengHuo opened a new pull request, #6085: [HUDI-4384] fix hive style partition and record key prefix missing in bulk_insert
TengHuo opened a new pull request, #6085:
URL: https://github.com/apache/hudi/pull/6085
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
Fix [HUDI-4384] hive style partition and record key prefix missing in bulk_insert in Spark
## Brief change log
- Remove `ComplexKeyGenerator` bypass in `HoodieDatasetBulkInsertHelper.prepareHoodieDatasetForBulkInsert`
- Add a new unit test method named `testBulkInsertWithHiveStylePartition` in `TestHoodieDatasetBulkInsertHelper`
## Verify this pull request
This change added tests and can be verified as follows:
- Added a new unit test method named `testBulkInsertWithHiveStylePartition` in `TestHoodieDatasetBulkInsertHelper`
- Manually verified the change by running a job locally.
## Committer checklist
- [x] Has a corresponding JIRA in PR title & commit
- [x] Commit message is descriptive of the change
- [ ] CI is green
- [x] Necessary doc changes done or have another open PR (no need)
- [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. (no need)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6085: [HUDI-4384] fix hive style partition and record key prefix missing in bulk_insert
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6085:
URL: https://github.com/apache/hudi/pull/6085#issuecomment-1181651546
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5b64362e190b0b51c9aa81f03960ff08d1e8ec71",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9866",
"triggerID" : "5b64362e190b0b51c9aa81f03960ff08d1e8ec71",
"triggerType" : "PUSH"
}, {
"hash" : "5b64362e190b0b51c9aa81f03960ff08d1e8ec71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1181647662",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 5b64362e190b0b51c9aa81f03960ff08d1e8ec71 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9866)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on a diff in pull request #6085: [HUDI-4384] fix hive style partition and record key prefix missing in bulk_insert
Posted by GitBox <gi...@apache.org>.
TengHuo commented on code in PR #6085:
URL: https://github.com/apache/hudi/pull/6085#discussion_r923249802
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/HoodieDatasetBulkInsertHelper.java:
##########
@@ -87,18 +86,15 @@ public static Dataset<Row> prepareHoodieDatasetForBulkInsert(SQLContext sqlConte
BuiltinKeyGenerator keyGenerator = (BuiltinKeyGenerator) ReflectionUtils.loadClass(keyGeneratorClass, properties);
Dataset<Row> rowDatasetWithRecordKeysAndPartitionPath;
- if (keyGeneratorClass.equals(NonpartitionedKeyGenerator.class.getName())) {
+ if (keyGeneratorClass.equals(NonpartitionedKeyGenerator.class.getName())
+ || (keyGeneratorClass.equals(SimpleKeyGenerator.class.getName()) && !config.isHiveStylePartitioningEnabled())) {
Review Comment:
yeah, this PR will fall back to UDF if hive style partition enabled, same logic as 0.10.
think PR #6049 is a better fix which can improve performance by using withColumn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6085: [HUDI-4384] fix hive style partition and record key prefix missing in bulk_insert
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6085:
URL: https://github.com/apache/hudi/pull/6085#issuecomment-1181533300
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5b64362e190b0b51c9aa81f03960ff08d1e8ec71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5b64362e190b0b51c9aa81f03960ff08d1e8ec71",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5b64362e190b0b51c9aa81f03960ff08d1e8ec71 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #6085: [HUDI-4384] fix hive style partition and record key prefix missing in bulk_insert
Posted by GitBox <gi...@apache.org>.
TengHuo commented on PR #6085:
URL: https://github.com/apache/hudi/pull/6085#issuecomment-1181647662
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed pull request #6085: [HUDI-4384] fix hive style partition and record key prefix missing in bulk_insert
Posted by GitBox <gi...@apache.org>.
nsivabalan closed pull request #6085: [HUDI-4384] fix hive style partition and record key prefix missing in bulk_insert
URL: https://github.com/apache/hudi/pull/6085
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6085: [HUDI-4384] fix hive style partition and record key prefix missing in bulk_insert
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6085:
URL: https://github.com/apache/hudi/pull/6085#issuecomment-1181694668
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5b64362e190b0b51c9aa81f03960ff08d1e8ec71",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9866",
"triggerID" : "5b64362e190b0b51c9aa81f03960ff08d1e8ec71",
"triggerType" : "PUSH"
}, {
"hash" : "5b64362e190b0b51c9aa81f03960ff08d1e8ec71",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9866",
"triggerID" : "1181647662",
"triggerType" : "MANUAL"
}, {
"hash" : "5b64362e190b0b51c9aa81f03960ff08d1e8ec71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1181654190",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 5b64362e190b0b51c9aa81f03960ff08d1e8ec71 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9866)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6085: [HUDI-4384] fix hive style partition and record key prefix missing in bulk_insert
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6085:
URL: https://github.com/apache/hudi/pull/6085#issuecomment-1181577011
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5b64362e190b0b51c9aa81f03960ff08d1e8ec71",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9866",
"triggerID" : "5b64362e190b0b51c9aa81f03960ff08d1e8ec71",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5b64362e190b0b51c9aa81f03960ff08d1e8ec71 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9866)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #6085: [HUDI-4384] fix hive style partition and record key prefix missing in bulk_insert
Posted by GitBox <gi...@apache.org>.
TengHuo commented on PR #6085:
URL: https://github.com/apache/hudi/pull/6085#issuecomment-1182710127
The test `UT FT clients & cli & utilities & sync` failed due to an exception `java.net.ConnectException: Connection refused` in the test of hudi-hive-sync, it should not be caused by this PR.
May I know how I can fix it? or should I re-run the test again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #6085: [HUDI-4384] fix hive style partition and record key prefix missing in bulk_insert
Posted by GitBox <gi...@apache.org>.
TengHuo commented on PR #6085:
URL: https://github.com/apache/hudi/pull/6085#issuecomment-1181549566
Manually verified the change by running a job locally. The reproduce code can be found in JIRA ticket: HUDI-4384
Result After fix:
<img width="1195" alt="Screenshot 2022-07-12 at 17 39 15" src="https://user-images.githubusercontent.com/7539060/178461193-1c8b9cad-403b-4c47-9977-70945eda3ab9.png">
Result before fix:
<img width="1192" alt="Screenshot 2022-07-12 at 16 39 58" src="https://user-images.githubusercontent.com/7539060/178461254-adae169a-471b-4715-a1ec-551f34f06fb1.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #6085: [HUDI-4384] fix hive style partition and record key prefix missing in bulk_insert
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6085:
URL: https://github.com/apache/hudi/pull/6085#discussion_r923226460
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/HoodieDatasetBulkInsertHelper.java:
##########
@@ -87,18 +86,15 @@ public static Dataset<Row> prepareHoodieDatasetForBulkInsert(SQLContext sqlConte
BuiltinKeyGenerator keyGenerator = (BuiltinKeyGenerator) ReflectionUtils.loadClass(keyGeneratorClass, properties);
Dataset<Row> rowDatasetWithRecordKeysAndPartitionPath;
- if (keyGeneratorClass.equals(NonpartitionedKeyGenerator.class.getName())) {
+ if (keyGeneratorClass.equals(NonpartitionedKeyGenerator.class.getName())
+ || (keyGeneratorClass.equals(SimpleKeyGenerator.class.getName()) && !config.isHiveStylePartitioningEnabled())) {
Review Comment:
if hive style partitioning is enabled, we are falling back to using udf flow is it? guess the intent was to do use udf based key gen only for non simple use-cases. Can we honor the same even w/ hive style partitioning enabled please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6085: [HUDI-4384] fix hive style partition and record key prefix missing in bulk_insert
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6085:
URL: https://github.com/apache/hudi/pull/6085#issuecomment-1181698954
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5b64362e190b0b51c9aa81f03960ff08d1e8ec71",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9866",
"triggerID" : "5b64362e190b0b51c9aa81f03960ff08d1e8ec71",
"triggerType" : "PUSH"
}, {
"hash" : "5b64362e190b0b51c9aa81f03960ff08d1e8ec71",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9866",
"triggerID" : "1181647662",
"triggerType" : "MANUAL"
}, {
"hash" : "5b64362e190b0b51c9aa81f03960ff08d1e8ec71",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9866",
"triggerID" : "1181654190",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 5b64362e190b0b51c9aa81f03960ff08d1e8ec71 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9866)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #6085: [HUDI-4384] fix hive style partition and record key prefix missing in bulk_insert
Posted by GitBox <gi...@apache.org>.
TengHuo commented on PR #6085:
URL: https://github.com/apache/hudi/pull/6085#issuecomment-1181654190
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #6085: [HUDI-4384] fix hive style partition and record key prefix missing in bulk_insert
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #6085:
URL: https://github.com/apache/hudi/pull/6085#issuecomment-1187053167
since the other is marked as priority blocker, closing this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #6085: [HUDI-4384] fix hive style partition and record key prefix missing in bulk_insert
Posted by GitBox <gi...@apache.org>.
TengHuo commented on PR #6085:
URL: https://github.com/apache/hudi/pull/6085#issuecomment-1181532130
Fix https://github.com/apache/hudi/issues/6070
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TengHuo commented on pull request #6085: [HUDI-4384] fix hive style partition and record key prefix missing in bulk_insert
Posted by GitBox <gi...@apache.org>.
TengHuo commented on PR #6085:
URL: https://github.com/apache/hudi/pull/6085#issuecomment-1184137184
Duplicate with #6049, will close this PR if #6049 is merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org