You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/08/12 14:01:22 UTC
[GitHub] [hudi] codope opened a new pull request #3465: [HUDI-1363] Provide option to drop partition columns
codope opened a new pull request #3465:
URL: https://github.com/apache/hudi/pull/3465
## What is the purpose of the pull request
Rework of https://github.com/apache/hudi/pull/2225
Currently, default is set to True in order to check CI with other features. Will revert after CI passes.
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
Added a unit test and verified locally with spark datasource.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#issuecomment-897667019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1688",
"triggerID" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"triggerType" : "PUSH"
}, {
"hash" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1689",
"triggerID" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1688)
* 8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1689)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#issuecomment-897667019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1688",
"triggerID" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"triggerType" : "PUSH"
}, {
"hash" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1689",
"triggerID" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"triggerType" : "PUSH"
}, {
"hash" : "5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c",
"triggerType" : "PUSH"
}, {
"hash" : "0038a70398da3848ccccf20391542e918b841dae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1696",
"triggerID" : "0038a70398da3848ccccf20391542e918b841dae",
"triggerType" : "PUSH"
}, {
"hash" : "630708d139fedd503131d5c378663970913ddb93",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1715",
"triggerID" : "630708d139fedd503131d5c378663970913ddb93",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c UNKNOWN
* 630708d139fedd503131d5c378663970913ddb93 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1715)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#issuecomment-897667019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1688",
"triggerID" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"triggerType" : "PUSH"
}, {
"hash" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1689",
"triggerID" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"triggerType" : "PUSH"
}, {
"hash" : "5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c",
"triggerType" : "PUSH"
}, {
"hash" : "0038a70398da3848ccccf20391542e918b841dae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1696",
"triggerID" : "0038a70398da3848ccccf20391542e918b841dae",
"triggerType" : "PUSH"
}, {
"hash" : "630708d139fedd503131d5c378663970913ddb93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "630708d139fedd503131d5c378663970913ddb93",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c UNKNOWN
* 0038a70398da3848ccccf20391542e918b841dae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1696)
* 630708d139fedd503131d5c378663970913ddb93 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Ales-Semchenkov commented on a change in pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
Ales-Semchenkov commented on a change in pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#discussion_r691235858
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -223,21 +240,25 @@ object HoodieSparkSqlWriter {
operation.equals(WriteOperationType.UPSERT) ||
parameters.getOrElse(HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key(),
HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.defaultValue()).toBoolean
+ val partitionKey = hoodieConfig.getString(DataSourceWriteOptions.PARTITIONPATH_FIELD)
Review comment:
@xuzifu666
We have tried Hudi with the PR and it solved the problem with the BigQuery index conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#issuecomment-897667019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1688",
"triggerID" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1688)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
codope commented on pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#issuecomment-898530926
> @codope : Can you review this patch. I have updated w/ some changes
LGTM. Resolved conflicts and tested locally with spark datasource. Should be good to land once CI passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#issuecomment-897667019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1688",
"triggerID" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1688)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#discussion_r691889565
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##########
@@ -223,9 +223,13 @@ public static Schema addMetadataFields(Schema schema, boolean withOperationField
}
public static Schema removeMetadataFields(Schema schema) {
+ return removeFields(schema, HoodieRecord.HOODIE_META_COLUMNS);
+ }
+
+ public static Schema removeFields(Schema schema, List<String> fieldsToRemove) {
List<Schema.Field> filteredFields = schema.getFields()
.stream()
- .filter(field -> !HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION.contains(field.name()))
+ .filter(field -> !fieldsToRemove.contains(field.name()))
.map(field -> new Schema.Field(field.name(), field.schema(), field.doc(), field.defaultVal()))
Review comment:
`_hoodie_operation` is a metadata field !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#issuecomment-897667019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1688",
"triggerID" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"triggerType" : "PUSH"
}, {
"hash" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1689",
"triggerID" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"triggerType" : "PUSH"
}, {
"hash" : "5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c",
"triggerType" : "PUSH"
}, {
"hash" : "0038a70398da3848ccccf20391542e918b841dae",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0038a70398da3848ccccf20391542e918b841dae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1689)
* 5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c UNKNOWN
* 0038a70398da3848ccccf20391542e918b841dae UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#discussion_r691721654
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##########
@@ -223,9 +223,13 @@ public static Schema addMetadataFields(Schema schema, boolean withOperationField
}
public static Schema removeMetadataFields(Schema schema) {
+ return removeFields(schema, HoodieRecord.HOODIE_META_COLUMNS);
+ }
+
+ public static Schema removeFields(Schema schema, List<String> fieldsToRemove) {
List<Schema.Field> filteredFields = schema.getFields()
.stream()
- .filter(field -> !HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION.contains(field.name()))
+ .filter(field -> !fieldsToRemove.contains(field.name()))
.map(field -> new Schema.Field(field.name(), field.schema(), field.doc(), field.defaultVal()))
Review comment:
Hey, is this right ? the logic is not equivalent: `HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION` -> `HoodieRecord.HOODIE_META_COLUMNS`, i added the `HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION` because when flink turns on the 'changelog mode', there is another metadata field named '_hoodie_operation'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#issuecomment-897667019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1688",
"triggerID" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"triggerType" : "PUSH"
}, {
"hash" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1689",
"triggerID" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"triggerType" : "PUSH"
}, {
"hash" : "5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1689)
* 5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#issuecomment-897667019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#issuecomment-897667019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1688",
"triggerID" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"triggerType" : "PUSH"
}, {
"hash" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1689",
"triggerID" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"triggerType" : "PUSH"
}, {
"hash" : "5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c",
"triggerType" : "PUSH"
}, {
"hash" : "0038a70398da3848ccccf20391542e918b841dae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1696",
"triggerID" : "0038a70398da3848ccccf20391542e918b841dae",
"triggerType" : "PUSH"
}, {
"hash" : "630708d139fedd503131d5c378663970913ddb93",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1715",
"triggerID" : "630708d139fedd503131d5c378663970913ddb93",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c UNKNOWN
* 0038a70398da3848ccccf20391542e918b841dae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1696)
* 630708d139fedd503131d5c378663970913ddb93 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1715)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #3465:
URL: https://github.com/apache/hudi/pull/3465
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#issuecomment-897667019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1688",
"triggerID" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"triggerType" : "PUSH"
}, {
"hash" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1689",
"triggerID" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1689)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xuzifu666 commented on a change in pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
xuzifu666 commented on a change in pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#discussion_r690224337
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -223,21 +240,25 @@ object HoodieSparkSqlWriter {
operation.equals(WriteOperationType.UPSERT) ||
parameters.getOrElse(HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key(),
HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.defaultValue()).toBoolean
+ val partitionKey = hoodieConfig.getString(DataSourceWriteOptions.PARTITIONPATH_FIELD)
Review comment:
hi codope! ask a question, user can drop partition data through the pr ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#discussion_r691368980
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -223,21 +240,25 @@ object HoodieSparkSqlWriter {
operation.equals(WriteOperationType.UPSERT) ||
parameters.getOrElse(HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key(),
HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.defaultValue()).toBoolean
+ val partitionKey = hoodieConfig.getString(DataSourceWriteOptions.PARTITIONPATH_FIELD)
Review comment:
@xuzifu666 Yes, you can drop partition fields with this patch.
@Ales-Semchenkov Good to hear!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#discussion_r691368980
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -223,21 +240,25 @@ object HoodieSparkSqlWriter {
operation.equals(WriteOperationType.UPSERT) ||
parameters.getOrElse(HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key(),
HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.defaultValue()).toBoolean
+ val partitionKey = hoodieConfig.getString(DataSourceWriteOptions.PARTITIONPATH_FIELD)
Review comment:
@xuzifu666 Yes, you can drop partition fields with this patch.
@Ales-Semchenkov Good to hear!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#discussion_r691915751
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##########
@@ -223,9 +223,13 @@ public static Schema addMetadataFields(Schema schema, boolean withOperationField
}
public static Schema removeMetadataFields(Schema schema) {
+ return removeFields(schema, HoodieRecord.HOODIE_META_COLUMNS);
+ }
+
+ public static Schema removeFields(Schema schema, List<String> fieldsToRemove) {
List<Schema.Field> filteredFields = schema.getFields()
.stream()
- .filter(field -> !HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION.contains(field.name()))
+ .filter(field -> !fieldsToRemove.contains(field.name()))
.map(field -> new Schema.Field(field.name(), field.schema(), field.doc(), field.defaultVal()))
Review comment:
Thanks @danny0405 for pointing this out! I'll create a patch to fix this today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#discussion_r691721654
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##########
@@ -223,9 +223,13 @@ public static Schema addMetadataFields(Schema schema, boolean withOperationField
}
public static Schema removeMetadataFields(Schema schema) {
+ return removeFields(schema, HoodieRecord.HOODIE_META_COLUMNS);
+ }
+
+ public static Schema removeFields(Schema schema, List<String> fieldsToRemove) {
List<Schema.Field> filteredFields = schema.getFields()
.stream()
- .filter(field -> !HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION.contains(field.name()))
+ .filter(field -> !fieldsToRemove.contains(field.name()))
.map(field -> new Schema.Field(field.name(), field.schema(), field.doc(), field.defaultVal()))
Review comment:
Hey, is this right ? the logic is not equivalent: HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION -> HoodieRecord.HOODIE_META_COLUMNS, i added the HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION because when flink turns on the 'changelog mode', there is another metadata field named '_hoodie_operation'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#issuecomment-897667019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1688",
"triggerID" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"triggerType" : "PUSH"
}, {
"hash" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1688)
* 8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#discussion_r691791697
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##########
@@ -223,9 +223,13 @@ public static Schema addMetadataFields(Schema schema, boolean withOperationField
}
public static Schema removeMetadataFields(Schema schema) {
+ return removeFields(schema, HoodieRecord.HOODIE_META_COLUMNS);
+ }
+
+ public static Schema removeFields(Schema schema, List<String> fieldsToRemove) {
List<Schema.Field> filteredFields = schema.getFields()
.stream()
- .filter(field -> !HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION.contains(field.name()))
+ .filter(field -> !fieldsToRemove.contains(field.name()))
.map(field -> new Schema.Field(field.name(), field.schema(), field.doc(), field.defaultVal()))
Review comment:
is "_hoodie_operation" considered a meta field? then yeah, we need to fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
codope commented on pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#issuecomment-897756642
There are 38 failures/errors in the following tests:
```
TestCOWDataSource
TestMORDataSource
TestHoodieFileIndex
TestDataSourceForBootstrap
TestTimeTravelQuery
```
All of them are due to schema mismatch. Will revert the config and run CI again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#issuecomment-897667019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1688",
"triggerID" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"triggerType" : "PUSH"
}, {
"hash" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1689",
"triggerID" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"triggerType" : "PUSH"
}, {
"hash" : "5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c",
"triggerType" : "PUSH"
}, {
"hash" : "0038a70398da3848ccccf20391542e918b841dae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1696",
"triggerID" : "0038a70398da3848ccccf20391542e918b841dae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c UNKNOWN
* 0038a70398da3848ccccf20391542e918b841dae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1696)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#discussion_r688043954
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -461,6 +461,12 @@ object DataSourceWriteOptions {
.defaultValue("io.confluent.kafka.serializers.KafkaAvroDeserializer")
.sinceVersion("0.9.0")
.withDocumentation("This class is used by kafka client to deserialize the records")
+
+ val ENABLE_DROP_PARTITION_COLUMNS: ConfigProperty[String] = ConfigProperty
+ .key("hoodie.datasource.write.drop.partition.columns.enable")
+ .defaultValue("false")
+ .withDocumentation("When set to true, will not write the partition columns into parquet file. " +
Review comment:
minor. "will not write partition columns to hudi"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#issuecomment-897667019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1688",
"triggerID" : "b5008ea5c63ef90d1bc41f0c13a6adf0bf0ca700",
"triggerType" : "PUSH"
}, {
"hash" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1689",
"triggerID" : "8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f",
"triggerType" : "PUSH"
}, {
"hash" : "5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c",
"triggerType" : "PUSH"
}, {
"hash" : "0038a70398da3848ccccf20391542e918b841dae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1696",
"triggerID" : "0038a70398da3848ccccf20391542e918b841dae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8aa498d6ecff678aa0f6f48c1c06d76ae90e0e7f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1689)
* 5a059a4c7538b0d9e7ccd3bd717ade7ec2d54d8c UNKNOWN
* 0038a70398da3848ccccf20391542e918b841dae Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1696)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#issuecomment-897993554
@codope : Can you review this patch. I have updated w/ some changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Ales-Semchenkov commented on a change in pull request #3465: [HUDI-1363] Provide option to drop partition columns
Posted by GitBox <gi...@apache.org>.
Ales-Semchenkov commented on a change in pull request #3465:
URL: https://github.com/apache/hudi/pull/3465#discussion_r691235858
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -223,21 +240,25 @@ object HoodieSparkSqlWriter {
operation.equals(WriteOperationType.UPSERT) ||
parameters.getOrElse(HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.key(),
HoodieWriteConfig.COMBINE_BEFORE_INSERT_PROP.defaultValue()).toBoolean
+ val partitionKey = hoodieConfig.getString(DataSourceWriteOptions.PARTITIONPATH_FIELD)
Review comment:
@xuzifu666
We have tried Hudi with the PR and it solved the problem with the BigQuery index conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org