You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/08/04 09:20:20 UTC
[GitHub] [hudi] tsianglei opened a new pull request #3402: [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException
tsianglei opened a new pull request #3402:
URL: https://github.com/apache/hudi/pull/3402
## What is the purpose of the pull request
[HUDI-2167](https://issues.apache.org/jira/browse/HUDI-2167)
*Fix org.apache.hudi.util.StreamerUtil#getHoodieClientConfig does not set HoodieCleaningPolicy but org.apache.hudi.config.HoodieCompactionConfig.Builder#build check the policy. Then will got NullPointerException.
Exception stack is:*
```
Caused by: java.lang.NullPointerException: Name is null
at java.lang.Enum.valueOf(Enum.java:236) ~[?:1.8.0_221]
at org.apache.hudi.common.model.HoodieCleaningPolicy.valueOf(HoodieCleaningPolicy.java:24) ~[hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar:0.9.0-SNAPSHOT]
at org.apache.hudi.config.HoodieCompactionConfig$Builder.build(HoodieCompactionConfig.java:392) ~[hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar:0.9.0-SNAPSHOT]
at org.apache.hudi.util.StreamerUtil.getHoodieClientConfig(StreamerUtil.java:162) ~[hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar:0.9.0-SNAPSHOT]
at org.apache.hudi.util.StreamerUtil.createWriteClient(StreamerUtil.java:305) ~[hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar:0.9.0-SNAPSHOT]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.start(StreamWriteOperatorCoordinator.java:153) ~[hudi-flink-bundle_2.11-0.9.0-SNAPSHOT.jar:0.9.0-SNAPSHOT]
at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.start(OperatorCoordinatorHolder.java:189) ~[flink-dist_2.11-1.12.2.jar:1.12.2]
at org.apache.flink.runtime.scheduler.SchedulerBase.startAllOperatorCoordinators(SchedulerBase.java:1253) ~[flink-dist_2.11-1.12.2.jar:1.12.2]
at org.apache.flink.runtime.scheduler.SchedulerBase.startScheduling(SchedulerBase.java:624) ~[flink-dist_2.11-1.12.2.jar:1.12.2]
at org.apache.flink.runtime.jobmaster.JobMaster.startScheduling(JobMaster.java:1032) ~[flink-dist_2.11-1.12.2.jar:1.12.2]
at java.util.concurrent.CompletableFuture.uniRun(CompletableFuture.java:705) ~[?:1.8.0_221]
```
## Brief change log
- *add HoodieCleaningPolicy*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] tsianglei commented on a change in pull request #3402: [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException
Posted by GitBox <gi...@apache.org>.
tsianglei commented on a change in pull request #3402:
URL: https://github.com/apache/hudi/pull/3402#discussion_r682461965
##########
File path: hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java
##########
@@ -152,6 +153,7 @@ public static HoodieWriteConfig getHoodieClientConfig(Configuration conf) {
// actually Flink cleaning is always with parallelism 1 now
.withCleanerParallelism(20)
.archiveCommitsWith(conf.getInteger(FlinkOptions.ARCHIVE_MIN_COMMITS), conf.getInteger(FlinkOptions.ARCHIVE_MAX_COMMITS))
+ .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
.build())
Review comment:
build() method get policy use compactionConfig.getString(CLEANER_POLICY_PROP),this method doesn't return default value, but null. if not set then will return null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3402: [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3402:
URL: https://github.com/apache/hudi/pull/3402#discussion_r682456792
##########
File path: hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java
##########
@@ -152,6 +153,7 @@ public static HoodieWriteConfig getHoodieClientConfig(Configuration conf) {
// actually Flink cleaning is always with parallelism 1 now
.withCleanerParallelism(20)
.archiveCommitsWith(conf.getInteger(FlinkOptions.ARCHIVE_MIN_COMMITS), conf.getInteger(FlinkOptions.ARCHIVE_MAX_COMMITS))
+ .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
.build())
Review comment:
Isn't `HoodieCleaningPolicy.KEEP_LATEST_COMMITS` the default value ?
##########
File path: hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java
##########
@@ -152,6 +153,7 @@ public static HoodieWriteConfig getHoodieClientConfig(Configuration conf) {
// actually Flink cleaning is always with parallelism 1 now
.withCleanerParallelism(20)
.archiveCommitsWith(conf.getInteger(FlinkOptions.ARCHIVE_MIN_COMMITS), conf.getInteger(FlinkOptions.ARCHIVE_MAX_COMMITS))
+ .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
.build())
Review comment:
The `HoodieCompactionConfig` would `setDefaults` first before build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 closed pull request #3402: [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException
Posted by GitBox <gi...@apache.org>.
danny0405 closed pull request #3402:
URL: https://github.com/apache/hudi/pull/3402
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3402: [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3402:
URL: https://github.com/apache/hudi/pull/3402#issuecomment-892505211
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40aa4313a56c82473828865ddcef89b550499d1e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1359",
"triggerID" : "40aa4313a56c82473828865ddcef89b550499d1e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40aa4313a56c82473828865ddcef89b550499d1e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1359)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #3402: [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException
Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #3402:
URL: https://github.com/apache/hudi/pull/3402#issuecomment-896441671
Not a valid fix, close it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 closed pull request #3402: [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException
Posted by GitBox <gi...@apache.org>.
danny0405 closed pull request #3402:
URL: https://github.com/apache/hudi/pull/3402
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3402: [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3402:
URL: https://github.com/apache/hudi/pull/3402#discussion_r682456792
##########
File path: hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java
##########
@@ -152,6 +153,7 @@ public static HoodieWriteConfig getHoodieClientConfig(Configuration conf) {
// actually Flink cleaning is always with parallelism 1 now
.withCleanerParallelism(20)
.archiveCommitsWith(conf.getInteger(FlinkOptions.ARCHIVE_MIN_COMMITS), conf.getInteger(FlinkOptions.ARCHIVE_MAX_COMMITS))
+ .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
.build())
Review comment:
Isn't `HoodieCleaningPolicy.KEEP_LATEST_COMMITS` the default value ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3402: [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3402:
URL: https://github.com/apache/hudi/pull/3402#issuecomment-892505211
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40aa4313a56c82473828865ddcef89b550499d1e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1359",
"triggerID" : "40aa4313a56c82473828865ddcef89b550499d1e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40aa4313a56c82473828865ddcef89b550499d1e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1359)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 closed pull request #3402: [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException
Posted by GitBox <gi...@apache.org>.
danny0405 closed pull request #3402:
URL: https://github.com/apache/hudi/pull/3402
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3402: [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3402:
URL: https://github.com/apache/hudi/pull/3402#issuecomment-892505211
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40aa4313a56c82473828865ddcef89b550499d1e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "40aa4313a56c82473828865ddcef89b550499d1e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40aa4313a56c82473828865ddcef89b550499d1e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3402: [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3402:
URL: https://github.com/apache/hudi/pull/3402#issuecomment-892505211
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40aa4313a56c82473828865ddcef89b550499d1e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1359",
"triggerID" : "40aa4313a56c82473828865ddcef89b550499d1e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40aa4313a56c82473828865ddcef89b550499d1e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1359)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3402: [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3402:
URL: https://github.com/apache/hudi/pull/3402#discussion_r682464218
##########
File path: hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java
##########
@@ -152,6 +153,7 @@ public static HoodieWriteConfig getHoodieClientConfig(Configuration conf) {
// actually Flink cleaning is always with parallelism 1 now
.withCleanerParallelism(20)
.archiveCommitsWith(conf.getInteger(FlinkOptions.ARCHIVE_MIN_COMMITS), conf.getInteger(FlinkOptions.ARCHIVE_MAX_COMMITS))
+ .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
.build())
Review comment:
The `HoodieCompactionConfig` would `setDefaults` first before build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] tsianglei commented on a change in pull request #3402: [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException
Posted by GitBox <gi...@apache.org>.
tsianglei commented on a change in pull request #3402:
URL: https://github.com/apache/hudi/pull/3402#discussion_r682461965
##########
File path: hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java
##########
@@ -152,6 +153,7 @@ public static HoodieWriteConfig getHoodieClientConfig(Configuration conf) {
// actually Flink cleaning is always with parallelism 1 now
.withCleanerParallelism(20)
.archiveCommitsWith(conf.getInteger(FlinkOptions.ARCHIVE_MIN_COMMITS), conf.getInteger(FlinkOptions.ARCHIVE_MAX_COMMITS))
+ .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
.build())
Review comment:
build() method get policy use compactionConfig.getString(CLEANER_POLICY_PROP),this method doesn't return default value, but null. if not set then will return null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3402: [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3402:
URL: https://github.com/apache/hudi/pull/3402#issuecomment-892505211
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40aa4313a56c82473828865ddcef89b550499d1e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1359",
"triggerID" : "40aa4313a56c82473828865ddcef89b550499d1e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40aa4313a56c82473828865ddcef89b550499d1e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1359)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3402: [HUDI-2167] HoodieCompactionConfig get HoodieCleaningPolicy NullPointerException
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3402:
URL: https://github.com/apache/hudi/pull/3402#issuecomment-892505211
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "40aa4313a56c82473828865ddcef89b550499d1e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1359",
"triggerID" : "40aa4313a56c82473828865ddcef89b550499d1e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40aa4313a56c82473828865ddcef89b550499d1e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1359)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org