You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/04 07:46:53 UTC
[GitHub] [hudi] SteNicholas opened a new pull request, #6298: [HUDI-4536] ClusteringOperator causes the NullPointerException when writing with BulkInsertWriterHelper in clustering
SteNicholas opened a new pull request, #6298:
URL: https://github.com/apache/hudi/pull/6298
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
`ClusteringOperator` causes the `NullPointerException` when writing with `BulkInsertWriterHelper` for clustering, because the `BulkInsertWriterHelper` isn't set to null after close. `BulkInsertWriterHelper` should be set to null after calling `BulkInsertWriterHelper#getWriteStatuses`.
## Brief change log
- `BulkInsertWriterHelper` is set to null after calling `BulkInsertWriterHelper#getWriteStatuses`.
## Verify this pull request
- `testInsertAsyncClustering` in `TestWriteCopyOnWrite` verifies whether to cause the `NullPointerException` when doing clustering.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6298: [HUDI-4536] ClusteringOperator causes the NullPointerException when writing with BulkInsertWriterHelper in clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6298:
URL: https://github.com/apache/hudi/pull/6298#issuecomment-1205041743
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "348ceda14b3facf1dee4e79bdfb31d5b73617e1a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10578",
"triggerID" : "348ceda14b3facf1dee4e79bdfb31d5b73617e1a",
"triggerType" : "PUSH"
}, {
"hash" : "de618f97e20125a186eb4f46ba50be7273c369fc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10581",
"triggerID" : "de618f97e20125a186eb4f46ba50be7273c369fc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 348ceda14b3facf1dee4e79bdfb31d5b73617e1a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10578)
* de618f97e20125a186eb4f46ba50be7273c369fc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10581)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #6298: [HUDI-4536] ClusteringOperator causes the NullPointerException when writing with BulkInsertWriterHelper in clustering
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #6298:
URL: https://github.com/apache/hudi/pull/6298#issuecomment-1205186140
Thanks for the contribution @SteNicholas , we can add more tests for clustering in subsequent PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6298: [HUDI-4536] ClusteringOperator causes the NullPointerException when writing with BulkInsertWriterHelper in clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6298:
URL: https://github.com/apache/hudi/pull/6298#issuecomment-1204913649
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "348ceda14b3facf1dee4e79bdfb31d5b73617e1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10578",
"triggerID" : "348ceda14b3facf1dee4e79bdfb31d5b73617e1a",
"triggerType" : "PUSH"
}, {
"hash" : "de618f97e20125a186eb4f46ba50be7273c369fc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de618f97e20125a186eb4f46ba50be7273c369fc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 348ceda14b3facf1dee4e79bdfb31d5b73617e1a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10578)
* de618f97e20125a186eb4f46ba50be7273c369fc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6298: [HUDI-4536] ClusteringOperator causes the NullPointerException when writing with BulkInsertWriterHelper in clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6298:
URL: https://github.com/apache/hudi/pull/6298#issuecomment-1204903202
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "348ceda14b3facf1dee4e79bdfb31d5b73617e1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10578",
"triggerID" : "348ceda14b3facf1dee4e79bdfb31d5b73617e1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 348ceda14b3facf1dee4e79bdfb31d5b73617e1a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10578)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6298: [HUDI-4536] ClusteringOperator causes the NullPointerException when writing with BulkInsertWriterHelper in clustering
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6298:
URL: https://github.com/apache/hudi/pull/6298#discussion_r937465161
##########
hudi-common/src/main/java/org/apache/hudi/common/fs/HoodieWrapperFileSystem.java:
##########
@@ -1015,6 +1015,10 @@ public void createImmutableFileInPath(Path fullPath, Option<byte[]> content)
boolean needTempFile = needCreateTempFile();
try {
+ if (exists(fullPath)) {
+ return;
+ }
+
Review Comment:
Do we need this ? Why not just throws ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6298: [HUDI-4536] ClusteringOperator causes the NullPointerException when writing with BulkInsertWriterHelper in clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6298:
URL: https://github.com/apache/hudi/pull/6298#issuecomment-1204974064
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "348ceda14b3facf1dee4e79bdfb31d5b73617e1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10578",
"triggerID" : "348ceda14b3facf1dee4e79bdfb31d5b73617e1a",
"triggerType" : "PUSH"
}, {
"hash" : "de618f97e20125a186eb4f46ba50be7273c369fc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10581",
"triggerID" : "de618f97e20125a186eb4f46ba50be7273c369fc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 348ceda14b3facf1dee4e79bdfb31d5b73617e1a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10578)
* de618f97e20125a186eb4f46ba50be7273c369fc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10581)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6298: [HUDI-4536] ClusteringOperator causes the NullPointerException when writing with BulkInsertWriterHelper in clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6298:
URL: https://github.com/apache/hudi/pull/6298#issuecomment-1205116433
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "348ceda14b3facf1dee4e79bdfb31d5b73617e1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10578",
"triggerID" : "348ceda14b3facf1dee4e79bdfb31d5b73617e1a",
"triggerType" : "PUSH"
}, {
"hash" : "de618f97e20125a186eb4f46ba50be7273c369fc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10581",
"triggerID" : "de618f97e20125a186eb4f46ba50be7273c369fc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de618f97e20125a186eb4f46ba50be7273c369fc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10581)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6298: [HUDI-4536] ClusteringOperator causes the NullPointerException when writing with BulkInsertWriterHelper in clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6298:
URL: https://github.com/apache/hudi/pull/6298#issuecomment-1204898457
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "348ceda14b3facf1dee4e79bdfb31d5b73617e1a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "348ceda14b3facf1dee4e79bdfb31d5b73617e1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 348ceda14b3facf1dee4e79bdfb31d5b73617e1a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #6298: [HUDI-4536] ClusteringOperator causes the NullPointerException when writing with BulkInsertWriterHelper in clustering
Posted by GitBox <gi...@apache.org>.
danny0405 merged PR #6298:
URL: https://github.com/apache/hudi/pull/6298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6298: [HUDI-4536] ClusteringOperator causes the NullPointerException when writing with BulkInsertWriterHelper in clustering
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6298:
URL: https://github.com/apache/hudi/pull/6298#discussion_r937466031
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/ClusteringOperator.java:
##########
@@ -233,6 +233,7 @@ private void doClustering(String instantTime, ClusteringPlanEvent event) throws
List<WriteStatus> writeStatuses = this.writerHelper.getWriteStatuses(this.taskID);
collector.collect(new ClusteringCommitEvent(instantTime, writeStatuses, this.taskID));
+ this.writerHelper = null;
}
Review Comment:
Is there any possibility we write some tests, say when there are multiple clustering plan to execute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org