You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/10/07 11:15:23 UTC
[GitHub] [hudi] pramodbiligiri opened a new pull request, #6886: [HUDI-4994] Adding undo of soft-delete to upsert code flow
pramodbiligiri opened a new pull request, #6886:
URL: https://github.com/apache/hudi/pull/6886
### Change Logs
JIRA: https://issues.apache.org/jira/browse/HUDI-4994
Datahub has a notion of soft-deletes, which means the entity might still exist in the database after soft-delete. When DatahubSyncTool updates a property of a table, it can be assumed that users also want to change the soft-delete status of the entity. Or else the entity won't get reflected in the Datahub UI.
This PR adds this undo of soft-delete, via setting status="removed:false" into the metadata object. This will have no impact on new entities, or hard-deleted entities (which will have nothing inside Datahub anyway)
Ref: See sections on Soft Delete and Hard Delete in the Datahub docs: https://datahubproject.io/docs/how/delete-metadata/#soft-delete-the-default.
### Impact
_Describe any public API or user-facing feature change or any performance impact._
**Risk level: none | low | medium | high**
_Choose one. If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1273569334
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049",
"triggerID" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"triggerType" : "PUSH"
}, {
"hash" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12107",
"triggerID" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"triggerType" : "PUSH"
}, {
"hash" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12110",
"triggerID" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 663462ac17e9b2ebfa80273aeff97c3f4dcc62ff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12107)
* ca9a7da30843eb7505f91545b4175582882a3e5f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12110)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1274453447
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049",
"triggerID" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"triggerType" : "PUSH"
}, {
"hash" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12107",
"triggerID" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"triggerType" : "PUSH"
}, {
"hash" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12110",
"triggerID" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"triggerType" : "PUSH"
}, {
"hash" : "1b285166990772855a00bd404e0974093969cdf9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12123",
"triggerID" : "1b285166990772855a00bd404e0974093969cdf9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b285166990772855a00bd404e0974093969cdf9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12123)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1274967268
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049",
"triggerID" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"triggerType" : "PUSH"
}, {
"hash" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12107",
"triggerID" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"triggerType" : "PUSH"
}, {
"hash" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12110",
"triggerID" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"triggerType" : "PUSH"
}, {
"hash" : "1b285166990772855a00bd404e0974093969cdf9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12123",
"triggerID" : "1b285166990772855a00bd404e0974093969cdf9",
"triggerType" : "PUSH"
}, {
"hash" : "1b285166990772855a00bd404e0974093969cdf9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12131",
"triggerID" : "1274491913",
"triggerType" : "MANUAL"
}, {
"hash" : "1b285166990772855a00bd404e0974093969cdf9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12123",
"triggerID" : "1274491913",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 1b285166990772855a00bd404e0974093969cdf9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12131) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12123)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1273357552
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049",
"triggerID" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"triggerType" : "PUSH"
}, {
"hash" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 832c98a95c9482d24389ee2f2052893097bfdeda Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049)
* 663462ac17e9b2ebfa80273aeff97c3f4dcc62ff UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1273563444
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049",
"triggerID" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"triggerType" : "PUSH"
}, {
"hash" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12107",
"triggerID" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"triggerType" : "PUSH"
}, {
"hash" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 663462ac17e9b2ebfa80273aeff97c3f4dcc62ff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12107)
* ca9a7da30843eb7505f91545b4175582882a3e5f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6886: [HUDI-4994] Adding undo of soft-delete to upsert code flow
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1271468381
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049",
"triggerID" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 832c98a95c9482d24389ee2f2052893097bfdeda Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1273366909
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049",
"triggerID" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"triggerType" : "PUSH"
}, {
"hash" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12107",
"triggerID" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 832c98a95c9482d24389ee2f2052893097bfdeda Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049)
* 663462ac17e9b2ebfa80273aeff97c3f4dcc62ff Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12107)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1274983002
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049",
"triggerID" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"triggerType" : "PUSH"
}, {
"hash" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12107",
"triggerID" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"triggerType" : "PUSH"
}, {
"hash" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12110",
"triggerID" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"triggerType" : "PUSH"
}, {
"hash" : "1b285166990772855a00bd404e0974093969cdf9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12123",
"triggerID" : "1b285166990772855a00bd404e0974093969cdf9",
"triggerType" : "PUSH"
}, {
"hash" : "1b285166990772855a00bd404e0974093969cdf9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12131",
"triggerID" : "1274491913",
"triggerType" : "MANUAL"
}, {
"hash" : "1b285166990772855a00bd404e0974093969cdf9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12123",
"triggerID" : "1274491913",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 1b285166990772855a00bd404e0974093969cdf9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12131) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12123)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6886:
URL: https://github.com/apache/hudi/pull/6886#discussion_r990786831
##########
hudi-sync/hudi-datahub-sync/src/main/java/org/apache/hudi/sync/datahub/DataHubSyncClient.java:
##########
@@ -135,6 +133,37 @@ public void close() {
// no op;
}
+ // When updating an entity, it is ncessary to set its soft-delete status to false, or else the update won't get
+ // reflected in the UI.
+ private void undoSoftDelete(RestEmitter client, DatahubResponseLogger responseLogger) throws IOException, ExecutionException,
+ InterruptedException {
+ MetadataChangeProposalWrapper softDeleteUndoProposal = MetadataChangeProposalWrapper.builder()
+ .entityType("dataset")
+ .entityUrn(datasetUrn)
+ .upsert()
+ .aspect(SOFT_DELETE_FALSE)
+ .aspectName("status")
+ .build();
+
+ client.emit(softDeleteUndoProposal, responseLogger).get();
Review Comment:
i see this is a different pattern here: it takes in an emitter and sends request from within `undoSoftDelete()`, while `createSchemaMetadataUpdate()` returns the change proposal. Can we align these 2? how about having emitter sending requests in the try block so it's clear to see what were sent in one place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1274098278
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049",
"triggerID" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"triggerType" : "PUSH"
}, {
"hash" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12107",
"triggerID" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"triggerType" : "PUSH"
}, {
"hash" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12110",
"triggerID" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"triggerType" : "PUSH"
}, {
"hash" : "1b285166990772855a00bd404e0974093969cdf9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b285166990772855a00bd404e0974093969cdf9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca9a7da30843eb7505f91545b4175582882a3e5f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12110)
* 1b285166990772855a00bd404e0974093969cdf9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan merged pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
xushiyan merged PR #6886:
URL: https://github.com/apache/hudi/pull/6886
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1273557454
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049",
"triggerID" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"triggerType" : "PUSH"
}, {
"hash" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12107",
"triggerID" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 663462ac17e9b2ebfa80273aeff97c3f4dcc62ff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12107)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6886: [HUDI-4994] Adding undo of soft-delete to upsert code flow
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1271463068
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 832c98a95c9482d24389ee2f2052893097bfdeda UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pramodbiligiri commented on a diff in pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
pramodbiligiri commented on code in PR #6886:
URL: https://github.com/apache/hudi/pull/6886#discussion_r991465618
##########
hudi-sync/hudi-datahub-sync/src/main/java/org/apache/hudi/sync/datahub/DataHubSyncClient.java:
##########
@@ -135,6 +133,37 @@ public void close() {
// no op;
}
+ // When updating an entity, it is ncessary to set its soft-delete status to false, or else the update won't get
+ // reflected in the UI.
+ private void undoSoftDelete(RestEmitter client, DatahubResponseLogger responseLogger) throws IOException, ExecutionException,
+ InterruptedException {
+ MetadataChangeProposalWrapper softDeleteUndoProposal = MetadataChangeProposalWrapper.builder()
+ .entityType("dataset")
+ .entityUrn(datasetUrn)
+ .upsert()
+ .aspect(SOFT_DELETE_FALSE)
+ .aspectName("status")
+ .build();
+
+ client.emit(softDeleteUndoProposal, responseLogger).get();
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1274101335
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049",
"triggerID" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"triggerType" : "PUSH"
}, {
"hash" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12107",
"triggerID" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"triggerType" : "PUSH"
}, {
"hash" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12110",
"triggerID" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"triggerType" : "PUSH"
}, {
"hash" : "1b285166990772855a00bd404e0974093969cdf9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12123",
"triggerID" : "1b285166990772855a00bd404e0974093969cdf9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca9a7da30843eb7505f91545b4175582882a3e5f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12110)
* 1b285166990772855a00bd404e0974093969cdf9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12123)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
xushiyan commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1275725322
CI passed https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=12123&view=results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1271727870
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049",
"triggerID" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 832c98a95c9482d24389ee2f2052893097bfdeda Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pramodbiligiri commented on pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
pramodbiligiri commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1274491913
@hudi-bot run azure
Kicking off another run as it's some failure in hudi-flink module (untouched by this PR): https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/12123/logs/27
I saw this once recently. Looks like that module has flaky tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1274530795
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049",
"triggerID" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"triggerType" : "PUSH"
}, {
"hash" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12107",
"triggerID" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"triggerType" : "PUSH"
}, {
"hash" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12110",
"triggerID" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"triggerType" : "PUSH"
}, {
"hash" : "1b285166990772855a00bd404e0974093969cdf9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12123",
"triggerID" : "1b285166990772855a00bd404e0974093969cdf9",
"triggerType" : "PUSH"
}, {
"hash" : "1b285166990772855a00bd404e0974093969cdf9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12131",
"triggerID" : "1274491913",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 1b285166990772855a00bd404e0974093969cdf9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12123) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12131)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1273703406
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12049",
"triggerID" : "832c98a95c9482d24389ee2f2052893097bfdeda",
"triggerType" : "PUSH"
}, {
"hash" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12107",
"triggerID" : "663462ac17e9b2ebfa80273aeff97c3f4dcc62ff",
"triggerType" : "PUSH"
}, {
"hash" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12110",
"triggerID" : "ca9a7da30843eb7505f91545b4175582882a3e5f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca9a7da30843eb7505f91545b4175582882a3e5f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12110)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pramodbiligiri commented on pull request #6886: [HUDI-4994] Fix bug that prevents re-ingestion of soft-deleted Datahub entities
Posted by GitBox <gi...@apache.org>.
pramodbiligiri commented on PR #6886:
URL: https://github.com/apache/hudi/pull/6886#issuecomment-1273559887
>
Added a test as of now. I might add more on the same lines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org