You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "danielfordfc (via GitHub)" <gi...@apache.org> on 2023/03/20 17:16:20 UTC
[GitHub] [hudi] danielfordfc opened a new pull request, #8246: Add cachedSchema per batch, fix idempotency with getSourceSchema calls
danielfordfc opened a new pull request, #8246:
URL: https://github.com/apache/hudi/pull/8246
### Change Logs
Addressing the issues discussed in https://github.com/apache/hudi/issues/8065
@nsivabalan's thinking:
The issue is. getSourceScheme in case of SchemaRegistry provider is not idempotent. even within a single batch of write, if we call getSourceSchema multiple times, it could return latest schema from the schema registry. ideally we want it to return one schema for one batch of write.
so, the fix is to add a new api to Source abstract class called "clearCaches" or "cleanupResources". also add similar apis to SchemaProvider. and so within source.clearCaches, we will call schemaProvider.clearCaches.
Incase of SchemaRegistryProvider, for every batch, we will fetch from remote schema registry and cache is locally. for subsequent calls to getsourceSchema, we will be returning the same value. before moving onto next batch of consume, we will have to call clearCaches which will invalidate the local cache of source schema.
### Impact
Deltastreamer jobs running in --continuous mode sometimes fail to gracefully evolve their schemas if the schema registry schema's change mid-way through a batch (presents itself as transient failures, as discussed in the above GH Issue)
Now, when a `SchemaRegistryProvider` calls `getSourceSchema`, it will cache its first schema called, and then re-use the cache on subsequent calls, to maintain that schema per batch.
Also added a clearCarches method to the KafkaSource as per @nsivabalan's recommendation, to "invalidate the local cache of source schema."
This should ensure that when the Source is instantiated and (KafkaSource and it's children) call `fetchNewData()`, any existing cache from the `SchemaProvider` (Currently only `SchemaRegistryProvider`) will be dropped.
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8246: [HUDI-5973] Add cachedSchema per batch, fix idempotency with getSourceSchema calls
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8246:
URL: https://github.com/apache/hudi/pull/8246#issuecomment-1540765618
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15814",
"triggerID" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"triggerType" : "PUSH"
}, {
"hash" : "26956ff23fcbd6806cca21a420158675f53a2265",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16921",
"triggerID" : "26956ff23fcbd6806cca21a420158675f53a2265",
"triggerType" : "PUSH"
}, {
"hash" : "869dbe6a349700c3a1f3c8686c078edae139bdd9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "869dbe6a349700c3a1f3c8686c078edae139bdd9",
"triggerType" : "PUSH"
}, {
"hash" : "f2b991d31006109e54ca98499e91ad9ea4bfdd30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16923",
"triggerID" : "f2b991d31006109e54ca98499e91ad9ea4bfdd30",
"triggerType" : "PUSH"
}, {
"hash" : "95d3eee7618a635cc34fbc134fd8c9444973beac",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16975",
"triggerID" : "95d3eee7618a635cc34fbc134fd8c9444973beac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 869dbe6a349700c3a1f3c8686c078edae139bdd9 UNKNOWN
* 95d3eee7618a635cc34fbc134fd8c9444973beac Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16975)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5973] Add cachedSchema per batch, fix idempotency with getSourceSchema calls [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua closed pull request #8246: [HUDI-5973] Add cachedSchema per batch, fix idempotency with getSourceSchema calls
URL: https://github.com/apache/hudi/pull/8246
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danielfordfc commented on pull request #8246: [HUDI-5973] Add cachedSchema per batch, fix idempotency with getSourceSchema calls
Posted by "danielfordfc (via GitHub)" <gi...@apache.org>.
danielfordfc commented on PR #8246:
URL: https://github.com/apache/hudi/pull/8246#issuecomment-1506557300
Bump @nsivabalan. Does this look along the right lines?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8246: [HUDI-5973] Add cachedSchema per batch, fix idempotency with getSourceSchema calls
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8246:
URL: https://github.com/apache/hudi/pull/8246#issuecomment-1537452543
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15814",
"triggerID" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"triggerType" : "PUSH"
}, {
"hash" : "26956ff23fcbd6806cca21a420158675f53a2265",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26956ff23fcbd6806cca21a420158675f53a2265",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab3c9eb91ad9711c66f8ed2727e2a004d1485385 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15814)
* 26956ff23fcbd6806cca21a420158675f53a2265 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8246: [HUDI-5973] Add cachedSchema per batch, fix idempotency with getSourceSchema calls
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8246:
URL: https://github.com/apache/hudi/pull/8246#issuecomment-1540046429
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15814",
"triggerID" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"triggerType" : "PUSH"
}, {
"hash" : "26956ff23fcbd6806cca21a420158675f53a2265",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16921",
"triggerID" : "26956ff23fcbd6806cca21a420158675f53a2265",
"triggerType" : "PUSH"
}, {
"hash" : "869dbe6a349700c3a1f3c8686c078edae139bdd9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "869dbe6a349700c3a1f3c8686c078edae139bdd9",
"triggerType" : "PUSH"
}, {
"hash" : "f2b991d31006109e54ca98499e91ad9ea4bfdd30",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16923",
"triggerID" : "f2b991d31006109e54ca98499e91ad9ea4bfdd30",
"triggerType" : "PUSH"
}, {
"hash" : "95d3eee7618a635cc34fbc134fd8c9444973beac",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "95d3eee7618a635cc34fbc134fd8c9444973beac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 869dbe6a349700c3a1f3c8686c078edae139bdd9 UNKNOWN
* f2b991d31006109e54ca98499e91ad9ea4bfdd30 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16923)
* 95d3eee7618a635cc34fbc134fd8c9444973beac UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8246: [HUDI-5973] Add cachedSchema per batch, fix idempotency with getSourceSchema calls
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8246:
URL: https://github.com/apache/hudi/pull/8246#issuecomment-1537465510
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15814",
"triggerID" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"triggerType" : "PUSH"
}, {
"hash" : "26956ff23fcbd6806cca21a420158675f53a2265",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16921",
"triggerID" : "26956ff23fcbd6806cca21a420158675f53a2265",
"triggerType" : "PUSH"
}, {
"hash" : "869dbe6a349700c3a1f3c8686c078edae139bdd9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "869dbe6a349700c3a1f3c8686c078edae139bdd9",
"triggerType" : "PUSH"
}, {
"hash" : "f2b991d31006109e54ca98499e91ad9ea4bfdd30",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f2b991d31006109e54ca98499e91ad9ea4bfdd30",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 26956ff23fcbd6806cca21a420158675f53a2265 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16921)
* 869dbe6a349700c3a1f3c8686c078edae139bdd9 UNKNOWN
* f2b991d31006109e54ca98499e91ad9ea4bfdd30 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5973] Add cachedSchema per batch, fix idempotency with getSourceSchema calls [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #8246:
URL: https://github.com/apache/hudi/pull/8246#issuecomment-1843581897
Cleaned this up and out out a new PR https://github.com/apache/hudi/pull/10261/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8246: Add cachedSchema per batch, fix idempotency with getSourceSchema calls
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8246:
URL: https://github.com/apache/hudi/pull/8246#issuecomment-1477014585
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15814",
"triggerID" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab3c9eb91ad9711c66f8ed2727e2a004d1485385 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15814)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8246: [HUDI-5973] Add cachedSchema per batch, fix idempotency with getSourceSchema calls
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8246:
URL: https://github.com/apache/hudi/pull/8246#issuecomment-1537467752
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15814",
"triggerID" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"triggerType" : "PUSH"
}, {
"hash" : "26956ff23fcbd6806cca21a420158675f53a2265",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16921",
"triggerID" : "26956ff23fcbd6806cca21a420158675f53a2265",
"triggerType" : "PUSH"
}, {
"hash" : "869dbe6a349700c3a1f3c8686c078edae139bdd9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "869dbe6a349700c3a1f3c8686c078edae139bdd9",
"triggerType" : "PUSH"
}, {
"hash" : "f2b991d31006109e54ca98499e91ad9ea4bfdd30",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16923",
"triggerID" : "f2b991d31006109e54ca98499e91ad9ea4bfdd30",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 869dbe6a349700c3a1f3c8686c078edae139bdd9 UNKNOWN
* f2b991d31006109e54ca98499e91ad9ea4bfdd30 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16923)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danielfordfc commented on pull request #8246: [HUDI-5973] Add cachedSchema per batch, fix idempotency with getSourceSchema calls
Posted by "danielfordfc (via GitHub)" <gi...@apache.org>.
danielfordfc commented on PR #8246:
URL: https://github.com/apache/hudi/pull/8246#issuecomment-1537458573
@nsivabalan I've taken your thoughts and moved the schemaprovider.refresh call into the SyncOnce call.
I've then added overrides to FileBasedSchemaProvider and SchemaRegistryProvider.
Is this how you envisioned it?
I guess where appropriate, individual providers can handle their refreshing in different ways (FileBased is simply re-polling the file, but SchemaRegistryProvider involves dropping the internal cachedSchema and refreshing). Is this along the right lines?
note: the FileBasedSchemaProvider has had a fair bit of refactoring since your branch off of 0.11.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8246: Add cachedSchema per batch, fix idempotency with getSourceSchema calls
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8246:
URL: https://github.com/apache/hudi/pull/8246#issuecomment-1476683508
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab3c9eb91ad9711c66f8ed2727e2a004d1485385 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8246: [HUDI-5973] Add cachedSchema per batch, fix idempotency with getSourceSchema calls
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8246:
URL: https://github.com/apache/hudi/pull/8246#issuecomment-1537463721
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15814",
"triggerID" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"triggerType" : "PUSH"
}, {
"hash" : "26956ff23fcbd6806cca21a420158675f53a2265",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16921",
"triggerID" : "26956ff23fcbd6806cca21a420158675f53a2265",
"triggerType" : "PUSH"
}, {
"hash" : "869dbe6a349700c3a1f3c8686c078edae139bdd9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "869dbe6a349700c3a1f3c8686c078edae139bdd9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab3c9eb91ad9711c66f8ed2727e2a004d1485385 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15814)
* 26956ff23fcbd6806cca21a420158675f53a2265 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16921)
* 869dbe6a349700c3a1f3c8686c078edae139bdd9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8246: [HUDI-5973] Add cachedSchema per batch, fix idempotency with getSourceSchema calls
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8246:
URL: https://github.com/apache/hudi/pull/8246#issuecomment-1540104080
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15814",
"triggerID" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"triggerType" : "PUSH"
}, {
"hash" : "26956ff23fcbd6806cca21a420158675f53a2265",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16921",
"triggerID" : "26956ff23fcbd6806cca21a420158675f53a2265",
"triggerType" : "PUSH"
}, {
"hash" : "869dbe6a349700c3a1f3c8686c078edae139bdd9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "869dbe6a349700c3a1f3c8686c078edae139bdd9",
"triggerType" : "PUSH"
}, {
"hash" : "f2b991d31006109e54ca98499e91ad9ea4bfdd30",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16923",
"triggerID" : "f2b991d31006109e54ca98499e91ad9ea4bfdd30",
"triggerType" : "PUSH"
}, {
"hash" : "95d3eee7618a635cc34fbc134fd8c9444973beac",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16975",
"triggerID" : "95d3eee7618a635cc34fbc134fd8c9444973beac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 869dbe6a349700c3a1f3c8686c078edae139bdd9 UNKNOWN
* f2b991d31006109e54ca98499e91ad9ea4bfdd30 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16923)
* 95d3eee7618a635cc34fbc134fd8c9444973beac Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16975)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8246: [HUDI-5973] Add cachedSchema per batch, fix idempotency with getSourceSchema calls
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8246:
URL: https://github.com/apache/hudi/pull/8246#issuecomment-1537454235
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15814",
"triggerID" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"triggerType" : "PUSH"
}, {
"hash" : "26956ff23fcbd6806cca21a420158675f53a2265",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16921",
"triggerID" : "26956ff23fcbd6806cca21a420158675f53a2265",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab3c9eb91ad9711c66f8ed2727e2a004d1485385 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15814)
* 26956ff23fcbd6806cca21a420158675f53a2265 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16921)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danielfordfc commented on pull request #8246: [HUDI-5973] Add cachedSchema per batch, fix idempotency with getSourceSchema calls
Posted by "danielfordfc (via GitHub)" <gi...@apache.org>.
danielfordfc commented on PR #8246:
URL: https://github.com/apache/hudi/pull/8246#issuecomment-1542222481
Have also extended the usage for the getTargetSchema calls for both of the providers. Is this also a valuable addition?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8246: Add cachedSchema per batch, fix idempotency with getSourceSchema calls
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8246:
URL: https://github.com/apache/hudi/pull/8246#issuecomment-1476694749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15814",
"triggerID" : "ab3c9eb91ad9711c66f8ed2727e2a004d1485385",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab3c9eb91ad9711c66f8ed2727e2a004d1485385 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15814)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5973] Add cachedSchema per batch, fix idempotency with getSourceSchema calls [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #8246:
URL: https://github.com/apache/hudi/pull/8246#issuecomment-1986761165
#10261 is landed. Closing this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org