You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "huangxiaopingRD (via GitHub)" <gi...@apache.org> on 2023/04/01 17:49:52 UTC
[GitHub] [hudi] huangxiaopingRD opened a new pull request, #8352: [HUDI-6015] Refresh the table after executing rollback to instantTime
huangxiaopingRD opened a new pull request, #8352:
URL: https://github.com/apache/hudi/pull/8352
### Change Logs
Spark will cache some meta information of the table. After the RollbackToInstantTimeProcedure is executed on the table, the meta information will change and the table needs to be refreshed. Otherwise, the following error will occur when querying the data again:
```
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://xxxxx/user/hive/warehouse/hudi_cow_nonpcf_tbl2/7a19abfb-35ab-40bb-9580-6b1af681506a-0_0-23-20_20230402002001284.parquet
It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:124)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:187)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:503)
```
### Impact
No
### Risk level (write none, low medium or high below)
none
### Documentation Update
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8352: [HUDI-6015] Refresh the table after executing rollback to instantTime
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8352:
URL: https://github.com/apache/hudi/pull/8352#issuecomment-1493146909
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16057",
"triggerID" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b24209938c245227cd6f1ccc1b428d50f3b51a9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16057)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8352: [HUDI-6015] Refresh the table after executing rollback to instantTime
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8352:
URL: https://github.com/apache/hudi/pull/8352#issuecomment-1493073594
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16057",
"triggerID" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b24209938c245227cd6f1ccc1b428d50f3b51a9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16057)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8352: [HUDI-6015] Refresh the table after executing rollback and restoreToSavepoint
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8352:
URL: https://github.com/apache/hudi/pull/8352#issuecomment-1513567941
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16057",
"triggerID" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"triggerType" : "PUSH"
}, {
"hash" : "254680e4dcb02cc0aff37bc3ac416f5d15b7c0f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16257",
"triggerID" : "254680e4dcb02cc0aff37bc3ac416f5d15b7c0f5",
"triggerType" : "PUSH"
}, {
"hash" : "b31681530640a9f8b3ccdd620978f21705e7905d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16425",
"triggerID" : "b31681530640a9f8b3ccdd620978f21705e7905d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b31681530640a9f8b3ccdd620978f21705e7905d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16425)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8352: [HUDI-6015] Refresh the table after executing rollback to instantTime
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8352:
URL: https://github.com/apache/hudi/pull/8352#issuecomment-1493071944
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b24209938c245227cd6f1ccc1b428d50f3b51a9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8352: [HUDI-6015] Refresh the table after executing rollback to instantTime
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8352:
URL: https://github.com/apache/hudi/pull/8352#issuecomment-1503353910
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16057",
"triggerID" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"triggerType" : "PUSH"
}, {
"hash" : "254680e4dcb02cc0aff37bc3ac416f5d15b7c0f5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "254680e4dcb02cc0aff37bc3ac416f5d15b7c0f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b24209938c245227cd6f1ccc1b428d50f3b51a9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16057)
* 254680e4dcb02cc0aff37bc3ac416f5d15b7c0f5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8352: [HUDI-6015] Refresh the table after executing rollback to instantTime
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8352:
URL: https://github.com/apache/hudi/pull/8352#issuecomment-1503625458
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16057",
"triggerID" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"triggerType" : "PUSH"
}, {
"hash" : "254680e4dcb02cc0aff37bc3ac416f5d15b7c0f5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16257",
"triggerID" : "254680e4dcb02cc0aff37bc3ac416f5d15b7c0f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 254680e4dcb02cc0aff37bc3ac416f5d15b7c0f5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16257)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8352: [HUDI-6015] Refresh the table after executing rollback and restoreToSavepoint
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8352:
URL: https://github.com/apache/hudi/pull/8352#issuecomment-1513061572
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16057",
"triggerID" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"triggerType" : "PUSH"
}, {
"hash" : "254680e4dcb02cc0aff37bc3ac416f5d15b7c0f5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16257",
"triggerID" : "254680e4dcb02cc0aff37bc3ac416f5d15b7c0f5",
"triggerType" : "PUSH"
}, {
"hash" : "b31681530640a9f8b3ccdd620978f21705e7905d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16425",
"triggerID" : "b31681530640a9f8b3ccdd620978f21705e7905d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 254680e4dcb02cc0aff37bc3ac416f5d15b7c0f5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16257)
* b31681530640a9f8b3ccdd620978f21705e7905d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16425)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8352: [HUDI-6015] Refresh the table after executing rollback to instantTime
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8352:
URL: https://github.com/apache/hudi/pull/8352#issuecomment-1512970923
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16057",
"triggerID" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"triggerType" : "PUSH"
}, {
"hash" : "254680e4dcb02cc0aff37bc3ac416f5d15b7c0f5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16257",
"triggerID" : "254680e4dcb02cc0aff37bc3ac416f5d15b7c0f5",
"triggerType" : "PUSH"
}, {
"hash" : "b31681530640a9f8b3ccdd620978f21705e7905d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b31681530640a9f8b3ccdd620978f21705e7905d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 254680e4dcb02cc0aff37bc3ac416f5d15b7c0f5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16257)
* b31681530640a9f8b3ccdd620978f21705e7905d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8352: [HUDI-6015] Refresh the table after executing rollback to instantTime
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8352:
URL: https://github.com/apache/hudi/pull/8352#issuecomment-1503368550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16057",
"triggerID" : "4b24209938c245227cd6f1ccc1b428d50f3b51a9",
"triggerType" : "PUSH"
}, {
"hash" : "254680e4dcb02cc0aff37bc3ac416f5d15b7c0f5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16257",
"triggerID" : "254680e4dcb02cc0aff37bc3ac416f5d15b7c0f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b24209938c245227cd6f1ccc1b428d50f3b51a9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16057)
* 254680e4dcb02cc0aff37bc3ac416f5d15b7c0f5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16257)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar merged pull request #8352: [HUDI-6015] Refresh the table after executing rollback and restoreToSavepoint
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar merged PR #8352:
URL: https://github.com/apache/hudi/pull/8352
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] huangxiaopingRD commented on pull request #8352: [HUDI-6015] Refresh the table after executing rollback to instantTime
Posted by "huangxiaopingRD (via GitHub)" <gi...@apache.org>.
huangxiaopingRD commented on PR #8352:
URL: https://github.com/apache/hudi/pull/8352#issuecomment-1512956650
> Good catch. Can you also add similar refresh in RollbackToSavepointProcedure.scala. Can you also add a test-case for this ?
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org