You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "alexeykudinkin (via GitHub)" <gi...@apache.org> on 2023/02/03 22:31:21 UTC
[GitHub] [hudi] alexeykudinkin opened a new pull request, #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
alexeykudinkin opened a new pull request, #7847:
URL: https://github.com/apache/hudi/pull/7847
### Change Logs
Currently, after most DML operations in Spark SQL, Hudi invokes `Catalog.refreshTable`
Prior to Spark 3.2, this was essentially doing the following:
1. Invalidating relation cache (forcing next time for relation to be re-resolved, creating new FileIndex, listing files, etc)
2. Trigger cascading invalidation (re-caching) of the cached data (in CacheManager)
As of Spark 3.2 it now additionally does `LogicalRelation.refresh` for ALL tables (previously this was only done for Temporary Views), therefore entailing
- Whole table to be re-listed again by triggering `FileIndex.refresh` which might be costly operation.
This change attempts to revert back to previous behavior from Spark 3.1, where `LogicalRelation` are not refreshed until they are actually read again.
### Impact
Avoiding costly `FileIndex.refresh` operations on the write path
### Risk level (write none, low medium or high below)
High (we need to carefully explore repercussions of adjusting refreshing behavior on all Spark versions)
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1432301821
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "28ce832318206166f9d72f58510d46b50ba652d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911",
"triggerID" : "28ce832318206166f9d72f58510d46b50ba652d2",
"triggerType" : "PUSH"
}, {
"hash" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044",
"triggerID" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"triggerType" : "PUSH"
}, {
"hash" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15220",
"triggerID" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044)
* aa44356a29f0de9c2a80e1f1c7b3ca956072797c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15220)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1432295056
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "28ce832318206166f9d72f58510d46b50ba652d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911",
"triggerID" : "28ce832318206166f9d72f58510d46b50ba652d2",
"triggerType" : "PUSH"
}, {
"hash" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044",
"triggerID" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"triggerType" : "PUSH"
}, {
"hash" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044)
* aa44356a29f0de9c2a80e1f1c7b3ca956072797c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
URL: https://github.com/apache/hudi/pull/7847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin commented on code in PR #7847:
URL: https://github.com/apache/hudi/pull/7847#discussion_r1116571566
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieCatalogUtils.scala:
##########
@@ -17,8 +17,76 @@
package org.apache.spark.sql
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+import org.apache.spark.sql.catalyst.{QualifiedTableName, TableIdentifier}
+
/**
* NOTE: Since support for [[TableCatalog]] was only added in Spark 3, this trait
* is going to be an empty one simply serving as a placeholder (for compatibility w/ Spark 2)
*/
trait HoodieCatalogUtils {}
+
+object HoodieCatalogUtils {
+
+ /**
+ * Please check scala-doc for other overloaded [[refreshTable()]] operation
+ */
+ def refreshTable(spark: SparkSession, qualifiedTableName: String): Unit = {
+ val tableId = spark.sessionState.sqlParser.parseTableIdentifier(qualifiedTableName)
+ refreshTable(spark, tableId)
+ }
+
+ /**
+ * Refreshes metadata and flushes cached data (resolved [[LogicalPlan]] representation,
+ * already loaded [[InMemoryRelation]]) for the table identified by [[tableId]].
+ *
+ * This method is usually invoked at the ond of the write operation to make sure cached
+ * data/metadata are synchronized with the state on storage.
+ *
+ * NOTE: PLEASE READ CAREFULLY BEFORE CHANGING
+ * This is borrowed from Spark 3.1.3 and modified to satisfy Hudi needs:
Review Comment:
Great question!
This seems to be the PR that changed it: https://github.com/apache/spark/pull/31206
I don't see any particular rationale for changing the part that triggers `relation.refresh()`. I guess the reason why Spark's core doesn't really care too much about it is simply b/c after listing of the (parquet) table, for ex, they simply create `InMemoryFileIndex` that is passed into `HadoopFsRelation` in that case you'd not notice the refresh as it actually just happens in memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "YannByron (via GitHub)" <gi...@apache.org>.
YannByron commented on code in PR #7847:
URL: https://github.com/apache/hudi/pull/7847#discussion_r1145600146
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieCatalogUtils.scala:
##########
@@ -17,8 +17,76 @@
package org.apache.spark.sql
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+import org.apache.spark.sql.catalyst.{QualifiedTableName, TableIdentifier}
+
/**
* NOTE: Since support for [[TableCatalog]] was only added in Spark 3, this trait
* is going to be an empty one simply serving as a placeholder (for compatibility w/ Spark 2)
*/
trait HoodieCatalogUtils {}
+
+object HoodieCatalogUtils {
+
+ /**
+ * Please check scala-doc for other overloaded [[refreshTable()]] operation
+ */
+ def refreshTable(spark: SparkSession, qualifiedTableName: String): Unit = {
+ val tableId = spark.sessionState.sqlParser.parseTableIdentifier(qualifiedTableName)
+ refreshTable(spark, tableId)
+ }
+
+ /**
+ * Refreshes metadata and flushes cached data (resolved [[LogicalPlan]] representation,
+ * already loaded [[InMemoryRelation]]) for the table identified by [[tableId]].
+ *
+ * This method is usually invoked at the end of the write operation to make sure cached
+ * data/metadata are synchronized with the state on storage.
+ *
+ * NOTE: PLEASE READ CAREFULLY BEFORE CHANGING
+ * This is borrowed from Spark 3.1.3 and modified to satisfy Hudi needs:
+ * - Unlike Spark canonical implementation, in case of Hudi this method is invoked
+ * after writes carried out via Spark DataSource integration as well and as such
+ * in these cases data might actually be missing from the caches, therefore
+ * actually re-triggering resolution phase (involving file-listing, etc) for the
+ * first time
+ * - Additionally, this method is modified to avoid refreshing [[LogicalRelation]]
+ * completely to make sure that we're not re-triggering the file-listing of the
+ * table, immediately after it's been written, instead deferring it to subsequent
+ * read operation
+ */
+ def refreshTable(spark: SparkSession, tableId: TableIdentifier): Unit = {
+ val sessionCatalog = spark.sessionState.catalog
+ val tableMetadata = sessionCatalog.getTempViewOrPermanentTableMetadata(tableId)
+
+ // Before proceeding we validate whether this table is actually cached w/in [[SessionCatalog]],
+ // since, for ex, in case of writing via Spark DataSource (V1) API, Spark wouldn't actually
+ // resort to caching the data
+ val cachedPlan = sessionCatalog.getCachedTable(
+ QualifiedTableName(tableId.database.getOrElse(tableMetadata.database), tableId.identifier))
+
+ if (cachedPlan != null) {
+ // NOTE: Provided that this table is still cached, following operation would not be
+ // triggering subsequent resolution and listing of the table
+ val table = spark.table(tableId)
+
+ if (tableMetadata.tableType == CatalogTableType.VIEW) {
Review Comment:
https://github.com/apache/spark/pull/31265 suggest not to invalidate view relation, so should we keep it and do nothing if `tableMetadata.tableType == CatalogTableType.VIEW`?
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieCatalogUtils.scala:
##########
@@ -17,8 +17,76 @@
package org.apache.spark.sql
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+import org.apache.spark.sql.catalyst.{QualifiedTableName, TableIdentifier}
+
/**
* NOTE: Since support for [[TableCatalog]] was only added in Spark 3, this trait
* is going to be an empty one simply serving as a placeholder (for compatibility w/ Spark 2)
*/
trait HoodieCatalogUtils {}
+
+object HoodieCatalogUtils {
+
+ /**
+ * Please check scala-doc for other overloaded [[refreshTable()]] operation
+ */
+ def refreshTable(spark: SparkSession, qualifiedTableName: String): Unit = {
+ val tableId = spark.sessionState.sqlParser.parseTableIdentifier(qualifiedTableName)
+ refreshTable(spark, tableId)
+ }
+
+ /**
+ * Refreshes metadata and flushes cached data (resolved [[LogicalPlan]] representation,
+ * already loaded [[InMemoryRelation]]) for the table identified by [[tableId]].
+ *
+ * This method is usually invoked at the end of the write operation to make sure cached
+ * data/metadata are synchronized with the state on storage.
+ *
+ * NOTE: PLEASE READ CAREFULLY BEFORE CHANGING
+ * This is borrowed from Spark 3.1.3 and modified to satisfy Hudi needs:
+ * - Unlike Spark canonical implementation, in case of Hudi this method is invoked
+ * after writes carried out via Spark DataSource integration as well and as such
+ * in these cases data might actually be missing from the caches, therefore
+ * actually re-triggering resolution phase (involving file-listing, etc) for the
+ * first time
+ * - Additionally, this method is modified to avoid refreshing [[LogicalRelation]]
+ * completely to make sure that we're not re-triggering the file-listing of the
+ * table, immediately after it's been written, instead deferring it to subsequent
+ * read operation
+ */
+ def refreshTable(spark: SparkSession, tableId: TableIdentifier): Unit = {
+ val sessionCatalog = spark.sessionState.catalog
+ val tableMetadata = sessionCatalog.getTempViewOrPermanentTableMetadata(tableId)
+
+ // Before proceeding we validate whether this table is actually cached w/in [[SessionCatalog]],
+ // since, for ex, in case of writing via Spark DataSource (V1) API, Spark wouldn't actually
+ // resort to caching the data
+ val cachedPlan = sessionCatalog.getCachedTable(
+ QualifiedTableName(tableId.database.getOrElse(tableMetadata.database), tableId.identifier))
+
+ if (cachedPlan != null) {
+ // NOTE: Provided that this table is still cached, following operation would not be
+ // triggering subsequent resolution and listing of the table
+ val table = spark.table(tableId)
+
+ if (tableMetadata.tableType == CatalogTableType.VIEW) {
Review Comment:
https://github.com/apache/spark/pull/31265 suggests not to invalidate view relation, so should we keep it and do nothing if `tableMetadata.tableType == CatalogTableType.VIEW`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #7847:
URL: https://github.com/apache/hudi/pull/7847#discussion_r1116550390
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieCatalogUtils.scala:
##########
@@ -17,8 +17,76 @@
package org.apache.spark.sql
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+import org.apache.spark.sql.catalyst.{QualifiedTableName, TableIdentifier}
+
/**
* NOTE: Since support for [[TableCatalog]] was only added in Spark 3, this trait
* is going to be an empty one simply serving as a placeholder (for compatibility w/ Spark 2)
*/
trait HoodieCatalogUtils {}
+
+object HoodieCatalogUtils {
+
+ /**
+ * Please check scala-doc for other overloaded [[refreshTable()]] operation
+ */
+ def refreshTable(spark: SparkSession, qualifiedTableName: String): Unit = {
+ val tableId = spark.sessionState.sqlParser.parseTableIdentifier(qualifiedTableName)
+ refreshTable(spark, tableId)
+ }
+
+ /**
+ * Refreshes metadata and flushes cached data (resolved [[LogicalPlan]] representation,
+ * already loaded [[InMemoryRelation]]) for the table identified by [[tableId]].
+ *
+ * This method is usually invoked at the ond of the write operation to make sure cached
Review Comment:
```suggestion
* This method is usually invoked at the end of the write operation to make sure cached
```
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieCatalogUtils.scala:
##########
@@ -17,8 +17,76 @@
package org.apache.spark.sql
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+import org.apache.spark.sql.catalyst.{QualifiedTableName, TableIdentifier}
+
/**
* NOTE: Since support for [[TableCatalog]] was only added in Spark 3, this trait
* is going to be an empty one simply serving as a placeholder (for compatibility w/ Spark 2)
*/
trait HoodieCatalogUtils {}
+
+object HoodieCatalogUtils {
+
+ /**
+ * Please check scala-doc for other overloaded [[refreshTable()]] operation
+ */
+ def refreshTable(spark: SparkSession, qualifiedTableName: String): Unit = {
+ val tableId = spark.sessionState.sqlParser.parseTableIdentifier(qualifiedTableName)
+ refreshTable(spark, tableId)
+ }
+
+ /**
+ * Refreshes metadata and flushes cached data (resolved [[LogicalPlan]] representation,
+ * already loaded [[InMemoryRelation]]) for the table identified by [[tableId]].
+ *
+ * This method is usually invoked at the ond of the write operation to make sure cached
+ * data/metadata are synchronized with the state on storage.
+ *
+ * NOTE: PLEASE READ CAREFULLY BEFORE CHANGING
+ * This is borrowed from Spark 3.1.3 and modified to satisfy Hudi needs:
Review Comment:
I am wondering what prompted Spark to change the behavior in version 3.2? IMO, whatever Spark 3.1.3 was doing is more reasonable. What do we lose by not sticking to 3.2 behavior?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1423367248
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "28ce832318206166f9d72f58510d46b50ba652d2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911",
"triggerID" : "28ce832318206166f9d72f58510d46b50ba652d2",
"triggerType" : "PUSH"
}, {
"hash" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044",
"triggerID" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 28ce832318206166f9d72f58510d46b50ba652d2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911)
* 51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1416520697
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "28ce832318206166f9d72f58510d46b50ba652d2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911",
"triggerID" : "28ce832318206166f9d72f58510d46b50ba652d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 28ce832318206166f9d72f58510d46b50ba652d2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1442597841
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "28ce832318206166f9d72f58510d46b50ba652d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911",
"triggerID" : "28ce832318206166f9d72f58510d46b50ba652d2",
"triggerType" : "PUSH"
}, {
"hash" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044",
"triggerID" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"triggerType" : "PUSH"
}, {
"hash" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15220",
"triggerID" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"triggerType" : "PUSH"
}, {
"hash" : "2e431b454879172e6042db57391c5345c24479f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15252",
"triggerID" : "2e431b454879172e6042db57391c5345c24479f9",
"triggerType" : "PUSH"
}, {
"hash" : "0241dc20d80d9f8d5fee38637ba1c3b5277ed55f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15369",
"triggerID" : "0241dc20d80d9f8d5fee38637ba1c3b5277ed55f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0241dc20d80d9f8d5fee38637ba1c3b5277ed55f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15369)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1423362089
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "28ce832318206166f9d72f58510d46b50ba652d2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911",
"triggerID" : "28ce832318206166f9d72f58510d46b50ba652d2",
"triggerType" : "PUSH"
}, {
"hash" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 28ce832318206166f9d72f58510d46b50ba652d2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911)
* 51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1423786708
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "28ce832318206166f9d72f58510d46b50ba652d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911",
"triggerID" : "28ce832318206166f9d72f58510d46b50ba652d2",
"triggerType" : "PUSH"
}, {
"hash" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044",
"triggerID" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1432501333
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "28ce832318206166f9d72f58510d46b50ba652d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911",
"triggerID" : "28ce832318206166f9d72f58510d46b50ba652d2",
"triggerType" : "PUSH"
}, {
"hash" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044",
"triggerID" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"triggerType" : "PUSH"
}, {
"hash" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15220",
"triggerID" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aa44356a29f0de9c2a80e1f1c7b3ca956072797c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15220)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1442930988
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "28ce832318206166f9d72f58510d46b50ba652d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911",
"triggerID" : "28ce832318206166f9d72f58510d46b50ba652d2",
"triggerType" : "PUSH"
}, {
"hash" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044",
"triggerID" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"triggerType" : "PUSH"
}, {
"hash" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15220",
"triggerID" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"triggerType" : "PUSH"
}, {
"hash" : "2e431b454879172e6042db57391c5345c24479f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15252",
"triggerID" : "2e431b454879172e6042db57391c5345c24479f9",
"triggerType" : "PUSH"
}, {
"hash" : "0241dc20d80d9f8d5fee38637ba1c3b5277ed55f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15369",
"triggerID" : "0241dc20d80d9f8d5fee38637ba1c3b5277ed55f",
"triggerType" : "PUSH"
}, {
"hash" : "24f50e8f624dd2b928cf1c6c4ca7db8b84c760fd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15379",
"triggerID" : "24f50e8f624dd2b928cf1c6c4ca7db8b84c760fd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0241dc20d80d9f8d5fee38637ba1c3b5277ed55f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15369)
* 24f50e8f624dd2b928cf1c6c4ca7db8b84c760fd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15379)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1433399717
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "28ce832318206166f9d72f58510d46b50ba652d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911",
"triggerID" : "28ce832318206166f9d72f58510d46b50ba652d2",
"triggerType" : "PUSH"
}, {
"hash" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044",
"triggerID" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"triggerType" : "PUSH"
}, {
"hash" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15220",
"triggerID" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"triggerType" : "PUSH"
}, {
"hash" : "2e431b454879172e6042db57391c5345c24479f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2e431b454879172e6042db57391c5345c24479f9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aa44356a29f0de9c2a80e1f1c7b3ca956072797c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15220)
* 2e431b454879172e6042db57391c5345c24479f9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] CTTY commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1433921836
Cherry-picked this fix on Hudi 0.12.1 and tested with Spark 3.3.0 and it looks good! No listing requests after the hive sync when Spark refresh the table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1435123694
CI is green:
<img width="1638" alt="Screenshot 2023-02-17 at 11 16 40 AM" src="https://user-images.githubusercontent.com/428277/219766427-86bee875-2f3c-4d7d-8308-45a43f5a1ebc.png">
https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=15252&view=results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1416516680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "28ce832318206166f9d72f58510d46b50ba652d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "28ce832318206166f9d72f58510d46b50ba652d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 28ce832318206166f9d72f58510d46b50ba652d2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1442908327
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "28ce832318206166f9d72f58510d46b50ba652d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911",
"triggerID" : "28ce832318206166f9d72f58510d46b50ba652d2",
"triggerType" : "PUSH"
}, {
"hash" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044",
"triggerID" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"triggerType" : "PUSH"
}, {
"hash" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15220",
"triggerID" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"triggerType" : "PUSH"
}, {
"hash" : "2e431b454879172e6042db57391c5345c24479f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15252",
"triggerID" : "2e431b454879172e6042db57391c5345c24479f9",
"triggerType" : "PUSH"
}, {
"hash" : "0241dc20d80d9f8d5fee38637ba1c3b5277ed55f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15369",
"triggerID" : "0241dc20d80d9f8d5fee38637ba1c3b5277ed55f",
"triggerType" : "PUSH"
}, {
"hash" : "24f50e8f624dd2b928cf1c6c4ca7db8b84c760fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24f50e8f624dd2b928cf1c6c4ca7db8b84c760fd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0241dc20d80d9f8d5fee38637ba1c3b5277ed55f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15369)
* 24f50e8f624dd2b928cf1c6c4ca7db8b84c760fd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #7847:
URL: https://github.com/apache/hudi/pull/7847#discussion_r1148306772
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieCatalogUtils.scala:
##########
@@ -17,8 +17,76 @@
package org.apache.spark.sql
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+import org.apache.spark.sql.catalyst.{QualifiedTableName, TableIdentifier}
+
/**
* NOTE: Since support for [[TableCatalog]] was only added in Spark 3, this trait
* is going to be an empty one simply serving as a placeholder (for compatibility w/ Spark 2)
*/
trait HoodieCatalogUtils {}
+
+object HoodieCatalogUtils {
+
+ /**
+ * Please check scala-doc for other overloaded [[refreshTable()]] operation
+ */
+ def refreshTable(spark: SparkSession, qualifiedTableName: String): Unit = {
+ val tableId = spark.sessionState.sqlParser.parseTableIdentifier(qualifiedTableName)
+ refreshTable(spark, tableId)
+ }
+
+ /**
+ * Refreshes metadata and flushes cached data (resolved [[LogicalPlan]] representation,
+ * already loaded [[InMemoryRelation]]) for the table identified by [[tableId]].
+ *
+ * This method is usually invoked at the end of the write operation to make sure cached
+ * data/metadata are synchronized with the state on storage.
+ *
+ * NOTE: PLEASE READ CAREFULLY BEFORE CHANGING
+ * This is borrowed from Spark 3.1.3 and modified to satisfy Hudi needs:
+ * - Unlike Spark canonical implementation, in case of Hudi this method is invoked
+ * after writes carried out via Spark DataSource integration as well and as such
+ * in these cases data might actually be missing from the caches, therefore
+ * actually re-triggering resolution phase (involving file-listing, etc) for the
+ * first time
+ * - Additionally, this method is modified to avoid refreshing [[LogicalRelation]]
+ * completely to make sure that we're not re-triggering the file-listing of the
+ * table, immediately after it's been written, instead deferring it to subsequent
+ * read operation
+ */
+ def refreshTable(spark: SparkSession, tableId: TableIdentifier): Unit = {
+ val sessionCatalog = spark.sessionState.catalog
+ val tableMetadata = sessionCatalog.getTempViewOrPermanentTableMetadata(tableId)
+
+ // Before proceeding we validate whether this table is actually cached w/in [[SessionCatalog]],
+ // since, for ex, in case of writing via Spark DataSource (V1) API, Spark wouldn't actually
+ // resort to caching the data
+ val cachedPlan = sessionCatalog.getCachedTable(
+ QualifiedTableName(tableId.database.getOrElse(tableMetadata.database), tableId.identifier))
+
+ if (cachedPlan != null) {
+ // NOTE: Provided that this table is still cached, following operation would not be
+ // triggering subsequent resolution and listing of the table
+ val table = spark.table(tableId)
+
+ if (tableMetadata.tableType == CatalogTableType.VIEW) {
Review Comment:
I concur with @YannByron comment.
@alexeykudinkin What do you think? IMO, not invalidating the relation cache could actually help performance slightly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1433413934
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "28ce832318206166f9d72f58510d46b50ba652d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911",
"triggerID" : "28ce832318206166f9d72f58510d46b50ba652d2",
"triggerType" : "PUSH"
}, {
"hash" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044",
"triggerID" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"triggerType" : "PUSH"
}, {
"hash" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15220",
"triggerID" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"triggerType" : "PUSH"
}, {
"hash" : "2e431b454879172e6042db57391c5345c24479f9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15252",
"triggerID" : "2e431b454879172e6042db57391c5345c24479f9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aa44356a29f0de9c2a80e1f1c7b3ca956072797c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15220)
* 2e431b454879172e6042db57391c5345c24479f9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15252)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1434094265
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "28ce832318206166f9d72f58510d46b50ba652d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911",
"triggerID" : "28ce832318206166f9d72f58510d46b50ba652d2",
"triggerType" : "PUSH"
}, {
"hash" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044",
"triggerID" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"triggerType" : "PUSH"
}, {
"hash" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15220",
"triggerID" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"triggerType" : "PUSH"
}, {
"hash" : "2e431b454879172e6042db57391c5345c24479f9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15252",
"triggerID" : "2e431b454879172e6042db57391c5345c24479f9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e431b454879172e6042db57391c5345c24479f9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15252)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1442495331
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "28ce832318206166f9d72f58510d46b50ba652d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911",
"triggerID" : "28ce832318206166f9d72f58510d46b50ba652d2",
"triggerType" : "PUSH"
}, {
"hash" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044",
"triggerID" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"triggerType" : "PUSH"
}, {
"hash" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15220",
"triggerID" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"triggerType" : "PUSH"
}, {
"hash" : "2e431b454879172e6042db57391c5345c24479f9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15252",
"triggerID" : "2e431b454879172e6042db57391c5345c24479f9",
"triggerType" : "PUSH"
}, {
"hash" : "0241dc20d80d9f8d5fee38637ba1c3b5277ed55f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15369",
"triggerID" : "0241dc20d80d9f8d5fee38637ba1c3b5277ed55f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e431b454879172e6042db57391c5345c24479f9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15252)
* 0241dc20d80d9f8d5fee38637ba1c3b5277ed55f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15369)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1443522421
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "28ce832318206166f9d72f58510d46b50ba652d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911",
"triggerID" : "28ce832318206166f9d72f58510d46b50ba652d2",
"triggerType" : "PUSH"
}, {
"hash" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044",
"triggerID" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"triggerType" : "PUSH"
}, {
"hash" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15220",
"triggerID" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"triggerType" : "PUSH"
}, {
"hash" : "2e431b454879172e6042db57391c5345c24479f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15252",
"triggerID" : "2e431b454879172e6042db57391c5345c24479f9",
"triggerType" : "PUSH"
}, {
"hash" : "0241dc20d80d9f8d5fee38637ba1c3b5277ed55f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15369",
"triggerID" : "0241dc20d80d9f8d5fee38637ba1c3b5277ed55f",
"triggerType" : "PUSH"
}, {
"hash" : "24f50e8f624dd2b928cf1c6c4ca7db8b84c760fd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15379",
"triggerID" : "24f50e8f624dd2b928cf1c6c4ca7db8b84c760fd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24f50e8f624dd2b928cf1c6c4ca7db8b84c760fd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15379)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7847: [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-1442457327
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "28ce832318206166f9d72f58510d46b50ba652d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14911",
"triggerID" : "28ce832318206166f9d72f58510d46b50ba652d2",
"triggerType" : "PUSH"
}, {
"hash" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15044",
"triggerID" : "51c7c7a8b8a5ce1fba2199d0a81a1722d3059c03",
"triggerType" : "PUSH"
}, {
"hash" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15220",
"triggerID" : "aa44356a29f0de9c2a80e1f1c7b3ca956072797c",
"triggerType" : "PUSH"
}, {
"hash" : "2e431b454879172e6042db57391c5345c24479f9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15252",
"triggerID" : "2e431b454879172e6042db57391c5345c24479f9",
"triggerType" : "PUSH"
}, {
"hash" : "0241dc20d80d9f8d5fee38637ba1c3b5277ed55f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0241dc20d80d9f8d5fee38637ba1c3b5277ed55f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e431b454879172e6042db57391c5345c24479f9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15252)
* 0241dc20d80d9f8d5fee38637ba1c3b5277ed55f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5697] Revisiting refreshing of Hudi relations after write operations on the tables [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #7847:
URL: https://github.com/apache/hudi/pull/7847#issuecomment-6700099583
Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org