You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2023/11/21 14:14:46 UTC
[PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
codope opened a new pull request, #10153:
URL: https://github.com/apache/hudi/pull/10153
### Change Logs
With files and functional index, it should be possible to read the table from spark-sql attached to an external catalog, say remote Hive Metastore, even when the partition metadata is not syned to the catalog.
- Check whether partitions are passed to `HoodieCatalog`. If not, then use the metadata file system view to get the partitions.
- Add a test for querying a table without syncing partition metadata.
NOTE:
1. The change is in `HoodieCatalog` which is dependent on Spark 3.2. So, this feature is not available for Spark versions < 3.2.0.
2. Currently, the PR assumes that the partition paths will be hive-style partitioned.
### Impact
Query a table from spark-sql that is syned to catalog without the partition metadata. This means we can make meta sync faster and do away with heavy partition metadata.
### Risk level (write none, low medium or high below)
medium
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #10153:
URL: https://github.com/apache/hudi/pull/10153
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1821942182
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 46a4c3344c79fd9a61db78620e8c40e7d98bcd36 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10153:
URL: https://github.com/apache/hudi/pull/10153#discussion_r1420366980
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -140,6 +147,75 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Functional Index With Hive Sync Non Partitioned Table") {
+ // There is a big difference between Java class loader architecture of versions 1.8 and 17.
+ // Hive 2.3.7 is compiled with Java 1.8, and the class loader used there throws error when Hive APIs are run on Java 17.
+ // So we special case this test only for Java 8.
+ if (HoodieSparkUtils.gteqSpark3_2 && HoodieTestUtils.getJavaVersion == 8) {
+ withTempDir { tmp =>
+ Seq("mor").foreach { tableType =>
+ val databaseName = "default"
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = '$tableType',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ // ts=1000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-01-01'
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ // ts=100000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-01-02'
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 100000)")
+ // ts=10000000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-04-26'
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 10000000)")
+
+ val createIndexSql = s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')"
+ spark.sql(createIndexSql)
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(basePath)
+ .setConf(spark.sessionState.newHadoopConf())
+ .build()
+ assertTrue(metaClient.getFunctionalIndexMetadata.isPresent)
+ val functionalIndexMetadata = metaClient.getFunctionalIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
+
+ // sync to hive without partition metadata
+ val hiveSyncProps = new TypedProperties()
+ hiveSyncProps.setProperty(HIVE_USER.key, "")
+ hiveSyncProps.setProperty(HIVE_PASS.key, "")
+ hiveSyncProps.setProperty(META_SYNC_DATABASE_NAME.key, databaseName)
+ hiveSyncProps.setProperty(META_SYNC_TABLE_NAME.key, tableName)
+ hiveSyncProps.setProperty(META_SYNC_BASE_PATH.key, basePath)
+ hiveSyncProps.setProperty(HIVE_USE_PRE_APACHE_INPUT_FORMAT.key, "false")
+ HiveTestUtil.setUp(Option.of(hiveSyncProps), false)
+ val tool = new HiveSyncTool(hiveSyncProps, HiveTestUtil.getHiveConf)
+ tool.syncHoodieTable()
Review Comment:
used the new config in tests and added validation for no partition metadata
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1849349982
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "76e1d519c42565b99b35837c3c767b600cdee311",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "76e1d519c42565b99b35837c3c767b600cdee311",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 76e1d519c42565b99b35837c3c767b600cdee311 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1848863968
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
}, {
"hash" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334",
"triggerID" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"triggerType" : "PUSH"
}, {
"hash" : "f631760188373ba756495f50d85421e4429b70c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21339",
"triggerID" : "f631760188373ba756495f50d85421e4429b70c9",
"triggerType" : "PUSH"
}, {
"hash" : "df20980253cf67ca2cb871e43a30142534854c5a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21405",
"triggerID" : "df20980253cf67ca2cb871e43a30142534854c5a",
"triggerType" : "PUSH"
}, {
"hash" : "4054985bed685d995f568217400a7f66bf591a72",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4054985bed685d995f568217400a7f66bf591a72",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df20980253cf67ca2cb871e43a30142534854c5a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21405)
* 4054985bed685d995f568217400a7f66bf591a72 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1842873490
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
}, {
"hash" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334",
"triggerID" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"triggerType" : "PUSH"
}, {
"hash" : "f631760188373ba756495f50d85421e4429b70c9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21339",
"triggerID" : "f631760188373ba756495f50d85421e4429b70c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d89786780e149be16784a44c7b4d319a2e756b6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334)
* f631760188373ba756495f50d85421e4429b70c9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21339)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10153:
URL: https://github.com/apache/hudi/pull/10153#discussion_r1416836312
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -140,6 +146,68 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Functional Index With Hive Sync Non Partitioned Table") {
+ if (HoodieSparkUtils.gteqSpark3_2) {
+ withTempDir { tmp =>
+ Seq("mor").foreach { tableType =>
+ val databaseName = "default"
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = '$tableType',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 1001)")
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 1002)")
+
+ val createIndexSql = s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')"
+ spark.sql(createIndexSql)
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(basePath)
+ .setConf(spark.sessionState.newHadoopConf())
+ .build()
+ assertTrue(metaClient.getFunctionalIndexMetadata.isPresent)
+ val functionalIndexMetadata = metaClient.getFunctionalIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
+
+ // sync to hive without partition metadata
+ val hiveSyncProps = new TypedProperties()
+ hiveSyncProps.setProperty(HIVE_URL.key, "jdbc:hive2://localhost:10000")
+ hiveSyncProps.setProperty(HIVE_USER.key, "")
+ hiveSyncProps.setProperty(HIVE_PASS.key, "")
+ hiveSyncProps.setProperty(META_SYNC_DATABASE_NAME.key, databaseName)
+ hiveSyncProps.setProperty(META_SYNC_TABLE_NAME.key, tableName)
+ hiveSyncProps.setProperty(META_SYNC_BASE_PATH.key, basePath)
+ hiveSyncProps.setProperty(HIVE_USE_PRE_APACHE_INPUT_FORMAT.key, "false")
+ // hiveSyncProps.setProperty(META_SYNC_PARTITION_FIELDS.key, "datestr")
+ hiveSyncProps.setProperty(META_SYNC_PARTITION_EXTRACTOR_CLASS.key, classOf[SlashEncodedDayPartitionValueExtractor].getName)
+ hiveSyncProps.setProperty(HIVE_BATCH_SYNC_PARTITION_NUM.key, "3")
+ HiveTestUtil.setUp(Option.of(hiveSyncProps))
Review Comment:
It's fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1849027683
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
}, {
"hash" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334",
"triggerID" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"triggerType" : "PUSH"
}, {
"hash" : "f631760188373ba756495f50d85421e4429b70c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21339",
"triggerID" : "f631760188373ba756495f50d85421e4429b70c9",
"triggerType" : "PUSH"
}, {
"hash" : "df20980253cf67ca2cb871e43a30142534854c5a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21405",
"triggerID" : "df20980253cf67ca2cb871e43a30142534854c5a",
"triggerType" : "PUSH"
}, {
"hash" : "4054985bed685d995f568217400a7f66bf591a72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21428",
"triggerID" : "4054985bed685d995f568217400a7f66bf591a72",
"triggerType" : "PUSH"
}, {
"hash" : "76e1d519c42565b99b35837c3c767b600cdee311",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21429",
"triggerID" : "76e1d519c42565b99b35837c3c767b600cdee311",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 76e1d519c42565b99b35837c3c767b600cdee311 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21429)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10153:
URL: https://github.com/apache/hudi/pull/10153#discussion_r1419864951
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala:
##########
@@ -54,7 +59,8 @@ class HoodieCatalog extends DelegatingCatalogExtension
override def stageCreate(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: util.Map[String, String]): StagedTable = {
if (sparkAdapter.isHoodieTable(properties)) {
val locUriAndTableType = deduceTableLocationURIAndTableType(ident, properties)
- HoodieStagedTable(ident, locUriAndTableType, this, schema, partitions,
+ val partitionTransforms = if (partitions.isEmpty) buildPartitionTransforms(spark, locUriAndTableType._1.getPath) else partitions
Review Comment:
oh yeah good point. I think `partitions` is just one element. I will test and handle this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1847092018
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
}, {
"hash" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334",
"triggerID" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"triggerType" : "PUSH"
}, {
"hash" : "f631760188373ba756495f50d85421e4429b70c9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21339",
"triggerID" : "f631760188373ba756495f50d85421e4429b70c9",
"triggerType" : "PUSH"
}, {
"hash" : "df20980253cf67ca2cb871e43a30142534854c5a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "df20980253cf67ca2cb871e43a30142534854c5a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f631760188373ba756495f50d85421e4429b70c9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21339)
* df20980253cf67ca2cb871e43a30142534854c5a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1848980984
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
}, {
"hash" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334",
"triggerID" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"triggerType" : "PUSH"
}, {
"hash" : "f631760188373ba756495f50d85421e4429b70c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21339",
"triggerID" : "f631760188373ba756495f50d85421e4429b70c9",
"triggerType" : "PUSH"
}, {
"hash" : "df20980253cf67ca2cb871e43a30142534854c5a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21405",
"triggerID" : "df20980253cf67ca2cb871e43a30142534854c5a",
"triggerType" : "PUSH"
}, {
"hash" : "4054985bed685d995f568217400a7f66bf591a72",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21428",
"triggerID" : "4054985bed685d995f568217400a7f66bf591a72",
"triggerType" : "PUSH"
}, {
"hash" : "76e1d519c42565b99b35837c3c767b600cdee311",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "76e1d519c42565b99b35837c3c767b600cdee311",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4054985bed685d995f568217400a7f66bf591a72 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21428)
* 76e1d519c42565b99b35837c3c767b600cdee311 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10153:
URL: https://github.com/apache/hudi/pull/10153#discussion_r1419864232
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -140,6 +147,75 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Functional Index With Hive Sync Non Partitioned Table") {
+ // There is a big difference between Java class loader architecture of versions 1.8 and 17.
+ // Hive 2.3.7 is compiled with Java 1.8, and the class loader used there throws error when Hive APIs are run on Java 17.
+ // So we special case this test only for Java 8.
+ if (HoodieSparkUtils.gteqSpark3_2 && HoodieTestUtils.getJavaVersion == 8) {
+ withTempDir { tmp =>
+ Seq("mor").foreach { tableType =>
+ val databaseName = "default"
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = '$tableType',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ // ts=1000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-01-01'
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ // ts=100000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-01-02'
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 100000)")
+ // ts=10000000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-04-26'
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 10000000)")
+
+ val createIndexSql = s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')"
+ spark.sql(createIndexSql)
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(basePath)
+ .setConf(spark.sessionState.newHadoopConf())
+ .build()
+ assertTrue(metaClient.getFunctionalIndexMetadata.isPresent)
+ val functionalIndexMetadata = metaClient.getFunctionalIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
+
+ // sync to hive without partition metadata
+ val hiveSyncProps = new TypedProperties()
Review Comment:
I am adding anew table config for logical partitioning in https://github.com/apache/hudi/pull/10274/files#diff-53ae78ff1f1bd5d8b0f87cb69853299e5228b44f30b770e27f60c0c3c27d4185R301
If logical partitioning is enabled for a table then neither the table will be physically partitioned, nor the partition metadata be synced to catalog. Do you think we should add a separate meta sync config, as in, user can still have physically partitioned table but no need to sync partition metadata to catalog?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10153:
URL: https://github.com/apache/hudi/pull/10153#discussion_r1419868357
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala:
##########
@@ -369,4 +378,28 @@ object HoodieCatalog {
(identityCols, bucketSpec)
}
+
+ def buildPartitionTransforms(spark: SparkSession,
+ basePath: String): Array[Transform] = {
+ val metaClient = HoodieTableMetaClient.builder()
+ .setConf(spark.sessionState.newHadoopConf())
+ .setBasePath(basePath)
+ .build()
Review Comment:
From debugging the tests, I see that this is called just once when the table is loaded. I think this should be fine. Even in other query engines, we build metaclient once per query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1847355964
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
}, {
"hash" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334",
"triggerID" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"triggerType" : "PUSH"
}, {
"hash" : "f631760188373ba756495f50d85421e4429b70c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21339",
"triggerID" : "f631760188373ba756495f50d85421e4429b70c9",
"triggerType" : "PUSH"
}, {
"hash" : "df20980253cf67ca2cb871e43a30142534854c5a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21405",
"triggerID" : "df20980253cf67ca2cb871e43a30142534854c5a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df20980253cf67ca2cb871e43a30142534854c5a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21405)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1847148101
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
}, {
"hash" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334",
"triggerID" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"triggerType" : "PUSH"
}, {
"hash" : "f631760188373ba756495f50d85421e4429b70c9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21339",
"triggerID" : "f631760188373ba756495f50d85421e4429b70c9",
"triggerType" : "PUSH"
}, {
"hash" : "df20980253cf67ca2cb871e43a30142534854c5a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21405",
"triggerID" : "df20980253cf67ca2cb871e43a30142534854c5a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f631760188373ba756495f50d85421e4429b70c9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21339)
* df20980253cf67ca2cb871e43a30142534854c5a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21405)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #10153:
URL: https://github.com/apache/hudi/pull/10153#discussion_r1419477488
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -140,6 +147,75 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Functional Index With Hive Sync Non Partitioned Table") {
+ // There is a big difference between Java class loader architecture of versions 1.8 and 17.
+ // Hive 2.3.7 is compiled with Java 1.8, and the class loader used there throws error when Hive APIs are run on Java 17.
+ // So we special case this test only for Java 8.
+ if (HoodieSparkUtils.gteqSpark3_2 && HoodieTestUtils.getJavaVersion == 8) {
+ withTempDir { tmp =>
+ Seq("mor").foreach { tableType =>
+ val databaseName = "default"
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = '$tableType',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ // ts=1000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-01-01'
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ // ts=100000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-01-02'
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 100000)")
+ // ts=10000000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-04-26'
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 10000000)")
+
+ val createIndexSql = s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')"
+ spark.sql(createIndexSql)
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(basePath)
+ .setConf(spark.sessionState.newHadoopConf())
+ .build()
+ assertTrue(metaClient.getFunctionalIndexMetadata.isPresent)
+ val functionalIndexMetadata = metaClient.getFunctionalIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
+
+ // sync to hive without partition metadata
+ val hiveSyncProps = new TypedProperties()
Review Comment:
Based on the logic, the user needs to unset the partition filed for meta sync, correct? Could such a change by user not take effect in the meta-sync logic in the write client (e.g., we automatically prepare the hive sync configs based on the partition field)? Should we introduce a new config like `hoodie.meta.sync.no_partition_metadata` to turn the new feature on or off?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -140,6 +147,75 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Functional Index With Hive Sync Non Partitioned Table") {
+ // There is a big difference between Java class loader architecture of versions 1.8 and 17.
+ // Hive 2.3.7 is compiled with Java 1.8, and the class loader used there throws error when Hive APIs are run on Java 17.
+ // So we special case this test only for Java 8.
+ if (HoodieSparkUtils.gteqSpark3_2 && HoodieTestUtils.getJavaVersion == 8) {
+ withTempDir { tmp =>
+ Seq("mor").foreach { tableType =>
+ val databaseName = "default"
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = '$tableType',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ // ts=1000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-01-01'
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ // ts=100000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-01-02'
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 100000)")
+ // ts=10000000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-04-26'
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 10000000)")
+
+ val createIndexSql = s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')"
+ spark.sql(createIndexSql)
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(basePath)
+ .setConf(spark.sessionState.newHadoopConf())
+ .build()
+ assertTrue(metaClient.getFunctionalIndexMetadata.isPresent)
+ val functionalIndexMetadata = metaClient.getFunctionalIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
+
+ // sync to hive without partition metadata
+ val hiveSyncProps = new TypedProperties()
+ hiveSyncProps.setProperty(HIVE_USER.key, "")
+ hiveSyncProps.setProperty(HIVE_PASS.key, "")
+ hiveSyncProps.setProperty(META_SYNC_DATABASE_NAME.key, databaseName)
+ hiveSyncProps.setProperty(META_SYNC_TABLE_NAME.key, tableName)
+ hiveSyncProps.setProperty(META_SYNC_BASE_PATH.key, basePath)
+ hiveSyncProps.setProperty(HIVE_USE_PRE_APACHE_INPUT_FORMAT.key, "false")
+ HiveTestUtil.setUp(Option.of(hiveSyncProps), false)
+ val tool = new HiveSyncTool(hiveSyncProps, HiveTestUtil.getHiveConf)
+ tool.syncHoodieTable()
Review Comment:
Add a validation here that the HMS does not have partition information of the table?
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala:
##########
@@ -369,4 +378,28 @@ object HoodieCatalog {
(identityCols, bucketSpec)
}
+
+ def buildPartitionTransforms(spark: SparkSession,
+ basePath: String): Array[Transform] = {
+ val metaClient = HoodieTableMetaClient.builder()
+ .setConf(spark.sessionState.newHadoopConf())
+ .setBasePath(basePath)
+ .build()
Review Comment:
Do you know under what conditions the stage change methods are called? This method has non-trivial latency. If often called, this may not scale.
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala:
##########
@@ -369,4 +378,28 @@ object HoodieCatalog {
(identityCols, bucketSpec)
}
+
+ def buildPartitionTransforms(spark: SparkSession,
+ basePath: String): Array[Transform] = {
+ val metaClient = HoodieTableMetaClient.builder()
+ .setConf(spark.sessionState.newHadoopConf())
+ .setBasePath(basePath)
+ .build()
+ val metadataConfig = HoodieMetadataConfig.newBuilder().enable(true).build()
+ val metadataFileSystemView = FileSystemViewManager.createInMemoryFileSystemView(
+ new HoodieSparkEngineContext(spark.sparkContext), metaClient, metadataConfig)
Review Comment:
For tables without MDT this can fail?
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala:
##########
@@ -54,7 +59,8 @@ class HoodieCatalog extends DelegatingCatalogExtension
override def stageCreate(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: util.Map[String, String]): StagedTable = {
if (sparkAdapter.isHoodieTable(properties)) {
val locUriAndTableType = deduceTableLocationURIAndTableType(ident, properties)
- HoodieStagedTable(ident, locUriAndTableType, this, schema, partitions,
+ val partitionTransforms = if (partitions.isEmpty) buildPartitionTransforms(spark, locUriAndTableType._1.getPath) else partitions
Review Comment:
When the table is non-partitioned, is `partitions` still empty, and can we add a short circuit to skip listing the table partition if non-partitioned?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1842410285
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
}, {
"hash" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334",
"triggerID" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 46a4c3344c79fd9a61db78620e8c40e7d98bcd36 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062)
* 5d89786780e149be16784a44c7b4d319a2e756b6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10153:
URL: https://github.com/apache/hudi/pull/10153#discussion_r1419865453
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala:
##########
@@ -369,4 +378,28 @@ object HoodieCatalog {
(identityCols, bucketSpec)
}
+
+ def buildPartitionTransforms(spark: SparkSession,
+ basePath: String): Array[Transform] = {
+ val metaClient = HoodieTableMetaClient.builder()
+ .setConf(spark.sessionState.newHadoopConf())
+ .setBasePath(basePath)
+ .build()
+ val metadataConfig = HoodieMetadataConfig.newBuilder().enable(true).build()
+ val metadataFileSystemView = FileSystemViewManager.createInMemoryFileSystemView(
+ new HoodieSparkEngineContext(spark.sparkContext), metaClient, metadataConfig)
Review Comment:
yeah the assumption is that MDT is enabled for all features that use indexes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #10153:
URL: https://github.com/apache/hudi/pull/10153#discussion_r1423224672
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala:
##########
@@ -140,7 +157,11 @@ class HoodieCatalog extends DelegatingCatalogExtension
properties: util.Map[String, String]): Table = {
if (sparkAdapter.isHoodieTable(properties)) {
val locUriAndTableType = deduceTableLocationURIAndTableType(ident, properties)
- createHoodieTable(ident, schema, locUriAndTableType, partitions, properties,
+ val partitionTransforms =
+ if (partitions.isEmpty && isTablePartitioned(getExistingTableIfExists(ident.asTableIdentifier)))
+ buildPartitionTransforms(spark, locUriAndTableType._1.getPath)
+ else partitions
Review Comment:
nit: this common logic can be extracted to a util method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1843319194
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
}, {
"hash" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334",
"triggerID" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"triggerType" : "PUSH"
}, {
"hash" : "f631760188373ba756495f50d85421e4429b70c9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21339",
"triggerID" : "f631760188373ba756495f50d85421e4429b70c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f631760188373ba756495f50d85421e4429b70c9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21339)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1849356018
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "76e1d519c42565b99b35837c3c767b600cdee311",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21429",
"triggerID" : "76e1d519c42565b99b35837c3c767b600cdee311",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 76e1d519c42565b99b35837c3c767b600cdee311 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21429)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1848865167
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
}, {
"hash" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334",
"triggerID" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"triggerType" : "PUSH"
}, {
"hash" : "f631760188373ba756495f50d85421e4429b70c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21339",
"triggerID" : "f631760188373ba756495f50d85421e4429b70c9",
"triggerType" : "PUSH"
}, {
"hash" : "df20980253cf67ca2cb871e43a30142534854c5a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21405",
"triggerID" : "df20980253cf67ca2cb871e43a30142534854c5a",
"triggerType" : "PUSH"
}, {
"hash" : "4054985bed685d995f568217400a7f66bf591a72",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21428",
"triggerID" : "4054985bed685d995f568217400a7f66bf591a72",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df20980253cf67ca2cb871e43a30142534854c5a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21405)
* 4054985bed685d995f568217400a7f66bf591a72 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21428)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10153:
URL: https://github.com/apache/hudi/pull/10153#discussion_r1423308916
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala:
##########
@@ -140,7 +157,11 @@ class HoodieCatalog extends DelegatingCatalogExtension
properties: util.Map[String, String]): Table = {
if (sparkAdapter.isHoodieTable(properties)) {
val locUriAndTableType = deduceTableLocationURIAndTableType(ident, properties)
- createHoodieTable(ident, schema, locUriAndTableType, partitions, properties,
+ val partitionTransforms =
+ if (partitions.isEmpty && isTablePartitioned(getExistingTableIfExists(ident.asTableIdentifier)))
+ buildPartitionTransforms(spark, locUriAndTableType._1.getPath)
+ else partitions
Review Comment:
Sure, I'll refactor it in a subsequent PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1842768129
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
}, {
"hash" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334",
"triggerID" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d89786780e149be16784a44c7b4d319a2e756b6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1821029238
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 46a4c3344c79fd9a61db78620e8c40e7d98bcd36 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10153:
URL: https://github.com/apache/hudi/pull/10153#discussion_r1400887742
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -140,6 +146,68 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Functional Index With Hive Sync Non Partitioned Table") {
+ if (HoodieSparkUtils.gteqSpark3_2) {
+ withTempDir { tmp =>
+ Seq("mor").foreach { tableType =>
+ val databaseName = "default"
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = '$tableType',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 1001)")
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 1002)")
+
+ val createIndexSql = s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')"
+ spark.sql(createIndexSql)
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(basePath)
+ .setConf(spark.sessionState.newHadoopConf())
+ .build()
+ assertTrue(metaClient.getFunctionalIndexMetadata.isPresent)
+ val functionalIndexMetadata = metaClient.getFunctionalIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
+
+ // sync to hive without partition metadata
+ val hiveSyncProps = new TypedProperties()
+ hiveSyncProps.setProperty(HIVE_URL.key, "jdbc:hive2://localhost:10000")
+ hiveSyncProps.setProperty(HIVE_USER.key, "")
+ hiveSyncProps.setProperty(HIVE_PASS.key, "")
+ hiveSyncProps.setProperty(META_SYNC_DATABASE_NAME.key, databaseName)
+ hiveSyncProps.setProperty(META_SYNC_TABLE_NAME.key, tableName)
+ hiveSyncProps.setProperty(META_SYNC_BASE_PATH.key, basePath)
+ hiveSyncProps.setProperty(HIVE_USE_PRE_APACHE_INPUT_FORMAT.key, "false")
+ // hiveSyncProps.setProperty(META_SYNC_PARTITION_FIELDS.key, "datestr")
+ hiveSyncProps.setProperty(META_SYNC_PARTITION_EXTRACTOR_CLASS.key, classOf[SlashEncodedDayPartitionValueExtractor].getName)
+ hiveSyncProps.setProperty(HIVE_BATCH_SYNC_PARTITION_NUM.key, "3")
+ HiveTestUtil.setUp(Option.of(hiveSyncProps))
Review Comment:
Need to fix this test. The setup call throws IllegalArgumentException. I've tested manually though.
##########
hudi-spark-datasource/hudi-spark/pom.xml:
##########
@@ -400,6 +400,14 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-hive-sync</artifactId>
Review Comment:
This is needed to setup hive for meta sync in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1821106645
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 46a4c3344c79fd9a61db78620e8c40e7d98bcd36 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1842388241
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
}, {
"hash" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 46a4c3344c79fd9a61db78620e8c40e7d98bcd36 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062)
* 5d89786780e149be16784a44c7b4d319a2e756b6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1842860997
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
}, {
"hash" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334",
"triggerID" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"triggerType" : "PUSH"
}, {
"hash" : "f631760188373ba756495f50d85421e4429b70c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f631760188373ba756495f50d85421e4429b70c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d89786780e149be16784a44c7b4d319a2e756b6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334)
* f631760188373ba756495f50d85421e4429b70c9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10153:
URL: https://github.com/apache/hudi/pull/10153#discussion_r1420366688
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -140,6 +147,75 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Functional Index With Hive Sync Non Partitioned Table") {
+ // There is a big difference between Java class loader architecture of versions 1.8 and 17.
+ // Hive 2.3.7 is compiled with Java 1.8, and the class loader used there throws error when Hive APIs are run on Java 17.
+ // So we special case this test only for Java 8.
+ if (HoodieSparkUtils.gteqSpark3_2 && HoodieTestUtils.getJavaVersion == 8) {
+ withTempDir { tmp =>
+ Seq("mor").foreach { tableType =>
+ val databaseName = "default"
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = '$tableType',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ // ts=1000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-01-01'
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ // ts=100000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-01-02'
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 100000)")
+ // ts=10000000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-04-26'
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 10000000)")
+
+ val createIndexSql = s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')"
+ spark.sql(createIndexSql)
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(basePath)
+ .setConf(spark.sessionState.newHadoopConf())
+ .build()
+ assertTrue(metaClient.getFunctionalIndexMetadata.isPresent)
+ val functionalIndexMetadata = metaClient.getFunctionalIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
+
+ // sync to hive without partition metadata
+ val hiveSyncProps = new TypedProperties()
Review Comment:
Added a new config as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10153:
URL: https://github.com/apache/hudi/pull/10153#discussion_r1420367409
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala:
##########
@@ -54,7 +59,8 @@ class HoodieCatalog extends DelegatingCatalogExtension
override def stageCreate(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: util.Map[String, String]): StagedTable = {
if (sparkAdapter.isHoodieTable(properties)) {
val locUriAndTableType = deduceTableLocationURIAndTableType(ident, properties)
- HoodieStagedTable(ident, locUriAndTableType, this, schema, partitions,
+ val partitionTransforms = if (partitions.isEmpty) buildPartitionTransforms(spark, locUriAndTableType._1.getPath) else partitions
Review Comment:
`partitions` was indeed empty, i've added a check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1848897427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
}, {
"hash" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334",
"triggerID" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"triggerType" : "PUSH"
}, {
"hash" : "f631760188373ba756495f50d85421e4429b70c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21339",
"triggerID" : "f631760188373ba756495f50d85421e4429b70c9",
"triggerType" : "PUSH"
}, {
"hash" : "df20980253cf67ca2cb871e43a30142534854c5a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21405",
"triggerID" : "df20980253cf67ca2cb871e43a30142534854c5a",
"triggerType" : "PUSH"
}, {
"hash" : "4054985bed685d995f568217400a7f66bf591a72",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21428",
"triggerID" : "4054985bed685d995f568217400a7f66bf591a72",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4054985bed685d995f568217400a7f66bf591a72 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21428)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10153:
URL: https://github.com/apache/hudi/pull/10153#issuecomment-1848982595
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21062",
"triggerID" : "46a4c3344c79fd9a61db78620e8c40e7d98bcd36",
"triggerType" : "PUSH"
}, {
"hash" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21334",
"triggerID" : "5d89786780e149be16784a44c7b4d319a2e756b6",
"triggerType" : "PUSH"
}, {
"hash" : "f631760188373ba756495f50d85421e4429b70c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21339",
"triggerID" : "f631760188373ba756495f50d85421e4429b70c9",
"triggerType" : "PUSH"
}, {
"hash" : "df20980253cf67ca2cb871e43a30142534854c5a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21405",
"triggerID" : "df20980253cf67ca2cb871e43a30142534854c5a",
"triggerType" : "PUSH"
}, {
"hash" : "4054985bed685d995f568217400a7f66bf591a72",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21428",
"triggerID" : "4054985bed685d995f568217400a7f66bf591a72",
"triggerType" : "PUSH"
}, {
"hash" : "76e1d519c42565b99b35837c3c767b600cdee311",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21429",
"triggerID" : "76e1d519c42565b99b35837c3c767b600cdee311",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4054985bed685d995f568217400a7f66bf591a72 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21428)
* 76e1d519c42565b99b35837c3c767b600cdee311 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21429)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #10153:
URL: https://github.com/apache/hudi/pull/10153#discussion_r1423219171
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -140,6 +147,75 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Functional Index With Hive Sync Non Partitioned Table") {
+ // There is a big difference between Java class loader architecture of versions 1.8 and 17.
+ // Hive 2.3.7 is compiled with Java 1.8, and the class loader used there throws error when Hive APIs are run on Java 17.
+ // So we special case this test only for Java 8.
+ if (HoodieSparkUtils.gteqSpark3_2 && HoodieTestUtils.getJavaVersion == 8) {
+ withTempDir { tmp =>
+ Seq("mor").foreach { tableType =>
+ val databaseName = "default"
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = '$tableType',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ // ts=1000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-01-01'
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ // ts=100000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-01-02'
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 100000)")
+ // ts=10000000 and from_unixtime(ts, 'yyyy-MM-dd') = '1970-04-26'
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 10000000)")
+
+ val createIndexSql = s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')"
+ spark.sql(createIndexSql)
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(basePath)
+ .setConf(spark.sessionState.newHadoopConf())
+ .build()
+ assertTrue(metaClient.getFunctionalIndexMetadata.isPresent)
+ val functionalIndexMetadata = metaClient.getFunctionalIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
+
+ // sync to hive without partition metadata
+ val hiveSyncProps = new TypedProperties()
Review Comment:
I think we can have this config for now and decouple the storage partitions/folders from partition metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org