You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2024/07/05 15:27:15 UTC
[PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
codope opened a new pull request, #11579:
URL: https://github.com/apache/hudi/pull/11579
### Change Logs
Just like column stats index, we can create partition stats index for all column if no columns configured by the user.
### Impact
Users don't necessarily have to configure columns to aggregate stats at partition level.
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2306446103
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
}, {
"hash" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152",
"triggerID" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"triggerType" : "PUSH"
}, {
"hash" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fbe86757380fb79162e440ba3260cd2c99c45ba Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152)
* 4da93fa946533fc98f5b4cbff8d946ca65515272 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #11579:
URL: https://github.com/apache/hudi/pull/11579
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2211130646
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25c6ae81e406de37846d79655e8949e49eef6806 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2211139635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25c6ae81e406de37846d79655e8949e49eef6806 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11579:
URL: https://github.com/apache/hudi/pull/11579#discussion_r1745777257
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestPartitionStatsIndexWithSql.scala:
##########
@@ -261,43 +262,89 @@ class TestPartitionStatsIndexWithSql extends HoodieSparkSqlTestBase {
|""".stripMargin
)
+ writeAndValidatePartitionStats(tableName, tablePath)
+ }
+ }
+ }
+
+ test(s"Test partition stats index without configuring columns to index") {
+ Seq("cow", "mor").foreach { tableType =>
+ withTempDir { tmp =>
+ val tableName = generateTableName
+ val tablePath = s"${tmp.getCanonicalPath}/$tableName"
+ // create table and enable partition stats without configuring columns to index
spark.sql(
s"""
- | insert into $tableName
- | values (1, 'a1', 1000, 10), (2, 'a2', 2000, 20), (3, 'a3', 3000, 30), (4, 'a4', 2000, 10), (5, 'a5', 3000, 20), (6, 'a6', 4000, 30)
- | """.stripMargin
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price int,
+ | ts long
+ |) using hudi
+ |partitioned by (ts)
+ |tblproperties (
+ | type = '$tableType',
+ | primaryKey = 'id',
+ | preCombineField = 'price',
+ | hoodie.metadata.index.partition.stats.enable = 'true'
+ |)
+ |location '$tablePath'
+ |""".stripMargin
)
- // Validate partition_stats index exists
- val metaClient = HoodieTableMetaClient.builder()
- .setBasePath(tablePath)
- .setConf(HoodieTestUtils.getDefaultStorageConf)
- .build()
- assertResult(tableName)(metaClient.getTableConfig.getTableName)
- assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains(PARTITION_STATS.getPartitionPath))
-
- spark.sql("set hoodie.metadata.enable=true")
- spark.sql("set hoodie.enable.data.skipping=true")
- spark.sql("set hoodie.fileIndex.dataSkippingFailureMode=strict")
- checkAnswer(s"select id, name, price, ts from $tableName where price>3000")(
- Seq(6, "a6", 4000, 30)
+ writeAndValidatePartitionStats(tableName, tablePath)
+ // validate partition stats index for id column
+ checkAnswer(s"select key, ColumnStatsMetadata.minValue.member1.value, ColumnStatsMetadata.maxValue.member1.value from hudi_metadata('$tableName') where type=3 and ColumnStatsMetadata.columnName='id'")(
+ Seq(getPartitionStatsIndexKey("ts=10", "id"), 1, 4),
+ Seq(getPartitionStatsIndexKey("ts=20", "id"), 2, 5),
+ Seq(getPartitionStatsIndexKey("ts=30", "id"), 3, 6)
)
-
- // Test price update, assert latest value and ensure file pruning
- spark.sql(s"update $tableName set price = price + 1 where id = 6")
- checkAnswer(s"select id, name, price, ts from $tableName where price>3000")(
- Seq(6, "a6", 4001, 30)
+ // validate partition stats index for name column
+ checkAnswer(s"select key, ColumnStatsMetadata.minValue.member6.value, ColumnStatsMetadata.maxValue.member6.value from hudi_metadata('$tableName') where type=3 and ColumnStatsMetadata.columnName='name'")(
+ Seq(getPartitionStatsIndexKey("ts=10", "name"), "a1", "a4"),
+ Seq(getPartitionStatsIndexKey("ts=20", "name"), "a2", "a5"),
+ Seq(getPartitionStatsIndexKey("ts=30", "name"), "a3", "a6")
)
-
- verifyFilePruning(
- Map.apply(DataSourceReadOptions.ENABLE_DATA_SKIPPING.key -> "true", HoodieMetadataConfig.ENABLE.key -> "true"),
- GreaterThan(AttributeReference("price", IntegerType)(), Literal(3000)),
- HoodieTableMetaClient.reload(metaClient),
- isDataSkippingExpected = true)
}
}
}
+ private def writeAndValidatePartitionStats(tableName: String, tablePath: String): Unit = {
+ spark.sql(
+ s"""
+ | insert into $tableName
+ | values (1, 'a1', 1000, 10), (2, 'a2', 2000, 20), (3, 'a3', 3000, 30), (4, 'a4', 2000, 10), (5, 'a5', 3000, 20), (6, 'a6', 4000, 30)
+ | """.stripMargin
+ )
+
+ // Validate partition_stats index exists
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(tablePath)
+ .setConf(HoodieTestUtils.getDefaultStorageConf)
+ .build()
+ assertResult(tableName)(metaClient.getTableConfig.getTableName)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains(PARTITION_STATS.getPartitionPath))
+
+ spark.sql("set hoodie.metadata.enable=true")
+ spark.sql("set hoodie.enable.data.skipping=true")
+ spark.sql("set hoodie.fileIndex.dataSkippingFailureMode=strict")
+ checkAnswer(s"select id, name, price, ts from $tableName where price>3000")(
+ Seq(6, "a6", 4000, 30)
+ )
+
+ // Test price update, assert latest value and ensure file pruning
+ spark.sql(s"update $tableName set price = price + 1 where id = 6")
+ checkAnswer(s"select id, name, price, ts from $tableName where price>3000")(
+ Seq(6, "a6", 4001, 30)
+ )
+
+ verifyFilePruning(
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11579:
URL: https://github.com/apache/hudi/pull/11579#discussion_r1746349635
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -332,7 +332,7 @@ public final class HoodieMetadataConfig extends HoodieConfig {
public static final ConfigProperty<Boolean> ENABLE_METADATA_INDEX_PARTITION_STATS = ConfigProperty
.key(METADATA_PREFIX + ".index.partition.stats.enable")
- .defaultValue(true)
+ .defaultValue(false)
Review Comment:
Should we have a follow-up to enable this by default?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2331332829
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
}, {
"hash" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152",
"triggerID" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"triggerType" : "PUSH"
}, {
"hash" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=154",
"triggerID" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"triggerType" : "PUSH"
}, {
"hash" : "a053027fb983874091cf4f029954226b94a0491f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=243",
"triggerID" : "a053027fb983874091cf4f029954226b94a0491f",
"triggerType" : "PUSH"
}, {
"hash" : "809a8651333a0d2dda16c19ebf87ac4dfeecfcfc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=271",
"triggerID" : "809a8651333a0d2dda16c19ebf87ac4dfeecfcfc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 809a8651333a0d2dda16c19ebf87ac4dfeecfcfc Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=271)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2306380821
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
}, {
"hash" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152",
"triggerID" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49f91bc146b2218410db8126b604934f2e79e4fe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091)
* 5fbe86757380fb79162e440ba3260cd2c99c45ba Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11579:
URL: https://github.com/apache/hudi/pull/11579#discussion_r1666935047
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -332,7 +332,7 @@ public final class HoodieMetadataConfig extends HoodieConfig {
public static final ConfigProperty<Boolean> ENABLE_METADATA_INDEX_PARTITION_STATS = ConfigProperty
.key(METADATA_PREFIX + ".index.partition.stats.enable")
- .defaultValue(true)
+ .defaultValue(false)
Review Comment:
With partition stats enabled by default and no columns specified, the partition stats would be calculated for mdt update for every commit for all columns, which can be costly. We will track enabling partition stats by default separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2270940631
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25c6ae81e406de37846d79655e8949e49eef6806 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730)
* eca39ac8726c593c26432dbbb580562f23832e5c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2276338798
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eca39ac8726c593c26432dbbb580562f23832e5c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073)
* 49f91bc146b2218410db8126b604934f2e79e4fe UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2211095975
I really feel we should cut down on the no of cols we generate stats out of the box. I have encountered OSS users give col stats a try and since it takes lot of time to populate col stats if their schema is wide, they give up on col stats. They don't know why its slow. just that the exp is not good, so they disable col stats and move on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2331655094
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
}, {
"hash" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152",
"triggerID" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"triggerType" : "PUSH"
}, {
"hash" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=154",
"triggerID" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"triggerType" : "PUSH"
}, {
"hash" : "a053027fb983874091cf4f029954226b94a0491f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=243",
"triggerID" : "a053027fb983874091cf4f029954226b94a0491f",
"triggerType" : "PUSH"
}, {
"hash" : "809a8651333a0d2dda16c19ebf87ac4dfeecfcfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=271",
"triggerID" : "809a8651333a0d2dda16c19ebf87ac4dfeecfcfc",
"triggerType" : "PUSH"
}, {
"hash" : "3854e3a27aa07675804e1e89b9bb5dd82c2c6485",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=273",
"triggerID" : "3854e3a27aa07675804e1e89b9bb5dd82c2c6485",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3854e3a27aa07675804e1e89b9bb5dd82c2c6485 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=273)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11579:
URL: https://github.com/apache/hudi/pull/11579#discussion_r1737442509
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestPartitionStatsIndexWithSql.scala:
##########
@@ -261,43 +262,89 @@ class TestPartitionStatsIndexWithSql extends HoodieSparkSqlTestBase {
|""".stripMargin
)
+ writeAndValidatePartitionStats(tableName, tablePath)
+ }
+ }
+ }
+
+ test(s"Test partition stats index without configuring columns to index") {
+ Seq("cow", "mor").foreach { tableType =>
+ withTempDir { tmp =>
+ val tableName = generateTableName
+ val tablePath = s"${tmp.getCanonicalPath}/$tableName"
+ // create table and enable partition stats without configuring columns to index
spark.sql(
s"""
- | insert into $tableName
- | values (1, 'a1', 1000, 10), (2, 'a2', 2000, 20), (3, 'a3', 3000, 30), (4, 'a4', 2000, 10), (5, 'a5', 3000, 20), (6, 'a6', 4000, 30)
- | """.stripMargin
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price int,
+ | ts long
+ |) using hudi
+ |partitioned by (ts)
+ |tblproperties (
+ | type = '$tableType',
+ | primaryKey = 'id',
+ | preCombineField = 'price',
+ | hoodie.metadata.index.partition.stats.enable = 'true'
+ |)
+ |location '$tablePath'
+ |""".stripMargin
)
- // Validate partition_stats index exists
- val metaClient = HoodieTableMetaClient.builder()
- .setBasePath(tablePath)
- .setConf(HoodieTestUtils.getDefaultStorageConf)
- .build()
- assertResult(tableName)(metaClient.getTableConfig.getTableName)
- assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains(PARTITION_STATS.getPartitionPath))
-
- spark.sql("set hoodie.metadata.enable=true")
- spark.sql("set hoodie.enable.data.skipping=true")
- spark.sql("set hoodie.fileIndex.dataSkippingFailureMode=strict")
- checkAnswer(s"select id, name, price, ts from $tableName where price>3000")(
- Seq(6, "a6", 4000, 30)
+ writeAndValidatePartitionStats(tableName, tablePath)
+ // validate partition stats index for id column
+ checkAnswer(s"select key, ColumnStatsMetadata.minValue.member1.value, ColumnStatsMetadata.maxValue.member1.value from hudi_metadata('$tableName') where type=3 and ColumnStatsMetadata.columnName='id'")(
+ Seq(getPartitionStatsIndexKey("ts=10", "id"), 1, 4),
+ Seq(getPartitionStatsIndexKey("ts=20", "id"), 2, 5),
+ Seq(getPartitionStatsIndexKey("ts=30", "id"), 3, 6)
)
-
- // Test price update, assert latest value and ensure file pruning
- spark.sql(s"update $tableName set price = price + 1 where id = 6")
- checkAnswer(s"select id, name, price, ts from $tableName where price>3000")(
- Seq(6, "a6", 4001, 30)
+ // validate partition stats index for name column
+ checkAnswer(s"select key, ColumnStatsMetadata.minValue.member6.value, ColumnStatsMetadata.maxValue.member6.value from hudi_metadata('$tableName') where type=3 and ColumnStatsMetadata.columnName='name'")(
+ Seq(getPartitionStatsIndexKey("ts=10", "name"), "a1", "a4"),
+ Seq(getPartitionStatsIndexKey("ts=20", "name"), "a2", "a5"),
+ Seq(getPartitionStatsIndexKey("ts=30", "name"), "a3", "a6")
)
-
- verifyFilePruning(
- Map.apply(DataSourceReadOptions.ENABLE_DATA_SKIPPING.key -> "true", HoodieMetadataConfig.ENABLE.key -> "true"),
- GreaterThan(AttributeReference("price", IntegerType)(), Literal(3000)),
- HoodieTableMetaClient.reload(metaClient),
- isDataSkippingExpected = true)
}
}
}
+ private def writeAndValidatePartitionStats(tableName: String, tablePath: String): Unit = {
+ spark.sql(
+ s"""
+ | insert into $tableName
+ | values (1, 'a1', 1000, 10), (2, 'a2', 2000, 20), (3, 'a3', 3000, 30), (4, 'a4', 2000, 10), (5, 'a5', 3000, 20), (6, 'a6', 4000, 30)
+ | """.stripMargin
+ )
+
+ // Validate partition_stats index exists
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(tablePath)
+ .setConf(HoodieTestUtils.getDefaultStorageConf)
+ .build()
+ assertResult(tableName)(metaClient.getTableConfig.getTableName)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains(PARTITION_STATS.getPartitionPath))
+
+ spark.sql("set hoodie.metadata.enable=true")
+ spark.sql("set hoodie.enable.data.skipping=true")
+ spark.sql("set hoodie.fileIndex.dataSkippingFailureMode=strict")
+ checkAnswer(s"select id, name, price, ts from $tableName where price>3000")(
+ Seq(6, "a6", 4000, 30)
+ )
+
+ // Test price update, assert latest value and ensure file pruning
+ spark.sql(s"update $tableName set price = price + 1 where id = 6")
+ checkAnswer(s"select id, name, price, ts from $tableName where price>3000")(
+ Seq(6, "a6", 4001, 30)
+ )
+
+ verifyFilePruning(
Review Comment:
Also call this with data skipping disabled to make sure partition stats is not used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11579:
URL: https://github.com/apache/hudi/pull/11579#discussion_r1745780330
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -332,7 +332,7 @@ public final class HoodieMetadataConfig extends HoodieConfig {
public static final ConfigProperty<Boolean> ENABLE_METADATA_INDEX_PARTITION_STATS = ConfigProperty
.key(METADATA_PREFIX + ".index.partition.stats.enable")
- .defaultValue(true)
+ .defaultValue(false)
Review Comment:
one can still enable partition stats index keeping col stats disabled. But the config to specify columns for which stats need to be aggregated is common - `hoodie.metadata.index.column.stats.column.list`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2276422725
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49f91bc146b2218410db8126b604934f2e79e4fe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2270956714
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eca39ac8726c593c26432dbbb580562f23832e5c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2331204563
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
}, {
"hash" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152",
"triggerID" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"triggerType" : "PUSH"
}, {
"hash" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=154",
"triggerID" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"triggerType" : "PUSH"
}, {
"hash" : "a053027fb983874091cf4f029954226b94a0491f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=243",
"triggerID" : "a053027fb983874091cf4f029954226b94a0491f",
"triggerType" : "PUSH"
}, {
"hash" : "809a8651333a0d2dda16c19ebf87ac4dfeecfcfc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "809a8651333a0d2dda16c19ebf87ac4dfeecfcfc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a053027fb983874091cf4f029954226b94a0491f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=243)
* 809a8651333a0d2dda16c19ebf87ac4dfeecfcfc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11579:
URL: https://github.com/apache/hudi/pull/11579#discussion_r1745776925
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1298,7 +1300,8 @@ private static Option<Schema> tryResolveSchemaForTable(HoodieTableMetaClient dat
TableSchemaResolver schemaResolver = new TableSchemaResolver(dataTableMetaClient);
return Option.of(schemaResolver.getTableAvroSchema());
} catch (Exception e) {
- throw new HoodieException("Failed to get latest columns for " + dataTableMetaClient.getBasePath(), e);
+ LOG.warn("Failed to get latest columns for {}", dataTableMetaClient.getBasePath());
+ return Option.empty();
Review Comment:
Makes sense. I don't recall why i changed to a warn log. Probably, I was debugging something. I have reverted now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #11579:
URL: https://github.com/apache/hudi/pull/11579#discussion_r1666960268
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestPartitionStatsIndexWithSql.scala:
##########
@@ -261,41 +261,75 @@ class TestPartitionStatsIndexWithSql extends HoodieSparkSqlTestBase {
|""".stripMargin
)
+ writeAndValidatePartitionStats(tableName, tablePath)
+ }
+ }
+ }
+
+ test(s"Test partition stats index without configuring columns to index") {
+ Seq("cow", "mor").foreach { tableType =>
+ withTempDir { tmp =>
+ val tableName = generateTableName
+ val tablePath = s"${tmp.getCanonicalPath}/$tableName"
+ // create table and enable partition stats without configuring columns to index
spark.sql(
s"""
- | insert into $tableName
- | values (1, 'a1', 1000, 10), (2, 'a2', 2000, 20), (3, 'a3', 3000, 30), (4, 'a4', 2000, 10), (5, 'a5', 3000, 20), (6, 'a6', 4000, 30)
- | """.stripMargin
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price int,
+ | ts long
+ |) using hudi
+ |partitioned by (ts)
+ |tblproperties (
+ | type = '$tableType',
+ | primaryKey = 'id',
+ | preCombineField = 'price',
+ | hoodie.metadata.index.partition.stats.enable = 'true'
+ |)
+ |location '$tablePath'
+ |""".stripMargin
)
- // Validate partition_stats index exists
- val metaClient = HoodieTableMetaClient.builder()
- .setBasePath(tablePath)
- .setConf(HoodieTestUtils.getDefaultStorageConf)
- .build()
- assertResult(tableName)(metaClient.getTableConfig.getTableName)
- assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains(PARTITION_STATS.getPartitionPath))
+ writeAndValidatePartitionStats(tableName, tablePath)
+ }
+ }
+ }
- spark.sql("set hoodie.metadata.enable=true")
- spark.sql("set hoodie.enable.data.skipping=true")
- spark.sql("set hoodie.fileIndex.dataSkippingFailureMode=strict")
- checkAnswer(s"select id, name, price, ts from $tableName where price>3000")(
- Seq(6, "a6", 4000, 30)
- )
+ private def writeAndValidatePartitionStats(tableName: String, tablePath: String): Unit = {
+ spark.sql(
+ s"""
+ | insert into $tableName
+ | values (1, 'a1', 1000, 10), (2, 'a2', 2000, 20), (3, 'a3', 3000, 30), (4, 'a4', 2000, 10), (5, 'a5', 3000, 20), (6, 'a6', 4000, 30)
+ | """.stripMargin
+ )
- // Test price update, assert latest value and ensure file pruning
- spark.sql(s"update $tableName set price = price + 1 where id = 6")
- checkAnswer(s"select id, name, price, ts from $tableName where price>3000")(
- Seq(6, "a6", 4001, 30)
- )
+ // Validate partition_stats index exists
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(tablePath)
+ .setConf(HoodieTestUtils.getDefaultStorageConf)
+ .build()
+ assertResult(tableName)(metaClient.getTableConfig.getTableName)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains(PARTITION_STATS.getPartitionPath))
- verifyFilePruning(
- Map.apply(DataSourceReadOptions.ENABLE_DATA_SKIPPING.key -> "true", HoodieMetadataConfig.ENABLE.key -> "true"),
- GreaterThan(AttributeReference("price", IntegerType)(), Literal(3000)),
- HoodieTableMetaClient.reload(metaClient),
- isDataSkippingExpected = true)
- }
- }
+ spark.sql("set hoodie.metadata.enable=true")
+ spark.sql("set hoodie.enable.data.skipping=true")
+ spark.sql("set hoodie.fileIndex.dataSkippingFailureMode=strict")
+ checkAnswer(s"select id, name, price, ts from $tableName where price>3000")(
+ Seq(6, "a6", 4000, 30)
+ )
+
+ // Test price update, assert latest value and ensure file pruning
+ spark.sql(s"update $tableName set price = price + 1 where id = 6")
+ checkAnswer(s"select id, name, price, ts from $tableName where price>3000")(
+ Seq(6, "a6", 4001, 30)
+ )
+
+ verifyFilePruning(
+ Map.apply(DataSourceReadOptions.ENABLE_DATA_SKIPPING.key -> "true", HoodieMetadataConfig.ENABLE.key -> "true"),
+ GreaterThan(AttributeReference("price", IntegerType)(), Literal(3000)),
+ HoodieTableMetaClient.reload(metaClient),
+ isDataSkippingExpected = true)
Review Comment:
a comment on L346. should we also need something like
cur code
```
if (isDataSkippingExpected) {
assertTrue(filteredFilesCount < latestDataFilesCount)
if (isNoScanExpected) {
assertTrue(filteredFilesCount == 0)
}
}
```
proposal:
```
if (isDataSkippingExpected) {
assertTrue(filteredFilesCount < latestDataFilesCount)
if (isNoScanExpected) {
assertTrue(filteredFilesCount == 0)
} else {
assertTrue(filteredFilesCount > 0)
}
}
```
i.e. assertTrue(filteredFilesCount > 0) when isNoScanExpected is false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2306444181
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
}, {
"hash" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152",
"triggerID" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"triggerType" : "PUSH"
}, {
"hash" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49f91bc146b2218410db8126b604934f2e79e4fe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091)
* 5fbe86757380fb79162e440ba3260cd2c99c45ba Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152)
* 4da93fa946533fc98f5b4cbff8d946ca65515272 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11579:
URL: https://github.com/apache/hudi/pull/11579#discussion_r1692356320
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -332,7 +332,7 @@ public final class HoodieMetadataConfig extends HoodieConfig {
public static final ConfigProperty<Boolean> ENABLE_METADATA_INDEX_PARTITION_STATS = ConfigProperty
.key(METADATA_PREFIX + ".index.partition.stats.enable")
- .defaultValue(true)
+ .defaultValue(false)
Review Comment:
Have you tried enabling this config? Do all tests still pass?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestPartitionStatsIndexWithSql.scala:
##########
@@ -261,41 +261,75 @@ class TestPartitionStatsIndexWithSql extends HoodieSparkSqlTestBase {
|""".stripMargin
)
+ writeAndValidatePartitionStats(tableName, tablePath)
+ }
+ }
+ }
+
+ test(s"Test partition stats index without configuring columns to index") {
+ Seq("cow", "mor").foreach { tableType =>
+ withTempDir { tmp =>
+ val tableName = generateTableName
+ val tablePath = s"${tmp.getCanonicalPath}/$tableName"
+ // create table and enable partition stats without configuring columns to index
spark.sql(
s"""
- | insert into $tableName
- | values (1, 'a1', 1000, 10), (2, 'a2', 2000, 20), (3, 'a3', 3000, 30), (4, 'a4', 2000, 10), (5, 'a5', 3000, 20), (6, 'a6', 4000, 30)
- | """.stripMargin
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price int,
+ | ts long
+ |) using hudi
+ |partitioned by (ts)
+ |tblproperties (
+ | type = '$tableType',
+ | primaryKey = 'id',
+ | preCombineField = 'price',
+ | hoodie.metadata.index.partition.stats.enable = 'true'
+ |)
+ |location '$tablePath'
+ |""".stripMargin
)
- // Validate partition_stats index exists
- val metaClient = HoodieTableMetaClient.builder()
- .setBasePath(tablePath)
- .setConf(HoodieTestUtils.getDefaultStorageConf)
- .build()
- assertResult(tableName)(metaClient.getTableConfig.getTableName)
- assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains(PARTITION_STATS.getPartitionPath))
+ writeAndValidatePartitionStats(tableName, tablePath)
+ }
+ }
+ }
- spark.sql("set hoodie.metadata.enable=true")
- spark.sql("set hoodie.enable.data.skipping=true")
- spark.sql("set hoodie.fileIndex.dataSkippingFailureMode=strict")
- checkAnswer(s"select id, name, price, ts from $tableName where price>3000")(
- Seq(6, "a6", 4000, 30)
- )
+ private def writeAndValidatePartitionStats(tableName: String, tablePath: String): Unit = {
+ spark.sql(
+ s"""
+ | insert into $tableName
+ | values (1, 'a1', 1000, 10), (2, 'a2', 2000, 20), (3, 'a3', 3000, 30), (4, 'a4', 2000, 10), (5, 'a5', 3000, 20), (6, 'a6', 4000, 30)
+ | """.stripMargin
+ )
- // Test price update, assert latest value and ensure file pruning
- spark.sql(s"update $tableName set price = price + 1 where id = 6")
- checkAnswer(s"select id, name, price, ts from $tableName where price>3000")(
- Seq(6, "a6", 4001, 30)
- )
+ // Validate partition_stats index exists
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(tablePath)
+ .setConf(HoodieTestUtils.getDefaultStorageConf)
+ .build()
+ assertResult(tableName)(metaClient.getTableConfig.getTableName)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains(PARTITION_STATS.getPartitionPath))
Review Comment:
Could we also directly read the partition_stats index and validate the index content?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2329558406
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
}, {
"hash" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152",
"triggerID" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"triggerType" : "PUSH"
}, {
"hash" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=154",
"triggerID" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"triggerType" : "PUSH"
}, {
"hash" : "a053027fb983874091cf4f029954226b94a0491f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=243",
"triggerID" : "a053027fb983874091cf4f029954226b94a0491f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4da93fa946533fc98f5b4cbff8d946ca65515272 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=154)
* a053027fb983874091cf4f029954226b94a0491f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=243)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2329697650
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
}, {
"hash" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152",
"triggerID" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"triggerType" : "PUSH"
}, {
"hash" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=154",
"triggerID" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"triggerType" : "PUSH"
}, {
"hash" : "a053027fb983874091cf4f029954226b94a0491f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=243",
"triggerID" : "a053027fb983874091cf4f029954226b94a0491f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a053027fb983874091cf4f029954226b94a0491f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=243)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2331206875
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
}, {
"hash" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152",
"triggerID" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"triggerType" : "PUSH"
}, {
"hash" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=154",
"triggerID" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"triggerType" : "PUSH"
}, {
"hash" : "a053027fb983874091cf4f029954226b94a0491f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=243",
"triggerID" : "a053027fb983874091cf4f029954226b94a0491f",
"triggerType" : "PUSH"
}, {
"hash" : "809a8651333a0d2dda16c19ebf87ac4dfeecfcfc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=271",
"triggerID" : "809a8651333a0d2dda16c19ebf87ac4dfeecfcfc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a053027fb983874091cf4f029954226b94a0491f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=243)
* 809a8651333a0d2dda16c19ebf87ac4dfeecfcfc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=271)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #11579:
URL: https://github.com/apache/hudi/pull/11579#discussion_r1666955280
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -332,7 +332,7 @@ public final class HoodieMetadataConfig extends HoodieConfig {
public static final ConfigProperty<Boolean> ENABLE_METADATA_INDEX_PARTITION_STATS = ConfigProperty
.key(METADATA_PREFIX + ".index.partition.stats.enable")
- .defaultValue(true)
+ .defaultValue(false)
Review Comment:
can someone not enable col stats, but enable only partition stats index. should we auto correct the config value for partition stats index in that case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2306466072
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
}, {
"hash" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152",
"triggerID" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"triggerType" : "PUSH"
}, {
"hash" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=154",
"triggerID" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fbe86757380fb79162e440ba3260cd2c99c45ba Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152)
* 4da93fa946533fc98f5b4cbff8d946ca65515272 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=154)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2331353084
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
}, {
"hash" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152",
"triggerID" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"triggerType" : "PUSH"
}, {
"hash" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=154",
"triggerID" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"triggerType" : "PUSH"
}, {
"hash" : "a053027fb983874091cf4f029954226b94a0491f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=243",
"triggerID" : "a053027fb983874091cf4f029954226b94a0491f",
"triggerType" : "PUSH"
}, {
"hash" : "809a8651333a0d2dda16c19ebf87ac4dfeecfcfc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=271",
"triggerID" : "809a8651333a0d2dda16c19ebf87ac4dfeecfcfc",
"triggerType" : "PUSH"
}, {
"hash" : "3854e3a27aa07675804e1e89b9bb5dd82c2c6485",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=273",
"triggerID" : "3854e3a27aa07675804e1e89b9bb5dd82c2c6485",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 809a8651333a0d2dda16c19ebf87ac4dfeecfcfc Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=271)
* 3854e3a27aa07675804e1e89b9bb5dd82c2c6485 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=273)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11579:
URL: https://github.com/apache/hudi/pull/11579#discussion_r1745776198
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -332,7 +332,7 @@ public final class HoodieMetadataConfig extends HoodieConfig {
public static final ConfigProperty<Boolean> ENABLE_METADATA_INDEX_PARTITION_STATS = ConfigProperty
.key(METADATA_PREFIX + ".index.partition.stats.enable")
- .defaultValue(true)
+ .defaultValue(false)
Review Comment:
yes, i ran all tests with this config true and fixed some issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11579:
URL: https://github.com/apache/hudi/pull/11579#discussion_r1746478329
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -332,7 +332,7 @@ public final class HoodieMetadataConfig extends HoodieConfig {
public static final ConfigProperty<Boolean> ENABLE_METADATA_INDEX_PARTITION_STATS = ConfigProperty
.key(METADATA_PREFIX + ".index.partition.stats.enable")
- .defaultValue(true)
+ .defaultValue(false)
Review Comment:
yes after fixing the colstats related issue - HUDI-8178
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11579:
URL: https://github.com/apache/hudi/pull/11579#discussion_r1737376626
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -389,6 +385,9 @@ public static MetadataPartitionType[] getValidValues() {
* Returns the list of metadata partition types enabled based on the metadata config and table config.
*/
public static List<MetadataPartitionType> getEnabledPartitions(TypedProperties writeConfig, HoodieTableMetaClient metaClient) {
+ if (!writeConfig.getBoolean(ENABLE.key(), ENABLE.defaultValue())) {
Review Comment:
Use `ConfigUtils.getBooleanWithAltKeys`
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1298,7 +1300,8 @@ private static Option<Schema> tryResolveSchemaForTable(HoodieTableMetaClient dat
TableSchemaResolver schemaResolver = new TableSchemaResolver(dataTableMetaClient);
return Option.of(schemaResolver.getTableAvroSchema());
} catch (Exception e) {
- throw new HoodieException("Failed to get latest columns for " + dataTableMetaClient.getBasePath(), e);
+ LOG.warn("Failed to get latest columns for {}", dataTableMetaClient.getBasePath());
+ return Option.empty();
Review Comment:
Should this still throw the exception? If there are completed commits, the table schema is present; any unexpected error should fail the job instead of returning an empty schema and empty list of columns, potentially causing the index to skip those columns unintentionally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2251769299
> I really feel we should cut down on the no of cols we generate stats out of the box. I have encountered OSS users give col stats a try and since it takes lot of time to populate col stats if their schema is wide, they give up on col stats. They don't know why its slow. just that the exp is not good, so they disable col stats and move on.
This PR makes the behavior of the col_stats and partition_stats index consistent in terms of what columns to generate the index. Before this PR, if no value is specified in `hoodie.metadata.index.column.stats.column.list`, column stats of all columns are generated, while the partition stats is not generated at all.
We can cut down the number of columns for generating columns stats by default. That should be tackled in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2211188134
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25c6ae81e406de37846d79655e8949e49eef6806 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2306586611
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
}, {
"hash" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152",
"triggerID" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"triggerType" : "PUSH"
}, {
"hash" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=154",
"triggerID" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4da93fa946533fc98f5b4cbff8d946ca65515272 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=154)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2306378978
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
}, {
"hash" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49f91bc146b2218410db8126b604934f2e79e4fe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091)
* 5fbe86757380fb79162e440ba3260cd2c99c45ba UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2331350456
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
}, {
"hash" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152",
"triggerID" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"triggerType" : "PUSH"
}, {
"hash" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=154",
"triggerID" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"triggerType" : "PUSH"
}, {
"hash" : "a053027fb983874091cf4f029954226b94a0491f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=243",
"triggerID" : "a053027fb983874091cf4f029954226b94a0491f",
"triggerType" : "PUSH"
}, {
"hash" : "809a8651333a0d2dda16c19ebf87ac4dfeecfcfc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=271",
"triggerID" : "809a8651333a0d2dda16c19ebf87ac4dfeecfcfc",
"triggerType" : "PUSH"
}, {
"hash" : "3854e3a27aa07675804e1e89b9bb5dd82c2c6485",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3854e3a27aa07675804e1e89b9bb5dd82c2c6485",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 809a8651333a0d2dda16c19ebf87ac4dfeecfcfc Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=271)
* 3854e3a27aa07675804e1e89b9bb5dd82c2c6485 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2329556365
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
}, {
"hash" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=152",
"triggerID" : "5fbe86757380fb79162e440ba3260cd2c99c45ba",
"triggerType" : "PUSH"
}, {
"hash" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=154",
"triggerID" : "4da93fa946533fc98f5b4cbff8d946ca65515272",
"triggerType" : "PUSH"
}, {
"hash" : "a053027fb983874091cf4f029954226b94a0491f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a053027fb983874091cf4f029954226b94a0491f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4da93fa946533fc98f5b4cbff8d946ca65515272 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=154)
* a053027fb983874091cf4f029954226b94a0491f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7958] Create partition stats index for all columns when no cols specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11579:
URL: https://github.com/apache/hudi/pull/11579#issuecomment-2276352659
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25c6ae81e406de37846d79655e8949e49eef6806",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24730",
"triggerID" : "25c6ae81e406de37846d79655e8949e49eef6806",
"triggerType" : "PUSH"
}, {
"hash" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073",
"triggerID" : "eca39ac8726c593c26432dbbb580562f23832e5c",
"triggerType" : "PUSH"
}, {
"hash" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091",
"triggerID" : "49f91bc146b2218410db8126b604934f2e79e4fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eca39ac8726c593c26432dbbb580562f23832e5c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25073)
* 49f91bc146b2218410db8126b604934f2e79e4fe Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25091)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org