You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2024/09/25 07:41:59 UTC
[PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
codope opened a new pull request, #12001:
URL: https://github.com/apache/hudi/pull/12001
### Change Logs
Add a metadata config to enable or disable functional index. Note that this is an uber config to disable all functional indexes. To build or drop each functional index individually, users still need to use CREATE/DROP INDEX SQL commands.
### Impact
Ease of use, users can easily disable func index with a simple toggle.
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2376353505
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
}, {
"hash" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778",
"triggerID" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"triggerType" : "PUSH"
}, {
"hash" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=779",
"triggerID" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"triggerType" : "PUSH"
}, {
"hash" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=784",
"triggerID" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"triggerType" : "PUSH"
}, {
"hash" : "94aabd627f91386dd41bfb1ce1ead7b0b866bdac",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "94aabd627f91386dd41bfb1ce1ead7b0b866bdac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=784)
* 94aabd627f91386dd41bfb1ce1ead7b0b866bdac UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12001:
URL: https://github.com/apache/hudi/pull/12001#discussion_r1776649737
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -424,6 +424,78 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Enable and Disable Functional Index") {
+ if (HoodieSparkUtils.gteqSpark3_3) {
+ withTempDir { tmp =>
+ // create a simple partitioned mor table and insert some records
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = 'mor',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 1001)")
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 1002)")
+ // create functional index and verify
+ spark.sql(s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')")
+ val metaClient = createMetaClient(spark, basePath)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains("func_index_idx_datestr"))
+ assertTrue(metaClient.getIndexMetadata.isPresent)
+ var functionalIndexMetadata = metaClient.getIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
+
+ // verify functional index records by querying metadata table
+ val result2DF = spark.sql(
+ s"select * from hudi_metadata('$tableName') where type=3"
+ )
+ assert(result2DF.count() == 1)
Review Comment:
Added min/max validation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2373799185
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
}, {
"hash" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778",
"triggerID" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3fe4adb70e750ab3972e8681fa3f2cafb1da206c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2374421053
> > uber config
>
> What do you mean? Not the company right?
No i meant like a super config for func indexes. Sorry for the confusion. I have rephrased.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2374327687
> uber config
What do you mean? Not the company right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12001:
URL: https://github.com/apache/hudi/pull/12001#discussion_r1775531824
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -424,6 +424,78 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Enable and Disable Functional Index") {
+ if (HoodieSparkUtils.gteqSpark3_3) {
+ withTempDir { tmp =>
+ // create a simple partitioned mor table and insert some records
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = 'mor',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 1001)")
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 1002)")
+ // create functional index and verify
+ spark.sql(s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')")
+ val metaClient = createMetaClient(spark, basePath)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains("func_index_idx_datestr"))
+ assertTrue(metaClient.getIndexMetadata.isPresent)
+ var functionalIndexMetadata = metaClient.getIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
Review Comment:
will remove this assertion. A bit unnecessary and already being asserted elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12001:
URL: https://github.com/apache/hudi/pull/12001#discussion_r1776648962
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -424,6 +424,78 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Enable and Disable Functional Index") {
+ if (HoodieSparkUtils.gteqSpark3_3) {
+ withTempDir { tmp =>
+ // create a simple partitioned mor table and insert some records
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = 'mor',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 1001)")
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 1002)")
+ // create functional index and verify
+ spark.sql(s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')")
+ val metaClient = createMetaClient(spark, basePath)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains("func_index_idx_datestr"))
+ assertTrue(metaClient.getIndexMetadata.isPresent)
+ var functionalIndexMetadata = metaClient.getIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
+
+ // verify functional index records by querying metadata table
+ val result2DF = spark.sql(
Review Comment:
I had result1DF previously which was simply asserting the records inserted. Then I removed it as it was redundant. But, did not rename result2DF and subsequentn DFs :)
However, now that I am asserting actual value in test so we don't need these DFs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2376530563
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
}, {
"hash" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778",
"triggerID" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"triggerType" : "PUSH"
}, {
"hash" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=779",
"triggerID" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"triggerType" : "PUSH"
}, {
"hash" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=784",
"triggerID" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"triggerType" : "PUSH"
}, {
"hash" : "94aabd627f91386dd41bfb1ce1ead7b0b866bdac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=797",
"triggerID" : "94aabd627f91386dd41bfb1ce1ead7b0b866bdac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 94aabd627f91386dd41bfb1ce1ead7b0b866bdac Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=797)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12001:
URL: https://github.com/apache/hudi/pull/12001#discussion_r1777307915
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java:
##########
@@ -179,40 +176,27 @@ protected HoodieData<HoodieRecord> getFunctionalIndexRecords(List<Pair<String, F
String columnToIndex = indexDefinition.getSourceFields().get(0);
SQLContext sqlContext = sparkEngineContext.getSqlContext();
String basePath = metaClient.getBasePath().toString();
- for (Pair<String, FileSlice> pair : partitionFileSlicePairs) {
- String partition = pair.getKey();
- FileSlice fileSlice = pair.getValue();
+
+ // Group FileSlices by partition
+ Map<String, List<FileSlice>> partitionToFileSlicesMap = partitionFileSlicePairs.stream()
+ .collect(Collectors.groupingBy(Pair::getKey, Collectors.mapping(Pair::getValue, Collectors.toList())));
+ List<HoodieRecord> allRecords = new ArrayList<>();
+ for (Map.Entry<String, List<FileSlice>> entry : partitionToFileSlicesMap.entrySet()) {
+ String partition = entry.getKey();
+ List<FileSlice> fileSlices = entry.getValue();
+ List<HoodieRecord> recordsForPartition = Collections.emptyList();
// For functional index using column_stats
if (indexDefinition.getIndexType().equalsIgnoreCase(PARTITION_NAME_COLUMN_STATS)) {
- return getFunctionalIndexRecordsUsingColumnStats(
- metaClient,
- parallelism,
- readerSchema,
- fileSlice,
- basePath,
- partition,
- functionalIndex,
- columnToIndex,
- sqlContext,
- sparkEngineContext);
+ recordsForPartition = getFunctionalIndexRecordsUsingColumnStats(metaClient, readerSchema, fileSlices, basePath, partition, functionalIndex, columnToIndex, sqlContext);
}
// For functional index using bloom_filters
if (indexDefinition.getIndexType().equalsIgnoreCase(PARTITION_NAME_BLOOM_FILTERS)) {
Review Comment:
Change this to an else if.
The two ifs are not both possible to be true. And since we are overwriting recordsForPartition in both of them it makes even more sense to not execute both of them.
```
indexDefinition.getIndexType().equalsIgnoreCase(PARTITION_NAME_BLOOM_FILTERS)
indexDefinition.getIndexType().equalsIgnoreCase(PARTITION_NAME_COLUMN_STATS)
```
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -70,118 +68,88 @@ public class SparkMetadataWriterUtils {
private static final String READ_PATHS_CONFIG = "hoodie.datasource.read.paths";
private static final String GLOB_PATHS_CONFIG = "glob.paths";
- public static HoodieJavaRDD<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(
- HoodieTableMetaClient metaClient,
- int parallelism,
- Schema readerSchema,
- FileSlice fileSlice,
- String basePath,
- String partition,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- HoodieSparkEngineContext sparkEngineContext) {
+ public static List<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(HoodieTableMetaClient metaClient,
+ Schema readerSchema,
+ List<FileSlice> fileSlices,
+ String basePath,
+ String partition,
+ HoodieFunctionalIndex<Column, Column> functionalIndex,
+ String columnToIndex,
+ SQLContext sqlContext) {
List<HoodieColumnRangeMetadata<Comparable>> columnRangeMetadataList = new ArrayList<>();
- if (fileSlice.getBaseFile().isPresent()) {
- HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
- String filename = baseFile.getFileName();
- long fileSize = baseFile.getFileSize();
- Path baseFilePath = filePath(basePath, partition, filename);
- buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, baseFilePath);
+ for (FileSlice fileSlice : fileSlices) {
+ if (fileSlice.getBaseFile().isPresent()) {
+ HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
+ String filename = baseFile.getFileName();
+ long fileSize = baseFile.getFileSize();
+ Path baseFilePath = filePath(basePath, partition, filename);
+ buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, baseFilePath);
+ }
+ // Handle log files
+ fileSlice.getLogFiles().forEach(logFile -> {
+ String fileName = logFile.getFileName();
+ Path logFilePath = filePath(basePath, partition, fileName);
+ long fileSize = logFile.getFileSize();
+ buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, logFilePath);
+ });
}
- // Handle log files
- fileSlice.getLogFiles().forEach(logFile -> {
- String fileName = logFile.getFileName();
- Path logFilePath = filePath(basePath, partition, fileName);
- long fileSize = logFile.getFileSize();
- buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, logFilePath);
- });
- return HoodieJavaRDD.of(createColumnStatsRecords(partition, columnRangeMetadataList, false).collect(Collectors.toList()), sparkEngineContext, parallelism);
+ return createColumnStatsRecords(partition, columnRangeMetadataList, false).collect(Collectors.toList());
}
- public static HoodieJavaRDD<HoodieRecord> getFunctionalIndexRecordsUsingBloomFilter(
- HoodieTableMetaClient metaClient,
- int parallelism,
- Schema readerSchema,
- FileSlice fileSlice,
- String basePath,
- String partition,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- HoodieSparkEngineContext sparkEngineContext,
- HoodieWriteConfig metadataWriteConfig) {
+ public static List<HoodieRecord> getFunctionalIndexRecordsUsingBloomFilter(HoodieTableMetaClient metaClient,
+ Schema readerSchema,
+ List<FileSlice> fileSlices,
+ String basePath,
+ String partition,
+ HoodieFunctionalIndex<Column, Column> functionalIndex,
+ String columnToIndex,
+ SQLContext sqlContext,
+ HoodieWriteConfig metadataWriteConfig) {
List<HoodieRecord> bloomFilterMetadataList = new ArrayList<>();
- if (fileSlice.getBaseFile().isPresent()) {
- HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
- String filename = baseFile.getFileName();
- Path baseFilePath = filePath(basePath, partition, filename);
- buildBloomFilterMetadata(
- metaClient,
- readerSchema,
- functionalIndex,
- columnToIndex,
- sqlContext,
- bloomFilterMetadataList,
- baseFilePath,
- metadataWriteConfig,
- partition,
- baseFile.getCommitTime());
+ for (FileSlice fileSlice : fileSlices) {
+ if (fileSlice.getBaseFile().isPresent()) {
+ HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
+ String filename = baseFile.getFileName();
+ Path baseFilePath = filePath(basePath, partition, filename);
+ buildBloomFilterMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, bloomFilterMetadataList, baseFilePath, metadataWriteConfig, partition, baseFile.getCommitTime());
+ }
+ // Handle log files
+ fileSlice.getLogFiles().forEach(logFile -> {
+ String fileName = logFile.getFileName();
+ Path logFilePath = filePath(basePath, partition, fileName);
+ buildBloomFilterMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, bloomFilterMetadataList, logFilePath, metadataWriteConfig, partition,
+ logFile.getDeltaCommitTime());
+ });
}
- // Handle log files
- fileSlice.getLogFiles().forEach(logFile -> {
- String fileName = logFile.getFileName();
- Path logFilePath = filePath(basePath, partition, fileName);
- buildBloomFilterMetadata(
- metaClient,
- readerSchema,
- functionalIndex,
- columnToIndex,
- sqlContext,
- bloomFilterMetadataList,
- logFilePath,
- metadataWriteConfig,
- partition,
- logFile.getDeltaCommitTime());
- });
- return HoodieJavaRDD.of(bloomFilterMetadataList, sparkEngineContext, parallelism);
+ return bloomFilterMetadataList;
}
- private static void buildColumnRangeMetadata(
- HoodieTableMetaClient metaClient,
- Schema readerSchema,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- List<HoodieColumnRangeMetadata<Comparable>> columnRangeMetadataList,
- long fileSize,
- Path filePath) {
- Dataset<Row> fileDf = readRecordsAsRow(
- new StoragePath[] {convertToStoragePath(filePath)},
- sqlContext,
- metaClient,
- readerSchema);
+ private static void buildColumnRangeMetadata(HoodieTableMetaClient metaClient,
+ Schema readerSchema,
+ HoodieFunctionalIndex<Column, Column> functionalIndex,
+ String columnToIndex,
+ SQLContext sqlContext,
+ List<HoodieColumnRangeMetadata<Comparable>> columnRangeMetadataList,
+ long fileSize,
+ Path filePath) {
Review Comment:
Should we just change this to storagepath?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -70,118 +68,88 @@ public class SparkMetadataWriterUtils {
private static final String READ_PATHS_CONFIG = "hoodie.datasource.read.paths";
private static final String GLOB_PATHS_CONFIG = "glob.paths";
- public static HoodieJavaRDD<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(
- HoodieTableMetaClient metaClient,
- int parallelism,
- Schema readerSchema,
- FileSlice fileSlice,
- String basePath,
- String partition,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- HoodieSparkEngineContext sparkEngineContext) {
+ public static List<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(HoodieTableMetaClient metaClient,
Review Comment:
This and getFunctionalIndexRecordsUsingBloomFilter look very similar. Is there a clean way to reuse code? There might not.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -70,118 +68,88 @@ public class SparkMetadataWriterUtils {
private static final String READ_PATHS_CONFIG = "hoodie.datasource.read.paths";
private static final String GLOB_PATHS_CONFIG = "glob.paths";
- public static HoodieJavaRDD<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(
- HoodieTableMetaClient metaClient,
- int parallelism,
- Schema readerSchema,
- FileSlice fileSlice,
- String basePath,
- String partition,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- HoodieSparkEngineContext sparkEngineContext) {
+ public static List<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(HoodieTableMetaClient metaClient,
+ Schema readerSchema,
+ List<FileSlice> fileSlices,
+ String basePath,
+ String partition,
+ HoodieFunctionalIndex<Column, Column> functionalIndex,
+ String columnToIndex,
+ SQLContext sqlContext) {
List<HoodieColumnRangeMetadata<Comparable>> columnRangeMetadataList = new ArrayList<>();
- if (fileSlice.getBaseFile().isPresent()) {
- HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
- String filename = baseFile.getFileName();
- long fileSize = baseFile.getFileSize();
- Path baseFilePath = filePath(basePath, partition, filename);
- buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, baseFilePath);
+ for (FileSlice fileSlice : fileSlices) {
+ if (fileSlice.getBaseFile().isPresent()) {
+ HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
+ String filename = baseFile.getFileName();
+ long fileSize = baseFile.getFileSize();
+ Path baseFilePath = filePath(basePath, partition, filename);
+ buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, baseFilePath);
+ }
+ // Handle log files
+ fileSlice.getLogFiles().forEach(logFile -> {
+ String fileName = logFile.getFileName();
+ Path logFilePath = filePath(basePath, partition, fileName);
+ long fileSize = logFile.getFileSize();
+ buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, logFilePath);
+ });
}
- // Handle log files
- fileSlice.getLogFiles().forEach(logFile -> {
- String fileName = logFile.getFileName();
- Path logFilePath = filePath(basePath, partition, fileName);
- long fileSize = logFile.getFileSize();
- buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, logFilePath);
- });
- return HoodieJavaRDD.of(createColumnStatsRecords(partition, columnRangeMetadataList, false).collect(Collectors.toList()), sparkEngineContext, parallelism);
+ return createColumnStatsRecords(partition, columnRangeMetadataList, false).collect(Collectors.toList());
}
- public static HoodieJavaRDD<HoodieRecord> getFunctionalIndexRecordsUsingBloomFilter(
- HoodieTableMetaClient metaClient,
- int parallelism,
- Schema readerSchema,
- FileSlice fileSlice,
- String basePath,
- String partition,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- HoodieSparkEngineContext sparkEngineContext,
- HoodieWriteConfig metadataWriteConfig) {
+ public static List<HoodieRecord> getFunctionalIndexRecordsUsingBloomFilter(HoodieTableMetaClient metaClient,
+ Schema readerSchema,
+ List<FileSlice> fileSlices,
+ String basePath,
+ String partition,
+ HoodieFunctionalIndex<Column, Column> functionalIndex,
+ String columnToIndex,
+ SQLContext sqlContext,
+ HoodieWriteConfig metadataWriteConfig) {
List<HoodieRecord> bloomFilterMetadataList = new ArrayList<>();
- if (fileSlice.getBaseFile().isPresent()) {
- HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
- String filename = baseFile.getFileName();
- Path baseFilePath = filePath(basePath, partition, filename);
- buildBloomFilterMetadata(
- metaClient,
- readerSchema,
- functionalIndex,
- columnToIndex,
- sqlContext,
- bloomFilterMetadataList,
- baseFilePath,
- metadataWriteConfig,
- partition,
- baseFile.getCommitTime());
+ for (FileSlice fileSlice : fileSlices) {
+ if (fileSlice.getBaseFile().isPresent()) {
Review Comment:
No bootstrap base handling? Do we need to create a followup ticket for that?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -70,118 +68,88 @@ public class SparkMetadataWriterUtils {
private static final String READ_PATHS_CONFIG = "hoodie.datasource.read.paths";
private static final String GLOB_PATHS_CONFIG = "glob.paths";
- public static HoodieJavaRDD<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(
- HoodieTableMetaClient metaClient,
- int parallelism,
- Schema readerSchema,
- FileSlice fileSlice,
- String basePath,
- String partition,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- HoodieSparkEngineContext sparkEngineContext) {
+ public static List<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(HoodieTableMetaClient metaClient,
+ Schema readerSchema,
+ List<FileSlice> fileSlices,
+ String basePath,
+ String partition,
+ HoodieFunctionalIndex<Column, Column> functionalIndex,
+ String columnToIndex,
+ SQLContext sqlContext) {
List<HoodieColumnRangeMetadata<Comparable>> columnRangeMetadataList = new ArrayList<>();
- if (fileSlice.getBaseFile().isPresent()) {
- HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
- String filename = baseFile.getFileName();
- long fileSize = baseFile.getFileSize();
- Path baseFilePath = filePath(basePath, partition, filename);
- buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, baseFilePath);
+ for (FileSlice fileSlice : fileSlices) {
+ if (fileSlice.getBaseFile().isPresent()) {
+ HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
+ String filename = baseFile.getFileName();
+ long fileSize = baseFile.getFileSize();
+ Path baseFilePath = filePath(basePath, partition, filename);
+ buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, baseFilePath);
+ }
+ // Handle log files
+ fileSlice.getLogFiles().forEach(logFile -> {
+ String fileName = logFile.getFileName();
+ Path logFilePath = filePath(basePath, partition, fileName);
+ long fileSize = logFile.getFileSize();
+ buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, logFilePath);
+ });
}
- // Handle log files
- fileSlice.getLogFiles().forEach(logFile -> {
- String fileName = logFile.getFileName();
- Path logFilePath = filePath(basePath, partition, fileName);
- long fileSize = logFile.getFileSize();
- buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, logFilePath);
- });
- return HoodieJavaRDD.of(createColumnStatsRecords(partition, columnRangeMetadataList, false).collect(Collectors.toList()), sparkEngineContext, parallelism);
+ return createColumnStatsRecords(partition, columnRangeMetadataList, false).collect(Collectors.toList());
}
- public static HoodieJavaRDD<HoodieRecord> getFunctionalIndexRecordsUsingBloomFilter(
- HoodieTableMetaClient metaClient,
- int parallelism,
- Schema readerSchema,
- FileSlice fileSlice,
- String basePath,
- String partition,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- HoodieSparkEngineContext sparkEngineContext,
- HoodieWriteConfig metadataWriteConfig) {
+ public static List<HoodieRecord> getFunctionalIndexRecordsUsingBloomFilter(HoodieTableMetaClient metaClient,
+ Schema readerSchema,
+ List<FileSlice> fileSlices,
+ String basePath,
+ String partition,
+ HoodieFunctionalIndex<Column, Column> functionalIndex,
+ String columnToIndex,
+ SQLContext sqlContext,
+ HoodieWriteConfig metadataWriteConfig) {
List<HoodieRecord> bloomFilterMetadataList = new ArrayList<>();
- if (fileSlice.getBaseFile().isPresent()) {
- HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
- String filename = baseFile.getFileName();
- Path baseFilePath = filePath(basePath, partition, filename);
- buildBloomFilterMetadata(
- metaClient,
- readerSchema,
- functionalIndex,
- columnToIndex,
- sqlContext,
- bloomFilterMetadataList,
- baseFilePath,
- metadataWriteConfig,
- partition,
- baseFile.getCommitTime());
+ for (FileSlice fileSlice : fileSlices) {
+ if (fileSlice.getBaseFile().isPresent()) {
+ HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
+ String filename = baseFile.getFileName();
+ Path baseFilePath = filePath(basePath, partition, filename);
+ buildBloomFilterMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, bloomFilterMetadataList, baseFilePath, metadataWriteConfig, partition, baseFile.getCommitTime());
+ }
+ // Handle log files
+ fileSlice.getLogFiles().forEach(logFile -> {
+ String fileName = logFile.getFileName();
+ Path logFilePath = filePath(basePath, partition, fileName);
Review Comment:
Shouldn't the log file already have the storagepath?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2373647462
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
}, {
"hash" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33609199fe8f7d355cfa760c04f2fe1c4b3fc723 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776)
* 3fe4adb70e750ab3972e8681fa3f2cafb1da206c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2374426357
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
}, {
"hash" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778",
"triggerID" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"triggerType" : "PUSH"
}, {
"hash" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=779",
"triggerID" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"triggerType" : "PUSH"
}, {
"hash" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=784",
"triggerID" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f328dade37789a968eed244f9f0aac52c114aa5c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=779)
* a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=784)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2374425472
> > > uber config
> >
> >
> > What do you mean? Not the company right?
>
> No i meant like a super config for func indexes. Sorry for the confusion. I have rephrased.
normally wouldn't be confusing, but Uber is pretty involved with Hudi 😂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2377742041
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
}, {
"hash" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778",
"triggerID" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"triggerType" : "PUSH"
}, {
"hash" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=779",
"triggerID" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"triggerType" : "PUSH"
}, {
"hash" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=784",
"triggerID" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"triggerType" : "PUSH"
}, {
"hash" : "94aabd627f91386dd41bfb1ce1ead7b0b866bdac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=797",
"triggerID" : "94aabd627f91386dd41bfb1ce1ead7b0b866bdac",
"triggerType" : "PUSH"
}, {
"hash" : "c066484cc5993428a6c6a42b2c7fc2e301332be5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=804",
"triggerID" : "c066484cc5993428a6c6a42b2c7fc2e301332be5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c066484cc5993428a6c6a42b2c7fc2e301332be5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=804)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #12001:
URL: https://github.com/apache/hudi/pull/12001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12001:
URL: https://github.com/apache/hudi/pull/12001#discussion_r1776653027
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -424,6 +424,78 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Enable and Disable Functional Index") {
+ if (HoodieSparkUtils.gteqSpark3_3) {
+ withTempDir { tmp =>
+ // create a simple partitioned mor table and insert some records
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = 'mor',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 1001)")
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 1002)")
+ // create functional index and verify
+ spark.sql(s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')")
+ val metaClient = createMetaClient(spark, basePath)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains("func_index_idx_datestr"))
+ assertTrue(metaClient.getIndexMetadata.isPresent)
+ var functionalIndexMetadata = metaClient.getIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
+
+ // verify functional index records by querying metadata table
+ val result2DF = spark.sql(
+ s"select * from hudi_metadata('$tableName') where type=3"
+ )
+ assert(result2DF.count() == 1)
+
+ // disable functional index
+ spark.sql("set hoodie.metadata.functional.index.enable=false")
+ // do another insert after initializing the index
+ spark.sql(s"insert into $tableName values(4, 'a4', 10, 10000000)")
+ // check query result
+ checkAnswer(s"select id, name from $tableName where from_unixtime(ts, 'yyyy-MM-dd') = '1970-04-26'")(
+ Seq(4, "a4")
+ )
+ // verify there are no new updates to functional index
+ val result3DF = spark.sql(
+ s"select * from hudi_metadata('$tableName') where type=3"
+ )
+ assert(result3DF.count() == 1)
+
+ // enable functional index
+ spark.sql("set hoodie.metadata.functional.index.enable=true")
+ // do another insert after initializing the index
+ spark.sql(s"insert into $tableName values(5, 'a5', 10, 10000000)")
Review Comment:
I have made it more realistic now by taking recent epoch times. I have also added a comment before inserting on the date that epoch translates to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2373650610
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
}, {
"hash" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778",
"triggerID" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33609199fe8f7d355cfa760c04f2fe1c4b3fc723 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776)
* 3fe4adb70e750ab3972e8681fa3f2cafb1da206c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12001:
URL: https://github.com/apache/hudi/pull/12001#discussion_r1775530964
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -316,6 +316,13 @@ public final class HoodieMetadataConfig extends HoodieConfig {
.withDocumentation("Initializes the metadata table by reading from the file system when the table is first created. Enabled by default. "
+ "Warning: This should only be disabled when manually constructing the metadata table outside of typical Hudi writer flows.");
+ public static final ConfigProperty<Boolean> FUNCTIONAL_INDEX_ENABLE_PROP = ConfigProperty
+ .key(METADATA_PREFIX + ".functional.index.enable")
Review Comment:
You raise a good point. I simply named the key that way because we keep calling functional index. I noticed that it is only the record index which deviates from the standard,e.g. bloom filter - `.index.bloom.filter.enable`, colstats - `.index.column.stats.enable`. So, I am going to rename it to `.index.functional.enable`. I can take up renaming record index enable key and use alias for compatibility in a followup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12001:
URL: https://github.com/apache/hudi/pull/12001#discussion_r1776667897
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1847,9 +1847,15 @@ public static HoodieData<HoodieRecord> readRecordKeysFromFileSlices(HoodieEngine
}
public static Schema getProjectedSchemaForFunctionalIndex(HoodieIndexDefinition indexDefinition, HoodieTableMetaClient metaClient) throws Exception {
- TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient);
- Schema tableSchema = schemaResolver.getTableAvroSchema();
- return addMetadataFields(getSchemaForFields(tableSchema, indexDefinition.getSourceFields()));
+ Schema tableSchema = new TableSchemaResolver(metaClient).getTableAvroSchema();
+ List<String> partitionFields = metaClient.getTableConfig().getPartitionFields()
+ .map(Arrays::asList)
+ .orElse(Collections.emptyList());
+ List<String> sourceFields = indexDefinition.getSourceFields();
+ List<String> mergedFields = new ArrayList<>(partitionFields.size() + sourceFields.size());
+ mergedFields.addAll(partitionFields);
+ mergedFields.addAll(sourceFields);
Review Comment:
this is a bug fix which appeared after building functional index for non-parititon column. While loading file slices, the relation expects partition field as well. So, we are projecting metadata fields + partition fields + fields on which func index is enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12001:
URL: https://github.com/apache/hudi/pull/12001#discussion_r1777522927
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -70,118 +68,88 @@ public class SparkMetadataWriterUtils {
private static final String READ_PATHS_CONFIG = "hoodie.datasource.read.paths";
private static final String GLOB_PATHS_CONFIG = "glob.paths";
- public static HoodieJavaRDD<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(
- HoodieTableMetaClient metaClient,
- int parallelism,
- Schema readerSchema,
- FileSlice fileSlice,
- String basePath,
- String partition,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- HoodieSparkEngineContext sparkEngineContext) {
+ public static List<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(HoodieTableMetaClient metaClient,
+ Schema readerSchema,
+ List<FileSlice> fileSlices,
+ String basePath,
+ String partition,
+ HoodieFunctionalIndex<Column, Column> functionalIndex,
+ String columnToIndex,
+ SQLContext sqlContext) {
List<HoodieColumnRangeMetadata<Comparable>> columnRangeMetadataList = new ArrayList<>();
- if (fileSlice.getBaseFile().isPresent()) {
- HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
- String filename = baseFile.getFileName();
- long fileSize = baseFile.getFileSize();
- Path baseFilePath = filePath(basePath, partition, filename);
- buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, baseFilePath);
+ for (FileSlice fileSlice : fileSlices) {
+ if (fileSlice.getBaseFile().isPresent()) {
+ HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
+ String filename = baseFile.getFileName();
+ long fileSize = baseFile.getFileSize();
+ Path baseFilePath = filePath(basePath, partition, filename);
+ buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, baseFilePath);
+ }
+ // Handle log files
+ fileSlice.getLogFiles().forEach(logFile -> {
+ String fileName = logFile.getFileName();
+ Path logFilePath = filePath(basePath, partition, fileName);
+ long fileSize = logFile.getFileSize();
+ buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, logFilePath);
+ });
}
- // Handle log files
- fileSlice.getLogFiles().forEach(logFile -> {
- String fileName = logFile.getFileName();
- Path logFilePath = filePath(basePath, partition, fileName);
- long fileSize = logFile.getFileSize();
- buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, logFilePath);
- });
- return HoodieJavaRDD.of(createColumnStatsRecords(partition, columnRangeMetadataList, false).collect(Collectors.toList()), sparkEngineContext, parallelism);
+ return createColumnStatsRecords(partition, columnRangeMetadataList, false).collect(Collectors.toList());
}
- public static HoodieJavaRDD<HoodieRecord> getFunctionalIndexRecordsUsingBloomFilter(
- HoodieTableMetaClient metaClient,
- int parallelism,
- Schema readerSchema,
- FileSlice fileSlice,
- String basePath,
- String partition,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- HoodieSparkEngineContext sparkEngineContext,
- HoodieWriteConfig metadataWriteConfig) {
+ public static List<HoodieRecord> getFunctionalIndexRecordsUsingBloomFilter(HoodieTableMetaClient metaClient,
+ Schema readerSchema,
+ List<FileSlice> fileSlices,
+ String basePath,
+ String partition,
+ HoodieFunctionalIndex<Column, Column> functionalIndex,
+ String columnToIndex,
+ SQLContext sqlContext,
+ HoodieWriteConfig metadataWriteConfig) {
List<HoodieRecord> bloomFilterMetadataList = new ArrayList<>();
- if (fileSlice.getBaseFile().isPresent()) {
- HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
- String filename = baseFile.getFileName();
- Path baseFilePath = filePath(basePath, partition, filename);
- buildBloomFilterMetadata(
- metaClient,
- readerSchema,
- functionalIndex,
- columnToIndex,
- sqlContext,
- bloomFilterMetadataList,
- baseFilePath,
- metadataWriteConfig,
- partition,
- baseFile.getCommitTime());
+ for (FileSlice fileSlice : fileSlices) {
+ if (fileSlice.getBaseFile().isPresent()) {
Review Comment:
Currently, functional index cannot be created for bootstrap tables. Created HUDI-8265 to track.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -70,118 +68,88 @@ public class SparkMetadataWriterUtils {
private static final String READ_PATHS_CONFIG = "hoodie.datasource.read.paths";
private static final String GLOB_PATHS_CONFIG = "glob.paths";
- public static HoodieJavaRDD<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(
- HoodieTableMetaClient metaClient,
- int parallelism,
- Schema readerSchema,
- FileSlice fileSlice,
- String basePath,
- String partition,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- HoodieSparkEngineContext sparkEngineContext) {
+ public static List<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(HoodieTableMetaClient metaClient,
+ Schema readerSchema,
+ List<FileSlice> fileSlices,
+ String basePath,
+ String partition,
+ HoodieFunctionalIndex<Column, Column> functionalIndex,
+ String columnToIndex,
+ SQLContext sqlContext) {
List<HoodieColumnRangeMetadata<Comparable>> columnRangeMetadataList = new ArrayList<>();
- if (fileSlice.getBaseFile().isPresent()) {
- HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
- String filename = baseFile.getFileName();
- long fileSize = baseFile.getFileSize();
- Path baseFilePath = filePath(basePath, partition, filename);
- buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, baseFilePath);
+ for (FileSlice fileSlice : fileSlices) {
+ if (fileSlice.getBaseFile().isPresent()) {
+ HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
+ String filename = baseFile.getFileName();
+ long fileSize = baseFile.getFileSize();
+ Path baseFilePath = filePath(basePath, partition, filename);
+ buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, baseFilePath);
+ }
+ // Handle log files
+ fileSlice.getLogFiles().forEach(logFile -> {
+ String fileName = logFile.getFileName();
+ Path logFilePath = filePath(basePath, partition, fileName);
+ long fileSize = logFile.getFileSize();
+ buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, logFilePath);
+ });
}
- // Handle log files
- fileSlice.getLogFiles().forEach(logFile -> {
- String fileName = logFile.getFileName();
- Path logFilePath = filePath(basePath, partition, fileName);
- long fileSize = logFile.getFileSize();
- buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, logFilePath);
- });
- return HoodieJavaRDD.of(createColumnStatsRecords(partition, columnRangeMetadataList, false).collect(Collectors.toList()), sparkEngineContext, parallelism);
+ return createColumnStatsRecords(partition, columnRangeMetadataList, false).collect(Collectors.toList());
}
- public static HoodieJavaRDD<HoodieRecord> getFunctionalIndexRecordsUsingBloomFilter(
- HoodieTableMetaClient metaClient,
- int parallelism,
- Schema readerSchema,
- FileSlice fileSlice,
- String basePath,
- String partition,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- HoodieSparkEngineContext sparkEngineContext,
- HoodieWriteConfig metadataWriteConfig) {
+ public static List<HoodieRecord> getFunctionalIndexRecordsUsingBloomFilter(HoodieTableMetaClient metaClient,
+ Schema readerSchema,
+ List<FileSlice> fileSlices,
+ String basePath,
+ String partition,
+ HoodieFunctionalIndex<Column, Column> functionalIndex,
+ String columnToIndex,
+ SQLContext sqlContext,
+ HoodieWriteConfig metadataWriteConfig) {
List<HoodieRecord> bloomFilterMetadataList = new ArrayList<>();
- if (fileSlice.getBaseFile().isPresent()) {
- HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
- String filename = baseFile.getFileName();
- Path baseFilePath = filePath(basePath, partition, filename);
- buildBloomFilterMetadata(
- metaClient,
- readerSchema,
- functionalIndex,
- columnToIndex,
- sqlContext,
- bloomFilterMetadataList,
- baseFilePath,
- metadataWriteConfig,
- partition,
- baseFile.getCommitTime());
+ for (FileSlice fileSlice : fileSlices) {
+ if (fileSlice.getBaseFile().isPresent()) {
+ HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
+ String filename = baseFile.getFileName();
+ Path baseFilePath = filePath(basePath, partition, filename);
+ buildBloomFilterMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, bloomFilterMetadataList, baseFilePath, metadataWriteConfig, partition, baseFile.getCommitTime());
+ }
+ // Handle log files
+ fileSlice.getLogFiles().forEach(logFile -> {
+ String fileName = logFile.getFileName();
+ Path logFilePath = filePath(basePath, partition, fileName);
+ buildBloomFilterMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, bloomFilterMetadataList, logFilePath, metadataWriteConfig, partition,
+ logFile.getDeltaCommitTime());
+ });
}
- // Handle log files
- fileSlice.getLogFiles().forEach(logFile -> {
- String fileName = logFile.getFileName();
- Path logFilePath = filePath(basePath, partition, fileName);
- buildBloomFilterMetadata(
- metaClient,
- readerSchema,
- functionalIndex,
- columnToIndex,
- sqlContext,
- bloomFilterMetadataList,
- logFilePath,
- metadataWriteConfig,
- partition,
- logFile.getDeltaCommitTime());
- });
- return HoodieJavaRDD.of(bloomFilterMetadataList, sparkEngineContext, parallelism);
+ return bloomFilterMetadataList;
}
- private static void buildColumnRangeMetadata(
- HoodieTableMetaClient metaClient,
- Schema readerSchema,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- List<HoodieColumnRangeMetadata<Comparable>> columnRangeMetadataList,
- long fileSize,
- Path filePath) {
- Dataset<Row> fileDf = readRecordsAsRow(
- new StoragePath[] {convertToStoragePath(filePath)},
- sqlContext,
- metaClient,
- readerSchema);
+ private static void buildColumnRangeMetadata(HoodieTableMetaClient metaClient,
+ Schema readerSchema,
+ HoodieFunctionalIndex<Column, Column> functionalIndex,
+ String columnToIndex,
+ SQLContext sqlContext,
+ List<HoodieColumnRangeMetadata<Comparable>> columnRangeMetadataList,
+ long fileSize,
+ Path filePath) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12001:
URL: https://github.com/apache/hudi/pull/12001#discussion_r1777524200
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -70,118 +68,88 @@ public class SparkMetadataWriterUtils {
private static final String READ_PATHS_CONFIG = "hoodie.datasource.read.paths";
private static final String GLOB_PATHS_CONFIG = "glob.paths";
- public static HoodieJavaRDD<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(
- HoodieTableMetaClient metaClient,
- int parallelism,
- Schema readerSchema,
- FileSlice fileSlice,
- String basePath,
- String partition,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- HoodieSparkEngineContext sparkEngineContext) {
+ public static List<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(HoodieTableMetaClient metaClient,
Review Comment:
We gain 3 lines.. i kinda feel that those 3 lines read better inline with the method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12001:
URL: https://github.com/apache/hudi/pull/12001#discussion_r1776646587
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -316,6 +316,13 @@ public final class HoodieMetadataConfig extends HoodieConfig {
.withDocumentation("Initializes the metadata table by reading from the file system when the table is first created. Enabled by default. "
+ "Warning: This should only be disabled when manually constructing the metadata table outside of typical Hudi writer flows.");
+ public static final ConfigProperty<Boolean> FUNCTIONAL_INDEX_ENABLE_PROP = ConfigProperty
Review Comment:
marked advanced, following the convention. Let's revisit if we can exclude certain configs from advanced list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12001:
URL: https://github.com/apache/hudi/pull/12001#discussion_r1776665013
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -70,118 +68,88 @@ public class SparkMetadataWriterUtils {
private static final String READ_PATHS_CONFIG = "hoodie.datasource.read.paths";
private static final String GLOB_PATHS_CONFIG = "glob.paths";
- public static HoodieJavaRDD<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(
- HoodieTableMetaClient metaClient,
- int parallelism,
- Schema readerSchema,
- FileSlice fileSlice,
- String basePath,
- String partition,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- HoodieSparkEngineContext sparkEngineContext) {
+ public static List<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(HoodieTableMetaClient metaClient,
+ Schema readerSchema,
+ List<FileSlice> fileSlices,
+ String basePath,
+ String partition,
+ HoodieFunctionalIndex<Column, Column> functionalIndex,
+ String columnToIndex,
+ SQLContext sqlContext) {
List<HoodieColumnRangeMetadata<Comparable>> columnRangeMetadataList = new ArrayList<>();
- if (fileSlice.getBaseFile().isPresent()) {
- HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
- String filename = baseFile.getFileName();
- long fileSize = baseFile.getFileSize();
- Path baseFilePath = filePath(basePath, partition, filename);
- buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, baseFilePath);
+ for (FileSlice fileSlice : fileSlices) {
+ if (fileSlice.getBaseFile().isPresent()) {
+ HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
+ String filename = baseFile.getFileName();
+ long fileSize = baseFile.getFileSize();
+ Path baseFilePath = filePath(basePath, partition, filename);
+ buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, baseFilePath);
+ }
+ // Handle log files
+ fileSlice.getLogFiles().forEach(logFile -> {
+ String fileName = logFile.getFileName();
+ Path logFilePath = filePath(basePath, partition, fileName);
+ long fileSize = logFile.getFileSize();
+ buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, logFilePath);
+ });
}
- // Handle log files
- fileSlice.getLogFiles().forEach(logFile -> {
- String fileName = logFile.getFileName();
- Path logFilePath = filePath(basePath, partition, fileName);
- long fileSize = logFile.getFileSize();
- buildColumnRangeMetadata(metaClient, readerSchema, functionalIndex, columnToIndex, sqlContext, columnRangeMetadataList, fileSize, logFilePath);
- });
- return HoodieJavaRDD.of(createColumnStatsRecords(partition, columnRangeMetadataList, false).collect(Collectors.toList()), sparkEngineContext, parallelism);
+ return createColumnStatsRecords(partition, columnRangeMetadataList, false).collect(Collectors.toList());
}
- public static HoodieJavaRDD<HoodieRecord> getFunctionalIndexRecordsUsingBloomFilter(
- HoodieTableMetaClient metaClient,
- int parallelism,
- Schema readerSchema,
- FileSlice fileSlice,
- String basePath,
- String partition,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- HoodieSparkEngineContext sparkEngineContext,
- HoodieWriteConfig metadataWriteConfig) {
+ public static List<HoodieRecord> getFunctionalIndexRecordsUsingBloomFilter(HoodieTableMetaClient metaClient,
+ Schema readerSchema,
+ List<FileSlice> fileSlices,
Review Comment:
same here.. collect bloom filter metadata for all file slices in a partition.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1847,9 +1847,15 @@ public static HoodieData<HoodieRecord> readRecordKeysFromFileSlices(HoodieEngine
}
public static Schema getProjectedSchemaForFunctionalIndex(HoodieIndexDefinition indexDefinition, HoodieTableMetaClient metaClient) throws Exception {
- TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient);
- Schema tableSchema = schemaResolver.getTableAvroSchema();
- return addMetadataFields(getSchemaForFields(tableSchema, indexDefinition.getSourceFields()));
+ Schema tableSchema = new TableSchemaResolver(metaClient).getTableAvroSchema();
+ List<String> partitionFields = metaClient.getTableConfig().getPartitionFields()
+ .map(Arrays::asList)
+ .orElse(Collections.emptyList());
+ List<String> sourceFields = indexDefinition.getSourceFields();
+ List<String> mergedFields = new ArrayList<>(partitionFields.size() + sourceFields.size());
+ mergedFields.addAll(partitionFields);
+ mergedFields.addAll(sourceFields);
Review Comment:
this is a bug fix which appeared after building functional index for non-parititon column. While loading file slices, the relation expects partition field as well. So, we are projecting metadat fields + partition fields + fields on which func index is enabled.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -70,118 +68,88 @@ public class SparkMetadataWriterUtils {
private static final String READ_PATHS_CONFIG = "hoodie.datasource.read.paths";
private static final String GLOB_PATHS_CONFIG = "glob.paths";
- public static HoodieJavaRDD<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(
- HoodieTableMetaClient metaClient,
- int parallelism,
- Schema readerSchema,
- FileSlice fileSlice,
- String basePath,
- String partition,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- HoodieSparkEngineContext sparkEngineContext) {
+ public static List<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(HoodieTableMetaClient metaClient,
+ Schema readerSchema,
+ List<FileSlice> fileSlices,
Review Comment:
Please ignore indentation changes. I had to bring the style in line with the convention we follow. The main change here is a refactoring that now instead of collecting stats for single file slice, it does so for all file slices in a given partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2377593495
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
}, {
"hash" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778",
"triggerID" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"triggerType" : "PUSH"
}, {
"hash" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=779",
"triggerID" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"triggerType" : "PUSH"
}, {
"hash" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=784",
"triggerID" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"triggerType" : "PUSH"
}, {
"hash" : "94aabd627f91386dd41bfb1ce1ead7b0b866bdac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=797",
"triggerID" : "94aabd627f91386dd41bfb1ce1ead7b0b866bdac",
"triggerType" : "PUSH"
}, {
"hash" : "c066484cc5993428a6c6a42b2c7fc2e301332be5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=804",
"triggerID" : "c066484cc5993428a6c6a42b2c7fc2e301332be5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 94aabd627f91386dd41bfb1ce1ead7b0b866bdac Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=797)
* c066484cc5993428a6c6a42b2c7fc2e301332be5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=804)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12001:
URL: https://github.com/apache/hudi/pull/12001#discussion_r1777933874
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -424,6 +424,85 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Enable and Disable Functional Index") {
+ if (HoodieSparkUtils.gteqSpark3_3) {
+ withTempDir { tmp =>
+ // create a simple partitioned mor table and insert some records
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | price double,
+ | ts long,
+ | name string
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = 'mor',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(name)
+ | location '$basePath'
+ """.stripMargin)
+ // a record with from_unixtime(ts, 'yyyy-MM-dd') = 2020-09-26
+ spark.sql(s"insert into $tableName values(1, 10, 1601098924, 'a1')")
+ // a record with from_unixtime(ts, 'yyyy-MM-dd') = 2021-09-26
+ spark.sql(s"insert into $tableName values(2, 10, 1632634924, 'a1')")
+ // a record with from_unixtime(ts, 'yyyy-MM-dd') = 2022-09-26
+ spark.sql(s"insert into $tableName values(3, 10, 1664170924, 'a2')")
+ // create functional index and verify
+ spark.sql(s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')")
+ val metaClient = createMetaClient(spark, basePath)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains("func_index_idx_datestr"))
+ assertTrue(metaClient.getIndexMetadata.isPresent)
+ assertEquals(1, metaClient.getIndexMetadata.get.getIndexDefinitions.size())
+
+ // verify functional index records by querying metadata table
+ val metadataSql = s"select ColumnStatsMetadata.minValue.member6.value, ColumnStatsMetadata.maxValue.member6.value from hudi_metadata('$tableName') where type=3"
+ checkAnswer(metadataSql)(
+ Seq("2020-09-26", "2021-09-26"), // for file in name=a1
+ Seq("2022-09-26", "2022-09-26") // for file in name=a2
+ )
+
+ // disable functional index
+ spark.sql(s"set ${HoodieMetadataConfig.FUNCTIONAL_INDEX_ENABLE_PROP.key}=false")
+ // do another insert after disabling the index
+ // a record with from_unixtime(ts, 'yyyy-MM-dd') = 2022-09-26
+ spark.sql(s"insert into $tableName values(4, 10, 1664170924, 'a2')")
+ // check query result
+ checkAnswer(s"select id, name from $tableName where from_unixtime(ts, 'yyyy-MM-dd') = '2022-09-26'")(
+ Seq(3, "a2"),
+ Seq(4, "a2")
+ )
+ // verify there are no new updates to functional index
+ checkAnswer(metadataSql)(
+ Seq("2020-09-26", "2021-09-26"),
+ Seq("2022-09-26", "2022-09-26")
+ )
+
+ // enable functional index
+ spark.sql(s"set ${HoodieMetadataConfig.FUNCTIONAL_INDEX_ENABLE_PROP.key}=true")
+ // do another insert after initializing the index
+ // a record with from_unixtime(ts, 'yyyy-MM-dd') = 2024-09-26
Review Comment:
We can disable using indexes by setting `hoodie.enable.data.skipping` to false. Query result with/without index should be same (assuming index is built correctly). But, we could argue for a case where one might want to disable a specific index even while reading, e.g. say some index gives correct results but performs slower because of poor implementation (though one can very easily just disable all indexes on the reader as a temp workaround).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12001:
URL: https://github.com/apache/hudi/pull/12001#discussion_r1777675977
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -424,6 +424,85 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Enable and Disable Functional Index") {
+ if (HoodieSparkUtils.gteqSpark3_3) {
+ withTempDir { tmp =>
+ // create a simple partitioned mor table and insert some records
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | price double,
+ | ts long,
+ | name string
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = 'mor',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(name)
+ | location '$basePath'
+ """.stripMargin)
+ // a record with from_unixtime(ts, 'yyyy-MM-dd') = 2020-09-26
+ spark.sql(s"insert into $tableName values(1, 10, 1601098924, 'a1')")
+ // a record with from_unixtime(ts, 'yyyy-MM-dd') = 2021-09-26
+ spark.sql(s"insert into $tableName values(2, 10, 1632634924, 'a1')")
+ // a record with from_unixtime(ts, 'yyyy-MM-dd') = 2022-09-26
+ spark.sql(s"insert into $tableName values(3, 10, 1664170924, 'a2')")
+ // create functional index and verify
+ spark.sql(s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')")
+ val metaClient = createMetaClient(spark, basePath)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains("func_index_idx_datestr"))
+ assertTrue(metaClient.getIndexMetadata.isPresent)
+ assertEquals(1, metaClient.getIndexMetadata.get.getIndexDefinitions.size())
+
+ // verify functional index records by querying metadata table
+ val metadataSql = s"select ColumnStatsMetadata.minValue.member6.value, ColumnStatsMetadata.maxValue.member6.value from hudi_metadata('$tableName') where type=3"
+ checkAnswer(metadataSql)(
+ Seq("2020-09-26", "2021-09-26"), // for file in name=a1
+ Seq("2022-09-26", "2022-09-26") // for file in name=a2
+ )
+
+ // disable functional index
+ spark.sql(s"set ${HoodieMetadataConfig.FUNCTIONAL_INDEX_ENABLE_PROP.key}=false")
+ // do another insert after disabling the index
+ // a record with from_unixtime(ts, 'yyyy-MM-dd') = 2022-09-26
+ spark.sql(s"insert into $tableName values(4, 10, 1664170924, 'a2')")
+ // check query result
+ checkAnswer(s"select id, name from $tableName where from_unixtime(ts, 'yyyy-MM-dd') = '2022-09-26'")(
+ Seq(3, "a2"),
+ Seq(4, "a2")
+ )
+ // verify there are no new updates to functional index
+ checkAnswer(metadataSql)(
+ Seq("2020-09-26", "2021-09-26"),
+ Seq("2022-09-26", "2022-09-26")
+ )
+
+ // enable functional index
+ spark.sql(s"set ${HoodieMetadataConfig.FUNCTIONAL_INDEX_ENABLE_PROP.key}=true")
+ // do another insert after initializing the index
+ // a record with from_unixtime(ts, 'yyyy-MM-dd') = 2024-09-26
Review Comment:
@codope I was suggesting you check the query result again here. But you only use the config on the write side, so it wouldn't make any sense that the read result would change by the config toggle.
Is there a way to disable using indexes on the read side? That would be out of the scope of this pr but that might be wanted. Right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2374422922
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
}, {
"hash" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778",
"triggerID" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"triggerType" : "PUSH"
}, {
"hash" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=779",
"triggerID" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"triggerType" : "PUSH"
}, {
"hash" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f328dade37789a968eed244f9f0aac52c114aa5c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=779)
* a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12001:
URL: https://github.com/apache/hudi/pull/12001#discussion_r1776645416
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -527,6 +527,9 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeFunctionalIndexPartiti
}
private Set<String> getFunctionalIndexPartitionsToInit() {
+ if (dataMetaClient.getIndexMetadata().isEmpty()) {
Review Comment:
yeah that's correct.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1050,6 +1053,10 @@ public void update(HoodieCommitMetadata commitMetadata, HoodieData<HoodieRecord>
* Update functional index from {@link HoodieCommitMetadata}.
*/
private void updateFunctionalIndexIfPresent(HoodieCommitMetadata commitMetadata, String instantTime, Map<String, HoodieData<HoodieRecord>> partitionToRecordMap) {
+ // if functional index is disabled, then just return
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12001:
URL: https://github.com/apache/hudi/pull/12001#discussion_r1776653325
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -424,6 +424,78 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Enable and Disable Functional Index") {
+ if (HoodieSparkUtils.gteqSpark3_3) {
+ withTempDir { tmp =>
+ // create a simple partitioned mor table and insert some records
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = 'mor',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 1001)")
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 1002)")
+ // create functional index and verify
+ spark.sql(s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')")
+ val metaClient = createMetaClient(spark, basePath)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains("func_index_idx_datestr"))
+ assertTrue(metaClient.getIndexMetadata.isPresent)
+ var functionalIndexMetadata = metaClient.getIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
+
+ // verify functional index records by querying metadata table
+ val result2DF = spark.sql(
+ s"select * from hudi_metadata('$tableName') where type=3"
+ )
+ assert(result2DF.count() == 1)
+
+ // disable functional index
+ spark.sql("set hoodie.metadata.functional.index.enable=false")
+ // do another insert after initializing the index
+ spark.sql(s"insert into $tableName values(4, 'a4', 10, 10000000)")
+ // check query result
+ checkAnswer(s"select id, name from $tableName where from_unixtime(ts, 'yyyy-MM-dd') = '1970-04-26'")(
+ Seq(4, "a4")
+ )
+ // verify there are no new updates to functional index
+ val result3DF = spark.sql(
+ s"select * from hudi_metadata('$tableName') where type=3"
+ )
+ assert(result3DF.count() == 1)
+
+ // enable functional index
+ spark.sql("set hoodie.metadata.functional.index.enable=true")
+ // do another insert after initializing the index
Review Comment:
fixed comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2377595905
> Still have the question about:
>
> > In the test lines 468-479 you disable the functional index and verify the query still works. What happens if you
> >
> > 1. disable index
> > 2. do write
> > 3. enable index
> > 4. check query result
> > Will the query still be correct?
Yes, the query result will be correct. Just that, data skipping may not be very efficient (skipping ratio might be lower). I have added a query with predicates including values when functional index was disabled and enabled, and we see correct results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12001:
URL: https://github.com/apache/hudi/pull/12001#discussion_r1775418456
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -316,6 +316,13 @@ public final class HoodieMetadataConfig extends HoodieConfig {
.withDocumentation("Initializes the metadata table by reading from the file system when the table is first created. Enabled by default. "
+ "Warning: This should only be disabled when manually constructing the metadata table outside of typical Hudi writer flows.");
+ public static final ConfigProperty<Boolean> FUNCTIONAL_INDEX_ENABLE_PROP = ConfigProperty
+ .key(METADATA_PREFIX + ".functional.index.enable")
Review Comment:
the equivalent record index config is formatted as `.record.index.enable`
So that matches this config. However, for the functional index, the configs are
`.index.functional.file.group.count` `.index.functional.parallelism`
and the equivalent partition stats config is `.index.partition.stats.enable`
I think since the record index configs are 0.14 but the other index configs are 1.0, we should probably just change them all to mach the record index ones. So `.partition.stats.index.enable` `.functional.index.file.group.count` `.partition.index.stats.enable` maybe we can create some more static prefix strings?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1050,6 +1053,10 @@ public void update(HoodieCommitMetadata commitMetadata, HoodieData<HoodieRecord>
* Update functional index from {@link HoodieCommitMetadata}.
*/
private void updateFunctionalIndexIfPresent(HoodieCommitMetadata commitMetadata, String instantTime, Map<String, HoodieData<HoodieRecord>> partitionToRecordMap) {
+ // if functional index is disabled, then just return
Review Comment:
I don't think this comment is useful. But pretty harmless. You can decide if you want it to stay
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -424,6 +424,78 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Enable and Disable Functional Index") {
+ if (HoodieSparkUtils.gteqSpark3_3) {
+ withTempDir { tmp =>
+ // create a simple partitioned mor table and insert some records
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = 'mor',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 1001)")
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 1002)")
+ // create functional index and verify
+ spark.sql(s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')")
+ val metaClient = createMetaClient(spark, basePath)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains("func_index_idx_datestr"))
+ assertTrue(metaClient.getIndexMetadata.isPresent)
+ var functionalIndexMetadata = metaClient.getIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
+
+ // verify functional index records by querying metadata table
+ val result2DF = spark.sql(
+ s"select * from hudi_metadata('$tableName') where type=3"
+ )
+ assert(result2DF.count() == 1)
Review Comment:
Would it be difficult to also validate the data in addition to the count?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -527,6 +527,9 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeFunctionalIndexPartiti
}
private Set<String> getFunctionalIndexPartitionsToInit() {
+ if (dataMetaClient.getIndexMetadata().isEmpty()) {
Review Comment:
We are adding this now because before it was always assumed to exist because we couldn't disable the index?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -424,6 +424,78 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Enable and Disable Functional Index") {
+ if (HoodieSparkUtils.gteqSpark3_3) {
+ withTempDir { tmp =>
+ // create a simple partitioned mor table and insert some records
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = 'mor',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 1001)")
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 1002)")
+ // create functional index and verify
+ spark.sql(s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')")
+ val metaClient = createMetaClient(spark, basePath)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains("func_index_idx_datestr"))
+ assertTrue(metaClient.getIndexMetadata.isPresent)
+ var functionalIndexMetadata = metaClient.getIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
+
+ // verify functional index records by querying metadata table
+ val result2DF = spark.sql(
+ s"select * from hudi_metadata('$tableName') where type=3"
+ )
+ assert(result2DF.count() == 1)
+
+ // disable functional index
+ spark.sql("set hoodie.metadata.functional.index.enable=false")
+ // do another insert after initializing the index
+ spark.sql(s"insert into $tableName values(4, 'a4', 10, 10000000)")
+ // check query result
+ checkAnswer(s"select id, name from $tableName where from_unixtime(ts, 'yyyy-MM-dd') = '1970-04-26'")(
+ Seq(4, "a4")
+ )
+ // verify there are no new updates to functional index
+ val result3DF = spark.sql(
+ s"select * from hudi_metadata('$tableName') where type=3"
+ )
+ assert(result3DF.count() == 1)
+
+ // enable functional index
+ spark.sql("set hoodie.metadata.functional.index.enable=true")
+ // do another insert after initializing the index
+ spark.sql(s"insert into $tableName values(5, 'a5', 10, 10000000)")
Review Comment:
10000000 is hard to visually tell that they are the same unless they are right next to each other. Which they are not
```
spark.sql(s"insert into $tableName values(4, 'a4', 10, 10000000)")
spark.sql(s"insert into $tableName values(5, 'a5', 10, 10000000)")
```
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -424,6 +424,78 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Enable and Disable Functional Index") {
+ if (HoodieSparkUtils.gteqSpark3_3) {
+ withTempDir { tmp =>
+ // create a simple partitioned mor table and insert some records
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = 'mor',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 1001)")
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 1002)")
+ // create functional index and verify
+ spark.sql(s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')")
+ val metaClient = createMetaClient(spark, basePath)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains("func_index_idx_datestr"))
+ assertTrue(metaClient.getIndexMetadata.isPresent)
+ var functionalIndexMetadata = metaClient.getIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
Review Comment:
```
assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
```
This seems kinda reflexive?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -424,6 +424,78 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Enable and Disable Functional Index") {
+ if (HoodieSparkUtils.gteqSpark3_3) {
+ withTempDir { tmp =>
+ // create a simple partitioned mor table and insert some records
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = 'mor',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 1001)")
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 1002)")
+ // create functional index and verify
+ spark.sql(s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')")
+ val metaClient = createMetaClient(spark, basePath)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains("func_index_idx_datestr"))
+ assertTrue(metaClient.getIndexMetadata.isPresent)
+ var functionalIndexMetadata = metaClient.getIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
+
+ // verify functional index records by querying metadata table
+ val result2DF = spark.sql(
Review Comment:
there's no result1DF? also usually see the f lowercase when we do Df (but I don't really care if you want to keep it uppercase)
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -316,6 +316,13 @@ public final class HoodieMetadataConfig extends HoodieConfig {
.withDocumentation("Initializes the metadata table by reading from the file system when the table is first created. Enabled by default. "
+ "Warning: This should only be disabled when manually constructing the metadata table outside of typical Hudi writer flows.");
+ public static final ConfigProperty<Boolean> FUNCTIONAL_INDEX_ENABLE_PROP = ConfigProperty
Review Comment:
Mark advanced? I feel like every mdt config will be on the advanced side, so should we have a higher threshold (meaning that we only mark really advanced configs as advanced in HoodieMetadataConfig )?
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -316,6 +316,13 @@ public final class HoodieMetadataConfig extends HoodieConfig {
.withDocumentation("Initializes the metadata table by reading from the file system when the table is first created. Enabled by default. "
+ "Warning: This should only be disabled when manually constructing the metadata table outside of typical Hudi writer flows.");
+ public static final ConfigProperty<Boolean> FUNCTIONAL_INDEX_ENABLE_PROP = ConfigProperty
+ .key(METADATA_PREFIX + ".functional.index.enable")
Review Comment:
But I think there is an argument that the index part should go first. So all the mdt index configs have the same prefix. WDYT?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -424,6 +424,78 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Enable and Disable Functional Index") {
+ if (HoodieSparkUtils.gteqSpark3_3) {
+ withTempDir { tmp =>
+ // create a simple partitioned mor table and insert some records
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = 'mor',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 1001)")
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 1002)")
+ // create functional index and verify
+ spark.sql(s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')")
+ val metaClient = createMetaClient(spark, basePath)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains("func_index_idx_datestr"))
+ assertTrue(metaClient.getIndexMetadata.isPresent)
+ var functionalIndexMetadata = metaClient.getIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
+
+ // verify functional index records by querying metadata table
+ val result2DF = spark.sql(
+ s"select * from hudi_metadata('$tableName') where type=3"
+ )
+ assert(result2DF.count() == 1)
+
+ // disable functional index
+ spark.sql("set hoodie.metadata.functional.index.enable=false")
+ // do another insert after initializing the index
+ spark.sql(s"insert into $tableName values(4, 'a4', 10, 10000000)")
+ // check query result
+ checkAnswer(s"select id, name from $tableName where from_unixtime(ts, 'yyyy-MM-dd') = '1970-04-26'")(
+ Seq(4, "a4")
+ )
+ // verify there are no new updates to functional index
+ val result3DF = spark.sql(
+ s"select * from hudi_metadata('$tableName') where type=3"
+ )
+ assert(result3DF.count() == 1)
+
+ // enable functional index
+ spark.sql("set hoodie.metadata.functional.index.enable=true")
+ // do another insert after initializing the index
Review Comment:
`do another insert after initializing the index`
on line 469 and 483 confuse me a bit. Seems like maybe 1 of them should be different
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -424,6 +424,78 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test Enable and Disable Functional Index") {
+ if (HoodieSparkUtils.gteqSpark3_3) {
+ withTempDir { tmp =>
+ // create a simple partitioned mor table and insert some records
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | options (
+ | primaryKey ='id',
+ | type = 'mor',
+ | preCombineField = 'ts'
+ | )
+ | partitioned by(ts)
+ | location '$basePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 1001)")
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 1002)")
+ // create functional index and verify
+ spark.sql(s"create index idx_datestr on $tableName using column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')")
+ val metaClient = createMetaClient(spark, basePath)
+ assertTrue(metaClient.getTableConfig.getMetadataPartitions.contains("func_index_idx_datestr"))
+ assertTrue(metaClient.getIndexMetadata.isPresent)
+ var functionalIndexMetadata = metaClient.getIndexMetadata.get()
+ assertEquals(1, functionalIndexMetadata.getIndexDefinitions.size())
+ assertEquals("func_index_idx_datestr", functionalIndexMetadata.getIndexDefinitions.get("func_index_idx_datestr").getIndexName)
+
+ // verify functional index records by querying metadata table
+ val result2DF = spark.sql(
+ s"select * from hudi_metadata('$tableName') where type=3"
+ )
+ assert(result2DF.count() == 1)
+
+ // disable functional index
+ spark.sql("set hoodie.metadata.functional.index.enable=false")
+ // do another insert after initializing the index
+ spark.sql(s"insert into $tableName values(4, 'a4', 10, 10000000)")
+ // check query result
+ checkAnswer(s"select id, name from $tableName where from_unixtime(ts, 'yyyy-MM-dd') = '1970-04-26'")(
+ Seq(4, "a4")
+ )
+ // verify there are no new updates to functional index
+ val result3DF = spark.sql(
+ s"select * from hudi_metadata('$tableName') where type=3"
+ )
+ assert(result3DF.count() == 1)
Review Comment:
> Would it be difficult to also validate the data in addition to the count?
here as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2373463550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33609199fe8f7d355cfa760c04f2fe1c4b3fc723 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2373311376
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33609199fe8f7d355cfa760c04f2fe1c4b3fc723 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2373308648
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33609199fe8f7d355cfa760c04f2fe1c4b3fc723 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12001:
URL: https://github.com/apache/hudi/pull/12001#discussion_r1777666886
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -70,118 +68,88 @@ public class SparkMetadataWriterUtils {
private static final String READ_PATHS_CONFIG = "hoodie.datasource.read.paths";
private static final String GLOB_PATHS_CONFIG = "glob.paths";
- public static HoodieJavaRDD<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(
- HoodieTableMetaClient metaClient,
- int parallelism,
- Schema readerSchema,
- FileSlice fileSlice,
- String basePath,
- String partition,
- HoodieFunctionalIndex<Column, Column> functionalIndex,
- String columnToIndex,
- SQLContext sqlContext,
- HoodieSparkEngineContext sparkEngineContext) {
+ public static List<HoodieRecord> getFunctionalIndexRecordsUsingColumnStats(HoodieTableMetaClient metaClient,
Review Comment:
Yeah, that's how it often goes. Thanks for trying
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2376356608
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
}, {
"hash" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778",
"triggerID" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"triggerType" : "PUSH"
}, {
"hash" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=779",
"triggerID" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"triggerType" : "PUSH"
}, {
"hash" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=784",
"triggerID" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"triggerType" : "PUSH"
}, {
"hash" : "94aabd627f91386dd41bfb1ce1ead7b0b866bdac",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=797",
"triggerID" : "94aabd627f91386dd41bfb1ce1ead7b0b866bdac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=784)
* 94aabd627f91386dd41bfb1ce1ead7b0b866bdac Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=797)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2377591146
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
}, {
"hash" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778",
"triggerID" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"triggerType" : "PUSH"
}, {
"hash" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=779",
"triggerID" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"triggerType" : "PUSH"
}, {
"hash" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=784",
"triggerID" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"triggerType" : "PUSH"
}, {
"hash" : "94aabd627f91386dd41bfb1ce1ead7b0b866bdac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=797",
"triggerID" : "94aabd627f91386dd41bfb1ce1ead7b0b866bdac",
"triggerType" : "PUSH"
}, {
"hash" : "c066484cc5993428a6c6a42b2c7fc2e301332be5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c066484cc5993428a6c6a42b2c7fc2e301332be5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 94aabd627f91386dd41bfb1ce1ead7b0b866bdac Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=797)
* c066484cc5993428a6c6a42b2c7fc2e301332be5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2374597240
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
}, {
"hash" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778",
"triggerID" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"triggerType" : "PUSH"
}, {
"hash" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=779",
"triggerID" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"triggerType" : "PUSH"
}, {
"hash" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=784",
"triggerID" : "a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9ecc9a8fc6e28cd2ebdcfc6e1773625673f940a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=784)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2373825666
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
}, {
"hash" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778",
"triggerID" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"triggerType" : "PUSH"
}, {
"hash" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=779",
"triggerID" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3fe4adb70e750ab3972e8681fa3f2cafb1da206c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778)
* f328dade37789a968eed244f9f0aac52c114aa5c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=779)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2373822686
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
}, {
"hash" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778",
"triggerID" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"triggerType" : "PUSH"
}, {
"hash" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3fe4adb70e750ab3972e8681fa3f2cafb1da206c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778)
* f328dade37789a968eed244f9f0aac52c114aa5c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7662] Add a metadata config to enable or disable functional index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12001:
URL: https://github.com/apache/hudi/pull/12001#issuecomment-2373973804
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=776",
"triggerID" : "33609199fe8f7d355cfa760c04f2fe1c4b3fc723",
"triggerType" : "PUSH"
}, {
"hash" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=778",
"triggerID" : "3fe4adb70e750ab3972e8681fa3f2cafb1da206c",
"triggerType" : "PUSH"
}, {
"hash" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=779",
"triggerID" : "f328dade37789a968eed244f9f0aac52c114aa5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f328dade37789a968eed244f9f0aac52c114aa5c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=779)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org