You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/06/09 01:50:27 UTC
[GitHub] [hudi] Reimus opened a new issue, #5804: [SUPPORT] Data skipping using Column Stats index only seems to work on root level columns
Reimus opened a new issue, #5804:
URL: https://github.com/apache/hudi/issues/5804
After writing Hudi table using spark
```
ds.write
.format("hudi")
.mode(SaveMode.Append)
.option(DataSourceWriteOptions.PRECOMBINE_FIELD.key, "ts")
.option(DataSourceWriteOptions.RECORDKEY_FIELD.key, "id")
.option(DataSourceWriteOptions.PARTITIONPATH_FIELD.key, "ym")
.option(DataSourceWriteOptions.OPERATION.key, DataSourceWriteOptions.UPSERT_OPERATION_OPT_VAL)
.option(HoodieWriteConfig.TBL_NAME.key, tableName)
.option(DataSourceWriteOptions.RECONCILE_SCHEMA.key, "true")
.option(DataSourceWriteOptions.META_SYNC_CLIENT_TOOL_CLASS_NAME.key,s"${classOf[HiveSyncTool].getName},${classOf[DataHubSyncTool].getName})
.option(DataSourceWriteOptions.TABLE_TYPE.key, DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL)
.option(DataSourceWriteOptions.OPERATION.key, DataSourceWriteOptions.UPSERT_OPERATION_OPT_VAL)
.option(HoodieTableConfig.TIMELINE_TIMEZONE.key, HoodieTimelineTimeZone.UTC.name)
.option(HoodieWriteConfig.SCHEMA_EVOLUTION_ENABLE.key, "true")
.option(HoodieWriteConfig.AVRO_SCHEMA_VALIDATE_ENABLE.key, "true")
.option(HoodieWriteConfig.WRITE_CONCURRENCY_MODE.key, WriteConcurrencyMode.OPTIMISTIC_CONCURRENCY_CONTROL.name())
.option(HoodieIndexConfig.BLOOM_FILTER_TYPE.key, BloomFilterTypeCode.DYNAMIC_V0.name)
.option(HoodieIndexConfig.BLOOM_FILTER_NUM_ENTRIES_VALUE.key, String.valueOf(100000))
.option(HoodieIndexConfig.BLOOM_INDEX_USE_METADATA.key, "true")
.option(HoodieIndexConfig.BLOOM_INDEX_FILTER_DYNAMIC_MAX_ENTRIES.key, String.valueOf(1000000))
.option(HoodieLockConfig.HIVE_DATABASE_NAME.key, databaseName)
.option(HoodieLockConfig.HIVE_TABLE_NAME.key, tableName)
.option(HoodieLockConfig.HIVE_METASTORE_URI.key, env.spark.hiveMetastore)
.option(HoodieLockConfig.LOCK_PROVIDER_CLASS_NAME.key, classOf[org.apache.hudi.hive.HiveMetastoreBasedLockProvider].getName)
.option(HoodieStorageConfig.PARQUET_MAX_FILE_SIZE.key, String.valueOf(256 * 1024 * 1024))
.option(HoodieStorageConfig.PARQUET_BLOCK_SIZE.key, String.valueOf(256 * 1024 * 1024))
.option(HoodieCompactionConfig.AUTO_CLEAN.key, "true")
.option(HoodieCompactionConfig.FAILED_WRITES_CLEANER_POLICY.key, HoodieFailedWritesCleaningPolicy.LAZY.name)
.option(HoodieCompactionConfig.CLEANER_POLICY.key, HoodieCleaningPolicy.KEEP_LATEST_BY_HOURS.name())
.option(HoodieCompactionConfig.CLEANER_HOURS_RETAINED.key, String.valueOf(24))
.option(HoodieCompactionConfig.PARQUET_SMALL_FILE_LIMIT.key, String.valueOf(104857600))
.option(HoodieMetadataConfig.COLUMN_STATS_INDEX_FOR_COLUMNS.key, "ym,ymd,date,ts,lvl1.ymd,lvl1.lvl2.date")
.option(HoodieMetadataConfig.BLOOM_FILTER_INDEX_FOR_COLUMNS.key, "id,col1,col2")
.option(HoodieMetadataConfig.POPULATE_META_FIELDS.key, "true")
.option(HoodieMetadataConfig.ENABLE_METADATA_INDEX_COLUMN_STATS.key, "true")
.option(HoodieMetadataConfig.ENABLE_METADATA_INDEX_BLOOM_FILTER.key, "true")
.option(HoodieMetadataConfig.ENABLE.key, "true")
.save("/tmp/hudi")
```
And reading said table using:
```
val s = spark.read.format("hudi").option("hoodie.datasource.query.type","read_optimized").option("hoodie.file.index.enable","true").option("hoodie.enable.data.skipping","true").option("hoodie.metadata.enable","true").
option("hoodie.metadata.index.column.stats.enable","true").option("","true").option("hoodie.datasource.read.extract.partition.values.from.path","true").load("/tmp/hudi")
```
I am finding that the data skipping works only queries using root level fields.
ym, ymd, date and ts.
I've set up lvl1.ymd == ymd, and lvl1.lvl2.date == date to make apples to apples comparison.
When I try to filter on lvl.ymd (same field as root level ymd) - spark performs full table scan, instead of limiting to files containing that ymd.
**Expected behavior**
Nested fields also work with data skipping.
**Environment Description**
* Hudi version : 0.11
* Spark version : 3.1.2
* Hadoop version : 3.0.X
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on issue #5804: [SUPPORT] Data skipping using Column Stats index only seems to work on root level columns
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on issue #5804:
URL: https://github.com/apache/hudi/issues/5804#issuecomment-1154434634
Created HUDI-4245 to track
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #5804: [SUPPORT] Data skipping using Column Stats index only seems to work on root level columns
Posted by GitBox <gi...@apache.org>.
codope closed issue #5804: [SUPPORT] Data skipping using Column Stats index only seems to work on root level columns
URL: https://github.com/apache/hudi/issues/5804
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #5804: [SUPPORT] Data skipping using Column Stats index only seems to work on root level columns
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5804:
URL: https://github.com/apache/hudi/issues/5804#issuecomment-1152983029
@alexeykudinkin : can you follow up on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on issue #5804: [SUPPORT] Data skipping using Column Stats index only seems to work on root level columns
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on issue #5804:
URL: https://github.com/apache/hudi/issues/5804#issuecomment-1154206281
@Reimus thanks for reporting!
Internally Data Skipping is relying on Column Stats Index that currently builds statistics only for the top-level columns, and therefore Data Skipping is only able to prune based on the top-level columns.
If you feel this could be generalized to all nested fields please feel free to create a feature-request or (even better!) kickstart the conversation on the dev-list and maybe kickoff the RFC process to extend this to general case of nested columns in the table as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #5804: [SUPPORT] Data skipping using Column Stats index only seems to work on root level columns
Posted by GitBox <gi...@apache.org>.
codope commented on issue #5804:
URL: https://github.com/apache/hudi/issues/5804#issuecomment-1160553029
Closing the issue as there is a tracking jira matching priority with fix version 0.12.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org