You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "noahtaite (via GitHub)" <gi...@apache.org> on 2023/12/15 15:19:57 UTC
[I] [SUPPORT] Incremental read query performance [hudi]
noahtaite opened a new issue, #10334:
URL: https://github.com/apache/hudi/issues/10334
**Describe the problem you faced**
I'm running an application that reads from 4 medium-sized (few hundred GB) Hudi MoR tables which are compacted weekly.
When running incremental queries to look over 3~5 commits with `hoodie.datasource.read.incr.fallback.fulltablescan.enable -> true`, I am seeing a 5 hour gap in my Spark History Server:
Incremental read config:
```
Scan MergeOnReadIncrementalRelation(
org.apache.spark.sql.SQLContext@3b457d8a,Map(
hoodie.datasource.read.incr.fallback.fulltablescan.enable -> true,
path -> s3://lake/names.all_hudi,
hoodie.write.lock.zookeeper.url -> ip-10-80-36-225.ec2.internal,
hoodie.write.lock.zookeeper.base_path -> /hudi,
hoodie.metadata.enable -> true,
hoodie.datasource.hive_sync.jdbcurl -> jdbc:hive2://ip-10-xx-xx-xxx.ec2.internal:10000,
hoodie.datasource.read.begin.instanttime -> 20231207170214000,
hoodie.datasource.query.type -> incremental,
hoodie.cleaner.policy.failed.writes -> EAGER,
hoodie.write.lock.zookeeper.port -> 2181,
hoodie.write.lock.provider -> org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider,
hoodie.write.concurrency.mode -> single_writer),HoodieTableMetaClient{basePath='s3://lake/names.all_hudi',
metaPath='s3://lake/names.all_hudi/.hoodie',
tableType=MERGE_ON_READ},None,None
)
Job time: 2023/12/12 00:47:50
=> 5 commits for incremental read
number of output rows: 19,926,780
```
During the gap, my driver kills all 40 of my executor instances and I see the following logs on driver: [log.txt](https://github.com/apache/hudi/files/13686484/log.txt)
When I use snapshot query + filter on _hoodie_commit_time, the gap reduces to ~20mins:
Snapshot read config:
```
Scan MergeOnReadSnapshotRelation(org.apache.spark.sql.SQLContext@24a4d185,
Map(
path -> s3://lake/names.all_hudi,
hoodie.write.lock.zookeeper.url -> ip-xx-xx-xxx.ec2.internal,
hoodie.write.lock.zookeeper.base_path -> /hudi,
hoodie.metadata.enable -> true,
hoodie.datasource.hive_sync.jdbcurl -> jdbc:hive2://ip-10-xx-xx-xxx.ec2.internal:10000,
hoodie.datasource.query.type -> snapshot,
hoodie.cleaner.policy.failed.writes -> EAGER,
hoodie.write.lock.zookeeper.port -> 2181,
hoodie.write.lock.provider -> org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider,
hoodie.write.concurrency.mode -> single_writer),
HoodieTableMetaClient{basePath='s3://lake/names.all_hudi',
metaPath='s3://lake/names.all_hudi/.hoodie',
tableType=MERGE_ON_READ},
List(),
None,
None
)
```
When I do incremental read with `hoodie.datasource.read.incr.fallback.fulltablescan.enable -> false`, I see the incremental query gap go from 5 hours -> 5 mins 😮 :
Incremental read config (no fulltable fallback):
```
Scan MergeOnReadIncrementalRelation(org.apache.spark.sql.SQLContext@dcb6efd,
Map(
path -> s3://lake/names.all_hudi,
hoodie.write.lock.zookeeper.url -> ip-10-80-39-205.ec2.internal,
hoodie.write.lock.zookeeper.base_path -> /hudi,
hoodie.metadata.enable -> true,
hoodie.datasource.hive_sync.jdbcurl -> jdbc:hive2://ip-10-80-39-205.ec2.internal:10000,
hoodie.datasource.read.begin.instanttime -> 20231212232646000,
hoodie.datasource.query.type -> incremental,
hoodie.cleaner.policy.failed.writes -> EAGER,
hoodie.write.lock.zookeeper.port -> 2181,
hoodie.write.lock.provider -> org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider,
hoodie.write.concurrency.mode -> single_writer),
HoodieTableMetaClient{basePath='s3://lake/names.all_hudi',
metaPath='s3://lake/names.all_hudi/.hoodie',
tableType=MERGE_ON_READ},
None,
None
)
number of output rows: 12,541,221
```
This leads me to believe the incremental query was degenerating into a full table scan when it shouldn't... (all my commits are active and uncleaned):
https://github.com/apache/hudi/blob/7a6543958368540d221ddc18e0c12b8d526b6859/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala#L142-L151
Wondering if someone can help me understand the following:
1. Why is the incremental load degenerating into a full table scan when I can run the same job with the flag disabled and do not run into File Not Found?
2. Is the full table scan expected to happen on only one instance (the driver?)
3. Can I safely leave this flag false, and expect to only run into FileNotFound if running cleaner?
**Expected behavior**
Only fall back to full table scan in incremental load when necessary (file not found issue) and not all the time.
**Environment Description**
* Hudi version : 0.13.1
* Spark version : 3.4.0
* Hive version : 3.1.3
* Hadoop version : 3.3.3
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental read query performance [hudi]
Posted by "Hans-Raintree (via GitHub)" <gi...@apache.org>.
Hans-Raintree commented on issue #10334:
URL: https://github.com/apache/hudi/issues/10334#issuecomment-5773062869
Hey @danny0405
I'm using table version 9, MOR. hoodie.file.group.reader.enabled is not set, so I believe it should default to True.
The queried range did include archived instants.
I believe it's existence-check overhead since when I reviewed Spark UI there was a long gap, where no 'jobs' or 'stages' were running, then when it started actually reading the data it took only seconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental read query performance [hudi]
Posted by "Hans-Raintree (via GitHub)" <gi...@apache.org>.
Hans-Raintree commented on issue #10334:
URL: https://github.com/apache/hudi/issues/10334#issuecomment-5725930454
I'm having a similar issue on Hudi 1.1.0. (S3, EMR, Metadata table off)
I had cleaner.commits.retained and hoodie.keep.max.commits and hoodie.keep.min.commits too low. I was using hoodie.datasource.read.incr.path.glob to read a single partition and it took 105 seconds, when I turned fallback off it produced the same result in 2.7 seconds.
As this option is now true by default, I think this code path should be optimized.
Some options would be:
1. Filter affected files before checking existence (when using hoodie.datasource.read.incr.path.glob)
2. Check only file slices actually needed by the query
3. Run existence checks concurrently (currently it's sequential)
4. Use Hudi cleaner metadata instead of probing S3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental read query performance [hudi]
Posted by "Hans-Raintree (via GitHub)" <gi...@apache.org>.
Hans-Raintree commented on issue #10334:
URL: https://github.com/apache/hudi/issues/10334#issuecomment-5790304339
For reference, this table is relatively big, 334GB, 227012 objects in S3. The table has a three-level partition hierarchy: 435 on the first level, 703 on the second, and 3216 on the third.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental read query performance [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10334:
URL: https://github.com/apache/hudi/issues/10334#issuecomment-5739192506
@Hans-Raintree I checked the 1.1.0 source, and your concern is valid for the incremental path used by the file-group reader.
In [`HoodieIncrementalRelationV2Trait.fullTableScan`](https://github.com/apache/hudi/blob/release-1.1.0/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV2.scala#L193-L198), `affectedFilesInCommits.asScala.exists(...)` performs sequential storage existence checks on the driver. The path glob is applied only afterward, when filtering file slices. A query for one partition can therefore probe files from other partitions touched by the selected commits, and a missing file outside the requested partition can trigger fallback.
There is also a distinction between affected files and files actually needed by the query: [`listAffectedFilesForCommits`](https://github.com/apache/hudi/blob/release-1.1.0/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java#L488-L507) deduplicates by full path, whereas the reader subsequently selects the latest merged file slices. That leaves room for unnecessary checks of older file versions.
I would prioritize selecting the query-relevant slices first, preserving the existing glob and base/log-file semantics, then checking their required files with bounded concurrency. Cleaner metadata could help identify known deletions, but using it to replace existence checks would need guarantees about metadata coverage and concurrent cleaning.
Two qualifications for diagnosing the 105s vs. 2.7s difference:
- Enabling fallback can incur these checks even when no full scan ultimately happens. If the selected range includes archived instants, however, the condition short-circuits directly to fallback and skips the existence checks. The retention settings make this distinction worth checking.
- The sequential behavior depends on the reader path. The file-group reader reuses this relation for both COW and MOR, while the [legacy COW reader](https://github.com/apache/hudi/blob/release-1.1.0/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelationV2.scala#L174-L245) already filters by glob and distributes existence checks through Spark.
Could you share the table type/version, `hoodie.file.group.reader.enabled` setting, and whether the queried range included archived instants? A driver profile or storage-request metrics from the slow run would help distinguish existence-check overhead from an actual fallback scan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental read query performance [hudi]
Posted by "noahtaite (via GitHub)" <gi...@apache.org>.
noahtaite commented on issue #10334:
URL: https://github.com/apache/hudi/issues/10334#issuecomment-1875740582
Thanks @danny0405 for response. @linliu-code let me know if you would share some insight or need details. tks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental read query performance [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10334:
URL: https://github.com/apache/hudi/issues/10334#issuecomment-1859487213
cc @linliu-code , maybe he could give some investigations of spark inc read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org