You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "foxtail463 (via GitHub)" <gi...@apache.org> on 2026/05/26 12:52:32 UTC
[I] [Enhancement] Avoid duplicate Hive metadata loading when estimating HMS external table row count [doris]
foxtail463 opened a new issue, #63694:
URL: https://github.com/apache/doris/issues/63694
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Description
For HMS external tables, Doris may estimate table row count by listing Hive files when HMS table parameters do not contain row count and `enable_get_row_count_from_file_list` is enabled.
Currently, this row-count estimation path may read Hive partition and file metadata without filling Doris' Hive external metadata cache. In a normal query planning flow, the scan planning phase still needs the same partition and file metadata later, so Doris can read the same HMS/file metadata twice in one query planning process.
This is inefficient for Hive tables with many partitions or files, especially when HMS access is expensive.
Expected behavior:
- Query planning should be able to reuse Hive metadata fetched during row-count estimation.
- Non-query metadata display paths, such as `SHOW TABLE STATUS`, `SHOW STATS`, or `information_schema.tables`, should still avoid filling heavy Hive metadata caches just for displaying cached row count.
### Solution
Introduce a separate row-count loading mode for query planning and metadata display paths.
- `ExternalTable.getRowCount()` should load row count in a query-planning mode that may fill external metadata cache.
- `ExternalTable.getCachedRowCount()` and display-oriented paths should keep the lightweight behavior and avoid filling heavy metadata cache.
- `HMSExternalTable` should choose cached or non-cached Hive metadata APIs when estimating row count from file list.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Enhancement] Avoid duplicate Hive metadata loading when estimating HMS external table row count [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on issue #63694:
URL: https://github.com/apache/doris/issues/63694#issuecomment-4544186906
Breakwater-GitHub-Analysis-Slot: slot_75b5f7a31eb3
Initial triage: this looks like a valid performance/caching enhancement for the HMS external table planning path, not a correctness bug.
I checked current `origin/master` locally at `4483daf9f03`. The relevant flow matches the report:
- `ExternalTable.getRowCount()` initializes the table and loads the row-count cache through `ExternalRowCountCache.getCachedRowCount(...)`.
- `ExternalRowCountCache.RowCountCacheLoader` calls `table.fetchRowCount()`.
- `HMSExternalTable.fetchRowCount()` first tries HMS table parameters, then falls back to `getRowCountFromFileList()` when the row count is unknown and `enable_get_row_count_from_file_list` is enabled.
- In that fallback, `getFilesForPartitions(...)` explicitly uses `cache.getAllPartitionsWithoutCache(...)` and then `cache.getFilesByPartitions(..., withCache=false, ...)`.
- Later scan planning for the same HMS table uses the normal cached path: `HiveScanNode.getPartitions()` calls `getAllPartitionsWithCache(...)`, and file split generation calls `getFilesByPartitions(...)` with `withCache = Config.max_external_file_cache_num > 0`.
So the same Hive partition/file metadata can indeed be fetched once for row-count estimation and then fetched again for scan planning in one planning process. The issue description is consistent with the current code.
Suggested implementation direction:
- Add an explicit row-count loading mode, for example `QUERY_PLANNING` vs `DISPLAY_ONLY`, rather than inferring intent inside `HMSExternalTable`.
- Use the planning mode from `ExternalTable.getRowCount()` / optimizer stats paths, and allow the HMS file-list row-count fallback to populate or reuse the normal Hive partition/file caches in that mode.
- Keep display-oriented paths such as `getCachedRowCount()`, `SHOW TABLE STATUS`, `SHOW STATS`, and `information_schema.tables` lightweight. If the intent is strictly "cached value only", consider wiring them to `ExternalRowCountCache.getCachedRowCountIfPresent(...)` or equivalent non-loading behavior; the method already exists but is not currently used.
- For HMS row-count estimation, make the partition and file metadata APIs mode-dependent: cached APIs for query planning, non-cached APIs for display/background-only loading.
Recommended validation:
- Add an FE unit test or mock-cache test proving that planning-time row-count estimation for an HMS table uses the cached partition/file path.
- Add a companion test proving that display-only calls do not populate heavy Hive partition/file caches just to render row count.
- If possible, include a simple before/after planning profile or FE log snippet for a partitioned HMS table without `numRows` in HMS parameters and with `enable_get_row_count_from_file_list=true`, showing that partition/file listing happens once after the change.
No additional user logs are required to accept this as an enhancement, but a concrete reproduction table shape would help measure impact: Doris commit/version, catalog properties, partition count, file count, `hive_stats_partition_sample_size`, `max_external_file_cache_num`, `fetch_hive_row_count_sync`, and a representative `EXPLAIN` or planning profile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Enhancement] Avoid duplicate Hive metadata loading when estimating HMS external table row count [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow closed issue #63694: [Enhancement] Avoid duplicate Hive metadata loading when estimating HMS external table row count
URL: https://github.com/apache/doris/issues/63694
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org