[jira] [Updated] (IMPALA-15435) [DOCS] Document turning off block location loading (impala.preload-block-locations-for-scheduling)

["Noémi Pap-Takács (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-15435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Noémi Pap-Takács updated IMPALA-15435:
--------------------------------------
    Fix Version/s: Impala 5.0.0

> [DOCS] Document turning off block location loading (impala.preload-block-locations-for-scheduling)
> --------------------------------------------------------------------------------------------------
>
>                 Key: IMPALA-15435
>                 URL: https://issues.apache.org/jira/browse/IMPALA-15435
>             Project: IMPALA
>          Issue Type: Documentation
>            Reporter: Zoltán Borók-Nagy
>            Assignee: Zoltán Borók-Nagy
>            Priority: Major
>             Fix For: Impala 5.0.0
>
>
> IMPALA-14138 lets admins turn off block location loading for block-based filesystems through Hadoop configuration. Nothing in docs/ mentions it (git grep -i preload docs/ is empty). These are core-site.xml keys, not gflags, so they don't show up in --help or /varz either; only /hadoop-varz shows the value in effect.
> Add a concept — impala_scalability.xml, or next to "Block Location Tracking" in impala_config_performance.xml — covering:
> - The keys: impala.preload-block-locations-for-scheduling (all filesystems), ...scheme.<scheme> and ...authority.<authority>. The authority is the bare URI authority, such as mycluster or nn-host:8020. The default is true, and a false at any level wins.
> - Affected schemes: hdfs, ofs, o3fs, alluxio. Object stores never load block locations.
> - The keys are read once at JVM startup, so a restart is needed.
> - A warning that catalogd and every coordinator must agree. With catalogd off and a coordinator on, files arrive with no blocks and get skipped, so queries can silently return too few rows. The opposite mismatch hits a Preconditions check.
> - Effects when off: no locality and no short-circuit reads, every scan range remote, synthetic splits (Parquet follows PARQUET_OBJECT_STORE_SPLIT_SIZE, other formats the filesystem's default block size), the loading pool sized by max_nonhdfs_partitions_parallel_load, and much cheaper table loading for large tables. Point at the data
>   cache for remote reads.
> - When to use it: Impala not co-located with HDFS/Ozone, for example in containers.
> - For Iceberg tables in this mode, SHOW FILES shows 'N/A' in the EC Policy column (IMPALA-15291), and erasure-coding detection (ALLOW_ERASURE_CODED_FILES, the EC scan counters) doesn't apply to those files.
> Also update:
> - impala_parquet_object_store_split_size.xml: it says the option controls split sizes "on non-block based stores, e.g. S3, ADLS", which is now incomplete — it also applies to HDFS and Ozone when preloading is off.
> - impala_config_performance.xml, section block_location_tracking: "Mandatory: Block Location Tracking ... Impala will not start unless this setting is enabled" is obsolete (a Hadoop 2 claim) and now contradicts a supported opt-out. Say that locality only matters when Impala runs next to the DataNodes. The same page links to that section
>   as "Optional: Block Location Tracking", so the two titles already disagree.
> - Optional: one-line xrefs from impala_ozone.xml and impala_data_cache.xml, and a short explanation of the EC Policy column values (NONE, the policy name, MIXED, N/A) in the SHOW FILES usage notes in impala_show.xml.
> The default is unchanged, so no incompatible-changes entry is needed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org