[jira] [Created] (IMPALA-15435) [DOCS] Document turning off block location loading (impala.preload-block-locations-for-scheduling)

["Zoltán Borók-Nagy (Jira)" <ji...@apache.org>]

Zoltán Borók-Nagy created IMPALA-15435:
------------------------------------------

             Summary: [DOCS] Document turning off block location loading (impala.preload-block-locations-for-scheduling)
                 Key: IMPALA-15435
                 URL: https://issues.apache.org/jira/browse/IMPALA-15435
             Project: IMPALA
          Issue Type: Documentation
            Reporter: Zoltán Borók-Nagy
            Assignee: Zoltán Borók-Nagy


IMPALA-14138 lets admins turn off block location loading for block-based filesystems through Hadoop configuration. Nothing in docs/ mentions it (git grep -i preload docs/ is empty). These are core-site.xml keys, not gflags, so they don't show up in --help or /varz either; only /hadoop-varz shows the value in effect.

Add a concept — impala_scalability.xml, or next to "Block Location Tracking" in impala_config_performance.xml — covering:

- The keys: impala.preload-block-locations-for-scheduling (all filesystems), ...scheme.<scheme> and ...authority.<authority>. The authority is the bare URI authority, such as mycluster or nn-host:8020. The default is true, and a false at any level wins.
- Affected schemes: hdfs, ofs, o3fs, alluxio. Object stores never load block locations.
- The keys are read once at JVM startup, so a restart is needed.
- A warning that catalogd and every coordinator must agree. With catalogd off and a coordinator on, files arrive with no blocks and get skipped, so queries can silently return too few rows. The opposite mismatch hits a Preconditions check.
- Effects when off: no locality and no short-circuit reads, every scan range remote, synthetic splits (Parquet follows PARQUET_OBJECT_STORE_SPLIT_SIZE, other formats the filesystem's default block size), the loading pool sized by max_nonhdfs_partitions_parallel_load, and much cheaper table loading for large tables. Point at the data
  cache for remote reads.
- When to use it: Impala not co-located with HDFS/Ozone, for example in containers.
- For Iceberg tables in this mode, SHOW FILES shows 'N/A' in the EC Policy column (IMPALA-15291), and erasure-coding detection (ALLOW_ERASURE_CODED_FILES, the EC scan counters) doesn't apply to those files.

Also update:
- impala_parquet_object_store_split_size.xml: it says the option controls split sizes "on non-block based stores, e.g. S3, ADLS", which is now incomplete — it also applies to HDFS and Ozone when preloading is off.
- impala_config_performance.xml, section block_location_tracking: "Mandatory: Block Location Tracking ... Impala will not start unless this setting is enabled" is obsolete (a Hadoop 2 claim) and now contradicts a supported opt-out. Say that locality only matters when Impala runs next to the DataNodes. The same page links to that section
  as "Optional: Block Location Tracking", so the two titles already disagree.
- Optional: one-line xrefs from impala_ozone.xml and impala_data_cache.xml, and a short explanation of the EC Policy column values (NONE, the policy name, MIXED, N/A) in the SHOW FILES usage notes in impala_show.xml.

The default is unchanged, so no incompatible-changes entry is needed.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org