[jira] [Commented] (IMPALA-10213) Handle block location for Ozone

["Michael Smith (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-10213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572603#comment-17572603 ] 

Michael Smith commented on IMPALA-10213:
----------------------------------------

This is mostly already enabled, because Ozone support uses Impala's HDFS implementation. Co-located storage uses local paths (which violates the DCHECK at https://github.com/apache/impala/blob/4.1.0/be/src/runtime/io/disk-io-mgr.cc#L843). We don't run into it in local testing much because Ozone returns storage addresses as {{localhost}} (where-as HDFS converts them to {{127.0.0.1}}) and our local dev environments look for the hostname or {{127.0.0.1}}. Explicitly setting {{--hostname=localhost}} (as done in {{test_client_ssl.py}}) triggers the local filesystem path (and errors out on the DCHECK).

The main thing we're missing is that Ozone returns an array of NULLs for https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/BlockLocation.html#getStorageIds--, which results in warnings about "missing disk id information", and we can't use separate queues for individual disks on an Ozone backend.

> Handle block location for Ozone
> -------------------------------
>
>                 Key: IMPALA-10213
>                 URL: https://issues.apache.org/jira/browse/IMPALA-10213
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Backend
>            Reporter: Attila Doroszlai
>            Priority: Major
>
> Currently Impala treats Ozone as a remote filesystem, similar to S3A, ADLS etc.  Ozone provides block location info in its Hadoop-compatible FS implementations.  Also, Ozone can be colocated with Impala daemons.  It would be nice if Impala could be improved to use Ozone's location info to support locality of execution.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org