You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2022/12/08 17:54:00 UTC

[jira] [Commented] (IMPALA-11730) Add support for spilling to Ozone

    [ https://issues.apache.org/jira/browse/IMPALA-11730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644926#comment-17644926 ] 

ASF subversion and git services commented on IMPALA-11730:
----------------------------------------------------------

Commit a469a9cf19a34d28a7d29d1b0fac0aeb0662ca8e in impala's branch refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=a469a9cf1 ]

IMPALA-11730: Add support for spilling to Ozone

Adds support for spilling to Ozone (ofs and o3fs schemes) for parity
with HDFS. Note that ofs paths start with <volume>/<bucket>, which have
naming restrictions; tmp/impala-scratch is a valid name, so something
like ofs://localhost:9862/tmp would work as a scratch directory (volume
tmp, implicit bucket impala-scratch).

Updates tests to determine the correct path from the environment. Fixes
backend tests to work with Ozone as well. Guards test_scratch_disk.py
behind a new flag for filesystems that support spilling. Updates metric
verification to wait for scratch-space-bytes-used to be non-zero, as it
seems to update slower with Ozone.

Refactors TmpDir to remove extraneous variables and functions. Each
implementation is expected to handle its own token parsing.

Initializes default_fs in ExecEnv when using TestEnv. Previously it was
uninitialized, and uses of default_fs would return an empty string.

Testing:
- Ran backend, end-to-end, and custom cluster tests with Ozone.
- Ran test_scratch_disk.py exhaustive runs with Ozone and HDFS.

Change-Id: I5837c30357363f727ca832fb94169f2474fb4f6f
Reviewed-on: http://gerrit.cloudera.org:8080/19251
Reviewed-by: Michael Smith <mi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Add support for spilling to Ozone
> ---------------------------------
>
>                 Key: IMPALA-11730
>                 URL: https://issues.apache.org/jira/browse/IMPALA-11730
>             Project: IMPALA
>          Issue Type: New Feature
>          Components: Backend
>            Reporter: Michael Smith
>            Assignee: Michael Smith
>            Priority: Major
>              Labels: ozone
>             Fix For: Impala 4.3.0
>
>
> Attempting to use Ozone as a scratch dir via something like
> {code:java}
> -scratch_dirs=/tmp/scratch,ofs://localhost:9862/impala/tmp'{code}
> fails to use the Ozone path with
> {code:java}
> I1116 16:14:03.494846 919843 status.cc:129] Malformed scratch directory capacity configuration 'ofs://localhost:9862/impala/tmp'
>     @           0xf9189f  impala::Status::Status()
>     @          0x1377524  impala::TmpDir::ParseByteLimit()
>     @          0x138e4ba  impala::TmpDir::ParseTokens()
>     @          0x138e609  impala::TmpDir::Parse()
>     @          0x13901bd  impala::TmpFileMgr::InitCustom()
>     @          0x13916f3  impala::TmpFileMgr::InitCustom()
>     @          0x13917dd  impala::TmpFileMgr::Init()
>     @          0x15a295f  impala::ImpalaServer::ImpalaServer()
>     @          0x156df01  ImpaladMain()
>     @           0xe7cc4c  main
>     @     0x7f0ac6a74083  __libc_start_main
>     @           0xe7caae  _start
> W1116 16:14:03.508049 919843 tmp-file-mgr.cc:309] Directory ofs://localhost:9862/impala/tmp is not used because Malformed scratch directory capacity configuration 'ofs://localhost:9862/impala/tmp'{code}
> We're working on feature parity between HDFS and Ozone, so we should support this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org