You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2021/10/08 00:51:00 UTC
[jira] [Commented] (IMPALA-10945) Unfavorable change on the S3
scratch path behavior
[ https://issues.apache.org/jira/browse/IMPALA-10945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425885#comment-17425885 ]
ASF subversion and git services commented on IMPALA-10945:
----------------------------------------------------------
Commit 45fd3320ad4f68ca86998dff0c9504aa896a278a in impala's branch refs/heads/master from Yida Wu
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=45fd332 ]
IMPALA-10945: Fix S3 scratch path behavior
IMPALA-10429 "Support Spill to HDFS" introduces a new behavior to
S3 scratch path. It added a path verification for the S3 path,
however, the HdfsFsCache::GetNameNodeFromPath in the verification
forces the input to have at least a directory after the authority,
like "s3a://authority/dir", otherwise it will return an error and
lead to a failure on the TmpFileMgr initialization. Therefore, it
changes the previous behavior which was able to support
"s3a://authority", and may affect current users.
This patch resumes the behavior of the s3 scratch path to allow
"s3a://authority". The solution is to pass the path of the user's
input combined with a scratch suffix "/impala-scratch" to the
verification function, therefore, at least one directory is contained
in the path.
Tests:
Ran core tests.
Added logic to run two types of path format in TmpFileMgrTest:
"s3a://authority" and "s3a://authority/dir".
Change-Id: I028f375b9f535f8641261cc02f921497e076aa9b
Reviewed-on: http://gerrit.cloudera.org:8080/17901
Reviewed-by: Abhishek Rawat <ar...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Unfavorable change on the S3 scratch path behavior
> --------------------------------------------------
>
> Key: IMPALA-10945
> URL: https://issues.apache.org/jira/browse/IMPALA-10945
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.1.0
> Reporter: Yida Wu
> Assignee: Yida Wu
> Priority: Major
>
> IMPALA-10429 "Support Spill to HDFS" introduces a new behavior to S3 scratch path.
> It added a connection verification logic for the S3 path (for optimization), but the HdfsFsCache requires the path to have a directory behind, like "s3a://host/dir", otherwise it will throw an error.
> [https://github.com/apache/impala/blob/1a61a8025c87c37921a1bba4c49f754d8bd10bcc/be/src/runtime/tmp-file-mgr.cc#L333]
> The change brings a problem, that may require the user to have a different s3a scratch path. It would be good to be compatible with the path we support in the previous version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org