You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by dh...@apache.org on 2018/06/08 00:44:16 UTC

[1/2] impala git commit: IMPALA-7071: make get_fs_path() idempotent

Repository: impala
Updated Branches:
  refs/heads/master c72c5dc66 -> 9d0ae238d


IMPALA-7071: make get_fs_path() idempotent

This avoids future errors like IMPALA-7068.

Change-Id: I10c7cbb36119883718e60660db2ac7313f14d388
Reviewed-on: http://gerrit.cloudera.org:8080/10517
Reviewed-by: Dan Hecht <dh...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/495d7781
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/495d7781
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/495d7781

Branch: refs/heads/master
Commit: 495d7781898c1189c199740702c4801a7a245640
Parents: c72c5dc
Author: Dan Hecht <dh...@cloudera.com>
Authored: Thu May 24 14:03:50 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Fri Jun 8 00:14:37 2018 +0000

----------------------------------------------------------------------
 tests/infra/test_utils.py      | 31 +++++++++++++++++++++++++++++++
 tests/util/filesystem_utils.py |  8 ++++++--
 2 files changed, 37 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/impala/blob/495d7781/tests/infra/test_utils.py
----------------------------------------------------------------------
diff --git a/tests/infra/test_utils.py b/tests/infra/test_utils.py
new file mode 100644
index 0000000..c998703
--- /dev/null
+++ b/tests/infra/test_utils.py
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This module contains tests for some of the tests/util code.
+
+from tests.util.filesystem_utils import prepend_with_fs
+
+def test_filesystem_utils():
+  # Verify that empty FS prefix gives back the same path.
+  path = "/fake-warehouse"
+  assert prepend_with_fs("", path) == path
+
+  # Verify that prepend_with_fs() is idempotent.
+  fs = "fakeFs://bucket"
+  path = "/fake-warehouse"
+  assert prepend_with_fs(fs, path) == fs + path
+  assert prepend_with_fs(fs, prepend_with_fs(fs, path)) == fs + path

http://git-wip-us.apache.org/repos/asf/impala/blob/495d7781/tests/util/filesystem_utils.py
----------------------------------------------------------------------
diff --git a/tests/util/filesystem_utils.py b/tests/util/filesystem_utils.py
index 82f6584..aef08a3 100644
--- a/tests/util/filesystem_utils.py
+++ b/tests/util/filesystem_utils.py
@@ -50,10 +50,14 @@ ADLS_CLIENT_ID = os.getenv("azure_client_id")
 ADLS_TENANT_ID = os.getenv("azure_tenant_id")
 ADLS_CLIENT_SECRET = os.getenv("azure_client_secret")
 
+def prepend_with_fs(fs, path):
+  """Prepend 'path' with 'fs' if it's not already the prefix."""
+  return path if path.startswith(fs) else "%s%s" % (fs, path)
+
 def get_fs_path(path):
-  return "%s%s" % (FILESYSTEM_PREFIX, path)
+  return prepend_with_fs(FILESYSTEM_PREFIX, path)
 
 def get_secondary_fs_path(path):
-  return "%s%s" % (SECONDARY_FILESYSTEM, path)
+  return prepend_with_fs(SECONDARY_FILESYSTEM, path)
 
 WAREHOUSE = get_fs_path('/test-warehouse')


[2/2] impala git commit: IMPALA-7099: Don't set FILESYSTEM_PREFIX for s3

Posted by dh...@apache.org.
IMPALA-7099: Don't set FILESYSTEM_PREFIX for s3

When S3 support was originally added to Impala, Impala still required
that the defaultFs point to HDFS, and so the S3 tests were originally
run as a "secondary" filesystem (which meant that all paths had to be
fully qualified).

Since then, the restriction on defaultFs has been addressed, and
additionally, the S3 test configuration had been changed to run with S3
as the defaultFs. So, setting FILESYSTEM_PREFIX for S3 is no longer
necessary.

Let's remove that so that S3 tests run in a configuration more similar
to HDFS, which will help prevent them from breaking as often (since
precheckin tests run only in the HDFS configuration).

Incidently, this addresses the problem with
test_unsupported_text_compression on S3 since the problem is with the
handling of fully qualified paths.

Testing:
- core on S3

Change-Id: Iace5fc9557ff6e7d184573beafea694191424742
Reviewed-on: http://gerrit.cloudera.org:8080/10595
Reviewed-by: Dan Hecht <dh...@cloudera.com>
Tested-by: Dan Hecht <dh...@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/9d0ae238
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/9d0ae238
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/9d0ae238

Branch: refs/heads/master
Commit: 9d0ae238d41de78bd458055849276d40e0ee6fd5
Parents: 495d778
Author: Dan Hecht <dh...@cloudera.com>
Authored: Fri Jun 1 10:02:23 2018 -0700
Committer: Dan Hecht <dh...@cloudera.com>
Committed: Fri Jun 8 00:43:02 2018 +0000

----------------------------------------------------------------------
 bin/impala-config.sh | 1 -
 1 file changed, 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/impala/blob/9d0ae238/bin/impala-config.sh
----------------------------------------------------------------------
diff --git a/bin/impala-config.sh b/bin/impala-config.sh
index 670832c..2c11d2c 100755
--- a/bin/impala-config.sh
+++ b/bin/impala-config.sh
@@ -403,7 +403,6 @@ if [ "${TARGET_FILESYSTEM}" = "s3" ]; then
     if ${IMPALA_HOME}/bin/check-s3-access.sh; then
       export S3_ACCESS_VALIDATED=1
       export DEFAULT_FS="s3a://${S3_BUCKET}"
-      export FILESYSTEM_PREFIX="${DEFAULT_FS}"
     else
       return 1
     fi