You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Fredy Wijaya (Code Review)" <ge...@cloudera.org> on 2018/07/13 13:23:56 UTC

[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2 

Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10940


Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2
......................................................................

IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2

This patch removes the use of IMPALA_MINICLUSTER_PROFILE. The code that
uses IMPALA_MINICLUSTER_PROFILE=2 is removed and it defaults to code from
IMPALA_MINICLUSTER_PROFILE=3. In order to reduce having too many code
changes in this patch, there is no code change for the shims. The shims
for IMPALA_MINICLUSTER_PROFILE=3 automatically become the default
implementation.

Testing:
- Ran core and exhaustive tests

Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
---
M bin/create-test-configuration.sh
M bin/impala-config.sh
M bin/jenkins/build-all-flag-combinations.sh
M fe/pom.xml
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetCatalogsReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetColumnsReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetFunctionsReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetInfoReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetSchemasReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/authorization/SentryAuthProvider.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/HdfsShim.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/MetastoreShim.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/MiniclusterProfile.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/util/SentryUtil.java
D fe/src/compat-minicluster-profile-2/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java
D fe/src/compat-minicluster-profile-3/java/org/apache/impala/compat/MiniclusterProfile.java
R fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java
R fe/src/main/java/org/apache/impala/authorization/ImpalaActionFactory.java
R fe/src/main/java/org/apache/impala/authorization/ImpalaPrivilegeModel.java
R fe/src/main/java/org/apache/impala/authorization/SentryAuthProvider.java
R fe/src/main/java/org/apache/impala/compat/HdfsShim.java
R fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
R fe/src/main/java/org/apache/impala/util/SentryUtil.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
R fe/src/test/java/org/apache/impala/authorization/ImpalaActionFactoryTest.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
R fe/src/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java
M impala-parent/pom.xml
M testdata/bin/run-hbase.sh
M testdata/bin/run-hive-server.sh
M testdata/bin/run-mini-dfs.sh
M testdata/cluster/node_templates/common/etc/init.d/common.tmpl
M testdata/cluster/node_templates/common/etc/init.d/yarn-common
M testdata/pom.xml
M testdata/workloads/functional-query/queries/QueryTest/views-compatibility.test
M tests/common/environ.py
M tests/metadata/test_views_compatibility.py
M tests/query_test/test_partitioning.py
39 files changed, 224 insertions(+), 1,123 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/10940/1
-- 
To view, visit http://gerrit.cloudera.org:8080/10940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Gerrit-Change-Number: 10940
Gerrit-PatchSet: 1
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>

[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 )

Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2
......................................................................


Patch Set 4: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/10940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Gerrit-Change-Number: 10940
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Sat, 14 Jul 2018 01:03:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 )

Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2816/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/10940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Gerrit-Change-Number: 10940
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Jul 2018 21:45:59 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 )

Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2
......................................................................


Patch Set 2:

Running GVO dry-run to make sure everything is fine.


-- 
To view, visit http://gerrit.cloudera.org:8080/10940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Gerrit-Change-Number: 10940
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Jul 2018 17:58:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 )

Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2
......................................................................


Patch Set 3: Code-Review+2

Carry Phil's + 2.


-- 
To view, visit http://gerrit.cloudera.org:8080/10940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Gerrit-Change-Number: 10940
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Jul 2018 18:04:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 )

Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2
......................................................................


Patch Set 2: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/10940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Gerrit-Change-Number: 10940
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Jul 2018 21:16:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 )

Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2814/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/10940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Gerrit-Change-Number: 10940
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Jul 2018 17:57:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 )

Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2
......................................................................


Patch Set 4: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/10940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Gerrit-Change-Number: 10940
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Jul 2018 21:45:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10940 )

Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2
......................................................................

IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2

This patch removes the use of IMPALA_MINICLUSTER_PROFILE. The code that
uses IMPALA_MINICLUSTER_PROFILE=2 is removed and it defaults to code from
IMPALA_MINICLUSTER_PROFILE=3. In order to reduce having too many code
changes in this patch, there is no code change for the shims. The shims
for IMPALA_MINICLUSTER_PROFILE=3 automatically become the default
implementation.

Testing:
- Ran core and exhaustive tests

Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Reviewed-on: http://gerrit.cloudera.org:8080/10940
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M bin/create-test-configuration.sh
M bin/impala-config.sh
M bin/jenkins/build-all-flag-combinations.sh
M fe/pom.xml
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetCatalogsReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetColumnsReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetFunctionsReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetInfoReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetSchemasReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/authorization/SentryAuthProvider.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/HdfsShim.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/MetastoreShim.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/MiniclusterProfile.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/util/SentryUtil.java
D fe/src/compat-minicluster-profile-2/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java
D fe/src/compat-minicluster-profile-3/java/org/apache/impala/compat/MiniclusterProfile.java
R fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java
R fe/src/main/java/org/apache/impala/authorization/ImpalaActionFactory.java
R fe/src/main/java/org/apache/impala/authorization/ImpalaPrivilegeModel.java
R fe/src/main/java/org/apache/impala/authorization/SentryAuthProvider.java
R fe/src/main/java/org/apache/impala/compat/HdfsShim.java
R fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
R fe/src/main/java/org/apache/impala/util/SentryUtil.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
R fe/src/test/java/org/apache/impala/authorization/ImpalaActionFactoryTest.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
R fe/src/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java
M impala-parent/pom.xml
M testdata/bin/run-hbase.sh
M testdata/bin/run-hive-server.sh
M testdata/bin/run-mini-dfs.sh
M testdata/cluster/node_templates/common/etc/init.d/common.tmpl
M testdata/cluster/node_templates/common/etc/init.d/yarn-common
M testdata/pom.xml
M testdata/workloads/functional-query/queries/QueryTest/views-compatibility.test
M tests/common/environ.py
M tests/metadata/test_views_compatibility.py
M tests/query_test/test_partitioning.py
39 files changed, 237 insertions(+), 1,122 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/10940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Gerrit-Change-Number: 10940
Gerrit-PatchSet: 5
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>

[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 )

Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2
......................................................................


Patch Set 1:

(8 comments)

Thanks. This looked pretty straight-forward to me. I've only got minor comments below.

http://gerrit.cloudera.org:8080/#/c/10940/1/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/10940/1/bin/impala-config.sh@a727
PS1, Line 727: 
             : 
             : 
             : 
             : 
             : 
             : 
             : 
             : 
             : 
             : 
             : 
             : 
             : 
             : 
I think this block should be kept. We presumably want to keep the Java != 1.7 assertion here.


http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh
File bin/jenkins/build-all-flag-combinations.sh:

http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh@a37
PS1, Line 37: 
This one is fine to remove, as "-so" is already covered on line39.


http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh@a42
PS1, Line 42: 
Keep this one, but remove -profile2. This is the only coverage for the ubsan build.


http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh@25
PS1, Line 25: # Usage: build-all-flag-combinations.sh [--dryrun]
GVO will run this for you, but since you're changing it, we should run it as part of our testing (or look explicitly at GVO).


http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh@33
PS1, Line 33: # "minicluster profile" where appropriate.
Please update this comment. (Remove "special sigil...")


http://gerrit.cloudera.org:8080/#/c/10940/1/fe/pom.xml
File fe/pom.xml:

http://gerrit.cloudera.org:8080/#/c/10940/1/fe/pom.xml@22
PS1, Line 22: <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
In addition to the testing you've already done, could you run "mvn help:effective-pom" (https://maven.apache.org/plugins/maven-help-plugin/effective-pom-mojo.html) with the relevant profile and compare before and after? Equivalently, "mvn dependency:tree"?

It would have been easy to drop something accidentally here.


http://gerrit.cloudera.org:8080/#/c/10940/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java:

http://gerrit.cloudera.org:8080/#/c/10940/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@826
PS1, Line 826:         // Hadoop 2 accepts kerberos names missing a realm, but insists
             :         // on having a terminating '@' even when the default realm
             :         // is intended.  Hadoop 3 now has more normal name convetions,
             :         // where to specify the default realm, everything after and
             :         // including the '@' character is omitted.
This comment is a little bit odd without the context of the if in old line 833. Perhaps add "Historical note: " to the comment or something...


http://gerrit.cloudera.org:8080/#/c/10940/1/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
File fe/src/test/java/org/apache/impala/common/FrontendTestBase.java:

http://gerrit.cloudera.org:8080/#/c/10940/1/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java@411
PS1, Line 411:       // Different versions of Hive have slightly different error messages;
             :       // we normalize here as follows:
             :       // 'No FileSystem for Scheme "x"' -> 'No FileSystem for scheme: x'
             :       if (errorString.contains("No FileSystem for scheme ")) {
             :         errorString = errorString.replace("\"", "");
             :         errorString = errorString.replace("No FileSystem for scheme ",
             :             "No FileSystem for scheme: ");
             :       }
I think it's possible to remove this in a future change. No need to do it as part of this change.



-- 
To view, visit http://gerrit.cloudera.org:8080/10940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Gerrit-Change-Number: 10940
Gerrit-PatchSet: 1
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Jul 2018 16:16:07 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 )

Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2
......................................................................


Patch Set 2:

(8 comments)

http://gerrit.cloudera.org:8080/#/c/10940/1/bin/impala-config.sh
File bin/impala-config.sh:

http://gerrit.cloudera.org:8080/#/c/10940/1/bin/impala-config.sh@a727
PS1, Line 727: 
             : 
             : 
             : 
             : 
             : 
             : 
             : 
             : 
             : 
             : 
             : 
             : 
             : 
             : 
> I think this block should be kept. We presumably want to keep the Java != 1
Good catch. Done.


http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh
File bin/jenkins/build-all-flag-combinations.sh:

http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh@a37
PS1, Line 37: 
> This one is fine to remove, as "-so" is already covered on line39.
Done


http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh@a42
PS1, Line 42: 
> Keep this one, but remove -profile2. This is the only coverage for the ubsa
Done


http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh@25
PS1, Line 25: # Usage: build-all-flag-combinations.sh [--dryrun]
> GVO will run this for you, but since you're changing it, we should run it a
Will test this with our dry-run GVO.


http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh@33
PS1, Line 33: CONFIGS=(
> Please update this comment. (Remove "special sigil...")
Done


http://gerrit.cloudera.org:8080/#/c/10940/1/fe/pom.xml
File fe/pom.xml:

http://gerrit.cloudera.org:8080/#/c/10940/1/fe/pom.xml@22
PS1, Line 22: <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
> In addition to the testing you've already done, could you run "mvn help:eff
Good idea. i ran a script to compare the before and after effective POMs and they have the same dependencies.


http://gerrit.cloudera.org:8080/#/c/10940/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java:

http://gerrit.cloudera.org:8080/#/c/10940/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@826
PS1, Line 826:         // Historial note: Hadoop 2 accepts kerberos names missing a realm, but insists
             :         // on having a terminating '@' even when the default realm is intended. Hadoop 3
             :         // now has more normal name conventions, where to specify the default realm,
             :         // everything after and including the '@' character is omitted.
             :         new User(USER.getName() + "/abc.host.com")
> This comment is a little bit odd without the context of the if in old line 
Done


http://gerrit.cloudera.org:8080/#/c/10940/1/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
File fe/src/test/java/org/apache/impala/common/FrontendTestBase.java:

http://gerrit.cloudera.org:8080/#/c/10940/1/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java@411
PS1, Line 411:       // TODO: This logic can be removed.
             :       // Different versions of Hive have slightly different error messages;
             :       // we normalize here as follows:
             :       // 'No FileSystem for Scheme "x"' -> 'No FileSystem for scheme: x'
             :       if (errorString.contains("No FileSystem for scheme ")) {
             :         errorString = errorString.replace("\"", "");
             :         errorString = errorString.replace("No FileSystem for scheme ",
             :        
> I think it's possible to remove this in a future change. No need to do it a
Will put a TODO.



-- 
To view, visit http://gerrit.cloudera.org:8080/10940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Gerrit-Change-Number: 10940
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Jul 2018 17:54:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 )

Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2
......................................................................


Patch Set 2: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/10940/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java:

http://gerrit.cloudera.org:8080/#/c/10940/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@826
PS2, Line 826:         // Historial note: Hadoop 2 accepts kerberos names missing a realm, but insists
nit: historical



-- 
To view, visit http://gerrit.cloudera.org:8080/10940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Gerrit-Change-Number: 10940
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Jul 2018 18:01:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/10940 )

Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2
......................................................................

IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2

This patch removes the use of IMPALA_MINICLUSTER_PROFILE. The code that
uses IMPALA_MINICLUSTER_PROFILE=2 is removed and it defaults to code from
IMPALA_MINICLUSTER_PROFILE=3. In order to reduce having too many code
changes in this patch, there is no code change for the shims. The shims
for IMPALA_MINICLUSTER_PROFILE=3 automatically become the default
implementation.

Testing:
- Ran core and exhaustive tests

Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
---
M bin/create-test-configuration.sh
M bin/impala-config.sh
M bin/jenkins/build-all-flag-combinations.sh
M fe/pom.xml
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetCatalogsReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetColumnsReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetFunctionsReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetInfoReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetSchemasReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/authorization/SentryAuthProvider.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/HdfsShim.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/MetastoreShim.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/MiniclusterProfile.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/util/SentryUtil.java
D fe/src/compat-minicluster-profile-2/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java
D fe/src/compat-minicluster-profile-3/java/org/apache/impala/compat/MiniclusterProfile.java
R fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java
R fe/src/main/java/org/apache/impala/authorization/ImpalaActionFactory.java
R fe/src/main/java/org/apache/impala/authorization/ImpalaPrivilegeModel.java
R fe/src/main/java/org/apache/impala/authorization/SentryAuthProvider.java
R fe/src/main/java/org/apache/impala/compat/HdfsShim.java
R fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
R fe/src/main/java/org/apache/impala/util/SentryUtil.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
R fe/src/test/java/org/apache/impala/authorization/ImpalaActionFactoryTest.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
R fe/src/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java
M impala-parent/pom.xml
M testdata/bin/run-hbase.sh
M testdata/bin/run-hive-server.sh
M testdata/bin/run-mini-dfs.sh
M testdata/cluster/node_templates/common/etc/init.d/common.tmpl
M testdata/cluster/node_templates/common/etc/init.d/yarn-common
M testdata/pom.xml
M testdata/workloads/functional-query/queries/QueryTest/views-compatibility.test
M tests/common/environ.py
M tests/metadata/test_views_compatibility.py
M tests/query_test/test_partitioning.py
39 files changed, 237 insertions(+), 1,122 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/10940/3
-- 
To view, visit http://gerrit.cloudera.org:8080/10940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Gerrit-Change-Number: 10940
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>

[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/10940 )

Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2
......................................................................

IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2

This patch removes the use of IMPALA_MINICLUSTER_PROFILE. The code that
uses IMPALA_MINICLUSTER_PROFILE=2 is removed and it defaults to code from
IMPALA_MINICLUSTER_PROFILE=3. In order to reduce having too many code
changes in this patch, there is no code change for the shims. The shims
for IMPALA_MINICLUSTER_PROFILE=3 automatically become the default
implementation.

Testing:
- Ran core and exhaustive tests

Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
---
M bin/create-test-configuration.sh
M bin/impala-config.sh
M bin/jenkins/build-all-flag-combinations.sh
M fe/pom.xml
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetCatalogsReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetColumnsReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetFunctionsReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetInfoReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetSchemasReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/authorization/SentryAuthProvider.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/HdfsShim.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/MetastoreShim.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/MiniclusterProfile.java
D fe/src/compat-minicluster-profile-2/java/org/apache/impala/util/SentryUtil.java
D fe/src/compat-minicluster-profile-2/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java
D fe/src/compat-minicluster-profile-3/java/org/apache/impala/compat/MiniclusterProfile.java
R fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java
R fe/src/main/java/org/apache/impala/authorization/ImpalaActionFactory.java
R fe/src/main/java/org/apache/impala/authorization/ImpalaPrivilegeModel.java
R fe/src/main/java/org/apache/impala/authorization/SentryAuthProvider.java
R fe/src/main/java/org/apache/impala/compat/HdfsShim.java
R fe/src/main/java/org/apache/impala/compat/MetastoreShim.java
R fe/src/main/java/org/apache/impala/util/SentryUtil.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
R fe/src/test/java/org/apache/impala/authorization/ImpalaActionFactoryTest.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
R fe/src/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java
M impala-parent/pom.xml
M testdata/bin/run-hbase.sh
M testdata/bin/run-hive-server.sh
M testdata/bin/run-mini-dfs.sh
M testdata/cluster/node_templates/common/etc/init.d/common.tmpl
M testdata/cluster/node_templates/common/etc/init.d/yarn-common
M testdata/pom.xml
M testdata/workloads/functional-query/queries/QueryTest/views-compatibility.test
M tests/common/environ.py
M tests/metadata/test_views_compatibility.py
M tests/query_test/test_partitioning.py
39 files changed, 237 insertions(+), 1,122 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/10940/2
-- 
To view, visit http://gerrit.cloudera.org:8080/10940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Gerrit-Change-Number: 10940
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>