You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Steve Carlin (Code Review)" <ge...@cloudera.org> on 2022/08/29 22:11:48 UTC
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Steve Carlin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18927
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
This commit handles the case for a specific kind of corrupt function
within the Hive Metastore in the following situation:
A valid Hive SQL function gets created in HMS. This UDF is written in Java and
must derive from the "UDF" class. After creating this function in Impala, we
then replace the underlying jar file with a class that does NOT derive from
the "UDF" class.
In this scenario, catalogd should reject the function and still start up
gracefully. Before this commit, catalogd wasn't coming up. The reason for this
was because the Hive function FunctionUtils.getUDFClassType() has a dependency
on UDAF and was throwing a LinkageError exception.
To fix this, we need to include UDAF in the shaded jar.
Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
---
M java/pom.xml
M java/shaded-deps/hive-exec/pom.xml
A java/test-corrupt-hive-udfs/pom.xml
A java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
M tests/custom_cluster/test_permanent_udfs.py
5 files changed, 117 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/18927/1
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 1
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function.
......................................................................
Patch Set 8: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 8
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Wed, 31 Aug 2022 15:27:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11245/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Aug 2022 23:16:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11262/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Tue, 30 Aug 2022 19:15:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 7:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/18927/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18927/7//COMMIT_MSG@12
PS7, Line 12: A valid Hive SQL function gets created in HMS. This UDF is written in Java and
nit: plz wrap commit message at 72
http://gerrit.cloudera.org:8080/#/c/18927/7//COMMIT_MSG@22
PS7, Line 22: To fix this, we need to include UDAF in the shaded jar.
This is a parallel fix for the case when the base class is Udaf, right?
http://gerrit.cloudera.org:8080/#/c/18927/7/tests/custom_cluster/test_permanent_udfs.py
File tests/custom_cluster/test_permanent_udfs.py:
http://gerrit.cloudera.org:8080/#/c/18927/7/tests/custom_cluster/test_permanent_udfs.py@a162
PS7, Line 162:
:
Why was this removed? This test should still not work if Hive doesn't support the FS.
+ the new test should also have these markings
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 7
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Wed, 31 Aug 2022 11:17:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Tamas Mate, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18927
to look at the new patch set (#4).
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
This commit handles the case for a specific kind of corrupt function
within the Hive Metastore in the following situation:
A valid Hive SQL function gets created in HMS. This UDF is written in Java and
must derive from the "UDF" class. After creating this function in Impala, we
then replace the underlying jar file with a class that does NOT derive from
the "UDF" class.
In this scenario, catalogd should reject the function and still start up
gracefully. Before this commit, catalogd wasn't coming up. The reason for this
was because the Hive function FunctionUtils.getUDFClassType() has a dependency
on UDAF and was throwing a LinkageError exception.
To fix this, we need to include UDAF in the shaded jar.
Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M java/pom.xml
M java/shaded-deps/hive-exec/pom.xml
A java/test-corrupt-hive-udfs/pom.xml
A java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
M testdata/bin/copy-udfs-udas.sh
M tests/custom_cluster/test_permanent_udfs.py
7 files changed, 122 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/18927/4
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 4
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18927/3/testdata/bin/copy-udfs-udas.sh
File testdata/bin/copy-udfs-udas.sh:
http://gerrit.cloudera.org:8080/#/c/18927/3/testdata/bin/copy-udfs-udas.sh@56
PS3, Line 56: cp target/test-corrupt-hive-udfs-1.0.jar "${IMPALA_HOME}/testdata/udfs/impala-corrupt-hive-udfs.jar"
line too long (102 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 3
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Tue, 30 Aug 2022 15:18:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11256/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 3
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Tue, 30 Aug 2022 15:38:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8507/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Aug 2022 23:01:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/18927/2/java/shaded-deps/hive-exec/pom.xml
File java/shaded-deps/hive-exec/pom.xml:
http://gerrit.cloudera.org:8080/#/c/18927/2/java/shaded-deps/hive-exec/pom.xml@103
PS2, Line 103: UDAF.class
> Shouldn't we add other UDAF classes like UDAFEvaluatorResolver.
The other classes are already imported from line 111.
But you did get me thinking that I should make this a little safer, so I added a catch for LinkageError so that in case anything new is added, we catch it and just skip the function.
http://gerrit.cloudera.org:8080/#/c/18927/2/java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
File java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java:
http://gerrit.cloudera.org:8080/#/c/18927/2/java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java@25
PS2, Line 25: We import the class NumberParseException from
: * Google's phone number library that won't be shaded with the UDF jar
> Where do we import it? CorruptUdf seems a valid udf to me at the first glan
Yeah, I was a bit too quick with the push. I missed a couple of files and I had it on my list to change this comment too.
This version is going to be a valid UDF. The test is going to use this version to create the function with HMS. Then I'm going to replace the jar file containing this with a version of this class that is invalid.
http://gerrit.cloudera.org:8080/#/c/18927/2/tests/custom_cluster/test_permanent_udfs.py
File tests/custom_cluster/test_permanent_udfs.py:
http://gerrit.cloudera.org:8080/#/c/18927/2/tests/custom_cluster/test_permanent_udfs.py@183
PS2, Line 183: funcction
> typo
Done
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Tue, 30 Aug 2022 18:57:49 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18927/6/java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
File java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java:
http://gerrit.cloudera.org:8080/#/c/18927/6/java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java@23
PS6, Line 23: /*
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18927/6/java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java@30
PS6, Line 30: */
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 6
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Tue, 30 Aug 2022 18:58:28 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function.
......................................................................
IMPALA-11528: Catalogd should start up with a corrupt Hive function.
This commit handles the case for a specific kind of corrupt function
within the Hive Metastore in the following situation:
A valid Hive SQL function gets created in HMS. This UDF is written in
Java and must derive from the "UDF" class. After creating this function
in Impala, we then replace the underlying jar file with a class that
does NOT derive from the "UDF" class.
In this scenario, catalogd should reject the function and still start
up gracefully. Before this commit, catalogd wasn't coming up. The
reason for this was because the Hive function
FunctionUtils.getUDFClassType() has a dependency on UDAF and was
throwing a LinkageError exception, so we need to include the UDAF
class in the shaded jar.
Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Reviewed-on: http://gerrit.cloudera.org:8080/18927
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Tamas Mate <tm...@apache.org>
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M java/pom.xml
M java/shaded-deps/hive-exec/pom.xml
A java/test-corrupt-hive-udfs/pom.xml
A java/test-corrupt-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
A java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
M testdata/bin/copy-udfs-udas.sh
M tests/custom_cluster/test_permanent_udfs.py
8 files changed, 153 insertions(+), 1 deletion(-)
Approvals:
Impala Public Jenkins: Verified
Tamas Mate: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 11
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Tamas Mate, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18927
to look at the new patch set (#10).
Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function.
......................................................................
IMPALA-11528: Catalogd should start up with a corrupt Hive function.
This commit handles the case for a specific kind of corrupt function
within the Hive Metastore in the following situation:
A valid Hive SQL function gets created in HMS. This UDF is written in
Java and must derive from the "UDF" class. After creating this function
in Impala, we then replace the underlying jar file with a class that
does NOT derive from the "UDF" class.
In this scenario, catalogd should reject the function and still start
up gracefully. Before this commit, catalogd wasn't coming up. The
reason for this was because the Hive function
FunctionUtils.getUDFClassType() has a dependency on UDAF and was
throwing a LinkageError exception, so we need to include the UDAF
class in the shaded jar.
Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M java/pom.xml
M java/shaded-deps/hive-exec/pom.xml
A java/test-corrupt-hive-udfs/pom.xml
A java/test-corrupt-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
A java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
M testdata/bin/copy-udfs-udas.sh
M tests/custom_cluster/test_permanent_udfs.py
8 files changed, 153 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/18927/10
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 10
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function.
......................................................................
Patch Set 10: Code-Review+2
(1 comment)
Thanks for fixing this Steve!
LGTM!
http://gerrit.cloudera.org:8080/#/c/18927/2/java/shaded-deps/hive-exec/pom.xml
File java/shaded-deps/hive-exec/pom.xml:
http://gerrit.cloudera.org:8080/#/c/18927/2/java/shaded-deps/hive-exec/pom.xml@103
PS2, Line 103: UDAF.class
> I'm a bit confused on this one.
Sorry, I confused myself, for some reason I thought that Impala supports executing UDAFs. Now I see that we won't instantiate any UDAF class, so UDAF dependencies like 'UDAFEvaluatorResolver' and 'DefaultUDAFEvaluatorResolver' won't be accessed
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 10
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Thu, 08 Sep 2022 07:46:16 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function.
......................................................................
Patch Set 7:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/18927/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18927/7//COMMIT_MSG@12
PS7, Line 12: A valid Hive SQL function gets created in HMS. This UDF is written in Java and
> nit: plz wrap commit message at 72
Done
http://gerrit.cloudera.org:8080/#/c/18927/7//COMMIT_MSG@22
PS7, Line 22: To fix this, we need to include UDAF in the shaded jar.
> This is a parallel fix for the case when the base class is Udaf, right?
Actually, this is needed because of what I stated on lines 19 and 20 here. The FunctionUtils.getUDFClassType() method has a line that says "UDAF.class.isAssignable()", so it needs the UDAF object in the shaded jar regardless of whether it derives from a UDAF.
I changed the message so hopefully it is more clear.
http://gerrit.cloudera.org:8080/#/c/18927/7/tests/custom_cluster/test_permanent_udfs.py
File tests/custom_cluster/test_permanent_udfs.py:
http://gerrit.cloudera.org:8080/#/c/18927/7/tests/custom_cluster/test_permanent_udfs.py@a162
PS7, Line 162:
:
> Why was this removed? This test should still not work if Hive doesn't suppo
No reason. This was removed as part of just general testing in my local environment while I was figuring out how to run this when it is not "exhaustive"
On the final push, I"ll put this back and I'll also add this before the "def test_corrupt_java_udf" line as well.
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 7
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Wed, 31 Aug 2022 14:51:59 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function.
......................................................................
Patch Set 8: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 8
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Wed, 31 Aug 2022 20:09:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function.
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11279/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 9
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Fri, 02 Sep 2022 13:57:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function.
......................................................................
Patch Set 10: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 10
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Wed, 07 Sep 2022 20:01:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Tamas Mate, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18927
to look at the new patch set (#6).
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
This commit handles the case for a specific kind of corrupt function
within the Hive Metastore in the following situation:
A valid Hive SQL function gets created in HMS. This UDF is written in Java and
must derive from the "UDF" class. After creating this function in Impala, we
then replace the underlying jar file with a class that does NOT derive from
the "UDF" class.
In this scenario, catalogd should reject the function and still start up
gracefully. Before this commit, catalogd wasn't coming up. The reason for this
was because the Hive function FunctionUtils.getUDFClassType() has a dependency
on UDAF and was throwing a LinkageError exception.
To fix this, we need to include UDAF in the shaded jar.
Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M java/pom.xml
M java/shaded-deps/hive-exec/pom.xml
A java/test-corrupt-hive-udfs/pom.xml
A java/test-corrupt-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
A java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
M testdata/bin/copy-udfs-udas.sh
M tests/custom_cluster/test_permanent_udfs.py
8 files changed, 148 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/18927/6
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 6
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Tamas Mate, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18927
to look at the new patch set (#3).
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
This commit handles the case for a specific kind of corrupt function
within the Hive Metastore in the following situation:
A valid Hive SQL function gets created in HMS. This UDF is written in Java and
must derive from the "UDF" class. After creating this function in Impala, we
then replace the underlying jar file with a class that does NOT derive from
the "UDF" class.
In this scenario, catalogd should reject the function and still start up
gracefully. Before this commit, catalogd wasn't coming up. The reason for this
was because the Hive function FunctionUtils.getUDFClassType() has a dependency
on UDAF and was throwing a LinkageError exception.
To fix this, we need to include UDAF in the shaded jar.
Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M java/pom.xml
M java/shaded-deps/hive-exec/pom.xml
A java/test-corrupt-hive-udfs/pom.xml
A java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
M testdata/bin/copy-udfs-udas.sh
M tests/custom_cluster/test_permanent_udfs.py
7 files changed, 122 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/18927/3
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 3
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11263/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 6
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Tue, 30 Aug 2022 19:20:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 7: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8513/
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 7
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Wed, 31 Aug 2022 03:32:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18927
to look at the new patch set (#2).
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
This commit handles the case for a specific kind of corrupt function
within the Hive Metastore in the following situation:
A valid Hive SQL function gets created in HMS. This UDF is written in Java and
must derive from the "UDF" class. After creating this function in Impala, we
then replace the underlying jar file with a class that does NOT derive from
the "UDF" class.
In this scenario, catalogd should reject the function and still start up
gracefully. Before this commit, catalogd wasn't coming up. The reason for this
was because the Hive function FunctionUtils.getUDFClassType() has a dependency
on UDAF and was throwing a LinkageError exception.
To fix this, we need to include UDAF in the shaded jar.
Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
---
M java/pom.xml
M java/shaded-deps/hive-exec/pom.xml
A java/test-corrupt-hive-udfs/pom.xml
A java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
M tests/custom_cluster/test_permanent_udfs.py
5 files changed, 117 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/18927/2
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11244/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 1
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Aug 2022 22:34:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11264/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 7
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Tue, 30 Aug 2022 21:00:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 2:
(1 comment)
Thanks for picking this up Steve.
http://gerrit.cloudera.org:8080/#/c/18927/2/java/shaded-deps/hive-exec/pom.xml
File java/shaded-deps/hive-exec/pom.xml:
http://gerrit.cloudera.org:8080/#/c/18927/2/java/shaded-deps/hive-exec/pom.xml@103
PS2, Line 103: UDAF.class
Shouldn't we add other UDAF classes like UDAFEvaluatorResolver.
https://github.com/apache/hive/blob/master/udf/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java#L78
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Tue, 30 Aug 2022 07:32:38 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 5:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/18927/5/java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
File java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java:
http://gerrit.cloudera.org:8080/#/c/18927/5/java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java@23
PS5, Line 23: /*
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18927/5/java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java@30
PS5, Line 30: */
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/18927/5/testdata/bin/copy-udfs-udas.sh
File testdata/bin/copy-udfs-udas.sh:
http://gerrit.cloudera.org:8080/#/c/18927/5/testdata/bin/copy-udfs-udas.sh@56
PS5, Line 56: cp target/test-corrupt-hive-udfs-1.0.jar "${IMPALA_HOME}/testdata/udfs/impala-corrupt-hive-udfs.jar"
line too long (102 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Tue, 30 Aug 2022 18:55:43 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Tamas Mate, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18927
to look at the new patch set (#9).
Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function.
......................................................................
IMPALA-11528: Catalogd should start up with a corrupt Hive function.
This commit handles the case for a specific kind of corrupt function
within the Hive Metastore in the following situation:
A valid Hive SQL function gets created in HMS. This UDF is written in
Java and must derive from the "UDF" class. After creating this function
in Impala, we then replace the underlying jar file with a class that
does NOT derive from the "UDF" class.
In this scenario, catalogd should reject the function and still start
up gracefully. Before this commit, catalogd wasn't coming up. The
reason for this was because the Hive function
FunctionUtils.getUDFClassType() has a dependency on UDAF and was
throwing a LinkageError exception, so we need to include the UDAF
class in the shaded jar.
Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M java/pom.xml
M java/shaded-deps/hive-exec/pom.xml
A java/test-corrupt-hive-udfs/pom.xml
A java/test-corrupt-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
A java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
M testdata/bin/copy-udfs-udas.sh
M tests/custom_cluster/test_permanent_udfs.py
8 files changed, 153 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/18927/9
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 9
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18927/4/testdata/bin/copy-udfs-udas.sh
File testdata/bin/copy-udfs-udas.sh:
http://gerrit.cloudera.org:8080/#/c/18927/4/testdata/bin/copy-udfs-udas.sh@56
PS4, Line 56: cp target/test-corrupt-hive-udfs-1.0.jar "${IMPALA_HOME}/testdata/udfs/impala-corrupt-hive-udfs.jar"
line too long (102 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 4
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Tue, 30 Aug 2022 15:19:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Tamas Mate, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18927
to look at the new patch set (#8).
Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function.
......................................................................
IMPALA-11528: Catalogd should start up with a corrupt Hive function.
This commit handles the case for a specific kind of corrupt function
within the Hive Metastore in the following situation:
A valid Hive SQL function gets created in HMS. This UDF is written in
Java and must derive from the "UDF" class. After creating this function
in Impala, we then replace the underlying jar file with a class that
does NOT derive from the "UDF" class.
In this scenario, catalogd should reject the function and still start
up gracefully. Before this commit, catalogd wasn't coming up. The
reason for this was because the Hive function
FunctionUtils.getUDFClassType() has a dependency on UDAF and was
throwing a LinkageError exception, so we need to include the UDAF
class in the shaded jar.
Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M java/pom.xml
M java/shaded-deps/hive-exec/pom.xml
A java/test-corrupt-hive-udfs/pom.xml
A java/test-corrupt-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
A java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
M testdata/bin/copy-udfs-udas.sh
M tests/custom_cluster/test_permanent_udfs.py
8 files changed, 150 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/18927/8
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 8
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function.
......................................................................
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/18927/9/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/18927/9/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1767
PS9, Line 1767: } catch (Exception e) {
: LOG.error("Skipping function load: " + function.getFunctionName(), e);
: }
: }
> nit: could be multi catch parameter:
Done
http://gerrit.cloudera.org:8080/#/c/18927/2/java/shaded-deps/hive-exec/pom.xml
File java/shaded-deps/hive-exec/pom.xml:
http://gerrit.cloudera.org:8080/#/c/18927/2/java/shaded-deps/hive-exec/pom.xml@103
PS2, Line 103: UDAF.class
> I think UDAFEvaluatorResolver won't be included, line 111 points to 'ql/udf
I'm a bit confused on this one.
So I looked in FunctionUtils.java on the Hive side here: https://github.com/apache/hive/blob/release-3.1.2-rc0/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionUtils.java#L93-L108
Am I looking at the wrong code? Can you point me to the code where you see this class being used?
http://gerrit.cloudera.org:8080/#/c/18927/9/java/test-corrupt-hive-udfs/pom.xml
File java/test-corrupt-hive-udfs/pom.xml:
http://gerrit.cloudera.org:8080/#/c/18927/9/java/test-corrupt-hive-udfs/pom.xml@1
PS9, Line 1: <!--
> nit: <?xml version="1.0" encoding="UTF-8"?>
Done
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Wed, 07 Sep 2022 14:58:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function.
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11308/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 10
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Wed, 07 Sep 2022 15:12:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 2: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8507/
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Aug 2022 03:51:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8513/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 7
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Tue, 30 Aug 2022 22:40:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function.
......................................................................
Patch Set 9:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/18927/9/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/18927/9/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1767
PS9, Line 1767: } catch (Exception e) {
: LOG.error("Skipping function load: " + function.getFunctionName(), e);
: } catch (LinkageError e) {
: LOG.error("Skipping function load: " + function.getFunctionName(), e);
nit: could be multi catch parameter:
} catch (Exception | LinkageError e) {
LOG.error("Skipping function load: " + function.getFunctionName(), e);
http://gerrit.cloudera.org:8080/#/c/18927/2/java/shaded-deps/hive-exec/pom.xml
File java/shaded-deps/hive-exec/pom.xml:
http://gerrit.cloudera.org:8080/#/c/18927/2/java/shaded-deps/hive-exec/pom.xml@103
PS2, Line 103: UDAF.class
> The other classes are already imported from line 111.
I think UDAFEvaluatorResolver won't be included, line 111 points to 'ql/udf' but UDAFEvaluatorResolver is under 'ql/exec' package.
http://gerrit.cloudera.org:8080/#/c/18927/9/java/test-corrupt-hive-udfs/pom.xml
File java/test-corrupt-hive-udfs/pom.xml:
http://gerrit.cloudera.org:8080/#/c/18927/9/java/test-corrupt-hive-udfs/pom.xml@1
PS9, Line 1: <!--
nit: <?xml version="1.0" encoding="UTF-8"?>
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 9
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Wed, 07 Sep 2022 08:34:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function.
......................................................................
Patch Set 10:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8540/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 10
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Wed, 07 Sep 2022 15:14:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function.
......................................................................
Patch Set 10:
Submitting as the exhaustive tests were run successfully.
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 10
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Tue, 13 Sep 2022 14:48:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Tamas Mate, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18927
to look at the new patch set (#5).
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
This commit handles the case for a specific kind of corrupt function
within the Hive Metastore in the following situation:
A valid Hive SQL function gets created in HMS. This UDF is written in Java and
must derive from the "UDF" class. After creating this function in Impala, we
then replace the underlying jar file with a class that does NOT derive from
the "UDF" class.
In this scenario, catalogd should reject the function and still start up
gracefully. Before this commit, catalogd wasn't coming up. The reason for this
was because the Hive function FunctionUtils.getUDFClassType() has a dependency
on UDAF and was throwing a LinkageError exception.
To fix this, we need to include UDAF in the shaded jar.
Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M java/pom.xml
M java/shaded-deps/hive-exec/pom.xml
A java/test-corrupt-hive-udfs/pom.xml
A java/test-corrupt-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
A java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
M testdata/bin/copy-udfs-udas.sh
M tests/custom_cluster/test_permanent_udfs.py
8 files changed, 147 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/18927/5
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 2:
(2 comments)
The build failure looks related:
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/17408/testReport/junit/custom_cluster.test_permanent_udfs/TestUdfPersistence/test_corrupt_java_bad_function/
http://gerrit.cloudera.org:8080/#/c/18927/2/java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
File java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java:
http://gerrit.cloudera.org:8080/#/c/18927/2/java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java@25
PS2, Line 25: We import the class NumberParseException from
: * Google's phone number library that won't be shaded with the UDF jar
Where do we import it? CorruptUdf seems a valid udf to me at the first glance.
http://gerrit.cloudera.org:8080/#/c/18927/2/tests/custom_cluster/test_permanent_udfs.py
File tests/custom_cluster/test_permanent_udfs.py:
http://gerrit.cloudera.org:8080/#/c/18927/2/tests/custom_cluster/test_permanent_udfs.py@183
PS2, Line 183: funcction
typo
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Aug 2022 04:18:48 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11257/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 4
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Tue, 30 Aug 2022 15:40:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function.
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11271/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 8
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Wed, 31 Aug 2022 15:10:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function.
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8518/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 8
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Wed, 31 Aug 2022 15:16:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Tamas Mate, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18927
to look at the new patch set (#7).
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
This commit handles the case for a specific kind of corrupt function
within the Hive Metastore in the following situation:
A valid Hive SQL function gets created in HMS. This UDF is written in Java and
must derive from the "UDF" class. After creating this function in Impala, we
then replace the underlying jar file with a class that does NOT derive from
the "UDF" class.
In this scenario, catalogd should reject the function and still start up
gracefully. Before this commit, catalogd wasn't coming up. The reason for this
was because the Hive function FunctionUtils.getUDFClassType() has a dependency
on UDAF and was throwing a LinkageError exception.
To fix this, we need to include UDAF in the shaded jar.
Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M java/pom.xml
M java/shaded-deps/hive-exec/pom.xml
A java/test-corrupt-hive-udfs/pom.xml
A java/test-corrupt-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
A java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java
M testdata/bin/copy-udfs-udas.sh
M tests/custom_cluster/test_permanent_udfs.py
8 files changed, 148 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/18927/7
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 7
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 )
Change subject: IMPALA-11528: Catalogd should start up even with a corrupt Hive function.
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18927/1/tests/custom_cluster/test_permanent_udfs.py
File tests/custom_cluster/test_permanent_udfs.py:
http://gerrit.cloudera.org:8080/#/c/18927/1/tests/custom_cluster/test_permanent_udfs.py@180
PS1, Line 180: =
flake8: E225 missing whitespace around operator
--
To view, visit http://gerrit.cloudera.org:8080/18927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112
Gerrit-Change-Number: 18927
Gerrit-PatchSet: 1
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Aug 2022 22:12:38 +0000
Gerrit-HasComments: Yes