You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "abstractdog (via GitHub)" <gi...@apache.org> on 2025/06/11 12:14:12 UTC
[PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
abstractdog opened a new pull request, #5859:
URL: https://github.com/apache/hive/pull/5859
### What changes were proposed in this pull request?
Include tez protobuf history logging jars into hive-exec fat jar.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #5859:
URL: https://github.com/apache/hive/pull/5859#issuecomment-2976272494
Yes, that is correct. After setting the `hive.hook.proto.base-directory `, i found another issue, which I filed under [HIVE-29012](https://issues.apache.org/jira/browse/HIVE-29012) but even after the potential fix in the respective PR, I am not seeing NoClassDefFoundError π
.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5859:
URL: https://github.com/apache/hive/pull/5859#issuecomment-2964610359
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5859) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5859&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5859&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5859&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5859&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5859&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5859)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2156721478
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
ATM in docker we only set the TEZ_HOME pointing to a TEZ distro.
However, we could include the needed plugin into Hive dist by adding the mvn dependency in the `packaging` module pom.xml.
Alternatively, we can add an extra step in Docker file to copy the needed plugin jar into Hive lib dir.
But we shouldn't embed it into the `hive-exec`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #5859:
URL: https://github.com/apache/hive/pull/5859#issuecomment-2963440019
@abstractdog, I am not able to repro this on hive master branch. in a fresh setup
I ran:
```
schematool -dbType mysql -initSchema
```
and started HS2 and HMS services and then ran
```
schematool -dbType hive -initSchema -metaDbType mysql
```
but the "CREATE DATABASE IF NOT EXISTS SYS" didn't went to HS2 like it happened based on the logs attached in the JIRA. I suspect schematool is running with embedded mode in my case. What am I missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on PR #5859:
URL: https://github.com/apache/hive/pull/5859#issuecomment-2979976869
@abstractdog Are you looking for this :-)
```
ayushsaxena@ayushsaxena apache-hive-4.1.0-SNAPSHOT-bin % bin/schematool -dbType hive -metaDbType derby -initSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/ayushsaxena/code/hive/packaging/target/apache-hive-4.1.0-SNAPSHOT-bin/apache-hive-4.1.0-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.24.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/ayushsaxena/code/incubator/hadoop-3.4.1/hadoop-3.4.1-RC3/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2025-06-16T20:51:52.848390Z main WARN The use of package scanning to locate Log4j plugins is deprecated.
Please remove the `packages` attribute from your configuration file.
See https://logging.apache.org/log4j/2.x/faq.html#package-scanning for details.
2025-06-16T20:51:52.908646Z main INFO Starting configuration org.apache.logging.log4j.core.config.properties.PropertiesConfiguration@56673b2c...
2025-06-16T20:51:52.908753Z main INFO Start watching for changes to jar:file:/Users/ayushsaxena/code/hive/packaging/target/apache-hive-4.1.0-SNAPSHOT-bin/apache-hive-4.1.0-SNAPSHOT-bin/lib/hive-common-4.1.0-SNAPSHOT.jar!/hive-log4j2.properties every 0 seconds
2025-06-16T20:51:52.908952Z main INFO Configuration org.apache.logging.log4j.core.config.properties.PropertiesConfiguration@56673b2c started.
2025-06-16T20:51:52.910037Z main INFO Stopping configuration org.apache.logging.log4j.core.config.DefaultConfiguration@3d5c822d...
2025-06-16T20:51:52.910209Z main INFO Configuration org.apache.logging.log4j.core.config.DefaultConfiguration@3d5c822d stopped.
Initializing the schema to: 4.1.0
Metastore connection URL: jdbc:hive2://?hive.conf.restricted.list=;hive.security.authorization.sqlstd.confwhitelist=.*;hive.security.authorization.sqlstd.confwhitelist.append=.*;hive.security.authorization.enabled=false;hive.metastore.uris=;hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactory;hive.support.concurrency=false;hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager;hive.metastore.rawstore.impl=org.apache.hadoop.hive.metastore.ObjectStore
Metastore connection Driver : org.apache.hive.jdbc.HiveDriver
Metastore connection User: APP
Hive Session ID = bd67b1bc-3dcb-4645-99da-d29d516920d5
Starting metastore schema initialization to 4.1.0
Initialization script hive-schema-4.1.0.hive.sql
Hive Session ID = 9479a44c-4250-4269-90a1-368b1c70f8aa
2025-06-16T20:51:56.103786Z main ERROR Log4j2 ConfigurationScheduler attempted to increment scheduled items after start
FAILED: Hive Internal Error: java.lang.NoClassDefFoundError(org/apache/tez/dag/history/logging/proto/DatePartitionedLogger)
java.lang.NoClassDefFoundError: org/apache/tez/dag/history/logging/proto/DatePartitionedLogger
at org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.<init>(HiveProtoLoggingHook.java:207)
at org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.getInstance(HiveProtoLoggingHook.java:545)
at org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook.run(HiveProtoLoggingHook.java:557)
at org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:269)
at org.apache.hadoop.hive.ql.HookRunner.runPreHooks(HookRunner.java:248)
at org.apache.hadoop.hive.ql.Executor.preExecutionActions(Executor.java:153)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:101)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:345)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:189)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:142)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:137)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:190)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:234)
at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:334)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:354)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.ClassNotFoundException: org.apache.tez.dag.history.logging.proto.DatePartitionedLogger
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
... 23 more
Error: Error while compiling statement: FAILED: Hive Internal Error: java.lang.NoClassDefFoundError(org/apache/tez/dag/history/logging/proto/DatePartitionedLogger); Query ID: ayushsaxena_20250617022156_76f0c477-9e01-47ae-a564-c3ab0a1f9472 (state=08S01,code=12)
Schema initialization FAILED! Metastore state would be inconsistent!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
Use --verbose for detailed stacktrace.
*** schemaTool failed ***
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #5859:
URL: https://github.com/apache/hive/pull/5859#issuecomment-2980591180
LGTM +1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2157519955
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
3) why do we need filter? that the only package in that jar
````
<filter>
<artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
<includes>
<include>org/apache/tez/dag/history/logging/proto/**</include>
</includes>
</filter>
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2158204496
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
@deniskuzZ : let me respond here the quesions above
> Why is the tez-protobuf-history-plugin needed in schematool to begin with? What purpose does it serve?
schematool runs hive-ql codepath, so it can hit anything which is used around the Driver, that's how query hooks came into the picture, see this trace below:
```
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:1877)
at org.apache.hadoop.hive.ql.ddl.database.use.SwitchDatabaseAnalyzer.analyzeInternal(SwitchDatabaseAnalyzer.java:45)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:324)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:227)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:108)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:203)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:659)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:604)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:598)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:210)
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:274)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:288)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:574)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:560)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:583)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:2168)
at com.sun.proxy.$Proxy47.ExecuteStatement(Unknown Source)
at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:374)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:316)
at org.apache.hive.jdbc.HiveConnection.setSchema(HiveConnection.java:2084)
at org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:90)
at org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:103)
at org.apache.hadoop.hive.metastore.CDHMetaStoreSchemaInfo.getMetaStoreSchemaVersion(CDHMetaStoreSchemaInfo.java:323)
at org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolTaskInitOrUpgrade.execute(SchemaToolTaskInitOrUpgrade.java:41)
at org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.run(MetastoreSchemaTool.java:486)
at org.apache.hive.beeline.schematool.HiveSchemaTool.main(HiveSchemaTool.java:143)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:321)
at org.apache.hadoop.util.RunJar.main(RunJar.java:237)
```
> in a root pom we define multiple exclusions for tez-protobuf-history-plugin, can't we exclude all the transitive dependencies like
let me check
> I noticed those classes were already part of hive-exec before [HIVE-28028](https://issues.apache.org/jira/browse/HIVE-28028), which removed them and replaced with the tez-protobuf-history-plugin. So, basically, this PR doesn't add any extra classes.
before HIVE-28028 hive had these classes as sources copied, so it's a major difference that now we shade the binaries (I didn't like that copied and version controlled them independently)
> why do we need filter? that's the only package in that artifact
good catch, I haven't checked, I'm removing the include
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DatePartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5859:
URL: https://github.com/apache/hive/pull/5859#issuecomment-2991468107
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5859) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5859&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5859&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5859&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5859&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5859&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5859)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2156721478
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
ATM in docker we only set the TEZ_HOME pointing to a TEZ distro, however, we could simply add a step to copy the needed plugin jar into Hive lib dir or include it into Hive dist by adding the plugin dependency in packaging module pom.xml
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
ATM in docker we only set the TEZ_HOME pointing to a TEZ distro, however, we could add a step to copy the needed plugin jar into Hive lib dir or include it into Hive dist by adding the plugin dependency in packaging module pom.xml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5859:
URL: https://github.com/apache/hive/pull/5859#issuecomment-2967642268
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5859) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5859&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5859&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5859&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5859&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5859&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5859)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #5859:
URL: https://github.com/apache/hive/pull/5859#issuecomment-2976487433
> > Yes, that is correct. After setting the `hive.hook.proto.base-directory `, i found another issue, which I filed under [HIVE-29012](https://issues.apache.org/jira/browse/HIVE-29012) but even after the potential fix in the respective PR, I am not seeing NoClassDefFoundError π
.
> > [updated_schemtool_logs.txt](https://github.com/user-attachments/files/20756399/updated_schemtool_logs.txt)
>
> hm, in this case, one of the jars on the classpath must contain DatePartitionedLogger, otherwise it should fail, can you please check that?
hmm, the following line in logs in schematool as well confirms that post hooks is getting triggered but in the classpath (attached above), using the find command with jar tvf , I am not able to find tez-protobuf-history-plugin jar or DatePartitionedLogger.class:
```
2025-06-16 17:34:20,489 INFO ql.Driver: Executing command(queryId=raghav_20250616173420_39d18e25-c007-483c-b692-0f6dfc39832f): SELECT t.SCHEMA_VERSION from VERSION t
2025-06-16 17:34:20,489 INFO hooks.HiveProtoLoggingHook: Received pre-hook notification for: raghav_20250616173420_39d18e25-c007-483c-b692-0f6dfc39832f
2025-06-16 17:34:20,490 INFO hooks.HiveProtoLoggingHook: Received post-hook notification for: raghav_20250616173420_39d18e25-c007-483c-b692-0f6dfc39832f
2025-06-16 17:34:20,490 INFO metadata.Hive: Dumping metastore api call timing information for : execution phase
```
I will also debug and see how schematool is working or if something else is happening altogether,
**but if you can request someone as well to test on hive4 setup that will be great, it will eliminate setup issues if any.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #5859:
URL: https://github.com/apache/hive/pull/5859#issuecomment-2967530284
@abstractdog , what you said above is correct but as per my understanding, schematool will launch 1 jvm which will have embedded HS2 and HMS and based on the HS2 logs attached in the JIRA, it looks like user tried to ran the content of the _hive-schema-*.hive.sql_ from the beeline.
1. I added `set -x` in `schematool.sh` and captured the classpath and captured
```schematool -dbType hive -initSchema -metaDbType mysql``` logs as well by adding
`org.apache.hadoop.hive.ql.hooks.HiveProtologgingHook` in HiveConf.java (pre, post and failure hooks) itself to be picked up by embedded HS2 and HMS but still didn't receive NoClassDefFoundError. Attaching the same.
[schema_logs.txt](https://github.com/user-attachments/files/20712770/schema_logs.txt)
[classpath.txt](https://github.com/user-attachments/files/20712771/classpath.txt)
2. Can you please share the steps to repro this issue in hive4 if possible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DatePartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog merged PR #5859:
URL: https://github.com/apache/hive/pull/5859
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2156721478
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
ATM in docker we only set the TEZ_HOME pointing to a TEZ distro, however, we could simply add a step to copy the needed plugin jar into Hive lib dir.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2157474971
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
Why is the tez-protobuf-history-plugin needed in schematool to begin with? What purpose does it serve?
I noticed those classes were already part of hive-exec before [HIVE-28028](https://issues.apache.org/jira/browse/HIVE-28028), which removed them and replaced with the tez-protobuf-history-plugin. So, basically, this PR doesn't add any extra classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2158204496
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
@deniskuzZ : let me respond here the quesions above
> Why is the tez-protobuf-history-plugin needed in schematool to begin with? What purpose does it serve?
schematool runs hive-ql codepath, so it can hit anything which is used around the Driver, that's how query hooks came into the picture, see this trace below:
```
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:1877)
at org.apache.hadoop.hive.ql.ddl.database.use.SwitchDatabaseAnalyzer.analyzeInternal(SwitchDatabaseAnalyzer.java:45)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:324)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:227)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:108)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:203)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:659)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:604)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:598)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:210)
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:274)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:288)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:574)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:560)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:583)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:2168)
at com.sun.proxy.$Proxy47.ExecuteStatement(Unknown Source)
at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:374)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:316)
at org.apache.hive.jdbc.HiveConnection.setSchema(HiveConnection.java:2084)
at org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:90)
at org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:103)
at org.apache.hadoop.hive.metastore.CDHMetaStoreSchemaInfo.getMetaStoreSchemaVersion(CDHMetaStoreSchemaInfo.java:323)
at org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolTaskInitOrUpgrade.execute(SchemaToolTaskInitOrUpgrade.java:41)
at org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.run(MetastoreSchemaTool.java:486)
at org.apache.hive.beeline.schematool.HiveSchemaTool.main(HiveSchemaTool.java:143)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:321)
at org.apache.hadoop.util.RunJar.main(RunJar.java:237)
```
> in a root pom we define multiple exclusions for tez-protobuf-history-plugin, can't we exclude all the transitive dependencies like
let me check
> I noticed those classes were already part of hive-exec before [HIVE-28028](https://issues.apache.org/jira/browse/HIVE-28028), which removed them and replaced with the tez-protobuf-history-plugin. So, basically, this PR doesn't add any extra classes.
before HIVE-28028 hive had these classes as sources copied, so it's a major difference that now we shade the binaries (I didn't like that copied and version controlled them independently, which is something we only do in crazy situations when there is no time for waiting for a tez release or something)
> why do we need filter? that's the only package in that artifact
good catch, I haven't checked, I'm removing the include
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2156721478
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
ATM in docker we only set the TEZ_HOME pointing to a TEZ distro.
However, we could include the needed plugin into Hive dist by adding the mvn dependency in the `packaging` module pom.xml.
Alternatively, we can add an extra step in Docker file to copy the needed plugin jar into Hive lib dir.
But, I think, we shouldn't embed it into the `hive-exec`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2157511186
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
can we drop `org.apache.hive.shims:hive-shims-0.23` from <include>, it's added twice
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
can we drop `org.apache.hive.shims:hive-shims-0.23` from `<include>`, it's added twice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5859:
URL: https://github.com/apache/hive/pull/5859#issuecomment-2976206982
> @abstractdog , what you said above is correct but as per my understanding, schematool will launch 1 jvm which will have embedded HS2 and HMS and based on the HS2 logs attached in the JIRA, it looks like user tried to ran the content of the _hive-schema-*.hive.sql_ from the beeline.
>
> 1. I added `set -x` in `schematool.sh` and captured the classpath and captured
> `schematool -dbType hive -initSchema -metaDbType mysql` logs as well by adding
> `org.apache.hadoop.hive.ql.hooks.HiveProtologgingHook` in HiveConf.java (pre, post and failure hooks) itself to be picked up by embedded HS2 and HMS but still didn't receive NoClassDefFoundError. Attaching the same.
>
> [schema_logs.txt](https://github.com/user-attachments/files/20712770/schema_logs.txt) [classpath.txt](https://github.com/user-attachments/files/20712771/classpath.txt)
>
> 2. Can you please share the steps to repro this issue in hive4 if possible?
thanks for taking a look
I can see this in the logs you attached:
```
2025-06-12 21:28:38,225 ERROR hooks.HiveProtoLoggingHook: hive.hook.proto.base-directory is not set, logging disabled.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5859:
URL: https://github.com/apache/hive/pull/5859#issuecomment-2976514465
> > > Yes, that is correct. After setting the `hive.hook.proto.base-directory `, i found another issue, which I filed under [HIVE-29012](https://issues.apache.org/jira/browse/HIVE-29012) but even after the potential fix in the respective PR, I am not seeing NoClassDefFoundError π
.
> > > [updated_schemtool_logs.txt](https://github.com/user-attachments/files/20756399/updated_schemtool_logs.txt)
> >
> >
> > hm, in this case, one of the jars on the classpath must contain DatePartitionedLogger, otherwise it should fail, can you please check that?
>
> hmm, the following line in logs in schematool as well confirms that post hooks is getting triggered but in the classpath (attached above), using the find command with jar tvf , I am not able to find tez-protobuf-history-plugin jar or DatePartitionedLogger.class:
>
> ```
> 2025-06-16 17:34:20,489 INFO ql.Driver: Executing command(queryId=raghav_20250616173420_39d18e25-c007-483c-b692-0f6dfc39832f): SELECT t.SCHEMA_VERSION from VERSION t
> 2025-06-16 17:34:20,489 INFO hooks.HiveProtoLoggingHook: Received pre-hook notification for: raghav_20250616173420_39d18e25-c007-483c-b692-0f6dfc39832f
> 2025-06-16 17:34:20,490 INFO hooks.HiveProtoLoggingHook: Received post-hook notification for: raghav_20250616173420_39d18e25-c007-483c-b692-0f6dfc39832f
> 2025-06-16 17:34:20,490 INFO metadata.Hive: Dumping metastore api call timing information for : execution phase
> ```
>
> I will also debug and see how schematool is working or if something else is happening altogether, **but if you can request someone as well to test on hive4 setup that will be great, it will eliminate setup issues if any.**
We could only test and encounter this issue on a downstream cluster, which has diverged from Hive 4.
In this case, I can only rely on the fact that this patch resolved the problem in our environment, but I agreeβa reproducible case on Hive master would definitely make the patch more trustworthy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5859:
URL: https://github.com/apache/hive/pull/5859#issuecomment-2965798025
> @abstractdog, I am not able to repro this on hive master branch. in a fresh setup I ran:
>
> ```
> schematool -dbType mysql -initSchema
> ```
>
> and started HS2 and HMS services and then ran
>
> ```
> schematool -dbType hive -initSchema -metaDbType mysql
> ```
>
> but the "CREATE DATABASE IF NOT EXISTS SYS" didn't went to HS2 like it happened based on the logs attached in the JIRA. I suspect schematool is running with embedded mode in my case. What am I missing?
the logs are confusing, I was shocked to see things like "HiveServer2-Background-Pool" in a metastore log or metastore-related log, but I guess as long as the hive executable runs, it will utilize a Hive CLI call to do things, and for that, the Driver is the entrypoint (so it runs the hooks too), and that process doesn't have the same classpath as a fully fledged HS2 process
on a Cloudera Manager based cluster, the tez-protobuf-history-plugin jar is already added for HS2 classpath, that's why we haven't hit that problem there
I'm wondering what's the classpath when you call schematool and if you actually hit the Driver classpath with the hooks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2155406124
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
That would include the Tez plugin into the `hive-exec` jar, isn't it? Should we include tez-protobuf-history-plugin into the final classpath in packaging or docker?
fat jar is only used in Hive; other components use the `core` classifier
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5859:
URL: https://github.com/apache/hive/pull/5859#issuecomment-2980020436
> bin/schematool -dbType hive -metaDbType derby -initSchema
nice repro! yes, this is what we hit downstream too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2156721478
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
ATM in docker we only set the TEZ_HOME pointing to a TEZ distro.
However, we could include the needed plugin into Hive dist by adding the mvn dependency in the `packaging` module pom.xml.
Or we can add an extra step in Docker file to copy the needed plugin jar into Hive lib dir.
But we shouldn't embed it into hive-exec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2157489271
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
in a root pom we define multiple exclusions for `tez-protobuf-history-plugin`, can't we exclude all the transitive dependencies like
````
<dependency>
<groupId>org.apache.tez</groupId>
<artifactId>tez-protobuf-history-plugin</artifactId>
<version>${tez.version}</version>
<exclusions>
<exclusion>
<groupId>*</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DatePartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5859:
URL: https://github.com/apache/hive/pull/5859#issuecomment-2992348800
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5859) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5859&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5859&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5859&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5859&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5859&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5859)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5859:
URL: https://github.com/apache/hive/pull/5859#issuecomment-2976322118
> Yes, that is correct. After setting the `hive.hook.proto.base-directory `, i found another issue, which I filed under [HIVE-29012](https://issues.apache.org/jira/browse/HIVE-29012) but even after the potential fix in the respective PR, I am not seeing NoClassDefFoundError π
.
>
> [updated_schemtool_logs.txt](https://github.com/user-attachments/files/20756399/updated_schemtool_logs.txt)
hm, in this case, one of the jars on the classpath must contain DatePartitionedLogger, otherwise it should fail, can you please check that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2157433154
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
hm, I don't get why we wouldn't simply add a few classes to hive-exec jar? that's what we've been doing for a long time:
https://github.com/apache/hive/blob/0c69f10f14b0f4b1983093de105ed6b35d4d483b/ql/pom.xml#L1047-L1106
basically, here we have 2 options:
1) add this include into hive-exec, making the classes available everywhere where hive ql is available
2) or take care of the classpath everywhere, this is at least 2-3 places:
2a) upstream packaging and/or Docker
2b) the classpath of the downstream process (schematool with hive ql usage embedded), where this issue was originally reported
2c) downstream HS2 classpaths theoretically: important to note that downstream clusters haven't started to fail immediately after HIVE-28028 just because (e.g. on the cluster I was working on) the HS2 classpath included tez-protobuf-history-plugin, no matter why it was done like that, looked strange as I expect only tez-api to be present on HS2 classpath
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2158204496
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
@deniskuzZ : let me respond here the quesions above
> Why is the tez-protobuf-history-plugin needed in schematool to begin with? What purpose does it serve?
schematool runs hive-ql codepath, so it can hit anything which is used around the Driver, that's how query hooks came into the picture, see this trace below:
```
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:1877)
at org.apache.hadoop.hive.ql.ddl.database.use.SwitchDatabaseAnalyzer.analyzeInternal(SwitchDatabaseAnalyzer.java:45)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:324)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:227)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:108)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:203)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:659)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:604)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:598)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:210)
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:274)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:288)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:574)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:560)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:583)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:2168)
at com.sun.proxy.$Proxy47.ExecuteStatement(Unknown Source)
at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:374)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:316)
at org.apache.hive.jdbc.HiveConnection.setSchema(HiveConnection.java:2084)
at org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:90)
at org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:103)
at org.apache.hadoop.hive.metastore.CDHMetaStoreSchemaInfo.getMetaStoreSchemaVersion(CDHMetaStoreSchemaInfo.java:323)
at org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolTaskInitOrUpgrade.execute(SchemaToolTaskInitOrUpgrade.java:41)
at org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.run(MetastoreSchemaTool.java:486)
at org.apache.hive.beeline.schematool.HiveSchemaTool.main(HiveSchemaTool.java:143)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:321)
at org.apache.hadoop.util.RunJar.main(RunJar.java:237)
```
_in a root pom we define multiple exclusions for tez-protobuf-history-plugin, can't we exclude all the transitive dependencies like_
let me check
_I noticed those classes were already part of hive-exec before [HIVE-28028](https://issues.apache.org/jira/browse/HIVE-28028), which removed them and replaced with the tez-protobuf-history-plugin. So, basically, this PR doesn't add any extra classes._
before HIVE-28028 hive had these classes as sources copied, so it's a major difference that now we shade the binaries (I didn't like that copied and version controlled them independently)
_why do we need filter? that's the only package in that artifact_
good catch, I haven't checked, I'm removing the include
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2158204496
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
@deniskuzZ : let me respond here the quesions above
> Why is the tez-protobuf-history-plugin needed in schematool to begin with? What purpose does it serve?
schematool runs hive-ql codepath, so it can hit anything which is used around the Driver, that's how query hooks came into the picture, see this trace below:
```
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:1877)
at org.apache.hadoop.hive.ql.ddl.database.use.SwitchDatabaseAnalyzer.analyzeInternal(SwitchDatabaseAnalyzer.java:45)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:324)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:227)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:108)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:203)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:659)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:604)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:598)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:210)
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:274)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:288)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:574)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:560)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:583)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:2168)
at com.sun.proxy.$Proxy47.ExecuteStatement(Unknown Source)
at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:374)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:316)
at org.apache.hive.jdbc.HiveConnection.setSchema(HiveConnection.java:2084)
at org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:90)
at org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:103)
at org.apache.hadoop.hive.metastore.CDHMetaStoreSchemaInfo.getMetaStoreSchemaVersion(CDHMetaStoreSchemaInfo.java:323)
at org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolTaskInitOrUpgrade.execute(SchemaToolTaskInitOrUpgrade.java:41)
at org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.run(MetastoreSchemaTool.java:486)
at org.apache.hive.beeline.schematool.HiveSchemaTool.main(HiveSchemaTool.java:143)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:321)
at org.apache.hadoop.util.RunJar.main(RunJar.java:237)
```
> in a root pom we define multiple exclusions for tez-protobuf-history-plugin, can't we exclude all the transitive dependencies like
let me check
> I noticed those classes were already part of hive-exec before [HIVE-28028](https://issues.apache.org/jira/browse/HIVE-28028), which removed them and replaced with the tez-protobuf-history-plugin. So, basically, this PR doesn't add any extra classes.
before HIVE-28028 hive had these classes as sources copied, so it's a major difference that now we shade the binaries (I didn't like that copied and version controlled them independently)
> why do we need filter? that's the only package in that artifact
good catch, I haven't checked, I'm removing the include
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2158204496
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
@deniskuzZ : let me respond here the quesions above
_Why is the tez-protobuf-history-plugin needed in schematool to begin with? What purpose does it serve?_
schematool runs hive-ql codepath, so it can hit anything which is used around the Driver, that's how query hooks came into the picture, see this trace below:
```
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:1877)
at org.apache.hadoop.hive.ql.ddl.database.use.SwitchDatabaseAnalyzer.analyzeInternal(SwitchDatabaseAnalyzer.java:45)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:324)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:227)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:108)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:203)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:659)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:604)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:598)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:210)
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:274)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:288)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:574)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:560)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:583)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:2168)
at com.sun.proxy.$Proxy47.ExecuteStatement(Unknown Source)
at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:374)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:316)
at org.apache.hive.jdbc.HiveConnection.setSchema(HiveConnection.java:2084)
at org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:90)
at org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:103)
at org.apache.hadoop.hive.metastore.CDHMetaStoreSchemaInfo.getMetaStoreSchemaVersion(CDHMetaStoreSchemaInfo.java:323)
at org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolTaskInitOrUpgrade.execute(SchemaToolTaskInitOrUpgrade.java:41)
at org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.run(MetastoreSchemaTool.java:486)
at org.apache.hive.beeline.schematool.HiveSchemaTool.main(HiveSchemaTool.java:143)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:321)
at org.apache.hadoop.util.RunJar.main(RunJar.java:237)
```
_in a root pom we define multiple exclusions for tez-protobuf-history-plugin, can't we exclude all the transitive dependencies like_
let me check
_I noticed those classes were already part of hive-exec before [HIVE-28028](https://issues.apache.org/jira/browse/HIVE-28028), which removed them and replaced with the tez-protobuf-history-plugin. So, basically, this PR doesn't add any extra classes._
before HIVE-28028 hive had these classes as sources copied, so it's a major difference that now we shade the binaries (I didn't like that copied and version controlled them independently)
_why do we need filter? that's the only package in that artifact_
good catch, I haven't checked, I'm removing the include
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2156696224
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
yeah, this includes the jar to hive-exec.jar so the problem is solved whereever it's used, I guess we assemble that jar to the docker image too, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2155406124
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
That would include the Tez plugin into the `hive-exec` jar, isn't it? Should we include tez-protobuf-history-plugin into the final classpath in packaging or docker?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2156721478
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
ATM in docker we only set the TEZ_HOME pointing to a TEZ distro, however, we could add a step to copy the needed plugin jar into Hive lib dir or include it into Hive dist by adding the plugin dependency in packaging module pom.xml. But we shouldn't embed it into hive-exec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2157489271
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
1) in a root pom we define multiple exclusions for `tez-protobuf-history-plugin`, can't we exclude all the transitive dependencies like
````
<dependency>
<groupId>org.apache.tez</groupId>
<artifactId>tez-protobuf-history-plugin</artifactId>
<version>${tez.version}</version>
<exclusions>
<exclusion>
<groupId>*</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
````
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
2) can we drop `org.apache.hive.shims:hive-shims-0.23` from `<include>`, it's added twice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2157519955
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
3) why do we need filter? that's the only package in that artifact
````
<filter>
<artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
<includes>
<include>org/apache/tez/dag/history/logging/proto/**</include>
</includes>
</filter>
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2157519955
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
3) why do we need filter? that the only package in that artifact
````
<filter>
<artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
<includes>
<include>org/apache/tez/dag/history/logging/proto/**</include>
</includes>
</filter>
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2157519955
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
3) why do we need filter
````
<filter>
<artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
<includes>
<include>org/apache/tez/dag/history/logging/proto/**</include>
</includes>
</filter>
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2156721478
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
ATM in docker we only set the TEZ_HOME pointing to a TEZ distro.
However, we could include the needed plugin into Hive dist by adding the mvn dependency in the `packaging` module pom.xml.
Alternatively, we can add an extra step in Docker file to copy the needed plugin jar into Hive lib dir.
But we shouldn't embed it into hive-exec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2156721478
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>org.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
ATM in docker we only set the TEZ_HOME pointing to a TEZ distro, however, we could include the plugin into Hive dist by adding the the dependency in packaging module pom.xml or add a step in Docker file to copy the needed plugin jar into Hive lib dir.
But we shouldn't embed it into hive-exec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2140558637
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>oorg.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
Shouldn't it be "org" instead of "oorg"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5859:
URL: https://github.com/apache/hive/pull/5859#discussion_r2141959852
##########
ql/pom.xml:
##########
@@ -1142,6 +1143,12 @@
<exclude>META-INF/licenses/slf4j*/**</exclude>
</excludes>
</filter>
+ <filter>
+ <artifact>oorg.apache.tez:tez-protobuf-history-plugin</artifact>
Review Comment:
yeah, silly typo :) let me fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29003: DataPartitionedLogger from Tez is always needed for hive logging hook [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #5859:
URL: https://github.com/apache/hive/pull/5859#issuecomment-2980590785
Thanks @ayushtkn, for provinding the repro.
@abstractdog, I debugged and found the reason why it is not happening in my setup.
**Setting `set -x` in schematool.sh is not the right way to identify the classpath**. I modified the _HiveSchemaTool#main()_ to include
```
String classpath = System.getProperty("java.class.path");
System.out.println("Classpath: " + classpath);
```
there I can see the tez-protobuf-history-plugin jar. This is coming because in my setup I have set the following in `$HADOOP_HOME/etc/hadoop/hadoop-env.sh`
```export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*:$HADOOP_CLASSPATH```
Attaching updated classpath for reference:
[classpath.txt](https://github.com/user-attachments/files/20777364/classpath.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org