You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Hive QA (JIRA)" <ji...@apache.org> on 2016/05/15 01:25:12 UTC
[jira] [Commented] (HIVE-13149) Remove some unnecessary HMS
connections from HS2
[ https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283749#comment-15283749 ]
Hive QA commented on HIVE-13149:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12803880/HIVE-13149.8.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/281/testReport
Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/281/console
Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-281/
Messages:
{noformat}
**** This message was trimmed, see log for full details ****
[INFO] Excluding org.apache.spark:spark-core_2.10:jar:1.6.0 from the shaded jar.
[INFO] Excluding com.twitter:chill_2.10:jar:0.5.0 from the shaded jar.
[INFO] Excluding com.twitter:chill-java:jar:0.5.0 from the shaded jar.
[INFO] Excluding org.apache.xbean:xbean-asm5-shaded:jar:4.4 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-client:jar:2.6.0 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.6.0 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.6.0 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.6.0 from the shaded jar.
[INFO] Excluding org.apache.spark:spark-launcher_2.10:jar:1.6.0 from the shaded jar.
[INFO] Excluding org.apache.spark:spark-network-common_2.10:jar:1.6.0 from the shaded jar.
[INFO] Excluding org.apache.spark:spark-network-shuffle_2.10:jar:1.6.0 from the shaded jar.
[INFO] Excluding org.apache.spark:spark-unsafe_2.10:jar:1.6.0 from the shaded jar.
[INFO] Excluding org.slf4j:jul-to-slf4j:jar:1.7.10 from the shaded jar.
[INFO] Excluding org.slf4j:jcl-over-slf4j:jar:1.7.10 from the shaded jar.
[INFO] Excluding com.ning:compress-lzf:jar:1.0.3 from the shaded jar.
[INFO] Excluding net.jpountz.lz4:lz4:jar:1.3.0 from the shaded jar.
[INFO] Excluding com.typesafe.akka:akka-remote_2.10:jar:2.3.11 from the shaded jar.
[INFO] Excluding com.typesafe.akka:akka-actor_2.10:jar:2.3.11 from the shaded jar.
[INFO] Excluding com.typesafe:config:jar:1.2.1 from the shaded jar.
[INFO] Excluding org.uncommons.maths:uncommons-maths:jar:1.2.2a from the shaded jar.
[INFO] Excluding com.typesafe.akka:akka-slf4j_2.10:jar:2.3.11 from the shaded jar.
[INFO] Excluding org.scala-lang:scala-library:jar:2.10.4 from the shaded jar.
[INFO] Excluding org.json4s:json4s-jackson_2.10:jar:3.2.10 from the shaded jar.
[INFO] Excluding org.json4s:json4s-core_2.10:jar:3.2.10 from the shaded jar.
[INFO] Excluding org.json4s:json4s-ast_2.10:jar:3.2.10 from the shaded jar.
[INFO] Excluding org.scala-lang:scalap:jar:2.10.0 from the shaded jar.
[INFO] Excluding org.scala-lang:scala-compiler:jar:2.10.0 from the shaded jar.
[INFO] Excluding org.apache.mesos:mesos:jar:shaded-protobuf:0.21.1 from the shaded jar.
[INFO] Excluding com.clearspring.analytics:stream:jar:2.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-graphite:jar:3.1.2 from the shaded jar.
[INFO] Excluding com.fasterxml.jackson.module:jackson-module-scala_2.10:jar:2.4.4 from the shaded jar.
[INFO] Excluding org.scala-lang:scala-reflect:jar:2.10.4 from the shaded jar.
[INFO] Excluding oro:oro:jar:2.0.8 from the shaded jar.
[INFO] Excluding org.tachyonproject:tachyon-client:jar:0.8.2 from the shaded jar.
[INFO] Excluding org.tachyonproject:tachyon-underfs-hdfs:jar:0.8.2 from the shaded jar.
[INFO] Excluding org.tachyonproject:tachyon-underfs-s3:jar:0.8.2 from the shaded jar.
[INFO] Excluding org.tachyonproject:tachyon-underfs-local:jar:0.8.2 from the shaded jar.
[INFO] Excluding net.razorvine:pyrolite:jar:4.9 from the shaded jar.
[INFO] Excluding net.sf.py4j:py4j:jar:0.9 from the shaded jar.
[INFO] Excluding org.spark-project.spark:unused:jar:1.0.0 from the shaded jar.
[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.10 from the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing /data/hive-ptest/working/apache-github-source-source/ql/target/hive-exec-2.1.0-SNAPSHOT.jar with /data/hive-ptest/working/apache-github-source-source/ql/target/hive-exec-2.1.0-SNAPSHOT-shaded.jar
[INFO] Dependency-reduced POM written at: /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at: /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at: /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at: /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at: /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at: /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-exec ---
[INFO] Installing /data/hive-ptest/working/apache-github-source-source/ql/target/hive-exec-2.1.0-SNAPSHOT.jar to /home/hiveptest/.m2/repository/org/apache/hive/hive-exec/2.1.0-SNAPSHOT/hive-exec-2.1.0-SNAPSHOT.jar
[INFO] Installing /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml to /home/hiveptest/.m2/repository/org/apache/hive/hive-exec/2.1.0-SNAPSHOT/hive-exec-2.1.0-SNAPSHOT.pom
[INFO] Installing /data/hive-ptest/working/apache-github-source-source/ql/target/hive-exec-2.1.0-SNAPSHOT-tests.jar to /home/hiveptest/.m2/repository/org/apache/hive/hive-exec/2.1.0-SNAPSHOT/hive-exec-2.1.0-SNAPSHOT-tests.jar
[INFO] Installing /data/hive-ptest/working/apache-github-source-source/ql/target/hive-exec-2.1.0-SNAPSHOT-core.jar to /home/hiveptest/.m2/repository/org/apache/hive/hive-exec/2.1.0-SNAPSHOT/hive-exec-2.1.0-SNAPSHOT-core.jar
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Hive Llap Server 2.1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-llap-server ---
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/llap-server/target
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/llap-server (includes = [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ hive-llap-server ---
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-llap-server ---
[INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/llap-server/src/gen/protobuf/gen-java added.
[INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/llap-server/src/gen/thrift/gen-javabean added.
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-llap-server ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-llap-server ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 19 resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-llap-server ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-llap-server ---
[INFO] Compiling 87 source files to /data/hive-ptest/working/apache-github-source-source/llap-server/target/classes
[WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleAllocator.java:[29,16] sun.misc.Cleaner is internal proprietary API and may be removed in a future release
[WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleAllocator.java:[29,16] sun.misc.Cleaner is internal proprietary API and may be removed in a future release
[WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleAllocator.java:[29,16] sun.misc.Cleaner is internal proprietary API and may be removed in a future release
[WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleAllocator.java:[74,9] sun.misc.Cleaner is internal proprietary API and may be removed in a future release
[WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java: Some input files use or override a deprecated API.
[WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java: Recompile with -Xlint:deprecation for details.
[WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/DirWatcher.java: Some input files use unchecked or unsafe operations.
[WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/DirWatcher.java: Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hive-llap-server ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-llap-server ---
[INFO] Executing tasks
main:
[mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/llap-server/target/tmp
[mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/llap-server/target/warehouse
[mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/llap-server/target/tmp/conf
[copy] Copying 15 files to /data/hive-ptest/working/apache-github-source-source/llap-server/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-llap-server ---
[INFO] Compiling 13 source files to /data/hive-ptest/working/apache-github-source-source/llap-server/target/test-classes
[WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java: Some input files use unchecked or unsafe operations.
[WARNING] /data/hive-ptest/working/apache-github-source-source/llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java: Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-llap-server ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-llap-server ---
[INFO] Building jar: /data/hive-ptest/working/apache-github-source-source/llap-server/target/hive-llap-server-2.1.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hive-llap-server ---
[INFO]
[INFO] --- maven-jar-plugin:2.2:test-jar (default) @ hive-llap-server ---
[INFO] Building jar: /data/hive-ptest/working/apache-github-source-source/llap-server/target/hive-llap-server-2.1.0-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-llap-server ---
[INFO] Installing /data/hive-ptest/working/apache-github-source-source/llap-server/target/hive-llap-server-2.1.0-SNAPSHOT.jar to /home/hiveptest/.m2/repository/org/apache/hive/hive-llap-server/2.1.0-SNAPSHOT/hive-llap-server-2.1.0-SNAPSHOT.jar
[INFO] Installing /data/hive-ptest/working/apache-github-source-source/llap-server/pom.xml to /home/hiveptest/.m2/repository/org/apache/hive/hive-llap-server/2.1.0-SNAPSHOT/hive-llap-server-2.1.0-SNAPSHOT.pom
[INFO] Installing /data/hive-ptest/working/apache-github-source-source/llap-server/target/hive-llap-server-2.1.0-SNAPSHOT-tests.jar to /home/hiveptest/.m2/repository/org/apache/hive/hive-llap-server/2.1.0-SNAPSHOT/hive-llap-server-2.1.0-SNAPSHOT-tests.jar
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Hive Service 2.1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
Downloading: http://repository.apache.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/maven-metadata.xml
[WARNING] Could not transfer metadata org.apache.directory.client.ldap:ldap-client-api:0.1-SNAPSHOT/maven-metadata.xml from/to apache.snapshots (http://repository.apache.org/snapshots): Failed to transfer file: http://repository.apache.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/maven-metadata.xml. Return code is: 503 , ReasonPhrase:Service Unavailable.
[WARNING] Failure to transfer org.apache.directory.client.ldap:ldap-client-api:0.1-SNAPSHOT/maven-metadata.xml from http://repository.apache.org/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced. Original error: Could not transfer metadata org.apache.directory.client.ldap:ldap-client-api:0.1-SNAPSHOT/maven-metadata.xml from/to apache.snapshots (http://repository.apache.org/snapshots): Failed to transfer file: http://repository.apache.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/maven-metadata.xml. Return code is: 503 , ReasonPhrase:Service Unavailable.
Downloading: http://repository.apache.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/ldap-client-api-0.1-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Hive .............................................. SUCCESS [2.605s]
[INFO] Hive Shims Common ................................. SUCCESS [4.784s]
[INFO] Hive Shims 0.23 ................................... SUCCESS [3.335s]
[INFO] Hive Shims Scheduler .............................. SUCCESS [0.910s]
[INFO] Hive Shims ........................................ SUCCESS [0.834s]
[INFO] Hive Storage API .................................. SUCCESS [1.603s]
[INFO] Hive ORC .......................................... SUCCESS [3.186s]
[INFO] Hive Common ....................................... SUCCESS [4.498s]
[INFO] Hive Service RPC .................................. SUCCESS [2.591s]
[INFO] Hive Serde ........................................ SUCCESS [3.850s]
[INFO] Hive Metastore .................................... SUCCESS [16.519s]
[INFO] Hive Ant Utilities ................................ SUCCESS [0.532s]
[INFO] Hive Llap Common .................................. SUCCESS [3.487s]
[INFO] Hive Llap Client .................................. SUCCESS [1.193s]
[INFO] Hive Llap Tez ..................................... SUCCESS [1.920s]
[INFO] Spark Remote Client ............................... SUCCESS [2.839s]
[INFO] Hive Query Language ............................... SUCCESS [51.477s]
[INFO] Hive Llap Server .................................. SUCCESS [2.717s]
[INFO] Hive Service ...................................... FAILURE [1:02.564s]
[INFO] Hive Accumulo Handler ............................. SKIPPED
[INFO] Hive JDBC ......................................... SKIPPED
[INFO] Hive Beeline ...................................... SKIPPED
[INFO] Hive CLI .......................................... SKIPPED
[INFO] Hive Contrib ...................................... SKIPPED
[INFO] Hive HBase Handler ................................ SKIPPED
[INFO] Hive HCatalog ..................................... SKIPPED
[INFO] Hive HCatalog Core ................................ SKIPPED
[INFO] Hive HCatalog Pig Adapter ......................... SKIPPED
[INFO] Hive HCatalog Server Extensions ................... SKIPPED
[INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED
[INFO] Hive HCatalog Webhcat ............................. SKIPPED
[INFO] Hive HCatalog Streaming ........................... SKIPPED
[INFO] Hive HPL/SQL ...................................... SKIPPED
[INFO] Hive HWI .......................................... SKIPPED
[INFO] Hive Llap External Client ......................... SKIPPED
[INFO] Hive Shims Aggregator ............................. SKIPPED
[INFO] Hive TestUtils .................................... SKIPPED
[INFO] Hive Packaging .................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2:52.399s
[INFO] Finished at: Sun May 15 01:25:29 GMT 2016
[INFO] Final Memory: 153M/1215M
[INFO] ------------------------------------------------------------------------
[WARNING] The requested profile "hadoop-1" could not be activated because it does not exist.
[ERROR] Failed to execute goal on project hive-service: Could not resolve dependencies for project org.apache.hive:hive-service:jar:2.1.0-SNAPSHOT: Failed to collect dependencies for [org.apache.hive:hive-exec:jar:2.1.0-SNAPSHOT (compile), org.apache.hive:hive-metastore:jar:2.1.0-SNAPSHOT (compile), org.apache.hive:hive-service-rpc:jar:2.1.0-SNAPSHOT (compile), org.apache.hive:hive-llap-server:jar:2.1.0-SNAPSHOT (compile), commons-codec:commons-codec:jar:1.4 (compile), commons-cli:commons-cli:jar:1.2 (compile), net.sf.jpam:jpam:jar:1.1 (compile), commons-lang:commons-lang:jar:2.6 (compile), org.eclipse.jetty.aggregate:jetty-all:jar:7.6.0.v20120127 (compile), tomcat:jasper-compiler:jar:5.5.23 (compile), tomcat:jasper-runtime:jar:5.5.23 (compile), org.apache.thrift:libfb303:jar:0.9.3 (compile), org.apache.thrift:libthrift:jar:0.9.3 (compile), org.apache.curator:curator-framework:jar:2.6.0 (compile), org.apache.curator:curator-recipes:jar:2.6.0 (compile), org.apache.hadoop:hadoop-common:jar:2.6.0 (compile?), org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.0 (compile?), org.jamon:jamon-runtime:jar:2.3.1 (compile), org.apache.hive:hive-exec:jar:tests:2.1.0-SNAPSHOT (test), org.apache.hive:hive-common:jar:tests:2.1.0-SNAPSHOT (test), junit:junit:jar:4.11 (test), org.apache.directory.client.ldap:ldap-client-api:jar:0.1 (test), org.apache.directory.server:apacheds-server-integ:jar:1.5.6 (test), org.apache.directory.server:apacheds-test-framework:jar:1.5.6 (test), org.slf4j:slf4j-api:jar:1.7.10 (compile)]: Failed to read artifact descriptor for org.apache.directory.client.ldap:ldap-client-api:jar:0.1-SNAPSHOT: Could not transfer artifact org.apache.directory.client.ldap:ldap-client-api:pom:0.1-SNAPSHOT from/to apache.snapshots (http://repository.apache.org/snapshots): Failed to transfer file: http://repository.apache.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/ldap-client-api-0.1-SNAPSHOT.pom. Return code is: 503 , ReasonPhrase:Service Unavailable. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hive-service
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12803880 - PreCommit-HIVE-MASTER-Build
> Remove some unnecessary HMS connections from HS2
> -------------------------------------------------
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
> Affects Versions: 2.0.0
> Reporter: Aihua Xu
> Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13149.1.patch, HIVE-13149.2.patch, HIVE-13149.3.patch, HIVE-13149.4.patch, HIVE-13149.5.patch, HIVE-13149.6.patch, HIVE-13149.7.patch, HIVE-13149.8.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection in {{start(SessionState startSs, boolean isAsync, LogHelper console)}} regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most of the tasks other than some like StatsTask, don't need to access HMS. Currently a new HMS connection will be established for each Task thread. If HiveServer2 is configured to run in parallel and the query involves many tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)