You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tephra.apache.org by "Francis Chuang (JIRA)" <ji...@apache.org> on 2016/09/02 00:35:20 UTC
[jira] [Created] (TEPHRA-179) Tephra transaction manager breaks on
zookeeper restart
Francis Chuang created TEPHRA-179:
-------------------------------------
Summary: Tephra transaction manager breaks on zookeeper restart
Key: TEPHRA-179
URL: https://issues.apache.org/jira/browse/TEPHRA-179
Project: Tephra
Issue Type: Bug
Components: manager
Affects Versions: 0.8.0-incubating
Environment: OpenJDK 8 (JDK) on Alpine Linux 3.4 in Docker
Reporter: Francis Chuang
Assignee: Poorna Chandra
I am running HBase 1.2.2 with Phoenix 4.8.0 with the tephra transaction server in 1 docker container. In another docker container, I have Zookeeper 3.4.8 manage by Netflix Exhibitor.
When everything first starts, I am able to create transactional table and run transactional queries.
However, once Exhibitor restarts zookeeper and tephra reconnects to zookeeper, it no longer works correctly.
Running transactional queries result in this error:
{code}
Error: Error -1 (00000) : Error while executing SQL "CREATE TABLE my_table321 (k BIGINT PRIMARY KEY, v VARCHAR) TRANSACTIONAL=true": Remote driver error: RuntimeException: java.lang.Exception: Thrift error for org.apache.tephra.distributed.TransactionServiceClient$2@2361d7ab: Internal error processing startShort -> Exception: Thrift error for org.apache.tephra.distributed.TransactionServiceClient$2@2361d7ab: Internal error processing startShort -> TApplicationException: Internal error processing startShort
SQLState: 00000
ErrorCode: -1
{code}
This is the full log:
{code}
Fri Sep 2 00:26:50 UTC 2016 Starting tephra service on m9edd51-hmaster1.m9edd51
-f: file size (blocks) unlimited
-t: cpu time (seconds) unlimited
-d: data seg size (kb) unlimited
-s: stack size (kb) 8192
-c: core file size (blocks) unlimited
-m: resident set size (kb) unlimited
-l: locked memory (kb) 64
-p: processes unlimited
-n: file descriptors 65536
-v: address space (kb) unlimited
-w: locks unlimited
-e: scheduling priority 0
-r: real-time priority 0
Command: /usr/lib/jvm/java-1.8-openjdk/bin/java -XX:+UseConcMarkSweepGC -cp /opt/hbase/bin/../lib/*:/opt/hbase/bin/../conf/:/opt/hbase/phoenix-client/target/*:/opt/hbase/conf:/usr/lib/jvm/java-1.8-openjdk/lib/tools.jar:/opt/hbase:/opt/hbase/lib/activation-1.1.jar:/opt/hbase/lib/antisamy-1.5.3.jar:/opt/hbase/lib/aopalliance-1.0.jar:/opt/hbase/lib/apacheds-i18n-2.0.0-M15.jar:/opt/hbase/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/hbase/lib/api-asn1-api-1.0.0-M20.jar:/opt/hbase/lib/api-util-1.0.0-M20.jar:/opt/hbase/lib/asm-3.1.jar:/opt/hbase/lib/avro-1.7.4.jar:/opt/hbase/lib/batik-css-1.8.jar:/opt/hbase/lib/batik-ext-1.8.jar:/opt/hbase/lib/batik-util-1.8.jar:/opt/hbase/lib/bsh-core-2.0b4.jar:/opt/hbase/lib/commons-beanutils-1.7.0.jar:/opt/hbase/lib/commons-beanutils-core-1.8.3.jar:/opt/hbase/lib/commons-cli-1.2.jar:/opt/hbase/lib/commons-codec-1.9.jar:/opt/hbase/lib/commons-collections-3.2.2.jar:/opt/hbase/lib/commons-compress-1.4.1.jar:/opt/hbase/lib/commons-configuration-1.6.jar:/opt/hbase/lib/commons-daemon-1.0.13.jar:/opt/hbase/lib/commons-digester-1.8.jar:/opt/hbase/lib/commons-el-1.0.jar:/opt/hbase/lib/commons-fileupload-1.3.1.jar:/opt/hbase/lib/commons-httpclient-3.1.jar:/opt/hbase/lib/commons-io-2.4.jar:/opt/hbase/lib/commons-lang-2.6.jar:/opt/hbase/lib/commons-logging-1.2.jar:/opt/hbase/lib/commons-math-2.2.jar:/opt/hbase/lib/commons-math3-3.1.1.jar:/opt/hbase/lib/commons-net-3.1.jar:/opt/hbase/lib/disruptor-3.3.0.jar:/opt/hbase/lib/esapi-2.1.0.1.jar:/opt/hbase/lib/findbugs-annotations-1.3.9-1.jar:/opt/hbase/lib/guava-12.0.1.jar:/opt/hbase/lib/guice-3.0.jar:/opt/hbase/lib/guice-servlet-3.0.jar:/opt/hbase/lib/hadoop-annotations-2.5.1.jar:/opt/hbase/lib/hadoop-auth-2.5.1.jar:/opt/hbase/lib/hadoop-client-2.5.1.jar:/opt/hbase/lib/hadoop-common-2.5.1.jar:/opt/hbase/lib/hadoop-hdfs-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-app-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-common-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-core-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/opt/hbase/lib/hadoop-yarn-api-2.5.1.jar:/opt/hbase/lib/hadoop-yarn-client-2.5.1.jar:/opt/hbase/lib/hadoop-yarn-common-2.5.1.jar:/opt/hbase/lib/hadoop-yarn-server-common-2.5.1.jar:/opt/hbase/lib/hbase-annotations-1.2.2-tests.jar:/opt/hbase/lib/hbase-annotations-1.2.2.jar:/opt/hbase/lib/hbase-client-1.2.2.jar:/opt/hbase/lib/hbase-common-1.2.2-tests.jar:/opt/hbase/lib/hbase-common-1.2.2.jar:/opt/hbase/lib/hbase-examples-1.2.2.jar:/opt/hbase/lib/hbase-external-blockcache-1.2.2.jar:/opt/hbase/lib/hbase-hadoop-compat-1.2.2.jar:/opt/hbase/lib/hbase-hadoop2-compat-1.2.2.jar:/opt/hbase/lib/hbase-it-1.2.2-tests.jar:/opt/hbase/lib/hbase-it-1.2.2.jar:/opt/hbase/lib/hbase-prefix-tree-1.2.2.jar:/opt/hbase/lib/hbase-procedure-1.2.2.jar:/opt/hbase/lib/hbase-protocol-1.2.2.jar:/opt/hbase/lib/hbase-resource-bundle-1.2.2.jar:/opt/hbase/lib/hbase-rest-1.2.2.jar:/opt/hbase/lib/hbase-server-1.2.2-tests.jar:/opt/hbase/lib/hbase-server-1.2.2.jar:/opt/hbase/lib/hbase-shell-1.2.2.jar:/opt/hbase/lib/hbase-thrift-1.2.2.jar:/opt/hbase/lib/htrace-core-3.1.0-incubating.jar:/opt/hbase/lib/httpclient-4.2.5.jar:/opt/hbase/lib/httpcore-4.4.1.jar:/opt/hbase/lib/jackson-core-asl-1.9.13.jar:/opt/hbase/lib/jackson-jaxrs-1.9.13.jar:/opt/hbase/lib/jackson-mapper-asl-1.9.13.jar:/opt/hbase/lib/jackson-xc-1.9.13.jar:/opt/hbase/lib/jamon-runtime-2.4.1.jar:/opt/hbase/lib/jasper-compiler-5.5.23.jar:/opt/hbase/lib/jasper-runtime-5.5.23.jar:/opt/hbase/lib/java-xmlbuilder-0.4.jar:/opt/hbase/lib/javax.inject-1.jar:/opt/hbase/lib/jaxb-api-2.2.2.jar:/opt/hbase/lib/jaxb-impl-2.2.3-1.jar:/opt/hbase/lib/jcodings-1.0.8.jar:/opt/hbase/lib/jersey-client-1.9.jar:/opt/hbase/lib/jersey-core-1.9.jar:/opt/hbase/lib/jersey-guice-1.9.jar:/opt/hbase/lib/jersey-json-1.9.jar:/opt/hbase/lib/jersey-server-1.9.jar:/opt/hbase/lib/jets3t-0.9.0.jar:/opt/hbase/lib/jettison-1.3.3.jar:/opt/hbase/lib/jetty-6.1.26.jar:/opt/hbase/lib/jetty-sslengine-6.1.26.jar:/opt/hbase/lib/jetty-util-6.1.26.jar:/opt/hbase/lib/joni-2.1.2.jar:/opt/hbase/lib/jruby-complete-1.6.8.jar:/opt/hbase/lib/jsch-0.1.42.jar:/opt/hbase/lib/jsp-2.1-6.1.14.jar:/opt/hbase/lib/jsp-api-2.1-6.1.14.jar:/opt/hbase/lib/jsr305-1.3.9.jar:/opt/hbase/lib/junit-4.12.jar:/opt/hbase/lib/leveldbjni-all-1.8.jar:/opt/hbase/lib/libthrift-0.9.3.jar:/opt/hbase/lib/log4j-1.2.17.jar:/opt/hbase/lib/metrics-core-2.2.0.jar:/opt/hbase/lib/nekohtml-1.9.16.jar:/opt/hbase/lib/netty-all-4.0.23.Final.jar:/opt/hbase/lib/paranamer-2.3.jar:/opt/hbase/lib/phoenix-4.8.0-HBase-1.2-client.jar:/opt/hbase/lib/phoenix-4.8.0-HBase-1.2-server.jar:/opt/hbase/lib/protobuf-java-2.5.0.jar:/opt/hbase/lib/servlet-api-2.5-6.1.14.jar:/opt/hbase/lib/servlet-api-2.5.jar:/opt/hbase/lib/slf4j-api-1.7.7.jar:/opt/hbase/lib/slf4j-log4j12-1.7.5.jar:/opt/hbase/lib/snappy-java-1.0.4.1.jar:/opt/hbase/lib/spymemcached-2.11.6.jar:/opt/hbase/lib/xalan-2.7.0.jar:/opt/hbase/lib/xml-apis-1.3.03.jar:/opt/hbase/lib/xml-apis-ext-1.3.04.jar:/opt/hbase/lib/xmlenc-0.52.jar:/opt/hbase/lib/xom-1.2.5.jar:/opt/hbase/lib/xz-1.0.jar:/opt/hbase/lib/zookeeper-3.4.6.jar:: org.apache.tephra.TransactionServiceMain
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hbase/lib/phoenix-4.8.0-HBase-1.2-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2016-09-02 00:26:51,571 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2016-09-02 00:26:51,571 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=m9edd51-hmaster1.m9edd51
2016-09-02 00:26:51,571 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_92-internal
2016-09-02 00:26:51,571 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2016-09-02 00:26:51,571 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8-openjdk/jre
2016-09-02 00:26:51,571 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/opt/hbase/bin/../lib/phoenix-4.8.0-HBase-1.2-server.jar:/opt/hbase/bin/../lib/phoenix-4.8.0-HBase-1.2-client.jar:/opt/hbase/bin/../lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/opt/hbase/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/hbase/bin/../lib/httpcore-4.4.1.jar:/opt/hbase/bin/../lib/libthrift-0.9.3.jar:/opt/hbase/bin/../lib/hbase-hadoop2-compat-1.2.2.jar:/opt/hbase/bin/../lib/hadoop-hdfs-2.5.1.jar:/opt/hbase/bin/../lib/hbase-server-1.2.2-tests.jar:/opt/hbase/bin/../lib/commons-collections-3.2.2.jar:/opt/hbase/bin/../lib/commons-compress-1.4.1.jar:/opt/hbase/bin/../lib/jackson-jaxrs-1.9.13.jar:/opt/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hbase/bin/../lib/jetty-util-6.1.26.jar:/opt/hbase/bin/../lib/hbase-annotations-1.2.2.jar:/opt/hbase/bin/../lib/jetty-6.1.26.jar:/opt/hbase/bin/../lib/jackson-core-asl-1.9.13.jar:/opt/hbase/bin/../lib/disruptor-3.3.0.jar:/opt/hbase/bin/../lib/commons-cli-1.2.jar:/opt/hbase/bin/../lib/hbase-thrift-1.2.2.jar:/opt/hbase/bin/../lib/xalan-2.7.0.jar:/opt/hbase/bin/../lib/hbase-shell-1.2.2.jar:/opt/hbase/bin/../lib/jackson-xc-1.9.13.jar:/opt/hbase/bin/../lib/activation-1.1.jar:/opt/hbase/bin/../lib/hadoop-yarn-server-common-2.5.1.jar:/opt/hbase/bin/../lib/jersey-server-1.9.jar:/opt/hbase/bin/../lib/commons-net-3.1.jar:/opt/hbase/bin/../lib/xmlenc-0.52.jar:/opt/hbase/bin/../lib/batik-ext-1.8.jar:/opt/hbase/bin/../lib/jersey-guice-1.9.jar:/opt/hbase/bin/../lib/spymemcached-2.11.6.jar:/opt/hbase/bin/../lib/bsh-core-2.0b4.jar:/opt/hbase/bin/../lib/hbase-server-1.2.2.jar:/opt/hbase/bin/../lib/hbase-rest-1.2.2.jar:/opt/hbase/bin/../lib/guava-12.0.1.jar:/opt/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:/opt/hbase/bin/../lib/xz-1.0.jar:/opt/hbase/bin/../lib/jsch-0.1.42.jar:/opt/hbase/bin/../lib/jackson-mapper-asl-1.9.13.jar:/opt/hbase/bin/../lib/hbase-examples-1.2.2.jar:/opt/hbase/bin/../lib/hbase-annotations-1.2.2-tests.jar:/opt/hbase/bin/../lib/hbase-resource-bundle-1.2.2.jar:/opt/hbase/bin/../lib/snappy-java-1.0.4.1.jar:/opt/hbase/bin/../lib/commons-daemon-1.0.13.jar:/opt/hbase/bin/../lib/commons-el-1.0.jar:/opt/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/opt/hbase/bin/../lib/hadoop-client-2.5.1.jar:/opt/hbase/bin/../lib/hadoop-mapreduce-client-core-2.5.1.jar:/opt/hbase/bin/../lib/servlet-api-2.5.jar:/opt/hbase/bin/../lib/httpclient-4.2.5.jar:/opt/hbase/bin/../lib/joni-2.1.2.jar:/opt/hbase/bin/../lib/hadoop-yarn-client-2.5.1.jar:/opt/hbase/bin/../lib/jaxb-api-2.2.2.jar:/opt/hbase/bin/../lib/hadoop-yarn-api-2.5.1.jar:/opt/hbase/bin/../lib/commons-digester-1.8.jar:/opt/hbase/bin/../lib/jersey-core-1.9.jar:/opt/hbase/bin/../lib/hbase-procedure-1.2.2.jar:/opt/hbase/bin/../lib/slf4j-api-1.7.7.jar:/opt/hbase/bin/../lib/commons-logging-1.2.jar:/opt/hbase/bin/../lib/hbase-it-1.2.2.jar:/opt/hbase/bin/../lib/guice-3.0.jar:/opt/hbase/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/opt/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/opt/hbase/bin/../lib/api-util-1.0.0-M20.jar:/opt/hbase/bin/../lib/asm-3.1.jar:/opt/hbase/bin/../lib/hbase-hadoop-compat-1.2.2.jar:/opt/hbase/bin/../lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/opt/hbase/bin/../lib/hadoop-mapreduce-client-common-2.5.1.jar:/opt/hbase/bin/../lib/hbase-common-1.2.2.jar:/opt/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/opt/hbase/bin/../lib/nekohtml-1.9.16.jar:/opt/hbase/bin/../lib/java-xmlbuilder-0.4.jar:/opt/hbase/bin/../lib/hbase-protocol-1.2.2.jar:/opt/hbase/bin/../lib/esapi-2.1.0.1.jar:/opt/hbase/bin/../lib/junit-4.12.jar:/opt/hbase/bin/../lib/commons-fileupload-1.3.1.jar:/opt/hbase/bin/../lib/jamon-runtime-2.4.1.jar:/opt/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/hbase/bin/../lib/hbase-it-1.2.2-tests.jar:/opt/hbase/bin/../lib/slf4j-log4j12-1.7.5.jar:/opt/hbase/bin/../lib/xom-1.2.5.jar:/opt/hbase/bin/../lib/avro-1.7.4.jar:/opt/hbase/bin/../lib/hadoop-common-2.5.1.jar:/opt/hbase/bin/../lib/leveldbjni-all-1.8.jar:/opt/hbase/bin/../lib/commons-io-2.4.jar:/opt/hbase/bin/../lib/batik-css-1.8.jar:/opt/hbase/bin/../lib/commons-codec-1.9.jar:/opt/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/opt/hbase/bin/../lib/hadoop-yarn-common-2.5.1.jar:/opt/hbase/bin/../lib/javax.inject-1.jar:/opt/hbase/bin/../lib/commons-httpclient-3.1.jar:/opt/hbase/bin/../lib/commons-configuration-1.6.jar:/opt/hbase/bin/../lib/zookeeper-3.4.6.jar:/opt/hbase/bin/../lib/jets3t-0.9.0.jar:/opt/hbase/bin/../lib/guice-servlet-3.0.jar:/opt/hbase/bin/../lib/jersey-client-1.9.jar:/opt/hbase/bin/../lib/hbase-common-1.2.2-tests.jar:/opt/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/opt/hbase/bin/../lib/hadoop-mapreduce-client-app-2.5.1.jar:/opt/hbase/bin/../lib/aopalliance-1.0.jar:/opt/hbase/bin/../lib/antisamy-1.5.3.jar:/opt/hbase/bin/../lib/jsr305-1.3.9.jar:/opt/hbase/bin/../lib/metrics-core-2.2.0.jar:/opt/hbase/bin/../lib/xml-apis-1.3.03.jar:/opt/hbase/bin/../lib/protobuf-java-2.5.0.jar:/opt/hbase/bin/../lib/commons-math-2.2.jar:/opt/hbase/bin/../lib/jettison-1.3.3.jar:/opt/hbase/bin/../lib/netty-all-4.0.23.Final.jar:/opt/hbase/bin/../lib/paranamer-2.3.jar:/opt/hbase/bin/../lib/xml-apis-ext-1.3.04.jar:/opt/hbase/bin/../lib/log4j-1.2.17.jar:/opt/hbase/bin/../lib/jersey-json-1.9.jar:/opt/hbase/bin/../lib/jruby-complete-1.6.8.jar:/opt/hbase/bin/../lib/hbase-prefix-tree-1.2.2.jar:/opt/hbase/bin/../lib/commons-math3-3.1.1.jar:/opt/hbase/bin/../lib/jcodings-1.0.8.jar:/opt/hbase/bin/../lib/api-asn1-api-1.0.0-M20.jar:/opt/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/opt/hbase/bin/../lib/commons-lang-2.6.jar:/opt/hbase/bin/../lib/hadoop-annotations-2.5.1.jar:/opt/hbase/bin/../lib/commons-beanutils-core-1.8.3.jar:/opt/hbase/bin/../lib/hbase-client-1.2.2.jar:/opt/hbase/bin/../lib/hadoop-auth-2.5.1.jar:/opt/hbase/bin/../lib/batik-util-1.8.jar:/opt/hbase/bin/../lib/htrace-core-3.1.0-incubating.jar:/opt/hbase/bin/../lib/hbase-external-blockcache-1.2.2.jar:/opt/hbase/bin/../conf/:/opt/hbase/phoenix-client/target/*:/opt/hbase/conf:/usr/lib/jvm/java-1.8-openjdk/lib/tools.jar:/opt/hbase:/opt/hbase/lib/activation-1.1.jar:/opt/hbase/lib/antisamy-1.5.3.jar:/opt/hbase/lib/aopalliance-1.0.jar:/opt/hbase/lib/apacheds-i18n-2.0.0-M15.jar:/opt/hbase/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/hbase/lib/api-asn1-api-1.0.0-M20.jar:/opt/hbase/lib/api-util-1.0.0-M20.jar:/opt/hbase/lib/asm-3.1.jar:/opt/hbase/lib/avro-1.7.4.jar:/opt/hbase/lib/batik-css-1.8.jar:/opt/hbase/lib/batik-ext-1.8.jar:/opt/hbase/lib/batik-util-1.8.jar:/opt/hbase/lib/bsh-core-2.0b4.jar:/opt/hbase/lib/commons-beanutils-1.7.0.jar:/opt/hbase/lib/commons-beanutils-core-1.8.3.jar:/opt/hbase/lib/commons-cli-1.2.jar:/opt/hbase/lib/commons-codec-1.9.jar:/opt/hbase/lib/commons-collections-3.2.2.jar:/opt/hbase/lib/commons-compress-1.4.1.jar:/opt/hbase/lib/commons-configuration-1.6.jar:/opt/hbase/lib/commons-daemon-1.0.13.jar:/opt/hbase/lib/commons-digester-1.8.jar:/opt/hbase/lib/commons-el-1.0.jar:/opt/hbase/lib/commons-fileupload-1.3.1.jar:/opt/hbase/lib/commons-httpclient-3.1.jar:/opt/hbase/lib/commons-io-2.4.jar:/opt/hbase/lib/commons-lang-2.6.jar:/opt/hbase/lib/commons-logging-1.2.jar:/opt/hbase/lib/commons-math-2.2.jar:/opt/hbase/lib/commons-math3-3.1.1.jar:/opt/hbase/lib/commons-net-3.1.jar:/opt/hbase/lib/disruptor-3.3.0.jar:/opt/hbase/lib/esapi-2.1.0.1.jar:/opt/hbase/lib/findbugs-annotations-1.3.9-1.jar:/opt/hbase/lib/guava-12.0.1.jar:/opt/hbase/lib/guice-3.0.jar:/opt/hbase/lib/guice-servlet-3.0.jar:/opt/hbase/lib/hadoop-annotations-2.5.1.jar:/opt/hbase/lib/hadoop-auth-2.5.1.jar:/opt/hbase/lib/hadoop-client-2.5.1.jar:/opt/hbase/lib/hadoop-common-2.5.1.jar:/opt/hbase/lib/hadoop-hdfs-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-app-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-common-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-core-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/opt/hbase/lib/hadoop-yarn-api-2.5.1.jar:/opt/hbase/lib/hadoop-yarn-client-2.5.1.jar:/opt/hbase/lib/hadoop-yarn-common-2.5.1.jar:/opt/hbase/lib/hadoop-yarn-server-common-2.5.1.jar:/opt/hbase/lib/hbase-annotations-1.2.2-tests.jar:/opt/hbase/lib/hbase-annotations-1.2.2.jar:/opt/hbase/lib/hbase-client-1.2.2.jar:/opt/hbase/lib/hbase-common-1.2.2-tests.jar:/opt/hbase/lib/hbase-common-1.2.2.jar:/opt/hbase/lib/hbase-examples-1.2.2.jar:/opt/hbase/lib/hbase-external-blockcache-1.2.2.jar:/opt/hbase/lib/hbase-hadoop-compat-1.2.2.jar:/opt/hbase/lib/hbase-hadoop2-compat-1.2.2.jar:/opt/hbase/lib/hbase-it-1.2.2-tests.jar:/opt/hbase/lib/hbase-it-1.2.2.jar:/opt/hbase/lib/hbase-prefix-tree-1.2.2.jar:/opt/hbase/lib/hbase-procedure-1.2.2.jar:/opt/hbase/lib/hbase-protocol-1.2.2.jar:/opt/hbase/lib/hbase-resource-bundle-1.2.2.jar:/opt/hbase/lib/hbase-rest-1.2.2.jar:/opt/hbase/lib/hbase-server-1.2.2-tests.jar:/opt/hbase/lib/hbase-server-1.2.2.jar:/opt/hbase/lib/hbase-shell-1.2.2.jar:/opt/hbase/lib/hbase-thrift-1.2.2.jar:/opt/hbase/lib/htrace-core-3.1.0-incubating.jar:/opt/hbase/lib/httpclient-4.2.5.jar:/opt/hbase/lib/httpcore-4.4.1.jar:/opt/hbase/lib/jackson-core-asl-1.9.13.jar:/opt/hbase/lib/jackson-jaxrs-1.9.13.jar:/opt/hbase/lib/jackson-mapper-asl-1.9.13.jar:/opt/hbase/lib/jackson-xc-1.9.13.jar:/opt/hbase/lib/jamon-runtime-2.4.1.jar:/opt/hbase/lib/jasper-compiler-5.5.23.jar:/opt/hbase/lib/jasper-runtime-5.5.23.jar:/opt/hbase/lib/java-xmlbuilder-0.4.jar:/opt/hbase/lib/javax.inject-1.jar:/opt/hbase/lib/jaxb-api-2.2.2.jar:/opt/hbase/lib/jaxb-impl-2.2.3-1.jar:/opt/hbase/lib/jcodings-1.0.8.jar:/opt/hbase/lib/jersey-client-1.9.jar:/opt/hbase/lib/jersey-core-1.9.jar:/opt/hbase/lib/jersey-guice-1.9.jar:/opt/hbase/lib/jersey-json-1.9.jar:/opt/hbase/lib/jersey-server-1.9.jar:/opt/hbase/lib/jets3t-0.9.0.jar:/opt/hbase/lib/jettison-1.3.3.jar:/opt/hbase/lib/jetty-6.1.26.jar:/opt/hbase/lib/jetty-sslengine-6.1.26.jar:/opt/hbase/lib/jetty-util-6.1.26.jar:/opt/hbase/lib/joni-2.1.2.jar:/opt/hbase/lib/jruby-complete-1.6.8.jar:/opt/hbase/lib/jsch-0.1.42.jar:/opt/hbase/lib/jsp-2.1-6.1.14.jar:/opt/hbase/lib/jsp-api-2.1-6.1.14.jar:/opt/hbase/lib/jsr305-1.3.9.jar:/opt/hbase/lib/junit-4.12.jar:/opt/hbase/lib/leveldbjni-all-1.8.jar:/opt/hbase/lib/libthrift-0.9.3.jar:/opt/hbase/lib/log4j-1.2.17.jar:/opt/hbase/lib/metrics-core-2.2.0.jar:/opt/hbase/lib/nekohtml-1.9.16.jar:/opt/hbase/lib/netty-all-4.0.23.Final.jar:/opt/hbase/lib/paranamer-2.3.jar:/opt/hbase/lib/phoenix-4.8.0-HBase-1.2-client.jar:/opt/hbase/lib/phoenix-4.8.0-HBase-1.2-server.jar:/opt/hbase/lib/protobuf-java-2.5.0.jar:/opt/hbase/lib/servlet-api-2.5-6.1.14.jar:/opt/hbase/lib/servlet-api-2.5.jar:/opt/hbase/lib/slf4j-api-1.7.7.jar:/opt/hbase/lib/slf4j-log4j12-1.7.5.jar:/opt/hbase/lib/snappy-java-1.0.4.1.jar:/opt/hbase/lib/spymemcached-2.11.6.jar:/opt/hbase/lib/xalan-2.7.0.jar:/opt/hbase/lib/xml-apis-1.3.03.jar:/opt/hbase/lib/xml-apis-ext-1.3.04.jar:/opt/hbase/lib/xmlenc-0.52.jar:/opt/hbase/lib/xom-1.2.5.jar:/opt/hbase/lib/xz-1.0.jar:/opt/hbase/lib/zookeeper-3.4.6.jar::
2016-09-02 00:26:51,572 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-09-02 00:26:51,572 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2016-09-02 00:26:51,572 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2016-09-02 00:26:51,572 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2016-09-02 00:26:51,572 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2016-09-02 00:26:51,572 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=4.4.0-36-generic
2016-09-02 00:26:51,572 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=hbase
2016-09-02 00:26:51,572 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/opt/hbase
2016-09-02 00:26:51,572 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/opt/hbase
2016-09-02 00:26:51,573 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=m9edd51-zookeeper.m9edd51 sessionTimeout=90000 watcher=org.apache.tephra.zookeeper.TephraZKClientService$5@45c7e403
2016-09-02 00:26:51,596 INFO [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Opening socket connection to server m9edd51-zookeeper.m9edd51/172.18.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
2016-09-02 00:26:51,600 INFO [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Socket connection established to m9edd51-zookeeper.m9edd51/172.18.0.2:2181, initiating session
2016-09-02 00:26:51,605 INFO [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Session establishment complete on server m9edd51-zookeeper.m9edd51/172.18.0.2:2181, sessionid = 0x156e849e51c0007, negotiated timeout = 40000
2016-09-02 00:26:51,625 INFO [main] inmemory.InMemoryTransactionService: Configuring TransactionService, address: 0.0.0.0, port: 15165, threads: 20, io threads: 2, max read buffer (bytes): 16777216
2016-09-02 00:26:51,626 INFO [main] tephra.TransactionServiceMain: Starting TransactionServiceMain
2016-09-02 00:26:51,633 INFO [main] zookeeper.LeaderElection: Start leader election on m9edd51-zookeeper.m9edd51/tx.service/leader with guid aa0cbcc9-5267-4618-9059-bb90a1e7e948
2016-09-02 00:26:51,694 INFO [leader-election-tx.service-leader] metrics.DefaultMetricsCollector: Configured metrics report to emit every 60 seconds
2016-09-02 00:26:51,863 INFO [ThriftRPCServer] tephra.TransactionManager: Starting transaction manager.
2016-09-02 00:26:51,874 INFO [DefaultMetricsCollector STARTING] metrics.DefaultMetricsCollector: Started metrics reporter
2016-09-02 00:26:51,954 WARN [HDFSTransactionStateStorage STARTING] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-09-02 00:26:52,554 INFO [HDFSTransactionStateStorage STARTING] persist.HDFSTransactionStateStorage: Using snapshot dir /tmp/tephra/snapshots
2016-09-02 00:26:52,594 INFO [ThriftRPCServer] persist.HDFSTransactionStateStorage: Creating snapshot dir at /tmp/tephra/snapshots
2016-09-02 00:26:52,620 INFO [ThriftRPCServer] persist.HDFSTransactionStateStorage: No snapshot files found in /tmp/tephra/snapshots
2016-09-02 00:26:52,624 INFO [ThriftRPCServer] tephra.TransactionManager: Starting periodic timed-out transaction cleanup every 10 seconds with default timeout of 60 seconds.
2016-09-02 00:26:52,624 INFO [ThriftRPCServer] tephra.TransactionManager: Starting periodic snapshot thread, frequency = 300 seconds, location = /tmp/tephra/snapshots
2016-09-02 00:26:52,624 INFO [ThriftRPCServer] tephra.TransactionManager: Starting periodic Metrics Emitter thread, frequency = 10000
2016-09-02 00:26:52,631 INFO [ThriftRPCServer] rpc.ThriftRPCServer: Starting RPC server for TTransactionServer
2016-09-02 00:26:52,641 INFO [ThriftRPCServer] rpc.ThriftRPCServer: Running RPC server for TTransactionServer
2016-09-02 00:26:52,641 INFO [ThriftRPCServer] server.TThreadedSelectorServerWithFix: Starting TThreadedSelectorServerWithFix
2016-09-02 00:26:52,720 INFO [leader-election-tx.service-leader] distributed.TransactionService: Transaction Thrift Service started successfully on m9edd51-hmaster1.m9edd51/172.18.0.12:15165
2016-09-02 00:27:38,642 INFO [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x156e849e51c0007, likely server has closed socket, closing socket connection and attempting reconnect
2016-09-02 00:27:38,746 INFO [leader-election-tx.service-leader] zookeeper.LeaderElection: Disconnected from ZK: m9edd51-zookeeper.m9edd51 for /tx.service/leader
2016-09-02 00:27:38,746 INFO [leader-election-tx.service-leader] zookeeper.LeaderElection: Stepping down as leader due to disconnect: m9edd51-zookeeper.m9edd51 for /tx.service/leader
2016-09-02 00:27:38,747 INFO [leader-election-tx.service-leader] rpc.ThriftRPCServer: Request to stop RPC server for TTransactionServer
2016-09-02 00:27:38,751 INFO [ThriftRPCServer] rpc.ThriftRPCServer: Done running RPC server for TTransactionServer
2016-09-02 00:27:38,751 INFO [ThriftRPCServer] tephra.TransactionManager: Shutting down gracefully...
2016-09-02 00:27:38,752 INFO [tx-clean-timeout] tephra.TransactionManager: Exiting thread tx-clean-timeout
2016-09-02 00:27:38,752 INFO [tx-metrics] tephra.TransactionManager: Exiting thread tx-metrics
2016-09-02 00:27:38,752 INFO [tx-snapshot] tephra.TransactionManager: Writing final snapshot prior to shutdown
2016-09-02 00:27:38,884 INFO [tx-snapshot] tephra.TransactionManager: Exiting thread tx-snapshot
2016-09-02 00:27:38,992 INFO [DefaultMetricsCollector STOPPING] metrics.DefaultMetricsCollector: Stopped metrics reporter
2016-09-02 00:27:38,992 INFO [ThriftRPCServer] tephra.TransactionManager: Took 240.2 ms to stop
2016-09-02 00:27:38,993 INFO [ThriftRPCServer] rpc.ThriftRPCServer: RPC server for TTransactionServer stopped.
2016-09-02 00:27:39,013 INFO [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Opening socket connection to server m9edd51-zookeeper.m9edd51/fe80:0:0:0:0:242:ac12:2:2181. Will not attempt to authenticate using SASL (unknown error)
2016-09-02 00:27:39,014 WARN [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Session 0x156e849e51c0007 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-09-02 00:27:40,462 INFO [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Opening socket connection to server m9edd51-zookeeper.m9edd51/172.18.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
2016-09-02 00:27:40,463 INFO [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Socket connection established to m9edd51-zookeeper.m9edd51/172.18.0.2:2181, initiating session
2016-09-02 00:27:40,464 INFO [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Session establishment complete on server m9edd51-zookeeper.m9edd51/172.18.0.2:2181, sessionid = 0x156e849e51c0007, negotiated timeout = 40000
2016-09-02 00:27:40,469 INFO [leader-election-tx.service-leader] zookeeper.LeaderElection: Connected to ZK, running election: m9edd51-zookeeper.m9edd51 for /tx.service/leader
2016-09-02 00:27:40,502 INFO [ThriftRPCServer] rpc.ThriftRPCServer: Starting RPC server for TTransactionServer
2016-09-02 00:27:40,502 INFO [ThriftRPCServer] rpc.ThriftRPCServer: Running RPC server for TTransactionServer
2016-09-02 00:27:40,502 INFO [ThriftRPCServer] server.TThreadedSelectorServerWithFix: Starting TThreadedSelectorServerWithFix
2016-09-02 00:27:40,509 INFO [leader-election-tx.service-leader] distributed.TransactionService: Transaction Thrift Service started successfully on m9edd51-hmaster1.m9edd51/172.18.0.12:15165
2016-09-02 00:28:52,892 ERROR [TTransactionServer-rpc-0] thrift.ProcessFunction: Internal error processing startShort
java.lang.IllegalStateException: Transaction Manager is not running.
at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
at org.apache.tephra.TransactionManager.ensureAvailable(TransactionManager.java:709)
at org.apache.tephra.TransactionManager.startTx(TransactionManager.java:768)
at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:728)
at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:716)
at org.apache.tephra.distributed.TransactionServiceThriftHandler.startShort(TransactionServiceThriftHandler.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServer.java:261)
at com.sun.proxy.$Proxy17.startShort(Unknown Source)
at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:974)
at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:959)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
at org.apache.thrift.server.Invocation.run(Invocation.java:18)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-09-02 00:28:53,297 ERROR [TTransactionServer-rpc-0] thrift.ProcessFunction: Internal error processing startShort
java.lang.IllegalStateException: Transaction Manager is not running.
at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
at org.apache.tephra.TransactionManager.ensureAvailable(TransactionManager.java:709)
at org.apache.tephra.TransactionManager.startTx(TransactionManager.java:768)
at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:728)
at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:716)
at org.apache.tephra.distributed.TransactionServiceThriftHandler.startShort(TransactionServiceThriftHandler.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServer.java:261)
at com.sun.proxy.$Proxy17.startShort(Unknown Source)
at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:974)
at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:959)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
at org.apache.thrift.server.Invocation.run(Invocation.java:18)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-09-02 00:28:54,901 ERROR [TTransactionServer-rpc-0] thrift.ProcessFunction: Internal error processing startShort
java.lang.IllegalStateException: Transaction Manager is not running.
at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
at org.apache.tephra.TransactionManager.ensureAvailable(TransactionManager.java:709)
at org.apache.tephra.TransactionManager.startTx(TransactionManager.java:768)
at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:728)
at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:716)
at org.apache.tephra.distributed.TransactionServiceThriftHandler.startShort(TransactionServiceThriftHandler.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServer.java:261)
at com.sun.proxy.$Proxy17.startShort(Unknown Source)
at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:974)
at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:959)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
at org.apache.thrift.server.Invocation.run(Invocation.java:18)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-09-02 00:29:01,313 ERROR [TTransactionServer-rpc-0] thrift.ProcessFunction: Internal error processing startShort
java.lang.IllegalStateException: Transaction Manager is not running.
at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
at org.apache.tephra.TransactionManager.ensureAvailable(TransactionManager.java:709)
at org.apache.tephra.TransactionManager.startTx(TransactionManager.java:768)
at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:728)
at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:716)
at org.apache.tephra.distributed.TransactionServiceThriftHandler.startShort(TransactionServiceThriftHandler.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServer.java:261)
at com.sun.proxy.$Proxy17.startShort(Unknown Source)
at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:974)
at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:959)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
at org.apache.thrift.server.Invocation.run(Invocation.java:18)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
Although it thinks that the transaction manager is not running, I can still see that it is running using jps:
{code}
bash-4.3# jps
138 HMaster
974 Jps
191 TransactionServiceMain
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)