You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2016/03/16 12:17:33 UTC
[jira] [Resolved] (SPARK-3544) SparkSQL thriftServer cannot release
locks correctly in Zookeeper
[ https://issues.apache.org/jira/browse/SPARK-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-3544.
------------------------------
Resolution: Incomplete
I think this is at least obsolete now; no follow up or other reports of this, but not particular resolution either.
> SparkSQL thriftServer cannot release locks correctly in Zookeeper
> -----------------------------------------------------------------
>
> Key: SPARK-3544
> URL: https://issues.apache.org/jira/browse/SPARK-3544
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.1.0
> Reporter: Patrick Liu
> Priority: Critical
>
> Bug description:
> The thriftServer cannot release locks correctly in Zookeeper.
> Once a thriftServer is started, the 1st SQL submited by Beeline or JDBC requiring locks can be successfully computed.
> However, the 2rd SQL requiring locks will be blocked. And the thriftServer log shows: INFO Driver: <PERFLOG method=acquireReadWriteLocks>.
> 2 Tests to illustrate the problem:
> Test 1:
> (0) Start thriftServer & use beeline to connect to it.
> (1) Switch database(require locks);(OK)
> (2) Drop table(require locks) “drop table src"; (Blocked)
> Test 2:
> (0) Start thriftServer & use beeline to connect to it.
> (1) Drop table(require locks) "drop table src"; (OK)
> (2) Drop another table(require locks) "drop table src2;"(Blocked)
> Basic Information:
> Spark 1.1.0
> Hadoop 2.0.0-cdh4.6.0
> Compile Command:
> export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
> mvn -Dhadoop.version=2.0.0-cdh4.6.0 -Phive -Pspark-ganglia-lgpl -DskipTests package
> hive-site.xml:
> <configuration>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://ns</value>
> </property>
> <property>
> <name>dfs.nameservices</name>
> <value>ns</value>
> </property>
>
> <property>
> <name>dfs.ha.namenodes.ns</name>
> <value>machine01,machine02</value>
> </property>
>
> <property>
> <name>dfs.namenode.rpc-address.ns.machine01</name>
> <value>machine01:54310</value>
>
> </property>
> <property>
> <name>dfs.namenode.rpc-address.ns.machine02</name>
> <value>machine02:54310</value>
>
> </property>
>
> <property>
> <name>dfs.client.failover.proxy.provider.ns</name>
>
> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionURL</name>
> <value>jdbc:mysql://localhost:3306/metastore</value>
> <description>JDBC connect string for a JDBC metastore</description>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionDriverName</name>
> <value>com.mysql.jdbc.Driver</value>
> <description>Driver class name for a JDBC metastore</description>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionUserName</name>
> <value>hive_user</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionPassword</name>
> <value>hive_123</value>
> </property>
> <property>
> <name>datanucleus.autoCreateSchema</name>
> <value>false</value>
> </property>
> <property>
> <name>datanucleus.autoCreateTables</name>
> <value>true</value>
> </property>
> <property>
> <name>datanucleus.fixedDatastore</name>
> <value>false</value>
> </property>
> <property>
> <name>hive.support.concurrency</name>
> <description>Enable Hive's Table Lock Manager Service</description>
> <value>true</value>
> </property>
> <property>
> <name>hive.zookeeper.quorum</name>
> <value>machine01,machine02,machine03</value>
> <description>Zookeeper quorum used by Hive's Table Lock Manager</description>
> </property>
> <property>
> <name>hive.metastore.warehouse.dir</name>
> <value>/user/hive/warehouse</value>
> <description>Hive warehouse directory</description>
> </property>
> <property>
> <name>mapred.job.tracker</name>
> <value>machine01:8032</value>
> </property>
> <property>
> <name>io.compression.codecs</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> </property>
> <property>
> <name>mapreduce.output.fileoutputformat.compress.codec</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> </property>
> <property>
> <name>mapreduce.output.fileoutputformat.compress.type</name>
> <value>BLOCK</value>
> </property>
> <property>
> <name>hive.exec.show.job.failure.debug.info</name>
> <value>true</value>
> <description>
> If a job fails, whether to provide a link in the CLI to the task with the
> most failures, along with debugging hints if applicable.
> </description>
> </property>
> <property>
> <name>hive.hwi.listen.host</name>
> <value>0.0.0.0</value>
> <description>This is the host address the Hive Web Interface will listen on</description>
> </property>
> <property>
> <name>hive.hwi.listen.port</name>
> <value>9999</value>
> <description>This is the port the Hive Web Interface will listen on</description>
> </property>
> <property>
> <name>hive.hwi.war.file</name>
> <value>/lib/hive-hwi-0.10.0-cdh4.2.0.war</value>
> <description>This is the WAR file with the jsp content for Hive Web Interface</description>
> </property>
> <property>
> <name>hive.aux.jars.path</name>
>
> <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.6.0.jar,file:///usr/lib/hbase/hbase-0.94.15-cdh4.6.0-security.jar,file:///usr/lib/zookeeper/zookeeper.jar</value>
> </property>
> <property>
> <name>hbase.zookeeper.quorum</name>
> <value>machine01,machine02,machine03</value>
> </property>
> <property>
> <name>hive.cli.print.header</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.metastore.execute.setugi</name>
> <value>true</value>
> <description>In unsecure mode, setting this property to true will cause the metastore to execute DFS operations using the client's reported user and group permissions. Note that this property must be set on both the client and server sides. Further note that its best effort. If client sets its to true and server sets it to false, client setting will be ignored.</description>
> </property>
> <property>
> <name>hive.security.authorization.enabled</name>
> <value>true</value>
> <description>enable or disable the hive client authorization</description>
> </property>
> <property>
> <name>hive.metastore.authorization.storage.checks</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.security.authorization.createtable.owner.grants</name>
> <value>ALL</value>
> <description>the privileges automatically granted to the owner whenever a table gets created.
> An example like "select,drop" will grant select and drop privilege to the owner of the table</description>
> </property>
> </configuration>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org