You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Suryansh Agnihotri <sa...@gmail.com> on 2021/05/13 15:22:49 UTC
beeline spark thrift server issue
Hi
I was trying to access spark sql through jdbc but facing some error. I am
trying to run beeline
! /usr/lib/spark/bin/beeline -u
'jdbc:hive2://host:10016/default;transportMode=binary' -e '' 2>&1| awk
'{print}'|grep -i -e 'Connection refused' -e 'Invalid URL' -e 'Error: Could
not open'
Error: Could not open client transport with JDBC Uri:
host:10016/default;transportMode=binary: java.net.ConnectException:
Connection refused (Connection refused) (state=08S01,code=0)
hive.server2.thrift.port=10016 and mode is binary.
I verified the process is running on this port.
I checked the spark thrift server logs https://pastebin.com/G5Mwaw7E
It says "java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient"
(pasted the logs in above link).
*I am using spark version 3.1.0 , hive 3.1.2 and hadoop 3.1.2.*
Following this guide
https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html#interacting-with-different-versions-of-hive-metastore
I set "spark.sql.hive.metastore.version" to 3.1.2 and set jars to point to
hivemetastore 3.1.2 jars but still getting the same error.
From logs I also got suspicious about a mismatch in the version of
datanucleus core in hive and spark but both use the same version.
https://github.com/apache/hive/blob/branch-3.1/pom.xml#L129
https://github.com/apache/spark/blob/branch-3.1/pom.xml#L184 (edited)
Is this a known issue , how should this be fixed. Let me know if anything
else is required.
Thanks
Re: beeline spark thrift server issue
Posted by Yuming Wang <wg...@gmail.com>.
Unable to access log(https://pastebin.com/G5Mwaw7E).
On Thu, May 13, 2021 at 11:23 PM Suryansh Agnihotri <
sagnihotri2k14@gmail.com> wrote:
> Hi
> I was trying to access spark sql through jdbc but facing some error. I am
> trying to run beeline
>
> ! /usr/lib/spark/bin/beeline -u
> 'jdbc:hive2://host:10016/default;transportMode=binary' -e '' 2>&1| awk
> '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL' -e 'Error: Could
> not open'
>
> Error: Could not open client transport with JDBC Uri:
> host:10016/default;transportMode=binary: java.net.ConnectException:
> Connection refused (Connection refused) (state=08S01,code=0)
>
> hive.server2.thrift.port=10016 and mode is binary.
> I verified the process is running on this port.
> I checked the spark thrift server logs https://pastebin.com/G5Mwaw7E
> It says "java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient"
> (pasted the logs in above link).
>
> *I am using spark version 3.1.0 , hive 3.1.2 and hadoop 3.1.2.*
> Following this guide
>
> https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html#interacting-with-different-versions-of-hive-metastore
>
> I set "spark.sql.hive.metastore.version" to 3.1.2 and set jars to point
> to hivemetastore 3.1.2 jars but still getting the same error.
>
> From logs I also got suspicious about a mismatch in the version of
> datanucleus core in hive and spark but both use the same version.
> https://github.com/apache/hive/blob/branch-3.1/pom.xml#L129
> https://github.com/apache/spark/blob/branch-3.1/pom.xml#L184 (edited)
>
> Is this a known issue , how should this be fixed. Let me know if anything
> else is required.
> Thanks
>
Re: beeline spark thrift server issue
Posted by Yuming Wang <wg...@gmail.com>.
Unable to access log(https://pastebin.com/G5Mwaw7E).
On Thu, May 13, 2021 at 11:23 PM Suryansh Agnihotri <
sagnihotri2k14@gmail.com> wrote:
> Hi
> I was trying to access spark sql through jdbc but facing some error. I am
> trying to run beeline
>
> ! /usr/lib/spark/bin/beeline -u
> 'jdbc:hive2://host:10016/default;transportMode=binary' -e '' 2>&1| awk
> '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL' -e 'Error: Could
> not open'
>
> Error: Could not open client transport with JDBC Uri:
> host:10016/default;transportMode=binary: java.net.ConnectException:
> Connection refused (Connection refused) (state=08S01,code=0)
>
> hive.server2.thrift.port=10016 and mode is binary.
> I verified the process is running on this port.
> I checked the spark thrift server logs https://pastebin.com/G5Mwaw7E
> It says "java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient"
> (pasted the logs in above link).
>
> *I am using spark version 3.1.0 , hive 3.1.2 and hadoop 3.1.2.*
> Following this guide
>
> https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html#interacting-with-different-versions-of-hive-metastore
>
> I set "spark.sql.hive.metastore.version" to 3.1.2 and set jars to point
> to hivemetastore 3.1.2 jars but still getting the same error.
>
> From logs I also got suspicious about a mismatch in the version of
> datanucleus core in hive and spark but both use the same version.
> https://github.com/apache/hive/blob/branch-3.1/pom.xml#L129
> https://github.com/apache/spark/blob/branch-3.1/pom.xml#L184 (edited)
>
> Is this a known issue , how should this be fixed. Let me know if anything
> else is required.
> Thanks
>