You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ambari.apache.org by Jan Hentschel <ja...@ultratendency.com> on 2019/12/12 19:03:30 UTC
Problem with Spark2 Thriftserver Alert in Ambari
Hello everybody,
we're having problems with the Ambari alert for the Spark Thriftserver, complaining about the Beeline connection to the Thriftserver (see the alert below):
Connection failed on host master01:10016 (Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts/alert_spark2_thrift_port.py", line 147, in execute
Execute(cmd, user=sparkuser, path=[beeline_cmd], timeout=CHECK_COMMAND_TIMEOUT_DEFAULT)
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
raise ExecutionFailed(err_msg, code, out, err)
ExecutionFailed: Execution of '! /usr/hdp/current/spark2-client/bin/beeline -u 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL' ' returned 1. Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)
)
Running it with a user the same problem comes up:
dr_who@master01:/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts$ ! /usr/hdp/current/spark2-client/bin/beeline -u 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary' -e ''
Connecting to jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary
19/09/30 10:40:26 INFO Utils: Supplied authorities: master01:10016
19/09/30 10:40:26 INFO Utils: Resolved authority: master01:10016
19/09/30 10:40:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/09/30 10:40:26 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary
19/09/30 10:40:27 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary
19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC connection: binary
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary: java.net.SocketException: Connection reset (state=08S01,code=0)
Beeline version 1.21.2.3.0.1.0-187 by Apache Hive
19/09/30 10:40:27 INFO Utils: Supplied authorities: master01:10016
19/09/30 10:40:27 INFO Utils: Resolved authority: master01:10016
19/09/30 10:40:27 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary
19/09/30 10:40:27 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary
19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC connection: binary
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary: java.net.SocketException: Broken pipe (Write failed) (state=08S01,code=0)
When trying to run the same with the Spark user from the command line, a similar issues arises:
spark@master01:~$ /usr/hdp/current/spark2-client/bin/beeline -u 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary' -e 'show databases'
Connecting to jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary
19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
19/11/13 14:42:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection: binary
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary: Invalid status 21 (state=08S01,code=0)
19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection: binary
No current connection
19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection: binary
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary: Invalid status 21 (state=08S01,code=0)
Looking at the Thriftserver logs reveals the following:
19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
at sun.security.ssl.InputRecord.read(InputRecord.java:527)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 16 more
19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
at sun.security.ssl.InputRecord.read(InputRecord.java:527)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 16 more
TLS is configured on the Hive side, but still it seems that instead of a TLS connection Beeline or Spark tries to establish a plaintext connection. Does anyone know how to resolve this problem? Besides the points mentioned above the Thriftserver, f.e. when connecting to it via ODBC.
Re: Problem with Spark2 Thriftserver Alert in Ambari
Posted by Matt Andruff <ma...@gmail.com>.
Dbevear as far as I know can't use zookeeper namespaces. (It can use host
and port)
I suggest using the value (master01:10000) if the dbeaver host can resolve
master001.
On Thu., Jan. 9, 2020, 11:51 Jan Hentschel, <ja...@ultratendency.com>
wrote:
> Hi Matt,
>
>
>
> thanks for your answer. We tried connecting to Hive via the ZooKeeper
> namespace in DBeaver with the JDBC URL
> jdbc:hive2://master01:2181,master02:2181,master03:2181/;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> and got a similar error:
>
>
>
> „Could not open client transport for any of the Server URI's in ZooKeeper:
> Invalid status 21“
>
>
>
> Wheoking into the namespace via the CLI, we get the correct address of our
> Hive instance returned:
>
>
>
> ls /hiveserver2
>
> [serverUri=;version=3.1.0.3.0.1.0-187;sequence=0000000123]
>
>
>
> Fun thing is that most of the clients, who are using the Thriftserver
> connection outside the cluster f.e. in Excel, are able to connect to it
> without the problem. We checked the truststores the clients have with the
> one’s inside the cluster and they are the same.
>
>
>
> Best regards, Jan
>
>
>
> *From: *Matt Andruff <ma...@gmail.com>
> *Reply-To: *"user@ambari.apache.org" <us...@ambari.apache.org>
> *Date: *Wednesday, December 18, 2019 at 3:49 AM
> *To: *"user@ambari.apache.org" <us...@ambari.apache.org>
> *Subject: *Re: Problem with Spark2 Thriftserver Alert in Ambari
>
>
>
> So if your not able to connect with beeline with that trustore info you
> have a typo. Either a typo in your connection string.(unlikely) or the
> trustore isn't valid.(a different type of typo)
>
>
>
> Can you rules either of them out as being a typo?
>
>
>
> (Does using the zookeeper namespace work connection string for hive
> work). You know the one that's in Ambari, on the hive summary page?
>
>
>
> Hive does actually right the connection string it wants you to use to the
> zookeeper namespace. Can you connect to zookeeper and check what string it
> wants you to use? (Send me a reply and I'll dig up how to do this if you
> don't know how to do it. If you have more than 1 hive instance their will
> be more than 1 entry... I'm not going to respond until tomorrow just so you
> know ).
>
>
>
>
>
>
>
> On Tue., Dec. 17, 2019, 12:00 Jan Hentschel, <
> jan.hentschel@ultratendency.com> wrote:
>
> We already tried the SSL configuration in the Beeline connection string
> with a similar result. Hive is in binary mode with TLS. Below the output
> when trying to use the Truststore parameters in the connection string
> (resulting in the same output in the Thriftserver logs):
>
>
>
> spark@master01:/$ /usr/hdp/current/spark2-client/bin/beeline -u
> 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
> ' -e 'show databases'
>
> Connecting to jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM
> ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
>
> 19/11/01 10:35:20 INFO Utils: Supplied authorities: master01:10016
>
> 19/11/01 10:35:20 INFO Utils: Resolved authority: master01:10016
>
> 19/11/01 10:35:20 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 19/11/01 10:35:20 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM
> ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
>
> 19/11/01 10:35:21 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM
> ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
>
> 19/11/01 10:35:21 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit:
> Invalid status 21 (state=08S01,code=0)
>
> 19/11/01 10:35:21 INFO Utils: Supplied authorities: master01:10016
>
> 19/11/01 10:35:21 INFO Utils: Resolved authority: master01:10016
>
> 19/11/01 10:35:21 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM
> ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
>
> 19/11/01 10:35:21 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM
> ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
>
> 19/11/01 10:35:21 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> No current connection
>
> 19/11/01 10:35:21 INFO Utils: Supplied authorities: master01:10016
>
> 19/11/01 10:35:21 INFO Utils: Resolved authority: master01:10016
>
> 19/11/01 10:35:21 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM
> ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
>
> 19/11/01 10:35:21 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM
> ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
>
> 19/11/01 10:35:21 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit:
> Invalid status 21 (state=08S01,code=0)
>
>
>
> *From: *Matt Andruff <ma...@gmail.com>
> *Reply-To: *"user@ambari.apache.org" <us...@ambari.apache.org>
> *Date: *Friday, December 13, 2019 at 8:17 PM
> *To: *"user@ambari.apache.org" <us...@ambari.apache.org>
> *Subject: *Re: Problem with Spark2 Thriftserver Alert in Ambari
>
>
>
> It's clear from the Ambati alert the thrift server connection it's trying
> to use binary connection (and missing the trustore). Are you using hive in
> binary with SSL or https?
>
>
>
> On Fri., Dec. 13, 2019, 14:13 Matt Andruff, <ma...@gmail.com>
> wrote:
>
> What happens when you try to connect via beeline with:
>
>
>
>
> jdbc:hive2://<host>:<port>/<db>;ssl=true;sslTrustStore=<trust_store_path>;trustStorePassword=<trust_store_password>
>
>
>
> ?
>
>
>
> Or are you just using the zookeeper namespace to connect?
>
>
>
> On Thu., Dec. 12, 2019, 14:03 Jan Hentschel, <
> jan.hentschel@ultratendency.com> wrote:
>
> Hello everybody,
>
>
>
> we're having problems with the Ambari alert for the Spark Thriftserver,
> complaining about the Beeline connection to the Thriftserver (see the alert
> below):
>
>
>
> Connection failed on host master01:10016 (Traceback (most recent call
> last):
>
> File
> "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts/alert_spark2_thrift_port.py",
> line 147, in execute
>
> Execute(cmd, user=sparkuser, path=[beeline_cmd],
> timeout=CHECK_COMMAND_TIMEOUT_DEFAULT)
>
> File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line
> 166, in __init__
>
> self.env.run()
>
> File
> "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line
> 160, in run
>
> self.run_action(resource, action)
>
> File
> "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line
> 124, in run_action
>
> provider_action()
>
> File
> "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py",
> line 263, in action_run
>
> returns=self.resource.returns)
>
> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
> 72, in inner
>
> result = function(command, **kwargs)
>
> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
> 102, in checked_call
>
> tries=tries, try_sleep=try_sleep,
> timeout_kill_strategy=timeout_kill_strategy, returns=returns)
>
> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
> 150, in _call_wrapper
>
> result = _call(command, **kwargs_copy)
>
> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
> 314, in _call
>
> raise ExecutionFailed(err_msg, code, out, err)
>
> ExecutionFailed: Execution of '!
> /usr/hdp/current/spark2-client/bin/beeline -u
> 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary'
> -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
> ' returned 1. Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> java.net.ConnectException: Connection refused (Connection refused)
> (state=08S01,code=0)
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> java.net.ConnectException: Connection refused (Connection refused)
> (state=08S01,code=0)
>
> )
>
>
>
> Running it with a user the same problem comes up:
>
>
>
> dr_who@master01:/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts$
> ! /usr/hdp/current/spark2-client/bin/beeline -u
> 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary'
> -e ''
>
> Connecting to jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/09/30 10:40:26 INFO Utils: Supplied authorities: master01:10016
>
> 19/09/30 10:40:26 INFO Utils: Resolved authority: master01:10016
>
> 19/09/30 10:40:26 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 19/09/30 10:40:26 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/09/30 10:40:27 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> java.net.SocketException: Connection reset (state=08S01,code=0)
>
> Beeline version 1.21.2.3.0.1.0-187 by Apache Hive
>
> 19/09/30 10:40:27 INFO Utils: Supplied authorities: master01:10016
>
> 19/09/30 10:40:27 INFO Utils: Resolved authority: master01:10016
>
> 19/09/30 10:40:27 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/09/30 10:40:27 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> java.net.SocketException: Broken pipe (Write failed) (state=08S01,code=0)
>
>
>
> When trying to run the same with the Spark user from the command line, a
> similar issues arises:
>
>
>
> spark@master01:~$ /usr/hdp/current/spark2-client/bin/beeline -u
> 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary'
> -e 'show databases'
>
> Connecting to jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
>
> 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
>
> 19/11/13 14:42:27 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> Invalid status 21 (state=08S01,code=0)
>
> 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
>
> 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
>
> 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> No current connection
>
> 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
>
> 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
>
> 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> Invalid status 21 (state=08S01,code=0)
>
>
>
> Looking at the Thriftserver logs reveals the following:
>
>
>
> 19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during
> processing of message.
>
> java.lang.RuntimeException:
> org.apache.thrift.transport.TTransportException:
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:360)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
>
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.thrift.transport.TTransportException:
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>
> at
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
>
> at
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
>
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>
> ... 10 more
>
> Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext
> connection?
>
> at
> sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
>
> at sun.security.ssl.InputRecord.read(InputRecord.java:527)
>
> at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
>
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
>
> at
> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
>
> at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
>
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>
> ... 16 more
>
> 19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during
> processing of message.
>
> java.lang.RuntimeException:
> org.apache.thrift.transport.TTransportException:
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:360)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
>
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.thrift.transport.TTransportException:
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>
> at
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
>
> at
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
>
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>
> ... 10 more
>
> Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext
> connection?
>
> at
> sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
>
> at sun.security.ssl.InputRecord.read(InputRecord.java:527)
>
> at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
>
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
>
> at
> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
>
> at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
>
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>
> ... 16 more
>
>
>
> TLS is configured on the Hive side, but still it seems that instead of a
> TLS connection Beeline or Spark tries to establish a plaintext connection.
> Does anyone know how to resolve this problem? Besides the points
> mentioned above the Thriftserver, f.e. when connecting to it via ODBC.
>
>
>
>
Re: Problem with Spark2 Thriftserver Alert in Ambari
Posted by Jan Hentschel <ja...@ultratendency.com>.
Hi Matt,
thanks for your answer. We tried connecting to Hive via the ZooKeeper namespace in DBeaver with the JDBC URL jdbc:hive2://master01:2181,master02:2181,master03:2181/;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2 and got a similar error:
„Could not open client transport for any of the Server URI's in ZooKeeper: Invalid status 21“
When looking into the namespace via the CLI, we get the correct address of our Hive instance returned:
ls /hiveserver2
[serverUri=master01:10000;version=3.1.0.3.0.1.0-187;sequence=0000000123]
Fun thing is that most of the clients, who are using the Thriftserver connection outside the cluster f.e. in Excel, are able to connect to it without the problem. We checked the truststores the clients have with the one’s inside the cluster and they are the same.
Best regards, Jan
From: Matt Andruff <ma...@gmail.com>
Reply-To: "user@ambari.apache.org" <us...@ambari.apache.org>
Date: Wednesday, December 18, 2019 at 3:49 AM
To: "user@ambari.apache.org" <us...@ambari.apache.org>
Subject: Re: Problem with Spark2 Thriftserver Alert in Ambari
So if your not able to connect with beeline with that trustore info you have a typo. Either a typo in your connection string.(unlikely) or the trustore isn't valid.(a different type of typo)
Can you rules either of them out as being a typo?
(Does using the zookeeper namespace work connection string for hive work). You know the one that's in Ambari, on the hive summary page?
Hive does actually right the connection string it wants you to use to the zookeeper namespace. Can you connect to zookeeper and check what string it wants you to use? (Send me a reply and I'll dig up how to do this if you don't know how to do it. If you have more than 1 hive instance their will be more than 1 entry... I'm not going to respond until tomorrow just so you know ).
On Tue., Dec. 17, 2019, 12:00 Jan Hentschel, <ja...@ultratendency.com>> wrote:
We already tried the SSL configuration in the Beeline connection string with a similar result. Hive is in binary mode with TLS. Below the output when trying to use the Truststore parameters in the connection string (resulting in the same output in the Thriftserver logs):
spark@master01:/$ /usr/hdp/current/spark2-client/bin/beeline -u 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit ' -e 'show databases'
Connecting to jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
19/11/01 10:35:20 INFO Utils: Supplied authorities: master01:10016
19/11/01 10:35:20 INFO Utils: Resolved authority: master01:10016
19/11/01 10:35:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/11/01 10:35:20 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
19/11/01 10:35:21 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
19/11/01 10:35:21 INFO HiveConnection: Transport Used for JDBC connection: binary
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit: Invalid status 21 (state=08S01,code=0)
19/11/01 10:35:21 INFO Utils: Supplied authorities: master01:10016
19/11/01 10:35:21 INFO Utils: Resolved authority: master01:10016
19/11/01 10:35:21 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
19/11/01 10:35:21 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
19/11/01 10:35:21 INFO HiveConnection: Transport Used for JDBC connection: binary
No current connection
19/11/01 10:35:21 INFO Utils: Supplied authorities: master01:10016
19/11/01 10:35:21 INFO Utils: Resolved authority: master01:10016
19/11/01 10:35:21 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
19/11/01 10:35:21 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
19/11/01 10:35:21 INFO HiveConnection: Transport Used for JDBC connection: binary
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit: Invalid status 21 (state=08S01,code=0)
From: Matt Andruff <ma...@gmail.com>>
Reply-To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Date: Friday, December 13, 2019 at 8:17 PM
To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Subject: Re: Problem with Spark2 Thriftserver Alert in Ambari
It's clear from the Ambati alert the thrift server connection it's trying to use binary connection (and missing the trustore). Are you using hive in binary with SSL or https?
On Fri., Dec. 13, 2019, 14:13 Matt Andruff, <ma...@gmail.com>> wrote:
What happens when you try to connect via beeline with:
jdbc:hive2://<host>:<port>/<db>;ssl=true;sslTrustStore=<trust_store_path>;trustStorePassword=<trust_store_password>
?
Or are you just using the zookeeper namespace to connect?
On Thu., Dec. 12, 2019, 14:03 Jan Hentschel, <ja...@ultratendency.com>> wrote:
Hello everybody,
we're having problems with the Ambari alert for the Spark Thriftserver, complaining about the Beeline connection to the Thriftserver (see the alert below):
Connection failed on host master01:10016 (Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts/alert_spark2_thrift_port.py", line 147, in execute
Execute(cmd, user=sparkuser, path=[beeline_cmd], timeout=CHECK_COMMAND_TIMEOUT_DEFAULT)
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
raise ExecutionFailed(err_msg, code, out, err)
ExecutionFailed: Execution of '! /usr/hdp/current/spark2-client/bin/beeline -u 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL' ' returned 1. Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)
)
Running it with a user the same problem comes up:
dr_who@master01:/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts$ ! /usr/hdp/current/spark2-client/bin/beeline -u 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary' -e ''
Connecting to jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/09/30 10:40:26 INFO Utils: Supplied authorities: master01:10016
19/09/30 10:40:26 INFO Utils: Resolved authority: master01:10016
19/09/30 10:40:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/09/30 10:40:26 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/09/30 10:40:27 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC connection: binary
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary: java.net.SocketException: Connection reset (state=08S01,code=0)
Beeline version 1.21.2.3.0.1.0-187 by Apache Hive
19/09/30 10:40:27 INFO Utils: Supplied authorities: master01:10016
19/09/30 10:40:27 INFO Utils: Resolved authority: master01:10016
19/09/30 10:40:27 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/09/30 10:40:27 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC connection: binary
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary: java.net.SocketException: Broken pipe (Write failed) (state=08S01,code=0)
When trying to run the same with the Spark user from the command line, a similar issues arises:
spark@master01:~$ /usr/hdp/current/spark2-client/bin/beeline -u 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary' -e 'show databases'
Connecting to jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
19/11/13 14:42:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection: binary
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary: Invalid status 21 (state=08S01,code=0)
19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection: binary
No current connection
19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection: binary
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary: Invalid status 21 (state=08S01,code=0)
Looking at the Thriftserver logs reveals the following:
19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
at sun.security.ssl.InputRecord.read(InputRecord.java:527)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 16 more
19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
at sun.security.ssl.InputRecord.read(InputRecord.java:527)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 16 more
TLS is configured on the Hive side, but still it seems that instead of a TLS connection Beeline or Spark tries to establish a plaintext connection. Does anyone know how to resolve this problem? Besides the points mentioned above the Thriftserver, f.e. when connecting to it via ODBC.
Re: Problem with Spark2 Thriftserver Alert in Ambari
Posted by Matt Andruff <ma...@gmail.com>.
So if your not able to connect with beeline with that trustore info you
have a typo. Either a typo in your connection string.(unlikely) or the
trustore isn't valid.(a different type of typo)
Can you rules either of them out as being a typo?
(Does using the zookeeper namespace work connection string for hive work).
You know the one that's in Ambari, on the hive summary page?
Hive does actually right the connection string it wants you to use to the
zookeeper namespace. Can you connect to zookeeper and check what string it
wants you to use? (Send me a reply and I'll dig up how to do this if you
don't know how to do it. If you have more than 1 hive instance their will
be more than 1 entry... I'm not going to respond until tomorrow just so you
know ).
On Tue., Dec. 17, 2019, 12:00 Jan Hentschel, <
jan.hentschel@ultratendency.com> wrote:
> We already tried the SSL configuration in the Beeline connection string
> with a similar result. Hive is in binary mode with TLS. Below the output
> when trying to use the Truststore parameters in the connection string
> (resulting in the same output in the Thriftserver logs):
>
>
>
> spark@master01:/$ /usr/hdp/current/spark2-client/bin/beeline -u
> 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
> ' -e 'show databases'
>
> Connecting to jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM
> ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
>
> 19/11/01 10:35:20 INFO Utils: Supplied authorities: master01:10016
>
> 19/11/01 10:35:20 INFO Utils: Resolved authority: master01:10016
>
> 19/11/01 10:35:20 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 19/11/01 10:35:20 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM
> ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
>
> 19/11/01 10:35:21 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM
> ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
>
> 19/11/01 10:35:21 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit:
> Invalid status 21 (state=08S01,code=0)
>
> 19/11/01 10:35:21 INFO Utils: Supplied authorities: master01:10016
>
> 19/11/01 10:35:21 INFO Utils: Resolved authority: master01:10016
>
> 19/11/01 10:35:21 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM
> ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
>
> 19/11/01 10:35:21 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM
> ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
>
> 19/11/01 10:35:21 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> No current connection
>
> 19/11/01 10:35:21 INFO Utils: Supplied authorities: master01:10016
>
> 19/11/01 10:35:21 INFO Utils: Resolved authority: master01:10016
>
> 19/11/01 10:35:21 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM
> ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
>
> 19/11/01 10:35:21 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM
> ;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
>
> 19/11/01 10:35:21 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit:
> Invalid status 21 (state=08S01,code=0)
>
>
>
> *From: *Matt Andruff <ma...@gmail.com>
> *Reply-To: *"user@ambari.apache.org" <us...@ambari.apache.org>
> *Date: *Friday, December 13, 2019 at 8:17 PM
> *To: *"user@ambari.apache.org" <us...@ambari.apache.org>
> *Subject: *Re: Problem with Spark2 Thriftserver Alert in Ambari
>
>
>
> It's clear from the Ambati alert the thrift server connection it's trying
> to use binary connection (and missing the trustore). Are you using hive in
> binary with SSL or https?
>
>
>
> On Fri., Dec. 13, 2019, 14:13 Matt Andruff, <ma...@gmail.com>
> wrote:
>
> What happens when you try to connect via beeline with:
>
>
>
>
> jdbc:hive2://<host>:<port>/<db>;ssl=true;sslTrustStore=<trust_store_path>;trustStorePassword=<trust_store_password>
>
>
>
> ?
>
>
>
> Or are you just using the zookeeper namespace to connect?
>
>
>
> On Thu., Dec. 12, 2019, 14:03 Jan Hentschel, <
> jan.hentschel@ultratendency.com> wrote:
>
> Hello everybody,
>
>
>
> we're having problems with the Ambari alert for the Spark Thriftserver,
> complaining about the Beeline connection to the Thriftserver (see the alert
> below):
>
>
>
> Connection failed on host master01:10016 (Traceback (most recent call
> last):
>
> File
> "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts/alert_spark2_thrift_port.py",
> line 147, in execute
>
> Execute(cmd, user=sparkuser, path=[beeline_cmd],
> timeout=CHECK_COMMAND_TIMEOUT_DEFAULT)
>
> File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line
> 166, in __init__
>
> self.env.run()
>
> File
> "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line
> 160, in run
>
> self.run_action(resource, action)
>
> File
> "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line
> 124, in run_action
>
> provider_action()
>
> File
> "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py",
> line 263, in action_run
>
> returns=self.resource.returns)
>
> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
> 72, in inner
>
> result = function(command, **kwargs)
>
> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
> 102, in checked_call
>
> tries=tries, try_sleep=try_sleep,
> timeout_kill_strategy=timeout_kill_strategy, returns=returns)
>
> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
> 150, in _call_wrapper
>
> result = _call(command, **kwargs_copy)
>
> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
> 314, in _call
>
> raise ExecutionFailed(err_msg, code, out, err)
>
> ExecutionFailed: Execution of '!
> /usr/hdp/current/spark2-client/bin/beeline -u
> 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary'
> -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
> ' returned 1. Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> java.net.ConnectException: Connection refused (Connection refused)
> (state=08S01,code=0)
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> java.net.ConnectException: Connection refused (Connection refused)
> (state=08S01,code=0)
>
> )
>
>
>
> Running it with a user the same problem comes up:
>
>
>
> dr_who@master01:/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts$
> ! /usr/hdp/current/spark2-client/bin/beeline -u
> 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary'
> -e ''
>
> Connecting to jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/09/30 10:40:26 INFO Utils: Supplied authorities: master01:10016
>
> 19/09/30 10:40:26 INFO Utils: Resolved authority: master01:10016
>
> 19/09/30 10:40:26 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 19/09/30 10:40:26 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/09/30 10:40:27 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> java.net.SocketException: Connection reset (state=08S01,code=0)
>
> Beeline version 1.21.2.3.0.1.0-187 by Apache Hive
>
> 19/09/30 10:40:27 INFO Utils: Supplied authorities: master01:10016
>
> 19/09/30 10:40:27 INFO Utils: Resolved authority: master01:10016
>
> 19/09/30 10:40:27 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/09/30 10:40:27 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> java.net.SocketException: Broken pipe (Write failed) (state=08S01,code=0)
>
>
>
> When trying to run the same with the Spark user from the command line, a
> similar issues arises:
>
>
>
> spark@master01:~$ /usr/hdp/current/spark2-client/bin/beeline -u
> 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary'
> -e 'show databases'
>
> Connecting to jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
>
> 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
>
> 19/11/13 14:42:27 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> Invalid status 21 (state=08S01,code=0)
>
> 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
>
> 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
>
> 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> No current connection
>
> 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
>
> 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
>
> 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> Invalid status 21 (state=08S01,code=0)
>
>
>
> Looking at the Thriftserver logs reveals the following:
>
>
>
> 19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during
> processing of message.
>
> java.lang.RuntimeException:
> org.apache.thrift.transport.TTransportException:
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:360)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
>
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.thrift.transport.TTransportException:
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>
> at
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
>
> at
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
>
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>
> ... 10 more
>
> Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext
> connection?
>
> at
> sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
>
> at sun.security.ssl.InputRecord.read(InputRecord.java:527)
>
> at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
>
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
>
> at
> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
>
> at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
>
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>
> ... 16 more
>
> 19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during
> processing of message.
>
> java.lang.RuntimeException:
> org.apache.thrift.transport.TTransportException:
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:360)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
>
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.thrift.transport.TTransportException:
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>
> at
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
>
> at
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
>
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>
> ... 10 more
>
> Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext
> connection?
>
> at
> sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
>
> at sun.security.ssl.InputRecord.read(InputRecord.java:527)
>
> at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
>
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
>
> at
> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
>
> at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
>
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>
> ... 16 more
>
>
>
> TLS is configured on the Hive side, but still it seems that instead of a
> TLS connection Beeline or Spark tries to establish a plaintext connection.
> Does anyone know how to resolve this problem? Besides the points
> mentioned above the Thriftserver, f.e. when connecting to it via ODBC.
>
>
>
>
Re: Problem with Spark2 Thriftserver Alert in Ambari
Posted by Jan Hentschel <ja...@ultratendency.com>.
We already tried the SSL configuration in the Beeline connection string with a similar result. Hive is in binary mode with TLS. Below the output when trying to use the Truststore parameters in the connection string (resulting in the same output in the Thriftserver logs):
spark@master01:/$ /usr/hdp/current/spark2-client/bin/beeline -u 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit ' -e 'show databases'
Connecting to jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
19/11/01 10:35:20 INFO Utils: Supplied authorities: master01:10016
19/11/01 10:35:20 INFO Utils: Resolved authority: master01:10016
19/11/01 10:35:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/11/01 10:35:20 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
19/11/01 10:35:21 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
19/11/01 10:35:21 INFO HiveConnection: Transport Used for JDBC connection: binary
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit: Invalid status 21 (state=08S01,code=0)
19/11/01 10:35:21 INFO Utils: Supplied authorities: master01:10016
19/11/01 10:35:21 INFO Utils: Resolved authority: master01:10016
19/11/01 10:35:21 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
19/11/01 10:35:21 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
19/11/01 10:35:21 INFO HiveConnection: Transport Used for JDBC connection: binary
No current connection
19/11/01 10:35:21 INFO Utils: Supplied authorities: master01:10016
19/11/01 10:35:21 INFO Utils: Resolved authority: master01:10016
19/11/01 10:35:21 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
19/11/01 10:35:21 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit
19/11/01 10:35:21 INFO HiveConnection: Transport Used for JDBC connection: binary
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary;ssl=true;sslTrustStore=/etc/ssl/jks/certs/truststore.jks;trustStorePassword=changeit: Invalid status 21 (state=08S01,code=0)
From: Matt Andruff <ma...@gmail.com>
Reply-To: "user@ambari.apache.org" <us...@ambari.apache.org>
Date: Friday, December 13, 2019 at 8:17 PM
To: "user@ambari.apache.org" <us...@ambari.apache.org>
Subject: Re: Problem with Spark2 Thriftserver Alert in Ambari
It's clear from the Ambati alert the thrift server connection it's trying to use binary connection (and missing the trustore). Are you using hive in binary with SSL or https?
On Fri., Dec. 13, 2019, 14:13 Matt Andruff, <ma...@gmail.com>> wrote:
What happens when you try to connect via beeline with:
jdbc:hive2://<host>:<port>/<db>;ssl=true;sslTrustStore=<trust_store_path>;trustStorePassword=<trust_store_password>
?
Or are you just using the zookeeper namespace to connect?
On Thu., Dec. 12, 2019, 14:03 Jan Hentschel, <ja...@ultratendency.com>> wrote:
Hello everybody,
we're having problems with the Ambari alert for the Spark Thriftserver, complaining about the Beeline connection to the Thriftserver (see the alert below):
Connection failed on host master01:10016 (Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts/alert_spark2_thrift_port.py", line 147, in execute
Execute(cmd, user=sparkuser, path=[beeline_cmd], timeout=CHECK_COMMAND_TIMEOUT_DEFAULT)
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
raise ExecutionFailed(err_msg, code, out, err)
ExecutionFailed: Execution of '! /usr/hdp/current/spark2-client/bin/beeline -u 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL' ' returned 1. Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)
)
Running it with a user the same problem comes up:
dr_who@master01:/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts$ ! /usr/hdp/current/spark2-client/bin/beeline -u 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary' -e ''
Connecting to jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/09/30 10:40:26 INFO Utils: Supplied authorities: master01:10016
19/09/30 10:40:26 INFO Utils: Resolved authority: master01:10016
19/09/30 10:40:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/09/30 10:40:26 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/09/30 10:40:27 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC connection: binary
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary: java.net.SocketException: Connection reset (state=08S01,code=0)
Beeline version 1.21.2.3.0.1.0-187 by Apache Hive
19/09/30 10:40:27 INFO Utils: Supplied authorities: master01:10016
19/09/30 10:40:27 INFO Utils: Resolved authority: master01:10016
19/09/30 10:40:27 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/09/30 10:40:27 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC connection: binary
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary: java.net.SocketException: Broken pipe (Write failed) (state=08S01,code=0)
When trying to run the same with the Spark user from the command line, a similar issues arises:
spark@master01:~$ /usr/hdp/current/spark2-client/bin/beeline -u 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary' -e 'show databases'
Connecting to jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
19/11/13 14:42:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection: binary
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary: Invalid status 21 (state=08S01,code=0)
19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection: binary
No current connection
19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary
19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection: binary
Error: Could not open client transport with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM<ma...@EXAMPLE.COM>;transportMode=binary: Invalid status 21 (state=08S01,code=0)
Looking at the Thriftserver logs reveals the following:
19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
at sun.security.ssl.InputRecord.read(InputRecord.java:527)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 16 more
19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
at sun.security.ssl.InputRecord.read(InputRecord.java:527)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 16 more
TLS is configured on the Hive side, but still it seems that instead of a TLS connection Beeline or Spark tries to establish a plaintext connection. Does anyone know how to resolve this problem? Besides the points mentioned above the Thriftserver, f.e. when connecting to it via ODBC.
Re: Problem with Spark2 Thriftserver Alert in Ambari
Posted by Matt Andruff <ma...@gmail.com>.
It's clear from the Ambati alert the thrift server connection it's trying
to use binary connection (and missing the trustore). Are you using hive in
binary with SSL or https?
On Fri., Dec. 13, 2019, 14:13 Matt Andruff, <ma...@gmail.com> wrote:
> What happens when you try to connect via beeline with:
>
>
> jdbc:hive2://<host>:<port>/<db>;ssl=true;sslTrustStore=<trust_store_path>;trustStorePassword=<trust_store_password>
>
> ?
>
> Or are you just using the zookeeper namespace to connect?
>
> On Thu., Dec. 12, 2019, 14:03 Jan Hentschel, <
> jan.hentschel@ultratendency.com> wrote:
>
>> Hello everybody,
>>
>>
>>
>> we're having problems with the Ambari alert for the Spark Thriftserver,
>> complaining about the Beeline connection to the Thriftserver (see the alert
>> below):
>>
>>
>>
>> Connection failed on host master01:10016 (Traceback (most recent call
>> last):
>>
>> File
>> "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts/alert_spark2_thrift_port.py",
>> line 147, in execute
>>
>> Execute(cmd, user=sparkuser, path=[beeline_cmd],
>> timeout=CHECK_COMMAND_TIMEOUT_DEFAULT)
>>
>> File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line
>> 166, in __init__
>>
>> self.env.run()
>>
>> File
>> "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line
>> 160, in run
>>
>> self.run_action(resource, action)
>>
>> File
>> "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line
>> 124, in run_action
>>
>> provider_action()
>>
>> File
>> "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py",
>> line 263, in action_run
>>
>> returns=self.resource.returns)
>>
>> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py",
>> line 72, in inner
>>
>> result = function(command, **kwargs)
>>
>> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py",
>> line 102, in checked_call
>>
>> tries=tries, try_sleep=try_sleep,
>> timeout_kill_strategy=timeout_kill_strategy, returns=returns)
>>
>> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py",
>> line 150, in _call_wrapper
>>
>> result = _call(command, **kwargs_copy)
>>
>> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py",
>> line 314, in _call
>>
>> raise ExecutionFailed(err_msg, code, out, err)
>>
>> ExecutionFailed: Execution of '!
>> /usr/hdp/current/spark2-client/bin/beeline -u
>> 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary'
>> -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
>> ' returned 1. Error: Could not open client transport with JDBC Uri:
>> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
>> java.net.ConnectException: Connection refused (Connection refused)
>> (state=08S01,code=0)
>>
>> Error: Could not open client transport with JDBC Uri:
>> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
>> java.net.ConnectException: Connection refused (Connection refused)
>> (state=08S01,code=0)
>>
>> )
>>
>>
>>
>> Running it with a user the same problem comes up:
>>
>>
>>
>> dr_who@master01:/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts$
>> ! /usr/hdp/current/spark2-client/bin/beeline -u
>> 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary'
>> -e ''
>>
>> Connecting to jdbc:hive2://master01:10016/default;principal=spark/
>> master01@EXAMPLE.COM;transportMode=binary
>>
>> 19/09/30 10:40:26 INFO Utils: Supplied authorities: master01:10016
>>
>> 19/09/30 10:40:26 INFO Utils: Resolved authority: master01:10016
>>
>> 19/09/30 10:40:26 WARN NativeCodeLoader: Unable to load native-hadoop
>> library for your platform... using builtin-java classes where applicable
>>
>> 19/09/30 10:40:26 INFO HiveConnection: Will try to open client transport
>> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
>> master01@EXAMPLE.COM;transportMode=binary
>>
>> 19/09/30 10:40:27 INFO HiveConnection: Could not open client transport
>> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
>> master01@EXAMPLE.COM;transportMode=binary
>>
>> 19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC
>> connection: binary
>>
>> Error: Could not open client transport with JDBC Uri:
>> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
>> java.net.SocketException: Connection reset (state=08S01,code=0)
>>
>> Beeline version 1.21.2.3.0.1.0-187 by Apache Hive
>>
>> 19/09/30 10:40:27 INFO Utils: Supplied authorities: master01:10016
>>
>> 19/09/30 10:40:27 INFO Utils: Resolved authority: master01:10016
>>
>> 19/09/30 10:40:27 INFO HiveConnection: Will try to open client transport
>> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
>> master01@EXAMPLE.COM;transportMode=binary
>>
>> 19/09/30 10:40:27 INFO HiveConnection: Could not open client transport
>> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
>> master01@EXAMPLE.COM;transportMode=binary
>>
>> 19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC
>> connection: binary
>>
>> Error: Could not open client transport with JDBC Uri:
>> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
>> java.net.SocketException: Broken pipe (Write failed) (state=08S01,code=0)
>>
>>
>>
>> When trying to run the same with the Spark user from the command line, a
>> similar issues arises:
>>
>>
>>
>> spark@master01:~$ /usr/hdp/current/spark2-client/bin/beeline -u
>> 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary'
>> -e 'show databases'
>>
>> Connecting to jdbc:hive2://master01:10016/default;principal=spark/
>> master01@EXAMPLE.COM;transportMode=binary
>>
>> 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
>>
>> 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
>>
>> 19/11/13 14:42:27 WARN NativeCodeLoader: Unable to load native-hadoop
>> library for your platform... using builtin-java classes where applicable
>>
>> 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport
>> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
>> master01@EXAMPLE.COM;transportMode=binary
>>
>> 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport
>> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
>> master01@EXAMPLE.COM;transportMode=binary
>>
>> 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC
>> connection: binary
>>
>> Error: Could not open client transport with JDBC Uri:
>> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
>> Invalid status 21 (state=08S01,code=0)
>>
>> 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
>>
>> 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
>>
>> 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport
>> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
>> master01@EXAMPLE.COM;transportMode=binary
>>
>> 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport
>> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
>> master01@EXAMPLE.COM;transportMode=binary
>>
>> 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC
>> connection: binary
>>
>> No current connection
>>
>> 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
>>
>> 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
>>
>> 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport
>> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
>> master01@EXAMPLE.COM;transportMode=binary
>>
>> 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport
>> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
>> master01@EXAMPLE.COM;transportMode=binary
>>
>> 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC
>> connection: binary
>>
>> Error: Could not open client transport with JDBC Uri:
>> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
>> Invalid status 21 (state=08S01,code=0)
>>
>>
>>
>> Looking at the Thriftserver logs reveals the following:
>>
>>
>>
>> 19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during
>> processing of message.
>>
>> java.lang.RuntimeException:
>> org.apache.thrift.transport.TTransportException:
>> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>>
>> at
>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>>
>> at
>> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
>>
>> at
>> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at javax.security.auth.Subject.doAs(Subject.java:360)
>>
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
>>
>> at
>> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
>>
>> at
>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Caused by: org.apache.thrift.transport.TTransportException:
>> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>>
>> at
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>>
>> at
>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>>
>> at
>> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
>>
>> at
>> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>>
>> at
>> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
>>
>> at
>> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>>
>> at
>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>>
>> ... 10 more
>>
>> Caused by: javax.net.ssl.SSLException: Unrecognized SSL message,
>> plaintext connection?
>>
>> at
>> sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
>>
>> at sun.security.ssl.InputRecord.read(InputRecord.java:527)
>>
>> at
>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
>>
>> at
>> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
>>
>> at
>> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
>>
>> at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
>>
>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>>
>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>>
>> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>>
>> at
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>>
>> ... 16 more
>>
>> 19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during
>> processing of message.
>>
>> java.lang.RuntimeException:
>> org.apache.thrift.transport.TTransportException:
>> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>>
>> at
>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>>
>> at
>> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
>>
>> at
>> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at javax.security.auth.Subject.doAs(Subject.java:360)
>>
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
>>
>> at
>> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
>>
>> at
>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Caused by: org.apache.thrift.transport.TTransportException:
>> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>>
>> at
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>>
>> at
>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>>
>> at
>> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
>>
>> at
>> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>>
>> at
>> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
>>
>> at
>> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>>
>> at
>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>>
>> ... 10 more
>>
>> Caused by: javax.net.ssl.SSLException: Unrecognized SSL message,
>> plaintext connection?
>>
>> at
>> sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
>>
>> at sun.security.ssl.InputRecord.read(InputRecord.java:527)
>>
>> at
>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
>>
>> at
>> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
>>
>> at
>> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
>>
>> at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
>>
>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>>
>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>>
>> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>>
>> at
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>>
>> ... 16 more
>>
>>
>>
>> TLS is configured on the Hive side, but still it seems that instead of a
>> TLS connection Beeline or Spark tries to establish a plaintext connection.
>> Does anyone know how to resolve this problem? Besides the points
>> mentioned above the Thriftserver, f.e. when connecting to it via ODBC.
>>
>>
>>
>
Re: Problem with Spark2 Thriftserver Alert in Ambari
Posted by Matt Andruff <ma...@gmail.com>.
What happens when you try to connect via beeline with:
jdbc:hive2://<host>:<port>/<db>;ssl=true;sslTrustStore=<trust_store_path>;trustStorePassword=<trust_store_password>
?
Or are you just using the zookeeper namespace to connect?
On Thu., Dec. 12, 2019, 14:03 Jan Hentschel, <
jan.hentschel@ultratendency.com> wrote:
> Hello everybody,
>
>
>
> we're having problems with the Ambari alert for the Spark Thriftserver,
> complaining about the Beeline connection to the Thriftserver (see the alert
> below):
>
>
>
> Connection failed on host master01:10016 (Traceback (most recent call
> last):
>
> File
> "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts/alert_spark2_thrift_port.py",
> line 147, in execute
>
> Execute(cmd, user=sparkuser, path=[beeline_cmd],
> timeout=CHECK_COMMAND_TIMEOUT_DEFAULT)
>
> File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line
> 166, in __init__
>
> self.env.run()
>
> File
> "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line
> 160, in run
>
> self.run_action(resource, action)
>
> File
> "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line
> 124, in run_action
>
> provider_action()
>
> File
> "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py",
> line 263, in action_run
>
> returns=self.resource.returns)
>
> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
> 72, in inner
>
> result = function(command, **kwargs)
>
> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
> 102, in checked_call
>
> tries=tries, try_sleep=try_sleep,
> timeout_kill_strategy=timeout_kill_strategy, returns=returns)
>
> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
> 150, in _call_wrapper
>
> result = _call(command, **kwargs_copy)
>
> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
> 314, in _call
>
> raise ExecutionFailed(err_msg, code, out, err)
>
> ExecutionFailed: Execution of '!
> /usr/hdp/current/spark2-client/bin/beeline -u
> 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary'
> -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
> ' returned 1. Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> java.net.ConnectException: Connection refused (Connection refused)
> (state=08S01,code=0)
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> java.net.ConnectException: Connection refused (Connection refused)
> (state=08S01,code=0)
>
> )
>
>
>
> Running it with a user the same problem comes up:
>
>
>
> dr_who@master01:/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts$
> ! /usr/hdp/current/spark2-client/bin/beeline -u
> 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary'
> -e ''
>
> Connecting to jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/09/30 10:40:26 INFO Utils: Supplied authorities: master01:10016
>
> 19/09/30 10:40:26 INFO Utils: Resolved authority: master01:10016
>
> 19/09/30 10:40:26 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 19/09/30 10:40:26 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/09/30 10:40:27 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> java.net.SocketException: Connection reset (state=08S01,code=0)
>
> Beeline version 1.21.2.3.0.1.0-187 by Apache Hive
>
> 19/09/30 10:40:27 INFO Utils: Supplied authorities: master01:10016
>
> 19/09/30 10:40:27 INFO Utils: Resolved authority: master01:10016
>
> 19/09/30 10:40:27 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/09/30 10:40:27 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> java.net.SocketException: Broken pipe (Write failed) (state=08S01,code=0)
>
>
>
> When trying to run the same with the Spark user from the command line, a
> similar issues arises:
>
>
>
> spark@master01:~$ /usr/hdp/current/spark2-client/bin/beeline -u
> 'jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary'
> -e 'show databases'
>
> Connecting to jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
>
> 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
>
> 19/11/13 14:42:27 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> Invalid status 21 (state=08S01,code=0)
>
> 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
>
> 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
>
> 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> No current connection
>
> 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016
>
> 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016
>
> 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport
> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/
> master01@EXAMPLE.COM;transportMode=binary
>
> 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC connection:
> binary
>
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://master01:10016/default;principal=spark/master01@EXAMPLE.COM;transportMode=binary:
> Invalid status 21 (state=08S01,code=0)
>
>
>
> Looking at the Thriftserver logs reveals the following:
>
>
>
> 19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during
> processing of message.
>
> java.lang.RuntimeException:
> org.apache.thrift.transport.TTransportException:
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:360)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
>
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.thrift.transport.TTransportException:
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>
> at
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
>
> at
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
>
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>
> ... 10 more
>
> Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext
> connection?
>
> at
> sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
>
> at sun.security.ssl.InputRecord.read(InputRecord.java:527)
>
> at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
>
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
>
> at
> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
>
> at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
>
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>
> ... 16 more
>
> 19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during
> processing of message.
>
> java.lang.RuntimeException:
> org.apache.thrift.transport.TTransportException:
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:360)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
>
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
>
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.thrift.transport.TTransportException:
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>
> at
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
>
> at
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
>
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>
> ... 10 more
>
> Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext
> connection?
>
> at
> sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
>
> at sun.security.ssl.InputRecord.read(InputRecord.java:527)
>
> at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
>
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
>
> at
> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
>
> at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
>
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>
> ... 16 more
>
>
>
> TLS is configured on the Hive side, but still it seems that instead of a
> TLS connection Beeline or Spark tries to establish a plaintext connection.
> Does anyone know how to resolve this problem? Besides the points
> mentioned above the Thriftserver, f.e. when connecting to it via ODBC.
>
>
>