You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by Aleksandr Saraseka <as...@callfire.com> on 2019/05/14 15:04:24 UTC
PQS + Kerberos problems
Hello, I have HBase + PQS 4.14.1
If I'm trying to connect by think client - everything works, but if I'm
using thin client in PQS logs I can see continuous INFO messages
2019-05-14 13:53:58,701 INFO
org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=10,
retries=35, started=48292 ms ago, cancelled=false, msg=
...
2019-05-14 14:18:41,446 INFO
org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=33,
retries=35, started=510325 ms ago, cancelled=false, msg=
2019-05-14 14:19:01,489 INFO
org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=34,
retries=35, started=530368 ms ago, cancelled=false, msg=
...
2019-05-14 14:18:41,446 INFO
org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=33,
retries=35, started=510325 ms ago, cancelled=false, msg=
2019-05-14 14:19:01,489 INFO
org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=34,
retries=35, started=530368 ms ago, cancelled=false, msg=
2019-05-14 14:19:50,139 INFO
org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=10,
retries=35, started=48480 ms ago, cancelled=false, msg=row
'SYSTEM:CATALOG,,' on table 'hbase:meta' at
region=hbase:meta,,1.1588230740,
hostname=datanode-001.fqdn.com,60020,1557323271824,
seqNum=0
2019-05-14 14:20:10,333 INFO
org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, tries=11,
retries=35, started=68676 ms ago, cancelled=false, msg=row
'SYSTEM:CATALOG,,' on table 'hbase:meta' at
region=hbase:meta,,1.1588230740,
hostname=datanode-001.fqdn.com,60020,1557323271824,
seqNum=0
*Hbase security logs:*
2019-05-14 14:42:19,524 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Auth successful for HTTP/phoenix-queryserver-fqdn.com@REALM.COM
(auth:KERBEROS)
2019-05-14 14:42:19,524 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 10.252.16.253 port: 41040 with version info: version:
"1.2.0-cdh5.14.2" url:
"file:///data/jenkins/workspace/generic-binary-tarball-and-maven-deploy/CDH5.14.2-Packaging-HBase-2018-03-27_13-15-05/hbase-1.2.0-cdh5.14.2"
revision: "Unknown" user: "jenkins" date: "Tue Mar 27 13:31:54 PDT 2018"
src_checksum: "05e6e90e06dd7796f56067208a9bf2aa"
2019-05-14 14:42:29,634 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Auth successful for HTTP/phoenix-queryserver-fqdn.com@REALM.COM
(auth:KERBEROS)
2019-05-14 14:42:29,635 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 10.252.16.253 port: 41046 with version info: version:
"1.2.0-cdh5.14.2" url:
"file:///data/jenkins/workspace/generic-binary-tarball-and-maven-deploy/CDH5.14.2-Packaging-HBase-2018-03-27_13-15-05/hbase-1.2.0-cdh5.14.2"
revision: "Unknown" user: "jenkins" date: "Tue Mar 27 13:31:54 PDT 2018"
src_checksum: "05e6e90e06dd7796f56067208a9bf2aa"
*thin client logs:*
19/05/14 14:10:08 DEBUG execchain.MainClientExec: Proxy auth state:
UNCHALLENGED
19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> POST / HTTP/1.1
19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Content-Length: 137
19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Content-Type:
application/octet-stream
19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Host:
host-fqdn.com:8765
19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Connection:
Keep-Alive
19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> User-Agent:
Apache-HttpClient/4.5.2 (Java/1.8.0_161)
19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Accept-Encoding:
gzip,deflate
19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Authorization:
Negotiate
YIICtQYGKwYBBQUCoIICqTCCAqWgDTALBgkqhkiG9xIBAgKhBAMCAXaiggKMBIICiGCCAoQGCSqGSIb3EgECAgEAboICczCCAm+gAwIBBaEDAgEOogcDBQAgAAAAo4IBgGGCAXwwggF4oAMCAQWhEBsOREFUQVNZUy5DRi5XVEaiQDA+oAMCAQChNzA1GwRIVFRQGy1kYXRhc3lzLXNlY3VyZS1odWUwMDEtc3RnLmMuY2Ytc3RhZ2UuaW50ZXJuYWyjggEbMIIBF6ADAgESoQMCAQOiggEJBIIBBTmyywjx8OcQBfIt2AccWAVPKmyf/UM5lDgRUs9cUixE35x6wl9EoIX6XTsw2hO4ESCt1fFrt4XU3iLadKSM2HtR1Zlp/Q4rw/sATQe2DOEQUquVg548kkvZ9c/M1PrigwZnv28Ew7CZd8FWtTB4+PFPMzbkBBXAI2pMAYDAvvUsrUXlgWBTdrg118uN07BApmK3qCB7BAjIJzTONxFmxXAx+PHlXAOiWMGxsi8V421411VCclGFG0txoSkf6aDdhzcZdQZhuHaL73+JfUXvS8j32cadLjhi1CKtAC8ddfG6NfBFciYNNa8khybXrxd/KsWb6TRAgV9EsZKkCNNOMga09Vv/7qSB1TCB0qADAgESooHKBIHHn1c00ERMjywGVzN7mBqq/2Kc+Is+Oxgs+L4t8A5uyx7/m9axITKA5Zr/wC2jLcVGH+CHEbcgb4zc9hCRIM7RS7MTHINGrh5oZAO6KS/LlrlfYYWuZNQEGuyToStLvV6UyuSs8T/DU9+l6+Y4jx5paoGx9/3Fuliy/RKTYWT4opWH9F8aVA4WIPJbLEOp+LNLmMjko8nzoeLEBkl7OLDcUTsbgYVy7WfnKnq05SKxZaLnVQPSTfWp0UoPj40r6qt0tqntxaJJBg==
19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "POST /
HTTP/1.1[\r][\n]"
19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Content-Length:
137[\r][\n]"
19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Content-Type:
application/octet-stream[\r][\n]"
19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Host:
host-fqdn.com:8765[\r][\n]"
19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Connection:
Keep-Alive[\r][\n]"
19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "User-Agent:
Apache-HttpClient/4.5.2 (Java/1.8.0_161)[\r][\n]"
19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Accept-Encoding:
gzip,deflate[\r][\n]"
19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Authorization:
Negotiate
YIICtQYGKwYBBQUCoIICqTCCAqWgDTALBgkqhkiG9xIBAgKhBAMCAXaiggKMBIICiGCCAoQGCSqGSIb3EgECAgEAboICczCCAm+gAwIBBaEDAgEOogcDBQAgAAAAo4IBgGGCAXwwggF4oAMCAQWhEBsOREFUQVNZUy5DRi5XVEaiQDA+oAMCAQChNzA1GwRIVFRQGy1kYXRhc3lzLXNlY3VyZS1odWUwMDEtc3RnLmMuY2Ytc3RhZ2UuaW50ZXJuYWyjggEbMIIBF6ADAgESoQMCAQOiggEJBIIBBTmyywjx8OcQBfIt2AccWAVPKmyf/UM5lDgRUs9cUixE35x6wl9EoIX6XTsw2hO4ESCt1fFrt4XU3iLadKSM2HtR1Zlp/Q4rw/sATQe2DOEQUquVg548kkvZ9c/M1PrigwZnv28Ew7CZd8FWtTB4+PFPMzbkBBXAI2pMAYDAvvUsrUXlgWBTdrg118uN07BApmK3qCB7BAjIJzTONxFmxXAx+PHlXAOiWMGxsi8V421411VCclGFG0txoSkf6aDdhzcZdQZhuHaL73+JfUXvS8j32cadLjhi1CKtAC8ddfG6NfBFciYNNa8khybXrxd/KsWb6TRAgV9EsZKkCNNOMga09Vv/7qSB1TCB0qADAgESooHKBIHHn1c00ERMjywGVzN7mBqq/2Kc+Is+Oxgs+L4t8A5uyx7/m9axITKA5Zr/wC2jLcVGH+CHEbcgb4zc9hCRIM7RS7MTHINGrh5oZAO6KS/LlrlfYYWuZNQEGuyToStLvV6UyuSs8T/DU9+l6+Y4jx5paoGx9/3Fuliy/RKTYWT4opWH9F8aVA4WIPJbLEOp+LNLmMjko8nzoeLEBkl7OLDcUTsbgYVy7WfnKnq05SKxZaLnVQPSTfWp0UoPj40r6qt0tqntxaJJBg==[\r][\n]"
19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]"
19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "[\n]"
19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >>
"?org.apache.calcite.avatica.proto.Requests$OpenConnectionRequest[0x12]F[\n]"
19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >>
"$5de75f3c-d53d-4a53-b78c-4167156a6b67[0x12][0x10][\n]"
19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >>
"[0x8]password[0x12][0x4]none[0x12][0xc][\n]"
19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >>
"[0x4]user[0x12][0x4]none"
*and thin client fails with:*
Tue May 14 14:59:43 UTC 2019,
RpcRetryingCaller{globalStartTime=1557845452306, pause=100, retries=35},
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to
data-node001.fqdn.com/ip:60020 failed on local exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection
to datasys-secure-hbase-data001-
stg.c.cf-stage.internal/10.252.20.182:60020 is closing. Call id=69,
waitTime=15
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException:
Call to data-node001.fqdn.com/ip:60020 failed on local exception: org.apac
he.hadoop.hbase.exceptions.ConnectionClosingException: Connection to
data-node001.fqdn.com/ip:60020 is closing. Call id=69, waitTime=15
Firewall is widely open from PQS to all HBase/Hadoop nodes.
Also can someone provide impersonal config for working PQS with Kerberos ?
Maybe I missed something.
--
Aleksandr Saraseka
DBA at EZ Texting
M 380997600401
E asaraseka@eztexting.com
W http://www.eztexting.com
<http://www.eztexting.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<http://facebook.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<http://linkedin.com/company/eztexting/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<http://twitter.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://www.facebook.com/alex.saraseka?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://www.linkedin.com/in/alexander-saraseka-32616076/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Re: PQS + Kerberos problems
Posted by Toshihiro Suzuki <br...@gmail.com>.
Can you please check the regionserver log on datanode-001.fqdn.com that has
hbase:meta region after enabling DEBUG log?
That will show the exact reason why the connections are failing.
On Wed, May 29, 2019 at 10:48 AM Aleksandr Saraseka <as...@eztexting.com>
wrote:
> Thank you Josh, that helps a lot.
> We have Query Server on a dedicated server and none of existing guides
> have an information that we need to have core-site.xml
> with hadoop.security.authentication option set to kerberos.
>
> On Tue, May 28, 2019 at 11:59 PM Josh Elser <el...@apache.org> wrote:
>
>> Make sure you have authorization set up correctly between PQS and HBase.
>>
>> Specifically, you must have the appropriate Hadoop proxyuser rules set
>> up in core-site.xml so that HBase will allow PQS to impersonate the PQS
>> end-user.
>>
>> On 5/14/19 11:04 AM, Aleksandr Saraseka wrote:
>> > Hello, I have HBase + PQS 4.14.1
>> > If I'm trying to connect by think client - everything works, but if I'm
>> > using thin client in PQS logs I can see continuous INFO messages
>> > 2019-05-14 13:53:58,701 INFO
>> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
>> > tries=10, retries=35, started=48292 ms ago, cancelled=false, msg=
>> > ...
>> > 2019-05-14 14:18:41,446 INFO
>> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
>> > tries=33, retries=35, started=510325 ms ago, cancelled=false, msg=
>> > 2019-05-14 14:19:01,489 INFO
>> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
>> > tries=34, retries=35, started=530368 ms ago, cancelled=false, msg=
>> > ...
>> > 2019-05-14 14:18:41,446 INFO
>> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
>> > tries=33, retries=35, started=510325 ms ago, cancelled=false, msg=
>> > 2019-05-14 14:19:01,489 INFO
>> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
>> > tries=34, retries=35, started=530368 ms ago, cancelled=false, msg=
>> > 2019-05-14 14:19:50,139 INFO
>> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
>> > tries=10, retries=35, started=48480 ms ago, cancelled=false, msg=row
>> > 'SYSTEM:CATALOG,,' on table 'hbase:meta' at
>> > region=hbase:meta,,1.1588230740, hostname=datanode-001.fqdn.com
>> > <http://datanode-001.fqdn.com>,60020,1557323271824, seqNum=0
>> > 2019-05-14 14:20:10,333 INFO
>> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
>> > tries=11, retries=35, started=68676 ms ago, cancelled=false, msg=row
>> > 'SYSTEM:CATALOG,,' on table 'hbase:meta' at
>> > region=hbase:meta,,1.1588230740, hostname=datanode-001.fqdn.com
>> > <http://datanode-001.fqdn.com>,60020,1557323271824, seqNum=0
>> >
>> > *Hbase security logs:*
>> > 2019-05-14 14:42:19,524 INFO
>> > SecurityLogger.org.apache.hadoop.hbase.Server: Auth successful for
>> > HTTP/phoenix-queryserver-fqdn.com@REALM.COM
>> > <ma...@REALM.COM> (auth:KERBEROS)
>> > 2019-05-14 14:42:19,524 INFO
>> > SecurityLogger.org.apache.hadoop.hbase.Server: Connection from
>> > 10.252.16.253 port: 41040 with version info: version: "1.2.0-cdh5.14.2"
>> > url:
>> >
>> "file:///data/jenkins/workspace/generic-binary-tarball-and-maven-deploy/CDH5.14.2-Packaging-HBase-2018-03-27_13-15-05/hbase-1.2.0-cdh5.14.2"
>>
>> > revision: "Unknown" user: "jenkins" date: "Tue Mar 27 13:31:54 PDT
>> 2018"
>> > src_checksum: "05e6e90e06dd7796f56067208a9bf2aa"
>> > 2019-05-14 14:42:29,634 INFO
>> > SecurityLogger.org.apache.hadoop.hbase.Server: Auth successful for
>> > HTTP/phoenix-queryserver-fqdn.com@REALM.COM
>> > <ma...@REALM.COM> (auth:KERBEROS)
>> > 2019-05-14 14:42:29,635 INFO
>> > SecurityLogger.org.apache.hadoop.hbase.Server: Connection from
>> > 10.252.16.253 port: 41046 with version info: version: "1.2.0-cdh5.14.2"
>> > url:
>> >
>> "file:///data/jenkins/workspace/generic-binary-tarball-and-maven-deploy/CDH5.14.2-Packaging-HBase-2018-03-27_13-15-05/hbase-1.2.0-cdh5.14.2"
>>
>> > revision: "Unknown" user: "jenkins" date: "Tue Mar 27 13:31:54 PDT
>> 2018"
>> > src_checksum: "05e6e90e06dd7796f56067208a9bf2aa"
>> >
>> >
>> > *thin client logs:*
>> > 19/05/14 14:10:08 DEBUG execchain.MainClientExec: Proxy auth state:
>> > UNCHALLENGED
>> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> POST / HTTP/1.1
>> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >>
>> Content-Length: 137
>> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Content-Type:
>> > application/octet-stream
>> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Host:
>> > host-fqdn.com:8765 <http://host-fqdn.com:8765>
>> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Connection:
>> > Keep-Alive
>> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> User-Agent:
>> > Apache-HttpClient/4.5.2 (Java/1.8.0_161)
>> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >>
>> > Accept-Encoding: gzip,deflate
>> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Authorization:
>> > Negotiate
>> >
>> YIICtQYGKwYBBQUCoIICqTCCAqWgDTALBgkqhkiG9xIBAgKhBAMCAXaiggKMBIICiGCCAoQGCSqGSIb3EgECAgEAboICczCCAm+gAwIBBaEDAgEOogcDBQAgAAAAo4IBgGGCAXwwggF4oAMCAQWhEBsOREFUQVNZUy5DRi5XVEaiQDA+oAMCAQChNzA1GwRIVFRQGy1kYXRhc3lzLXNlY3VyZS1odWUwMDEtc3RnLmMuY2Ytc3RhZ2UuaW50ZXJuYWyjggEbMIIBF6ADAgESoQMCAQOiggEJBIIBBTmyywjx8OcQBfIt2AccWAVPKmyf/UM5lDgRUs9cUixE35x6wl9EoIX6XTsw2hO4ESCt1fFrt4XU3iLadKSM2HtR1Zlp/Q4rw/sATQe2DOEQUquVg548kkvZ9c/M1PrigwZnv28Ew7CZd8FWtTB4+PFPMzbkBBXAI2pMAYDAvvUsrUXlgWBTdrg118uN07BApmK3qCB7BAjIJzTONxFmxXAx+PHlXAOiWMGxsi8V421411VCclGFG0txoSkf6aDdhzcZdQZhuHaL73+JfUXvS8j32cadLjhi1CKtAC8ddfG6NfBFciYNNa8khybXrxd/KsWb6TRAgV9EsZKkCNNOMga09Vv/7qSB1TCB0qADAgESooHKBIHHn1c00ERMjywGVzN7mBqq/2Kc+Is+Oxgs+L4t8A5uyx7/m9axITKA5Zr/wC2jLcVGH+CHEbcgb4zc9hCRIM7RS7MTHINGrh5oZAO6KS/LlrlfYYWuZNQEGuyToStLvV6UyuSs8T/DU9+l6+Y4jx5paoGx9/3Fuliy/RKTYWT4opWH9F8aVA4WIPJbLEOp+LNLmMjko8nzoeLEBkl7OLDcUTsbgYVy7WfnKnq05SKxZaLnVQPSTfWp0UoPj40r6qt0tqntxaJJBg==
>> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "POST /
>> > HTTP/1.1[\r][\n]"
>> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Content-Length:
>> > 137[\r][\n]"
>> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Content-Type:
>> > application/octet-stream[\r][\n]"
>> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Host:
>> > host-fqdn.com:8765[\r][\n]"
>> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Connection:
>> > Keep-Alive[\r][\n]"
>> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "User-Agent:
>> > Apache-HttpClient/4.5.2 (Java/1.8.0_161)[\r][\n]"
>> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Accept-Encoding:
>> > gzip,deflate[\r][\n]"
>> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Authorization:
>> > Negotiate
>> >
>> YIICtQYGKwYBBQUCoIICqTCCAqWgDTALBgkqhkiG9xIBAgKhBAMCAXaiggKMBIICiGCCAoQGCSqGSIb3EgECAgEAboICczCCAm+gAwIBBaEDAgEOogcDBQAgAAAAo4IBgGGCAXwwggF4oAMCAQWhEBsOREFUQVNZUy5DRi5XVEaiQDA+oAMCAQChNzA1GwRIVFRQGy1kYXRhc3lzLXNlY3VyZS1odWUwMDEtc3RnLmMuY2Ytc3RhZ2UuaW50ZXJuYWyjggEbMIIBF6ADAgESoQMCAQOiggEJBIIBBTmyywjx8OcQBfIt2AccWAVPKmyf/UM5lDgRUs9cUixE35x6wl9EoIX6XTsw2hO4ESCt1fFrt4XU3iLadKSM2HtR1Zlp/Q4rw/sATQe2DOEQUquVg548kkvZ9c/M1PrigwZnv28Ew7CZd8FWtTB4+PFPMzbkBBXAI2pMAYDAvvUsrUXlgWBTdrg118uN07BApmK3qCB7BAjIJzTONxFmxXAx+PHlXAOiWMGxsi8V421411VCclGFG0txoSkf6aDdhzcZdQZhuHaL73+JfUXvS8j32cadLjhi1CKtAC8ddfG6NfBFciYNNa8khybXrxd/KsWb6TRAgV9EsZKkCNNOMga09Vv/7qSB1TCB0qADAgESooHKBIHHn1c00ERMjywGVzN7mBqq/2Kc+Is+Oxgs+L4t8A5uyx7/m9axITKA5Zr/wC2jLcVGH+CHEbcgb4zc9hCRIM7RS7MTHINGrh5oZAO6KS/LlrlfYYWuZNQEGuyToStLvV6UyuSs8T/DU9+l6+Y4jx5paoGx9/3Fuliy/RKTYWT4opWH9F8aVA4WIPJbLEOp+LNLmMjko8nzoeLEBkl7OLDcUTsbgYVy7WfnKnq05SKxZaLnVQPSTfWp0UoPj40r6qt0tqntxaJJBg==[\r][\n]"
>> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]"
>> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "[\n]"
>> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >>
>> >
>> "?org.apache.calcite.avatica.proto.Requests$OpenConnectionRequest[0x12]F[\n]"
>> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >>
>> > "$5de75f3c-d53d-4a53-b78c-4167156a6b67[0x12][0x10][\n]"
>> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >>
>> > "[0x8]password[0x12][0x4]none[0x12][0xc][\n]"
>> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >>
>> > "[0x4]user[0x12][0x4]none"
>> >
>> > *and thin client fails with:*
>> > Tue May 14 14:59:43 UTC 2019,
>> > RpcRetryingCaller{globalStartTime=1557845452306, pause=100,
>> retries=35},
>> > org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to
>> > data-node001.fqdn.com/ip:60020 <http://data-node001.fqdn.com/ip:60020>
>> > failed on local exception:
>> > org.apache.hadoop.hbase.exceptions.ConnectionClosingException:
>> > Connection to datasys-secure-hbase-data001-
>> > stg.c.cf-stage.internal/10.252.20.182:60020 <http://10.252.20.182:60020>
>>
>> > is closing. Call id=69, waitTime=15
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157)
>> > at
>> >
>> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> > ... 1 more
>> > Caused by:
>> > org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to
>> > data-node001.fqdn.com/ip:60020 <http://data-node001.fqdn.com/ip:60020>
>> > failed on local exception: org.apac
>> > he.hadoop.hbase.exceptions.ConnectionClosingException: Connection to
>> > data-node001.fqdn.com/ip:60020 <http://data-node001.fqdn.com/ip:60020>
>> > is closing. Call id=69, waitTime=15
>> >
>> > Firewall is widely open from PQS to all HBase/Hadoop nodes.
>> > Also can someone provide impersonal config for working PQS with
>> Kerberos
>> > ? Maybe I missed something.
>> >
>> > --
>> >
>> >
>> > Aleksandr Saraseka
>> > DBA at EZ Texting
>> >
>> > M 380997600401 <tel:380997600401>
>> >
>> > E asaraseka@eztexting.com <ma...@eztexting.com>
>> >
>> > W http://www.eztexting.com
>> > <
>> http://www.eztexting.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>> >
>> >
>> > <
>> http://facebook.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>> > <
>> http://linkedin.com/company/eztexting/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>> > <
>> http://twitter.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>> > <
>> https://www.facebook.com/alex.saraseka?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>> > <
>> https://www.linkedin.com/in/alexander-saraseka-32616076/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature
>> >
>> >
>>
>
>
> --
> Aleksandr Saraseka
> DBA at EZ Texting
> M 380997600401
> E asaraseka@eztexting.com
> W http://www.eztexting.com
> <http://www.eztexting.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
>
> <http://facebook.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <http://linkedin.com/company/eztexting/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <http://twitter.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <https://www.facebook.com/alex.saraseka?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <https://www.linkedin.com/in/alexander-saraseka-32616076/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
Re: PQS + Kerberos problems
Posted by Aleksandr Saraseka <as...@eztexting.com>.
Thank you Josh, that helps a lot.
We have Query Server on a dedicated server and none of existing guides have
an information that we need to have core-site.xml
with hadoop.security.authentication option set to kerberos.
On Tue, May 28, 2019 at 11:59 PM Josh Elser <el...@apache.org> wrote:
> Make sure you have authorization set up correctly between PQS and HBase.
>
> Specifically, you must have the appropriate Hadoop proxyuser rules set
> up in core-site.xml so that HBase will allow PQS to impersonate the PQS
> end-user.
>
> On 5/14/19 11:04 AM, Aleksandr Saraseka wrote:
> > Hello, I have HBase + PQS 4.14.1
> > If I'm trying to connect by think client - everything works, but if I'm
> > using thin client in PQS logs I can see continuous INFO messages
> > 2019-05-14 13:53:58,701 INFO
> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> > tries=10, retries=35, started=48292 ms ago, cancelled=false, msg=
> > ...
> > 2019-05-14 14:18:41,446 INFO
> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> > tries=33, retries=35, started=510325 ms ago, cancelled=false, msg=
> > 2019-05-14 14:19:01,489 INFO
> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> > tries=34, retries=35, started=530368 ms ago, cancelled=false, msg=
> > ...
> > 2019-05-14 14:18:41,446 INFO
> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> > tries=33, retries=35, started=510325 ms ago, cancelled=false, msg=
> > 2019-05-14 14:19:01,489 INFO
> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> > tries=34, retries=35, started=530368 ms ago, cancelled=false, msg=
> > 2019-05-14 14:19:50,139 INFO
> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> > tries=10, retries=35, started=48480 ms ago, cancelled=false, msg=row
> > 'SYSTEM:CATALOG,,' on table 'hbase:meta' at
> > region=hbase:meta,,1.1588230740, hostname=datanode-001.fqdn.com
> > <http://datanode-001.fqdn.com>,60020,1557323271824, seqNum=0
> > 2019-05-14 14:20:10,333 INFO
> > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> > tries=11, retries=35, started=68676 ms ago, cancelled=false, msg=row
> > 'SYSTEM:CATALOG,,' on table 'hbase:meta' at
> > region=hbase:meta,,1.1588230740, hostname=datanode-001.fqdn.com
> > <http://datanode-001.fqdn.com>,60020,1557323271824, seqNum=0
> >
> > *Hbase security logs:*
> > 2019-05-14 14:42:19,524 INFO
> > SecurityLogger.org.apache.hadoop.hbase.Server: Auth successful for
> > HTTP/phoenix-queryserver-fqdn.com@REALM.COM
> > <ma...@REALM.COM> (auth:KERBEROS)
> > 2019-05-14 14:42:19,524 INFO
> > SecurityLogger.org.apache.hadoop.hbase.Server: Connection from
> > 10.252.16.253 port: 41040 with version info: version: "1.2.0-cdh5.14.2"
> > url:
> >
> "file:///data/jenkins/workspace/generic-binary-tarball-and-maven-deploy/CDH5.14.2-Packaging-HBase-2018-03-27_13-15-05/hbase-1.2.0-cdh5.14.2"
>
> > revision: "Unknown" user: "jenkins" date: "Tue Mar 27 13:31:54 PDT 2018"
> > src_checksum: "05e6e90e06dd7796f56067208a9bf2aa"
> > 2019-05-14 14:42:29,634 INFO
> > SecurityLogger.org.apache.hadoop.hbase.Server: Auth successful for
> > HTTP/phoenix-queryserver-fqdn.com@REALM.COM
> > <ma...@REALM.COM> (auth:KERBEROS)
> > 2019-05-14 14:42:29,635 INFO
> > SecurityLogger.org.apache.hadoop.hbase.Server: Connection from
> > 10.252.16.253 port: 41046 with version info: version: "1.2.0-cdh5.14.2"
> > url:
> >
> "file:///data/jenkins/workspace/generic-binary-tarball-and-maven-deploy/CDH5.14.2-Packaging-HBase-2018-03-27_13-15-05/hbase-1.2.0-cdh5.14.2"
>
> > revision: "Unknown" user: "jenkins" date: "Tue Mar 27 13:31:54 PDT 2018"
> > src_checksum: "05e6e90e06dd7796f56067208a9bf2aa"
> >
> >
> > *thin client logs:*
> > 19/05/14 14:10:08 DEBUG execchain.MainClientExec: Proxy auth state:
> > UNCHALLENGED
> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> POST / HTTP/1.1
> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Content-Length:
> 137
> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Content-Type:
> > application/octet-stream
> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Host:
> > host-fqdn.com:8765 <http://host-fqdn.com:8765>
> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Connection:
> > Keep-Alive
> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> User-Agent:
> > Apache-HttpClient/4.5.2 (Java/1.8.0_161)
> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >>
> > Accept-Encoding: gzip,deflate
> > 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Authorization:
> > Negotiate
> >
> YIICtQYGKwYBBQUCoIICqTCCAqWgDTALBgkqhkiG9xIBAgKhBAMCAXaiggKMBIICiGCCAoQGCSqGSIb3EgECAgEAboICczCCAm+gAwIBBaEDAgEOogcDBQAgAAAAo4IBgGGCAXwwggF4oAMCAQWhEBsOREFUQVNZUy5DRi5XVEaiQDA+oAMCAQChNzA1GwRIVFRQGy1kYXRhc3lzLXNlY3VyZS1odWUwMDEtc3RnLmMuY2Ytc3RhZ2UuaW50ZXJuYWyjggEbMIIBF6ADAgESoQMCAQOiggEJBIIBBTmyywjx8OcQBfIt2AccWAVPKmyf/UM5lDgRUs9cUixE35x6wl9EoIX6XTsw2hO4ESCt1fFrt4XU3iLadKSM2HtR1Zlp/Q4rw/sATQe2DOEQUquVg548kkvZ9c/M1PrigwZnv28Ew7CZd8FWtTB4+PFPMzbkBBXAI2pMAYDAvvUsrUXlgWBTdrg118uN07BApmK3qCB7BAjIJzTONxFmxXAx+PHlXAOiWMGxsi8V421411VCclGFG0txoSkf6aDdhzcZdQZhuHaL73+JfUXvS8j32cadLjhi1CKtAC8ddfG6NfBFciYNNa8khybXrxd/KsWb6TRAgV9EsZKkCNNOMga09Vv/7qSB1TCB0qADAgESooHKBIHHn1c00ERMjywGVzN7mBqq/2Kc+Is+Oxgs+L4t8A5uyx7/m9axITKA5Zr/wC2jLcVGH+CHEbcgb4zc9hCRIM7RS7MTHINGrh5oZAO6KS/LlrlfYYWuZNQEGuyToStLvV6UyuSs8T/DU9+l6+Y4jx5paoGx9/3Fuliy/RKTYWT4opWH9F8aVA4WIPJbLEOp+LNLmMjko8nzoeLEBkl7OLDcUTsbgYVy7WfnKnq05SKxZaLnVQPSTfWp0UoPj40r6qt0tqntxaJJBg==
> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "POST /
> > HTTP/1.1[\r][\n]"
> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Content-Length:
> > 137[\r][\n]"
> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Content-Type:
> > application/octet-stream[\r][\n]"
> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Host:
> > host-fqdn.com:8765[\r][\n]"
> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Connection:
> > Keep-Alive[\r][\n]"
> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "User-Agent:
> > Apache-HttpClient/4.5.2 (Java/1.8.0_161)[\r][\n]"
> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Accept-Encoding:
> > gzip,deflate[\r][\n]"
> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Authorization:
> > Negotiate
> >
> YIICtQYGKwYBBQUCoIICqTCCAqWgDTALBgkqhkiG9xIBAgKhBAMCAXaiggKMBIICiGCCAoQGCSqGSIb3EgECAgEAboICczCCAm+gAwIBBaEDAgEOogcDBQAgAAAAo4IBgGGCAXwwggF4oAMCAQWhEBsOREFUQVNZUy5DRi5XVEaiQDA+oAMCAQChNzA1GwRIVFRQGy1kYXRhc3lzLXNlY3VyZS1odWUwMDEtc3RnLmMuY2Ytc3RhZ2UuaW50ZXJuYWyjggEbMIIBF6ADAgESoQMCAQOiggEJBIIBBTmyywjx8OcQBfIt2AccWAVPKmyf/UM5lDgRUs9cUixE35x6wl9EoIX6XTsw2hO4ESCt1fFrt4XU3iLadKSM2HtR1Zlp/Q4rw/sATQe2DOEQUquVg548kkvZ9c/M1PrigwZnv28Ew7CZd8FWtTB4+PFPMzbkBBXAI2pMAYDAvvUsrUXlgWBTdrg118uN07BApmK3qCB7BAjIJzTONxFmxXAx+PHlXAOiWMGxsi8V421411VCclGFG0txoSkf6aDdhzcZdQZhuHaL73+JfUXvS8j32cadLjhi1CKtAC8ddfG6NfBFciYNNa8khybXrxd/KsWb6TRAgV9EsZKkCNNOMga09Vv/7qSB1TCB0qADAgESooHKBIHHn1c00ERMjywGVzN7mBqq/2Kc+Is+Oxgs+L4t8A5uyx7/m9axITKA5Zr/wC2jLcVGH+CHEbcgb4zc9hCRIM7RS7MTHINGrh5oZAO6KS/LlrlfYYWuZNQEGuyToStLvV6UyuSs8T/DU9+l6+Y4jx5paoGx9/3Fuliy/RKTYWT4opWH9F8aVA4WIPJbLEOp+LNLmMjko8nzoeLEBkl7OLDcUTsbgYVy7WfnKnq05SKxZaLnVQPSTfWp0UoPj40r6qt0tqntxaJJBg==[\r][\n]"
> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]"
> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "[\n]"
> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >>
> >
> "?org.apache.calcite.avatica.proto.Requests$OpenConnectionRequest[0x12]F[\n]"
> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >>
> > "$5de75f3c-d53d-4a53-b78c-4167156a6b67[0x12][0x10][\n]"
> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >>
> > "[0x8]password[0x12][0x4]none[0x12][0xc][\n]"
> > 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >>
> > "[0x4]user[0x12][0x4]none"
> >
> > *and thin client fails with:*
> > Tue May 14 14:59:43 UTC 2019,
> > RpcRetryingCaller{globalStartTime=1557845452306, pause=100, retries=35},
> > org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to
> > data-node001.fqdn.com/ip:60020 <http://data-node001.fqdn.com/ip:60020>
> > failed on local exception:
> > org.apache.hadoop.hbase.exceptions.ConnectionClosingException:
> > Connection to datasys-secure-hbase-data001-
> > stg.c.cf-stage.internal/10.252.20.182:60020 <http://10.252.20.182:60020>
>
> > is closing. Call id=69, waitTime=15
> >
> > at
> >
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157)
> > at
> >
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > ... 1 more
> > Caused by:
> > org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to
> > data-node001.fqdn.com/ip:60020 <http://data-node001.fqdn.com/ip:60020>
> > failed on local exception: org.apac
> > he.hadoop.hbase.exceptions.ConnectionClosingException: Connection to
> > data-node001.fqdn.com/ip:60020 <http://data-node001.fqdn.com/ip:60020>
> > is closing. Call id=69, waitTime=15
> >
> > Firewall is widely open from PQS to all HBase/Hadoop nodes.
> > Also can someone provide impersonal config for working PQS with Kerberos
> > ? Maybe I missed something.
> >
> > --
> >
> >
> > Aleksandr Saraseka
> > DBA at EZ Texting
> >
> > M 380997600401 <tel:380997600401>
> >
> > E asaraseka@eztexting.com <ma...@eztexting.com>
> >
> > W http://www.eztexting.com
> > <
> http://www.eztexting.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> >
> >
> > <
> http://facebook.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> > <
> http://linkedin.com/company/eztexting/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> > <
> http://twitter.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> > <
> https://www.facebook.com/alex.saraseka?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> > <
> https://www.linkedin.com/in/alexander-saraseka-32616076/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature
> >
> >
>
--
Aleksandr Saraseka
DBA at EZ Texting
M 380997600401
E asaraseka@eztexting.com
W http://www.eztexting.com
<http://www.eztexting.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<http://facebook.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<http://linkedin.com/company/eztexting/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<http://twitter.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://www.facebook.com/alex.saraseka?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://www.linkedin.com/in/alexander-saraseka-32616076/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Re: PQS + Kerberos problems
Posted by Josh Elser <el...@apache.org>.
Make sure you have authorization set up correctly between PQS and HBase.
Specifically, you must have the appropriate Hadoop proxyuser rules set
up in core-site.xml so that HBase will allow PQS to impersonate the PQS
end-user.
On 5/14/19 11:04 AM, Aleksandr Saraseka wrote:
> Hello, I have HBase + PQS 4.14.1
> If I'm trying to connect by think client - everything works, but if I'm
> using thin client in PQS logs I can see continuous INFO messages
> 2019-05-14 13:53:58,701 INFO
> org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> tries=10, retries=35, started=48292 ms ago, cancelled=false, msg=
> ...
> 2019-05-14 14:18:41,446 INFO
> org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> tries=33, retries=35, started=510325 ms ago, cancelled=false, msg=
> 2019-05-14 14:19:01,489 INFO
> org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> tries=34, retries=35, started=530368 ms ago, cancelled=false, msg=
> ...
> 2019-05-14 14:18:41,446 INFO
> org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> tries=33, retries=35, started=510325 ms ago, cancelled=false, msg=
> 2019-05-14 14:19:01,489 INFO
> org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> tries=34, retries=35, started=530368 ms ago, cancelled=false, msg=
> 2019-05-14 14:19:50,139 INFO
> org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> tries=10, retries=35, started=48480 ms ago, cancelled=false, msg=row
> 'SYSTEM:CATALOG,,' on table 'hbase:meta' at
> region=hbase:meta,,1.1588230740, hostname=datanode-001.fqdn.com
> <http://datanode-001.fqdn.com>,60020,1557323271824, seqNum=0
> 2019-05-14 14:20:10,333 INFO
> org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> tries=11, retries=35, started=68676 ms ago, cancelled=false, msg=row
> 'SYSTEM:CATALOG,,' on table 'hbase:meta' at
> region=hbase:meta,,1.1588230740, hostname=datanode-001.fqdn.com
> <http://datanode-001.fqdn.com>,60020,1557323271824, seqNum=0
>
> *Hbase security logs:*
> 2019-05-14 14:42:19,524 INFO
> SecurityLogger.org.apache.hadoop.hbase.Server: Auth successful for
> HTTP/phoenix-queryserver-fqdn.com@REALM.COM
> <ma...@REALM.COM> (auth:KERBEROS)
> 2019-05-14 14:42:19,524 INFO
> SecurityLogger.org.apache.hadoop.hbase.Server: Connection from
> 10.252.16.253 port: 41040 with version info: version: "1.2.0-cdh5.14.2"
> url:
> "file:///data/jenkins/workspace/generic-binary-tarball-and-maven-deploy/CDH5.14.2-Packaging-HBase-2018-03-27_13-15-05/hbase-1.2.0-cdh5.14.2"
> revision: "Unknown" user: "jenkins" date: "Tue Mar 27 13:31:54 PDT 2018"
> src_checksum: "05e6e90e06dd7796f56067208a9bf2aa"
> 2019-05-14 14:42:29,634 INFO
> SecurityLogger.org.apache.hadoop.hbase.Server: Auth successful for
> HTTP/phoenix-queryserver-fqdn.com@REALM.COM
> <ma...@REALM.COM> (auth:KERBEROS)
> 2019-05-14 14:42:29,635 INFO
> SecurityLogger.org.apache.hadoop.hbase.Server: Connection from
> 10.252.16.253 port: 41046 with version info: version: "1.2.0-cdh5.14.2"
> url:
> "file:///data/jenkins/workspace/generic-binary-tarball-and-maven-deploy/CDH5.14.2-Packaging-HBase-2018-03-27_13-15-05/hbase-1.2.0-cdh5.14.2"
> revision: "Unknown" user: "jenkins" date: "Tue Mar 27 13:31:54 PDT 2018"
> src_checksum: "05e6e90e06dd7796f56067208a9bf2aa"
>
>
> *thin client logs:*
> 19/05/14 14:10:08 DEBUG execchain.MainClientExec: Proxy auth state:
> UNCHALLENGED
> 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> POST / HTTP/1.1
> 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Content-Length: 137
> 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Content-Type:
> application/octet-stream
> 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Host:
> host-fqdn.com:8765 <http://host-fqdn.com:8765>
> 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Connection:
> Keep-Alive
> 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> User-Agent:
> Apache-HttpClient/4.5.2 (Java/1.8.0_161)
> 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >>
> Accept-Encoding: gzip,deflate
> 19/05/14 14:10:08 DEBUG http.headers: http-outgoing-0 >> Authorization:
> Negotiate
> YIICtQYGKwYBBQUCoIICqTCCAqWgDTALBgkqhkiG9xIBAgKhBAMCAXaiggKMBIICiGCCAoQGCSqGSIb3EgECAgEAboICczCCAm+gAwIBBaEDAgEOogcDBQAgAAAAo4IBgGGCAXwwggF4oAMCAQWhEBsOREFUQVNZUy5DRi5XVEaiQDA+oAMCAQChNzA1GwRIVFRQGy1kYXRhc3lzLXNlY3VyZS1odWUwMDEtc3RnLmMuY2Ytc3RhZ2UuaW50ZXJuYWyjggEbMIIBF6ADAgESoQMCAQOiggEJBIIBBTmyywjx8OcQBfIt2AccWAVPKmyf/UM5lDgRUs9cUixE35x6wl9EoIX6XTsw2hO4ESCt1fFrt4XU3iLadKSM2HtR1Zlp/Q4rw/sATQe2DOEQUquVg548kkvZ9c/M1PrigwZnv28Ew7CZd8FWtTB4+PFPMzbkBBXAI2pMAYDAvvUsrUXlgWBTdrg118uN07BApmK3qCB7BAjIJzTONxFmxXAx+PHlXAOiWMGxsi8V421411VCclGFG0txoSkf6aDdhzcZdQZhuHaL73+JfUXvS8j32cadLjhi1CKtAC8ddfG6NfBFciYNNa8khybXrxd/KsWb6TRAgV9EsZKkCNNOMga09Vv/7qSB1TCB0qADAgESooHKBIHHn1c00ERMjywGVzN7mBqq/2Kc+Is+Oxgs+L4t8A5uyx7/m9axITKA5Zr/wC2jLcVGH+CHEbcgb4zc9hCRIM7RS7MTHINGrh5oZAO6KS/LlrlfYYWuZNQEGuyToStLvV6UyuSs8T/DU9+l6+Y4jx5paoGx9/3Fuliy/RKTYWT4opWH9F8aVA4WIPJbLEOp+LNLmMjko8nzoeLEBkl7OLDcUTsbgYVy7WfnKnq05SKxZaLnVQPSTfWp0UoPj40r6qt0tqntxaJJBg==
> 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "POST /
> HTTP/1.1[\r][\n]"
> 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Content-Length:
> 137[\r][\n]"
> 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Content-Type:
> application/octet-stream[\r][\n]"
> 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Host:
> host-fqdn.com:8765[\r][\n]"
> 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Connection:
> Keep-Alive[\r][\n]"
> 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "User-Agent:
> Apache-HttpClient/4.5.2 (Java/1.8.0_161)[\r][\n]"
> 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Accept-Encoding:
> gzip,deflate[\r][\n]"
> 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "Authorization:
> Negotiate
> YIICtQYGKwYBBQUCoIICqTCCAqWgDTALBgkqhkiG9xIBAgKhBAMCAXaiggKMBIICiGCCAoQGCSqGSIb3EgECAgEAboICczCCAm+gAwIBBaEDAgEOogcDBQAgAAAAo4IBgGGCAXwwggF4oAMCAQWhEBsOREFUQVNZUy5DRi5XVEaiQDA+oAMCAQChNzA1GwRIVFRQGy1kYXRhc3lzLXNlY3VyZS1odWUwMDEtc3RnLmMuY2Ytc3RhZ2UuaW50ZXJuYWyjggEbMIIBF6ADAgESoQMCAQOiggEJBIIBBTmyywjx8OcQBfIt2AccWAVPKmyf/UM5lDgRUs9cUixE35x6wl9EoIX6XTsw2hO4ESCt1fFrt4XU3iLadKSM2HtR1Zlp/Q4rw/sATQe2DOEQUquVg548kkvZ9c/M1PrigwZnv28Ew7CZd8FWtTB4+PFPMzbkBBXAI2pMAYDAvvUsrUXlgWBTdrg118uN07BApmK3qCB7BAjIJzTONxFmxXAx+PHlXAOiWMGxsi8V421411VCclGFG0txoSkf6aDdhzcZdQZhuHaL73+JfUXvS8j32cadLjhi1CKtAC8ddfG6NfBFciYNNa8khybXrxd/KsWb6TRAgV9EsZKkCNNOMga09Vv/7qSB1TCB0qADAgESooHKBIHHn1c00ERMjywGVzN7mBqq/2Kc+Is+Oxgs+L4t8A5uyx7/m9axITKA5Zr/wC2jLcVGH+CHEbcgb4zc9hCRIM7RS7MTHINGrh5oZAO6KS/LlrlfYYWuZNQEGuyToStLvV6UyuSs8T/DU9+l6+Y4jx5paoGx9/3Fuliy/RKTYWT4opWH9F8aVA4WIPJbLEOp+LNLmMjko8nzoeLEBkl7OLDcUTsbgYVy7WfnKnq05SKxZaLnVQPSTfWp0UoPj40r6qt0tqntxaJJBg==[\r][\n]"
> 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "[\r][\n]"
> 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >> "[\n]"
> 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >>
> "?org.apache.calcite.avatica.proto.Requests$OpenConnectionRequest[0x12]F[\n]"
> 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >>
> "$5de75f3c-d53d-4a53-b78c-4167156a6b67[0x12][0x10][\n]"
> 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >>
> "[0x8]password[0x12][0x4]none[0x12][0xc][\n]"
> 19/05/14 14:10:08 DEBUG http.wire: http-outgoing-0 >>
> "[0x4]user[0x12][0x4]none"
>
> *and thin client fails with:*
> Tue May 14 14:59:43 UTC 2019,
> RpcRetryingCaller{globalStartTime=1557845452306, pause=100, retries=35},
> org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to
> data-node001.fqdn.com/ip:60020 <http://data-node001.fqdn.com/ip:60020>
> failed on local exception:
> org.apache.hadoop.hbase.exceptions.ConnectionClosingException:
> Connection to datasys-secure-hbase-data001-
> stg.c.cf-stage.internal/10.252.20.182:60020 <http://10.252.20.182:60020>
> is closing. Call id=69, waitTime=15
>
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ... 1 more
> Caused by:
> org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to
> data-node001.fqdn.com/ip:60020 <http://data-node001.fqdn.com/ip:60020>
> failed on local exception: org.apac
> he.hadoop.hbase.exceptions.ConnectionClosingException: Connection to
> data-node001.fqdn.com/ip:60020 <http://data-node001.fqdn.com/ip:60020>
> is closing. Call id=69, waitTime=15
>
> Firewall is widely open from PQS to all HBase/Hadoop nodes.
> Also can someone provide impersonal config for working PQS with Kerberos
> ? Maybe I missed something.
>
> --
>
>
> Aleksandr Saraseka
> DBA at EZ Texting
>
> M 380997600401 <tel:380997600401>
>
> E asaraseka@eztexting.com <ma...@eztexting.com>
>
> W http://www.eztexting.com
> <http://www.eztexting.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
>
> <http://facebook.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <http://linkedin.com/company/eztexting/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <http://twitter.com/eztexting?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <https://www.facebook.com/alex.saraseka?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <https://www.linkedin.com/in/alexander-saraseka-32616076/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>