You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by Reid Chan <re...@outlook.com> on 2019/07/01 09:53:49 UTC
Curl kerberized QueryServer using protobuf type
Hi team and other users,
Following is the script used for connecting to QS,
{code}
#!/usr/bin/env bash
set -u
AVATICA="hostname:8765"
echo $AVATICA
CONNECTION_ID="conn-$(whoami)-$(date +%s)"
echo "Open connection"
openConnectionReq="message OpenConnectionRequest {string connection_id = $CONNECTION_ID;}"
curl -i --negotiate -u : -w "\n" "$AVATICA" -H "Content-Type: application/protobuf" --data "$openConnectionReq"
{code}
But it ended with:
org.apache.calcite.avatica.proto.Responses$ErrorResponse�
�rg.apache.calcite.avatica.com.google.protobuf.InvalidProtocolBufferException$InvalidWireTypeException: Protocol message tag had invalid wire type.
at org.apache.calcite.avatica.com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:111)
at org.apache.calcite.avatica.com.google.protobuf.CodedInputStream$ArrayDecoder.skipField(CodedInputStream.java:591)
at org.apache.calcite.avatica.proto.Common$WireMessage.<init>(Common.java:12544)
at org.apache.calcite.avatica.proto.Common$WireMessage.<init>(Common.java:12511)
at org.apache.calcite.avatica.proto.Common$WireMessage$1.parsePartialFrom(Common.java:13054)
at org.apache.calcite.avatica.proto.Common$WireMessage$1.parsePartialFrom(Common.java:13049)
at org.apache.calcite.avatica.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:91)
at org.apache.calcite.avatica.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:96)
at org.apache.calcite.avatica.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
at org.apache.calcite.avatica.com.google.protobuf.GeneratedMessageV3.parseWithIOException(GeneratedMessageV3.java:311)
at org.apache.calcite.avatica.proto.Common$WireMessage.parseFrom(Common.java:12757)
at org.apache.calcite.avatica.remote.ProtobufTranslationImpl.parseRequest(ProtobufTranslationImpl.java:410)
at org.apache.calcite.avatica.remote.ProtobufHandler.decode(ProtobufHandler.java:51)
at org.apache.calcite.avatica.remote.ProtobufHandler.decode(ProtobufHandler.java:31)
at org.apache.calcite.avatica.remote.AbstractHandler.apply(AbstractHandler.java:93)
at org.apache.calcite.avatica.remote.ProtobufHandler.apply(ProtobufHandler.java:46)
at org.apache.calcite.avatica.server.AvaticaProtobufHandler$2.call(AvaticaProtobufHandler.java:123)
at org.apache.calcite.avatica.server.AvaticaProtobufHandler$2.call(AvaticaProtobufHandler.java:121)
at org.apache.phoenix.queryserver.server.QueryServer$PhoenixDoAsCallback$1.run(QueryServer.java:500)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
at org.apache.phoenix.queryserver.server.QueryServer$PhoenixDoAsCallback.doAsRemoteUser(QueryServer.java:497)
at org.apache.calcite.avatica.server.HttpServer$Builder$1.doAsRemoteUser(HttpServer.java:884)
at org.apache.calcite.avatica.server.AvaticaProtobufHandler.handle(AvaticaProtobufHandler.java:120)
at org.apache.phoenix.shaded.org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:542)
at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.apache.phoenix.shaded.org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.apache.phoenix.shaded.org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
EInvalidWireTypeException: Protocol message tag had invalid wire type. ����*000002
It looked like there's something wrong when decoding response.
I can only ensure kerberos authentication passed, because klist command showed the HTTP service principal after scripted executed.
Can't find any reference about how to curl QS using protobuf instead of json, hopefully can get some help from community!
--------------------------
Best regards,
R.C
Re: Curl kerberized QueryServer using protobuf type
Posted by Reid Chan <re...@outlook.com>.
Thanks Josh!
Finally, I switched to JSON serialization as a simple workaround.
--------------------------
Best regards,
R.C
________________________________________
From: Josh Elser <el...@apache.org>
Sent: 02 July 2019 22:55
To: user@phoenix.apache.org
Subject: Re: Curl kerberized QueryServer using protobuf type
Hey Reid,
Protobuf is a binary format -- this is error'ing out because you're
sending it plain-text.
You're going to have quite a hard time constructing messages in bash
alone. There are lots of language bindings[1]. You should be able to
pick any of these to help encode/decode messages (if you want to use
cURL as your "transport").
IMO, Avatica's protocol is too complex (by necessity of implementing all
of the JDBC API) to just throw some hand-constructed JSON at. I think
the better solution would be to think about some simpler API that
exposes just the bare bones if you want something developer focused.
[1] https://developers.google.com/protocol-buffers/docs/reference/overview
On 7/1/19 5:53 AM, Reid Chan wrote:
> Hi team and other users,
>
> Following is the script used for connecting to QS,
> {code}
> #!/usr/bin/env bash
>
> set -u
>
> AVATICA="hostname:8765"
> echo $AVATICA
> CONNECTION_ID="conn-$(whoami)-$(date +%s)"
>
> echo "Open connection"
> openConnectionReq="message OpenConnectionRequest {string connection_id = $CONNECTION_ID;}"
> curl -i --negotiate -u : -w "\n" "$AVATICA" -H "Content-Type: application/protobuf" --data "$openConnectionReq"
> {code}
>
> But it ended with:
> org.apache.calcite.avatica.proto.Responses$ErrorResponse�
> �rg.apache.calcite.avatica.com.google.protobuf.InvalidProtocolBufferException$InvalidWireTypeException: Protocol message tag had invalid wire type.
> at org.apache.calcite.avatica.com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:111)
> at org.apache.calcite.avatica.com.google.protobuf.CodedInputStream$ArrayDecoder.skipField(CodedInputStream.java:591)
> at org.apache.calcite.avatica.proto.Common$WireMessage.<init>(Common.java:12544)
> at org.apache.calcite.avatica.proto.Common$WireMessage.<init>(Common.java:12511)
> at org.apache.calcite.avatica.proto.Common$WireMessage$1.parsePartialFrom(Common.java:13054)
> at org.apache.calcite.avatica.proto.Common$WireMessage$1.parsePartialFrom(Common.java:13049)
> at org.apache.calcite.avatica.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:91)
> at org.apache.calcite.avatica.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:96)
> at org.apache.calcite.avatica.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> at org.apache.calcite.avatica.com.google.protobuf.GeneratedMessageV3.parseWithIOException(GeneratedMessageV3.java:311)
> at org.apache.calcite.avatica.proto.Common$WireMessage.parseFrom(Common.java:12757)
> at org.apache.calcite.avatica.remote.ProtobufTranslationImpl.parseRequest(ProtobufTranslationImpl.java:410)
> at org.apache.calcite.avatica.remote.ProtobufHandler.decode(ProtobufHandler.java:51)
> at org.apache.calcite.avatica.remote.ProtobufHandler.decode(ProtobufHandler.java:31)
> at org.apache.calcite.avatica.remote.AbstractHandler.apply(AbstractHandler.java:93)
> at org.apache.calcite.avatica.remote.ProtobufHandler.apply(ProtobufHandler.java:46)
> at org.apache.calcite.avatica.server.AvaticaProtobufHandler$2.call(AvaticaProtobufHandler.java:123)
> at org.apache.calcite.avatica.server.AvaticaProtobufHandler$2.call(AvaticaProtobufHandler.java:121)
> at org.apache.phoenix.queryserver.server.QueryServer$PhoenixDoAsCallback$1.run(QueryServer.java:500)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
> at org.apache.phoenix.queryserver.server.QueryServer$PhoenixDoAsCallback.doAsRemoteUser(QueryServer.java:497)
> at org.apache.calcite.avatica.server.HttpServer$Builder$1.doAsRemoteUser(HttpServer.java:884)
> at org.apache.calcite.avatica.server.AvaticaProtobufHandler.handle(AvaticaProtobufHandler.java:120)
> at org.apache.phoenix.shaded.org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:542)
> at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
> at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.apache.phoenix.shaded.org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
> at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at org.apache.phoenix.shaded.org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
> at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> EInvalidWireTypeException: Protocol message tag had invalid wire type. ����*000002
>
>
> It looked like there's something wrong when decoding response.
>
> I can only ensure kerberos authentication passed, because klist command showed the HTTP service principal after scripted executed.
>
> Can't find any reference about how to curl QS using protobuf instead of json, hopefully can get some help from community!
>
>
> --------------------------
>
> Best regards,
> R.C
>
>
Re: Curl kerberized QueryServer using protobuf type
Posted by Josh Elser <el...@apache.org>.
Hey Reid,
Protobuf is a binary format -- this is error'ing out because you're
sending it plain-text.
You're going to have quite a hard time constructing messages in bash
alone. There are lots of language bindings[1]. You should be able to
pick any of these to help encode/decode messages (if you want to use
cURL as your "transport").
IMO, Avatica's protocol is too complex (by necessity of implementing all
of the JDBC API) to just throw some hand-constructed JSON at. I think
the better solution would be to think about some simpler API that
exposes just the bare bones if you want something developer focused.
[1] https://developers.google.com/protocol-buffers/docs/reference/overview
On 7/1/19 5:53 AM, Reid Chan wrote:
> Hi team and other users,
>
> Following is the script used for connecting to QS,
> {code}
> #!/usr/bin/env bash
>
> set -u
>
> AVATICA="hostname:8765"
> echo $AVATICA
> CONNECTION_ID="conn-$(whoami)-$(date +%s)"
>
> echo "Open connection"
> openConnectionReq="message OpenConnectionRequest {string connection_id = $CONNECTION_ID;}"
> curl -i --negotiate -u : -w "\n" "$AVATICA" -H "Content-Type: application/protobuf" --data "$openConnectionReq"
> {code}
>
> But it ended with:
> org.apache.calcite.avatica.proto.Responses$ErrorResponse�
> �rg.apache.calcite.avatica.com.google.protobuf.InvalidProtocolBufferException$InvalidWireTypeException: Protocol message tag had invalid wire type.
> at org.apache.calcite.avatica.com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:111)
> at org.apache.calcite.avatica.com.google.protobuf.CodedInputStream$ArrayDecoder.skipField(CodedInputStream.java:591)
> at org.apache.calcite.avatica.proto.Common$WireMessage.<init>(Common.java:12544)
> at org.apache.calcite.avatica.proto.Common$WireMessage.<init>(Common.java:12511)
> at org.apache.calcite.avatica.proto.Common$WireMessage$1.parsePartialFrom(Common.java:13054)
> at org.apache.calcite.avatica.proto.Common$WireMessage$1.parsePartialFrom(Common.java:13049)
> at org.apache.calcite.avatica.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:91)
> at org.apache.calcite.avatica.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:96)
> at org.apache.calcite.avatica.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> at org.apache.calcite.avatica.com.google.protobuf.GeneratedMessageV3.parseWithIOException(GeneratedMessageV3.java:311)
> at org.apache.calcite.avatica.proto.Common$WireMessage.parseFrom(Common.java:12757)
> at org.apache.calcite.avatica.remote.ProtobufTranslationImpl.parseRequest(ProtobufTranslationImpl.java:410)
> at org.apache.calcite.avatica.remote.ProtobufHandler.decode(ProtobufHandler.java:51)
> at org.apache.calcite.avatica.remote.ProtobufHandler.decode(ProtobufHandler.java:31)
> at org.apache.calcite.avatica.remote.AbstractHandler.apply(AbstractHandler.java:93)
> at org.apache.calcite.avatica.remote.ProtobufHandler.apply(ProtobufHandler.java:46)
> at org.apache.calcite.avatica.server.AvaticaProtobufHandler$2.call(AvaticaProtobufHandler.java:123)
> at org.apache.calcite.avatica.server.AvaticaProtobufHandler$2.call(AvaticaProtobufHandler.java:121)
> at org.apache.phoenix.queryserver.server.QueryServer$PhoenixDoAsCallback$1.run(QueryServer.java:500)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
> at org.apache.phoenix.queryserver.server.QueryServer$PhoenixDoAsCallback.doAsRemoteUser(QueryServer.java:497)
> at org.apache.calcite.avatica.server.HttpServer$Builder$1.doAsRemoteUser(HttpServer.java:884)
> at org.apache.calcite.avatica.server.AvaticaProtobufHandler.handle(AvaticaProtobufHandler.java:120)
> at org.apache.phoenix.shaded.org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:542)
> at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
> at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.apache.phoenix.shaded.org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
> at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at org.apache.phoenix.shaded.org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
> at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> EInvalidWireTypeException: Protocol message tag had invalid wire type. ����*000002
>
>
> It looked like there's something wrong when decoding response.
>
> I can only ensure kerberos authentication passed, because klist command showed the HTTP service principal after scripted executed.
>
> Can't find any reference about how to curl QS using protobuf instead of json, hopefully can get some help from community!
>
>
> --------------------------
>
> Best regards,
> R.C
>
>