You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@quickstep.apache.org by Dylan Bacon <db...@wisc.edu> on 2017/11/30 22:49:21 UTC
Quickstep Network Mode and C++ Sockets
Hello,
I am attempting to interface with Quickstep using its NetworkCliClient
and it's not working as I would expect. I have the default port and IP
set to 3000 and 0.0.0.0 and am attempting to send single queries to be
processed over in my test harness. From what I could tell of the code
when QS is in network mode it accepts a socket connection and string
input from that function and processes it in NetworkCliClient.hpp and
NetworkCliClientMain.cpp, and yet this is not happening with my test
code. The connection is being established but Quickstep does not seem to
be doing anything with the queries that come in.
Attached is the test code that I am using. test is just a table by that
name, I'm selecting a literal from it so the contents shouldn't matter.
I've also attempted to create a table with this but Quickstep did not
process that.
--
Regards,
Dylan Bacon
University of Wisconsin - Madison
Department of Computer Sciences
dbacon@wisc.edu
Re: Quickstep Network Mode and C++ Sockets
Posted by Dylan Bacon <db...@wisc.edu>.
This should help, thank you. I'll try to take in that csv code into my
own work. And you're correct, we're using C++ for this project.
Quickstep thankfully has a native network module through that.
On 12/4/17 2:39 PM, Robert Claus wrote:
> Hey Dylan,
>
> Check out this branch in my fork:
> https://github.com/robertclaus/incubator-quickstep/tree/network-client-apis
>
> The latest commit in the branch allows you to change the default output of
> the CLI to csv instead of the table format. (The table format is fixed
> length-ish, so it's hard to parse without access to the table schema)
>
> There is also a trick to output CSV using COPY in the SQL, but it requires
> modifying your queries.
>
> Note this branch also has some libraries I wrote for accessing the network
> CLI from python and javascript if you still need them. (I got the
> impression you were working in a different language)
>
> -Robert
>
> On Mon, Dec 4, 2017 at 2:30 PM, Dylan Bacon <db...@wisc.edu> wrote:
>
>> Got this end of things working and am now talking to Quickstep remotely.
>> Is there a simple or built in way to parse our query result format or do I
>> just have to manually parse through all the column delimiters?
>>
>>
>> On 11/30/17 6:24 PM, Jianqiao wrote:
>>
>>> Yes. NetworkCliClient is quite standalone. The dependencies are:
>>> ----
>>> (1) #include <grpc++/grpc++.h>
>>> (2) #include "cli/NetworkCli.grpc.pb.h"
>>> (3) #include "cli/NetworkCli.pb.h"
>>> (4) #include "utility/Macros.hpp"
>>> ----
>>>
>>> To write your own client:
>>> ----
>>> (A) Setup grpc so that you can include the header files and link to it.
>>>
>>> (B) Grab NetworkCli.proto (https://github.com/apache/
>>> incubator-quickstep/blob/master/cli/NetworkCli.proto), change the package
>>> name if necessary (originally quickstep).
>>> Either
>>> (B.1) use grpc/protobuf tools to compile NetworkCli.proto to generate (2)
>>> and (3) -- see https://github.com/apache/incubator-quickstep/blob/
>>> master/cli/CMakeLists.txt#L53
>>> or
>>> (B.2) compile quickstep and grab the files from build/cli/
>>>
>>> (C) Copy the NetworkCliClient class from QS into your client code.
>>> ----
>>>
>>> (A)/(B) may be somehow annoying to handle as you need to search through
>>> various documentations ...
>>>
>>>
>>> Best,
>>> Jianqiao
>>>
>>> 2017-11-30 17:07 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
>>>
>>> So NetworkCliClient should be something I'm able to include in my program
>>>> along with the appropriate dependencies and use as the API? I was
>>>> thinking
>>>> about needing to do that but I wasn't sure if that was a standalone API
>>>> QS
>>>> has implemented or a core part of the system. Unless I'm being mistaken
>>>> and
>>>> you're talking about something from gRPC. This is my first time working
>>>> with it.
>>>>
>>>>
>>>>
>>>> On 11/30/17 4:58 PM, Jianqiao wrote:
>>>>
>>>> Hi Dylan,
>>>>> Currently the network mode is using gRPC, so you probably need to use
>>>>> the
>>>>> corresponding API (see
>>>>> https://github.com/apache/incubator-quickstep/blob/master/
>>>>> cli/NetworkCliClientMain.cpp#L42
>>>>> as an example). The raw socket connection won't work unless you hack
>>>>> gRPC's
>>>>> message exchange protocol ..
>>>>>
>>>>> Best,
>>>>> Jianqiao
>>>>>
>>>>> 2017-11-30 16:49 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
>>>>>
>>>>> Hello,
>>>>>
>>>>>> I am attempting to interface with Quickstep using its NetworkCliClient
>>>>>> and
>>>>>> it's not working as I would expect. I have the default port and IP set
>>>>>> to
>>>>>> 3000 and 0.0.0.0 and am attempting to send single queries to be
>>>>>> processed
>>>>>> over in my test harness. From what I could tell of the code when QS is
>>>>>> in
>>>>>> network mode it accepts a socket connection and string input from that
>>>>>> function and processes it in NetworkCliClient.hpp and
>>>>>> NetworkCliClientMain.cpp, and yet this is not happening with my test
>>>>>> code.
>>>>>> The connection is being established but Quickstep does not seem to be
>>>>>> doing
>>>>>> anything with the queries that come in.
>>>>>>
>>>>>> Attached is the test code that I am using. test is just a table by that
>>>>>> name, I'm selecting a literal from it so the contents shouldn't matter.
>>>>>> I've also attempted to create a table with this but Quickstep did not
>>>>>> process that.
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>>
>>>>>> Dylan Bacon
>>>>>> University of Wisconsin - Madison
>>>>>> Department of Computer Sciences
>>>>>> dbacon@wisc.edu
>>>>>>
>>>>>>
>>>>>>
>>>>>>
Re: Quickstep Network Mode and C++ Sockets
Posted by Robert Claus <ro...@gmail.com>.
Hey Dylan,
Check out this branch in my fork:
https://github.com/robertclaus/incubator-quickstep/tree/network-client-apis
The latest commit in the branch allows you to change the default output of
the CLI to csv instead of the table format. (The table format is fixed
length-ish, so it's hard to parse without access to the table schema)
There is also a trick to output CSV using COPY in the SQL, but it requires
modifying your queries.
Note this branch also has some libraries I wrote for accessing the network
CLI from python and javascript if you still need them. (I got the
impression you were working in a different language)
-Robert
On Mon, Dec 4, 2017 at 2:30 PM, Dylan Bacon <db...@wisc.edu> wrote:
> Got this end of things working and am now talking to Quickstep remotely.
> Is there a simple or built in way to parse our query result format or do I
> just have to manually parse through all the column delimiters?
>
>
> On 11/30/17 6:24 PM, Jianqiao wrote:
>
>> Yes. NetworkCliClient is quite standalone. The dependencies are:
>> ----
>> (1) #include <grpc++/grpc++.h>
>> (2) #include "cli/NetworkCli.grpc.pb.h"
>> (3) #include "cli/NetworkCli.pb.h"
>> (4) #include "utility/Macros.hpp"
>> ----
>>
>> To write your own client:
>> ----
>> (A) Setup grpc so that you can include the header files and link to it.
>>
>> (B) Grab NetworkCli.proto (https://github.com/apache/
>> incubator-quickstep/blob/master/cli/NetworkCli.proto), change the package
>> name if necessary (originally quickstep).
>> Either
>> (B.1) use grpc/protobuf tools to compile NetworkCli.proto to generate (2)
>> and (3) -- see https://github.com/apache/incubator-quickstep/blob/
>> master/cli/CMakeLists.txt#L53
>> or
>> (B.2) compile quickstep and grab the files from build/cli/
>>
>> (C) Copy the NetworkCliClient class from QS into your client code.
>> ----
>>
>> (A)/(B) may be somehow annoying to handle as you need to search through
>> various documentations ...
>>
>>
>> Best,
>> Jianqiao
>>
>> 2017-11-30 17:07 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
>>
>> So NetworkCliClient should be something I'm able to include in my program
>>> along with the appropriate dependencies and use as the API? I was
>>> thinking
>>> about needing to do that but I wasn't sure if that was a standalone API
>>> QS
>>> has implemented or a core part of the system. Unless I'm being mistaken
>>> and
>>> you're talking about something from gRPC. This is my first time working
>>> with it.
>>>
>>>
>>>
>>> On 11/30/17 4:58 PM, Jianqiao wrote:
>>>
>>> Hi Dylan,
>>>>
>>>> Currently the network mode is using gRPC, so you probably need to use
>>>> the
>>>> corresponding API (see
>>>> https://github.com/apache/incubator-quickstep/blob/master/
>>>> cli/NetworkCliClientMain.cpp#L42
>>>> as an example). The raw socket connection won't work unless you hack
>>>> gRPC's
>>>> message exchange protocol ..
>>>>
>>>> Best,
>>>> Jianqiao
>>>>
>>>> 2017-11-30 16:49 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
>>>>
>>>> Hello,
>>>>
>>>>> I am attempting to interface with Quickstep using its NetworkCliClient
>>>>> and
>>>>> it's not working as I would expect. I have the default port and IP set
>>>>> to
>>>>> 3000 and 0.0.0.0 and am attempting to send single queries to be
>>>>> processed
>>>>> over in my test harness. From what I could tell of the code when QS is
>>>>> in
>>>>> network mode it accepts a socket connection and string input from that
>>>>> function and processes it in NetworkCliClient.hpp and
>>>>> NetworkCliClientMain.cpp, and yet this is not happening with my test
>>>>> code.
>>>>> The connection is being established but Quickstep does not seem to be
>>>>> doing
>>>>> anything with the queries that come in.
>>>>>
>>>>> Attached is the test code that I am using. test is just a table by that
>>>>> name, I'm selecting a literal from it so the contents shouldn't matter.
>>>>> I've also attempted to create a table with this but Quickstep did not
>>>>> process that.
>>>>>
>>>>> --
>>>>> Regards,
>>>>>
>>>>> Dylan Bacon
>>>>> University of Wisconsin - Madison
>>>>> Department of Computer Sciences
>>>>> dbacon@wisc.edu
>>>>>
>>>>>
>>>>>
>>>>>
>
Re: Quickstep Network Mode and C++ Sockets
Posted by Dylan Bacon <db...@wisc.edu>.
Got this end of things working and am now talking to Quickstep remotely.
Is there a simple or built in way to parse our query result format or do
I just have to manually parse through all the column delimiters?
On 11/30/17 6:24 PM, Jianqiao wrote:
> Yes. NetworkCliClient is quite standalone. The dependencies are:
> ----
> (1) #include <grpc++/grpc++.h>
> (2) #include "cli/NetworkCli.grpc.pb.h"
> (3) #include "cli/NetworkCli.pb.h"
> (4) #include "utility/Macros.hpp"
> ----
>
> To write your own client:
> ----
> (A) Setup grpc so that you can include the header files and link to it.
>
> (B) Grab NetworkCli.proto (https://github.com/apache/
> incubator-quickstep/blob/master/cli/NetworkCli.proto), change the package
> name if necessary (originally quickstep).
> Either
> (B.1) use grpc/protobuf tools to compile NetworkCli.proto to generate (2)
> and (3) -- see https://github.com/apache/incubator-quickstep/blob/
> master/cli/CMakeLists.txt#L53
> or
> (B.2) compile quickstep and grab the files from build/cli/
>
> (C) Copy the NetworkCliClient class from QS into your client code.
> ----
>
> (A)/(B) may be somehow annoying to handle as you need to search through
> various documentations ...
>
>
> Best,
> Jianqiao
>
> 2017-11-30 17:07 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
>
>> So NetworkCliClient should be something I'm able to include in my program
>> along with the appropriate dependencies and use as the API? I was thinking
>> about needing to do that but I wasn't sure if that was a standalone API QS
>> has implemented or a core part of the system. Unless I'm being mistaken and
>> you're talking about something from gRPC. This is my first time working
>> with it.
>>
>>
>>
>> On 11/30/17 4:58 PM, Jianqiao wrote:
>>
>>> Hi Dylan,
>>>
>>> Currently the network mode is using gRPC, so you probably need to use the
>>> corresponding API (see
>>> https://github.com/apache/incubator-quickstep/blob/master/
>>> cli/NetworkCliClientMain.cpp#L42
>>> as an example). The raw socket connection won't work unless you hack
>>> gRPC's
>>> message exchange protocol ..
>>>
>>> Best,
>>> Jianqiao
>>>
>>> 2017-11-30 16:49 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
>>>
>>> Hello,
>>>> I am attempting to interface with Quickstep using its NetworkCliClient
>>>> and
>>>> it's not working as I would expect. I have the default port and IP set to
>>>> 3000 and 0.0.0.0 and am attempting to send single queries to be processed
>>>> over in my test harness. From what I could tell of the code when QS is in
>>>> network mode it accepts a socket connection and string input from that
>>>> function and processes it in NetworkCliClient.hpp and
>>>> NetworkCliClientMain.cpp, and yet this is not happening with my test
>>>> code.
>>>> The connection is being established but Quickstep does not seem to be
>>>> doing
>>>> anything with the queries that come in.
>>>>
>>>> Attached is the test code that I am using. test is just a table by that
>>>> name, I'm selecting a literal from it so the contents shouldn't matter.
>>>> I've also attempted to create a table with this but Quickstep did not
>>>> process that.
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> Dylan Bacon
>>>> University of Wisconsin - Madison
>>>> Department of Computer Sciences
>>>> dbacon@wisc.edu
>>>>
>>>>
>>>>
Re: Quickstep Network Mode and C++ Sockets
Posted by Jianqiao <ji...@apache.org>.
Yes. NetworkCliClient is quite standalone. The dependencies are:
----
(1) #include <grpc++/grpc++.h>
(2) #include "cli/NetworkCli.grpc.pb.h"
(3) #include "cli/NetworkCli.pb.h"
(4) #include "utility/Macros.hpp"
----
To write your own client:
----
(A) Setup grpc so that you can include the header files and link to it.
(B) Grab NetworkCli.proto (https://github.com/apache/
incubator-quickstep/blob/master/cli/NetworkCli.proto), change the package
name if necessary (originally quickstep).
Either
(B.1) use grpc/protobuf tools to compile NetworkCli.proto to generate (2)
and (3) -- see https://github.com/apache/incubator-quickstep/blob/
master/cli/CMakeLists.txt#L53
or
(B.2) compile quickstep and grab the files from build/cli/
(C) Copy the NetworkCliClient class from QS into your client code.
----
(A)/(B) may be somehow annoying to handle as you need to search through
various documentations ...
Best,
Jianqiao
2017-11-30 17:07 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
> So NetworkCliClient should be something I'm able to include in my program
> along with the appropriate dependencies and use as the API? I was thinking
> about needing to do that but I wasn't sure if that was a standalone API QS
> has implemented or a core part of the system. Unless I'm being mistaken and
> you're talking about something from gRPC. This is my first time working
> with it.
>
>
>
> On 11/30/17 4:58 PM, Jianqiao wrote:
>
>> Hi Dylan,
>>
>> Currently the network mode is using gRPC, so you probably need to use the
>> corresponding API (see
>> https://github.com/apache/incubator-quickstep/blob/master/
>> cli/NetworkCliClientMain.cpp#L42
>> as an example). The raw socket connection won't work unless you hack
>> gRPC's
>> message exchange protocol ..
>>
>> Best,
>> Jianqiao
>>
>> 2017-11-30 16:49 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
>>
>> Hello,
>>>
>>> I am attempting to interface with Quickstep using its NetworkCliClient
>>> and
>>> it's not working as I would expect. I have the default port and IP set to
>>> 3000 and 0.0.0.0 and am attempting to send single queries to be processed
>>> over in my test harness. From what I could tell of the code when QS is in
>>> network mode it accepts a socket connection and string input from that
>>> function and processes it in NetworkCliClient.hpp and
>>> NetworkCliClientMain.cpp, and yet this is not happening with my test
>>> code.
>>> The connection is being established but Quickstep does not seem to be
>>> doing
>>> anything with the queries that come in.
>>>
>>> Attached is the test code that I am using. test is just a table by that
>>> name, I'm selecting a literal from it so the contents shouldn't matter.
>>> I've also attempted to create a table with this but Quickstep did not
>>> process that.
>>>
>>> --
>>> Regards,
>>>
>>> Dylan Bacon
>>> University of Wisconsin - Madison
>>> Department of Computer Sciences
>>> dbacon@wisc.edu
>>>
>>>
>>>
>
Re: Quickstep Network Mode and C++ Sockets
Posted by Dylan Bacon <db...@wisc.edu>.
So NetworkCliClient should be something I'm able to include in my
program along with the appropriate dependencies and use as the API? I
was thinking about needing to do that but I wasn't sure if that was a
standalone API QS has implemented or a core part of the system. Unless
I'm being mistaken and you're talking about something from gRPC. This is
my first time working with it.
On 11/30/17 4:58 PM, Jianqiao wrote:
> Hi Dylan,
>
> Currently the network mode is using gRPC, so you probably need to use the
> corresponding API (see
> https://github.com/apache/incubator-quickstep/blob/master/cli/NetworkCliClientMain.cpp#L42
> as an example). The raw socket connection won't work unless you hack gRPC's
> message exchange protocol ..
>
> Best,
> Jianqiao
>
> 2017-11-30 16:49 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
>
>> Hello,
>>
>> I am attempting to interface with Quickstep using its NetworkCliClient and
>> it's not working as I would expect. I have the default port and IP set to
>> 3000 and 0.0.0.0 and am attempting to send single queries to be processed
>> over in my test harness. From what I could tell of the code when QS is in
>> network mode it accepts a socket connection and string input from that
>> function and processes it in NetworkCliClient.hpp and
>> NetworkCliClientMain.cpp, and yet this is not happening with my test code.
>> The connection is being established but Quickstep does not seem to be doing
>> anything with the queries that come in.
>>
>> Attached is the test code that I am using. test is just a table by that
>> name, I'm selecting a literal from it so the contents shouldn't matter.
>> I've also attempted to create a table with this but Quickstep did not
>> process that.
>>
>> --
>> Regards,
>>
>> Dylan Bacon
>> University of Wisconsin - Madison
>> Department of Computer Sciences
>> dbacon@wisc.edu
>>
>>
Re: Quickstep Network Mode and C++ Sockets
Posted by Jianqiao <ji...@apache.org>.
Hi Dylan,
Currently the network mode is using gRPC, so you probably need to use the
corresponding API (see
https://github.com/apache/incubator-quickstep/blob/master/cli/NetworkCliClientMain.cpp#L42
as an example). The raw socket connection won't work unless you hack gRPC's
message exchange protocol ..
Best,
Jianqiao
2017-11-30 16:49 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
> Hello,
>
> I am attempting to interface with Quickstep using its NetworkCliClient and
> it's not working as I would expect. I have the default port and IP set to
> 3000 and 0.0.0.0 and am attempting to send single queries to be processed
> over in my test harness. From what I could tell of the code when QS is in
> network mode it accepts a socket connection and string input from that
> function and processes it in NetworkCliClient.hpp and
> NetworkCliClientMain.cpp, and yet this is not happening with my test code.
> The connection is being established but Quickstep does not seem to be doing
> anything with the queries that come in.
>
> Attached is the test code that I am using. test is just a table by that
> name, I'm selecting a literal from it so the contents shouldn't matter.
> I've also attempted to create a table with this but Quickstep did not
> process that.
>
> --
> Regards,
>
> Dylan Bacon
> University of Wisconsin - Madison
> Department of Computer Sciences
> dbacon@wisc.edu
>
>