You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@quickstep.apache.org by Jianqiao <ji...@apache.org> on 2017/12/01 00:24:42 UTC
Re: Quickstep Network Mode and C++ Sockets
Yes. NetworkCliClient is quite standalone. The dependencies are:
----
(1) #include <grpc++/grpc++.h>
(2) #include "cli/NetworkCli.grpc.pb.h"
(3) #include "cli/NetworkCli.pb.h"
(4) #include "utility/Macros.hpp"
----
To write your own client:
----
(A) Setup grpc so that you can include the header files and link to it.
(B) Grab NetworkCli.proto (https://github.com/apache/
incubator-quickstep/blob/master/cli/NetworkCli.proto), change the package
name if necessary (originally quickstep).
Either
(B.1) use grpc/protobuf tools to compile NetworkCli.proto to generate (2)
and (3) -- see https://github.com/apache/incubator-quickstep/blob/
master/cli/CMakeLists.txt#L53
or
(B.2) compile quickstep and grab the files from build/cli/
(C) Copy the NetworkCliClient class from QS into your client code.
----
(A)/(B) may be somehow annoying to handle as you need to search through
various documentations ...
Best,
Jianqiao
2017-11-30 17:07 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
> So NetworkCliClient should be something I'm able to include in my program
> along with the appropriate dependencies and use as the API? I was thinking
> about needing to do that but I wasn't sure if that was a standalone API QS
> has implemented or a core part of the system. Unless I'm being mistaken and
> you're talking about something from gRPC. This is my first time working
> with it.
>
>
>
> On 11/30/17 4:58 PM, Jianqiao wrote:
>
>> Hi Dylan,
>>
>> Currently the network mode is using gRPC, so you probably need to use the
>> corresponding API (see
>> https://github.com/apache/incubator-quickstep/blob/master/
>> cli/NetworkCliClientMain.cpp#L42
>> as an example). The raw socket connection won't work unless you hack
>> gRPC's
>> message exchange protocol ..
>>
>> Best,
>> Jianqiao
>>
>> 2017-11-30 16:49 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
>>
>> Hello,
>>>
>>> I am attempting to interface with Quickstep using its NetworkCliClient
>>> and
>>> it's not working as I would expect. I have the default port and IP set to
>>> 3000 and 0.0.0.0 and am attempting to send single queries to be processed
>>> over in my test harness. From what I could tell of the code when QS is in
>>> network mode it accepts a socket connection and string input from that
>>> function and processes it in NetworkCliClient.hpp and
>>> NetworkCliClientMain.cpp, and yet this is not happening with my test
>>> code.
>>> The connection is being established but Quickstep does not seem to be
>>> doing
>>> anything with the queries that come in.
>>>
>>> Attached is the test code that I am using. test is just a table by that
>>> name, I'm selecting a literal from it so the contents shouldn't matter.
>>> I've also attempted to create a table with this but Quickstep did not
>>> process that.
>>>
>>> --
>>> Regards,
>>>
>>> Dylan Bacon
>>> University of Wisconsin - Madison
>>> Department of Computer Sciences
>>> dbacon@wisc.edu
>>>
>>>
>>>
>
Re: Quickstep Network Mode and C++ Sockets
Posted by Dylan Bacon <db...@wisc.edu>.
This should help, thank you. I'll try to take in that csv code into my
own work. And you're correct, we're using C++ for this project.
Quickstep thankfully has a native network module through that.
On 12/4/17 2:39 PM, Robert Claus wrote:
> Hey Dylan,
>
> Check out this branch in my fork:
> https://github.com/robertclaus/incubator-quickstep/tree/network-client-apis
>
> The latest commit in the branch allows you to change the default output of
> the CLI to csv instead of the table format. (The table format is fixed
> length-ish, so it's hard to parse without access to the table schema)
>
> There is also a trick to output CSV using COPY in the SQL, but it requires
> modifying your queries.
>
> Note this branch also has some libraries I wrote for accessing the network
> CLI from python and javascript if you still need them. (I got the
> impression you were working in a different language)
>
> -Robert
>
> On Mon, Dec 4, 2017 at 2:30 PM, Dylan Bacon <db...@wisc.edu> wrote:
>
>> Got this end of things working and am now talking to Quickstep remotely.
>> Is there a simple or built in way to parse our query result format or do I
>> just have to manually parse through all the column delimiters?
>>
>>
>> On 11/30/17 6:24 PM, Jianqiao wrote:
>>
>>> Yes. NetworkCliClient is quite standalone. The dependencies are:
>>> ----
>>> (1) #include <grpc++/grpc++.h>
>>> (2) #include "cli/NetworkCli.grpc.pb.h"
>>> (3) #include "cli/NetworkCli.pb.h"
>>> (4) #include "utility/Macros.hpp"
>>> ----
>>>
>>> To write your own client:
>>> ----
>>> (A) Setup grpc so that you can include the header files and link to it.
>>>
>>> (B) Grab NetworkCli.proto (https://github.com/apache/
>>> incubator-quickstep/blob/master/cli/NetworkCli.proto), change the package
>>> name if necessary (originally quickstep).
>>> Either
>>> (B.1) use grpc/protobuf tools to compile NetworkCli.proto to generate (2)
>>> and (3) -- see https://github.com/apache/incubator-quickstep/blob/
>>> master/cli/CMakeLists.txt#L53
>>> or
>>> (B.2) compile quickstep and grab the files from build/cli/
>>>
>>> (C) Copy the NetworkCliClient class from QS into your client code.
>>> ----
>>>
>>> (A)/(B) may be somehow annoying to handle as you need to search through
>>> various documentations ...
>>>
>>>
>>> Best,
>>> Jianqiao
>>>
>>> 2017-11-30 17:07 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
>>>
>>> So NetworkCliClient should be something I'm able to include in my program
>>>> along with the appropriate dependencies and use as the API? I was
>>>> thinking
>>>> about needing to do that but I wasn't sure if that was a standalone API
>>>> QS
>>>> has implemented or a core part of the system. Unless I'm being mistaken
>>>> and
>>>> you're talking about something from gRPC. This is my first time working
>>>> with it.
>>>>
>>>>
>>>>
>>>> On 11/30/17 4:58 PM, Jianqiao wrote:
>>>>
>>>> Hi Dylan,
>>>>> Currently the network mode is using gRPC, so you probably need to use
>>>>> the
>>>>> corresponding API (see
>>>>> https://github.com/apache/incubator-quickstep/blob/master/
>>>>> cli/NetworkCliClientMain.cpp#L42
>>>>> as an example). The raw socket connection won't work unless you hack
>>>>> gRPC's
>>>>> message exchange protocol ..
>>>>>
>>>>> Best,
>>>>> Jianqiao
>>>>>
>>>>> 2017-11-30 16:49 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
>>>>>
>>>>> Hello,
>>>>>
>>>>>> I am attempting to interface with Quickstep using its NetworkCliClient
>>>>>> and
>>>>>> it's not working as I would expect. I have the default port and IP set
>>>>>> to
>>>>>> 3000 and 0.0.0.0 and am attempting to send single queries to be
>>>>>> processed
>>>>>> over in my test harness. From what I could tell of the code when QS is
>>>>>> in
>>>>>> network mode it accepts a socket connection and string input from that
>>>>>> function and processes it in NetworkCliClient.hpp and
>>>>>> NetworkCliClientMain.cpp, and yet this is not happening with my test
>>>>>> code.
>>>>>> The connection is being established but Quickstep does not seem to be
>>>>>> doing
>>>>>> anything with the queries that come in.
>>>>>>
>>>>>> Attached is the test code that I am using. test is just a table by that
>>>>>> name, I'm selecting a literal from it so the contents shouldn't matter.
>>>>>> I've also attempted to create a table with this but Quickstep did not
>>>>>> process that.
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>>
>>>>>> Dylan Bacon
>>>>>> University of Wisconsin - Madison
>>>>>> Department of Computer Sciences
>>>>>> dbacon@wisc.edu
>>>>>>
>>>>>>
>>>>>>
>>>>>>
Re: Quickstep Network Mode and C++ Sockets
Posted by Robert Claus <ro...@gmail.com>.
Hey Dylan,
Check out this branch in my fork:
https://github.com/robertclaus/incubator-quickstep/tree/network-client-apis
The latest commit in the branch allows you to change the default output of
the CLI to csv instead of the table format. (The table format is fixed
length-ish, so it's hard to parse without access to the table schema)
There is also a trick to output CSV using COPY in the SQL, but it requires
modifying your queries.
Note this branch also has some libraries I wrote for accessing the network
CLI from python and javascript if you still need them. (I got the
impression you were working in a different language)
-Robert
On Mon, Dec 4, 2017 at 2:30 PM, Dylan Bacon <db...@wisc.edu> wrote:
> Got this end of things working and am now talking to Quickstep remotely.
> Is there a simple or built in way to parse our query result format or do I
> just have to manually parse through all the column delimiters?
>
>
> On 11/30/17 6:24 PM, Jianqiao wrote:
>
>> Yes. NetworkCliClient is quite standalone. The dependencies are:
>> ----
>> (1) #include <grpc++/grpc++.h>
>> (2) #include "cli/NetworkCli.grpc.pb.h"
>> (3) #include "cli/NetworkCli.pb.h"
>> (4) #include "utility/Macros.hpp"
>> ----
>>
>> To write your own client:
>> ----
>> (A) Setup grpc so that you can include the header files and link to it.
>>
>> (B) Grab NetworkCli.proto (https://github.com/apache/
>> incubator-quickstep/blob/master/cli/NetworkCli.proto), change the package
>> name if necessary (originally quickstep).
>> Either
>> (B.1) use grpc/protobuf tools to compile NetworkCli.proto to generate (2)
>> and (3) -- see https://github.com/apache/incubator-quickstep/blob/
>> master/cli/CMakeLists.txt#L53
>> or
>> (B.2) compile quickstep and grab the files from build/cli/
>>
>> (C) Copy the NetworkCliClient class from QS into your client code.
>> ----
>>
>> (A)/(B) may be somehow annoying to handle as you need to search through
>> various documentations ...
>>
>>
>> Best,
>> Jianqiao
>>
>> 2017-11-30 17:07 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
>>
>> So NetworkCliClient should be something I'm able to include in my program
>>> along with the appropriate dependencies and use as the API? I was
>>> thinking
>>> about needing to do that but I wasn't sure if that was a standalone API
>>> QS
>>> has implemented or a core part of the system. Unless I'm being mistaken
>>> and
>>> you're talking about something from gRPC. This is my first time working
>>> with it.
>>>
>>>
>>>
>>> On 11/30/17 4:58 PM, Jianqiao wrote:
>>>
>>> Hi Dylan,
>>>>
>>>> Currently the network mode is using gRPC, so you probably need to use
>>>> the
>>>> corresponding API (see
>>>> https://github.com/apache/incubator-quickstep/blob/master/
>>>> cli/NetworkCliClientMain.cpp#L42
>>>> as an example). The raw socket connection won't work unless you hack
>>>> gRPC's
>>>> message exchange protocol ..
>>>>
>>>> Best,
>>>> Jianqiao
>>>>
>>>> 2017-11-30 16:49 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
>>>>
>>>> Hello,
>>>>
>>>>> I am attempting to interface with Quickstep using its NetworkCliClient
>>>>> and
>>>>> it's not working as I would expect. I have the default port and IP set
>>>>> to
>>>>> 3000 and 0.0.0.0 and am attempting to send single queries to be
>>>>> processed
>>>>> over in my test harness. From what I could tell of the code when QS is
>>>>> in
>>>>> network mode it accepts a socket connection and string input from that
>>>>> function and processes it in NetworkCliClient.hpp and
>>>>> NetworkCliClientMain.cpp, and yet this is not happening with my test
>>>>> code.
>>>>> The connection is being established but Quickstep does not seem to be
>>>>> doing
>>>>> anything with the queries that come in.
>>>>>
>>>>> Attached is the test code that I am using. test is just a table by that
>>>>> name, I'm selecting a literal from it so the contents shouldn't matter.
>>>>> I've also attempted to create a table with this but Quickstep did not
>>>>> process that.
>>>>>
>>>>> --
>>>>> Regards,
>>>>>
>>>>> Dylan Bacon
>>>>> University of Wisconsin - Madison
>>>>> Department of Computer Sciences
>>>>> dbacon@wisc.edu
>>>>>
>>>>>
>>>>>
>>>>>
>
Re: Quickstep Network Mode and C++ Sockets
Posted by Dylan Bacon <db...@wisc.edu>.
Got this end of things working and am now talking to Quickstep remotely.
Is there a simple or built in way to parse our query result format or do
I just have to manually parse through all the column delimiters?
On 11/30/17 6:24 PM, Jianqiao wrote:
> Yes. NetworkCliClient is quite standalone. The dependencies are:
> ----
> (1) #include <grpc++/grpc++.h>
> (2) #include "cli/NetworkCli.grpc.pb.h"
> (3) #include "cli/NetworkCli.pb.h"
> (4) #include "utility/Macros.hpp"
> ----
>
> To write your own client:
> ----
> (A) Setup grpc so that you can include the header files and link to it.
>
> (B) Grab NetworkCli.proto (https://github.com/apache/
> incubator-quickstep/blob/master/cli/NetworkCli.proto), change the package
> name if necessary (originally quickstep).
> Either
> (B.1) use grpc/protobuf tools to compile NetworkCli.proto to generate (2)
> and (3) -- see https://github.com/apache/incubator-quickstep/blob/
> master/cli/CMakeLists.txt#L53
> or
> (B.2) compile quickstep and grab the files from build/cli/
>
> (C) Copy the NetworkCliClient class from QS into your client code.
> ----
>
> (A)/(B) may be somehow annoying to handle as you need to search through
> various documentations ...
>
>
> Best,
> Jianqiao
>
> 2017-11-30 17:07 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
>
>> So NetworkCliClient should be something I'm able to include in my program
>> along with the appropriate dependencies and use as the API? I was thinking
>> about needing to do that but I wasn't sure if that was a standalone API QS
>> has implemented or a core part of the system. Unless I'm being mistaken and
>> you're talking about something from gRPC. This is my first time working
>> with it.
>>
>>
>>
>> On 11/30/17 4:58 PM, Jianqiao wrote:
>>
>>> Hi Dylan,
>>>
>>> Currently the network mode is using gRPC, so you probably need to use the
>>> corresponding API (see
>>> https://github.com/apache/incubator-quickstep/blob/master/
>>> cli/NetworkCliClientMain.cpp#L42
>>> as an example). The raw socket connection won't work unless you hack
>>> gRPC's
>>> message exchange protocol ..
>>>
>>> Best,
>>> Jianqiao
>>>
>>> 2017-11-30 16:49 GMT-06:00 Dylan Bacon <db...@wisc.edu>:
>>>
>>> Hello,
>>>> I am attempting to interface with Quickstep using its NetworkCliClient
>>>> and
>>>> it's not working as I would expect. I have the default port and IP set to
>>>> 3000 and 0.0.0.0 and am attempting to send single queries to be processed
>>>> over in my test harness. From what I could tell of the code when QS is in
>>>> network mode it accepts a socket connection and string input from that
>>>> function and processes it in NetworkCliClient.hpp and
>>>> NetworkCliClientMain.cpp, and yet this is not happening with my test
>>>> code.
>>>> The connection is being established but Quickstep does not seem to be
>>>> doing
>>>> anything with the queries that come in.
>>>>
>>>> Attached is the test code that I am using. test is just a table by that
>>>> name, I'm selecting a literal from it so the contents shouldn't matter.
>>>> I've also attempted to create a table with this but Quickstep did not
>>>> process that.
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> Dylan Bacon
>>>> University of Wisconsin - Madison
>>>> Department of Computer Sciences
>>>> dbacon@wisc.edu
>>>>
>>>>
>>>>