You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@quickstep.apache.org by Marc Spehlmann <sp...@apache.org> on 2017/03/30 15:10:47 UTC
quickstep server
Hi dev,
I'm looking to make a simple TCP server interface to Quickstep so that we
can keep the db running while we communicate to it from a python program.
After investigating, I think the simplest way to do this is:
- create a Server executable which:
- opens a TCP socket
- forks a child with an open pipe
- std input of the child process is redirected from the pipe
- child process launches the main() code of QuickstepCli and goes on as
normal
- Server process channels messages from the TCP buffer to the pipe
- Server exits when child exits
Since I'd like to reuse the code in main() of quickstep_cli, I think this
means abstracting that code into a stand alone library.
Does anyone have any initial comments on this design?
--Marc
Re: quickstep server
Posted by Marc Spehlmann <sp...@gmail.com>.
I think that in the initial version, it will not be aware of multiple
clients, but in the second version, this will not be hard to accommodate.
On Sat, Apr 1, 2017 at 4:48 PM, Zuyu Zhang <zu...@cs.wisc.edu> wrote:
> Hi Marc,
>
> With the NetworkWrapper, does the QuickstepServer could serve queries from
> multiple clients?
>
> Cheers,
> Zuyu
>
Re: quickstep server
Posted by Zuyu Zhang <zu...@cs.wisc.edu>.
Hi Marc,
With the NetworkWrapper, does the QuickstepServer could serve queries from
multiple clients?
Cheers,
Zuyu
Re: quickstep server
Posted by Marc Spehlmann <sp...@gmail.com>.
Hi Zuyu,
Yes, that helps. I am making a prototype right now where I took the
QuickstepCli code and made another executable, QuickstepServer. The server
does all of the same initialization process. The main difference is that,
instead of communicating with a linereader, it must communicate with a
NetworkWrapper. Also, communication must not be one-way and rely on prints
to the screen, but it is two way and sends the results of STDOUT and ERR to
the NetworkWrapper.
Once the prototype is complete, we'll have a better idea if the CLI can be
refactored to save some code duplication.
--Marc
On Fri, Mar 31, 2017 at 3:06 PM, Zuyu Zhang <zu...@cs.wisc.edu> wrote:
> Hi Marc,
>
> The distributed version uses the network-based implementation of TMB to
> communicate among multiple components, including DistributedCli, Conductor
> (which contains Foreman), and Executor (which contains a Worker pool). We
> have defined additional TMB message types for the communication.
>
> For your purpose, you might want to split QuickstepCli.cpp into two new
> executables, one for the cli, and one the database core. Both need to link
> the Protobuf 3 and gRPC. And then, the python program could talk to the cli
> executable to send a query or a command.
>
> Hope it helps.
>
> Cheers,
> Zuyu
>
Re: quickstep server
Posted by Zuyu Zhang <zu...@cs.wisc.edu>.
Hi Marc,
The distributed version uses the network-based implementation of TMB to
communicate among multiple components, including DistributedCli, Conductor
(which contains Foreman), and Executor (which contains a Worker pool). We
have defined additional TMB message types for the communication.
For your purpose, you might want to split QuickstepCli.cpp into two new
executables, one for the cli, and one the database core. Both need to link
the Protobuf 3 and gRPC. And then, the python program could talk to the cli
executable to send a query or a command.
Hope it helps.
Cheers,
Zuyu
Re: quickstep server
Posted by Marc Spehlmann <sp...@gmail.com>.
Thank you. I spoke with Tianrun yesterday. Harshad brought up the fact that
Zuyu might also have made changes in the distributed version which can be
ported back for our use in the single node world.
@Zuyu, is there distributed code that would be good to borrow instead of
rolling our own version?
On Thu, Mar 30, 2017 at 12:58 PM, Jianqiao <ji...@cs.wisc.edu> wrote:
> Hi Marc,
>
> Tianrun had one implementation of the TCP Server interface when he was
> working on the GRAIL related stuff. The idea is to subclass the LineReader
> interface. This minimizes the changes needed inside QuickstepCli.cpp. The
> code might help and is available here:
>
> https://bitbucket.org/TianrunLi/tcpreader/src
>
> Best,
> Jianqiao
>
> 2017-03-30 10:10 GMT-05:00 Marc Spehlmann <sp...@apache.org>:
>
> > Hi dev,
> > I'm looking to make a simple TCP server interface to Quickstep so that we
> > can keep the db running while we communicate to it from a python program.
> > After investigating, I think the simplest way to do this is:
> >
> > - create a Server executable which:
> > - opens a TCP socket
> > - forks a child with an open pipe
> > - std input of the child process is redirected from the pipe
> > - child process launches the main() code of QuickstepCli and goes on
> as
> > normal
> > - Server process channels messages from the TCP buffer to the pipe
> > - Server exits when child exits
> >
> > Since I'd like to reuse the code in main() of quickstep_cli, I think this
> > means abstracting that code into a stand alone library.
> >
> > Does anyone have any initial comments on this design?
> >
> > --Marc
> >
>
Re: quickstep server
Posted by Jianqiao <ji...@cs.wisc.edu>.
Hi Marc,
Tianrun had one implementation of the TCP Server interface when he was
working on the GRAIL related stuff. The idea is to subclass the LineReader
interface. This minimizes the changes needed inside QuickstepCli.cpp. The
code might help and is available here:
https://bitbucket.org/TianrunLi/tcpreader/src
Best,
Jianqiao
2017-03-30 10:10 GMT-05:00 Marc Spehlmann <sp...@apache.org>:
> Hi dev,
> I'm looking to make a simple TCP server interface to Quickstep so that we
> can keep the db running while we communicate to it from a python program.
> After investigating, I think the simplest way to do this is:
>
> - create a Server executable which:
> - opens a TCP socket
> - forks a child with an open pipe
> - std input of the child process is redirected from the pipe
> - child process launches the main() code of QuickstepCli and goes on as
> normal
> - Server process channels messages from the TCP buffer to the pipe
> - Server exits when child exits
>
> Since I'd like to reuse the code in main() of quickstep_cli, I think this
> means abstracting that code into a stand alone library.
>
> Does anyone have any initial comments on this design?
>
> --Marc
>