You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@thrift.apache.org by chang liu <li...@gmail.com> on 2011/07/28 07:04:59 UTC
Does thrift support one-client communicating with multi-servers?
Hi,
I have a cluster of servers running the same service but with different
data, an aggregator acting as a client in front of the cluster communicats
to all of the servers and collects data from each service, so the client
gets a data set finally. I am going to use thrift as the RPC framework.
the service defination is below:
service MyService {
Data call(1:Param param);
}
But the generated code only has the method I defined, I wish some method
like this:
List<Data> callMulti(1:Param param);
to be generated from the defination above. Maybe there is also necessary to
have some TTransport implementation like TMultiServerTransport.
Is there any way to archive this?
Any Advice will be appropriated.
Re: Does thrift support one-client communicating with multi-servers?
Posted by chang liu <li...@gmail.com>.
Thanks very much, I think I know how to do it.
On Thu, Jul 28, 2011 at 9:59 PM, Chiku <ha...@gmail.com> wrote:
> I guess you do not need to implement something like Hadoop IPC Client
> class.
>
>
> You already have the logic for Client Server rpc mechanism through Thrift
> API.
>
> Are you looking into Async Clients or the Sync Clients?
>
> It should be a simple producer consumer problem. The Servers are the
> response generators or the producers. The responses from the various
> Servers
> are put into the queue. When you have responses from the all the Servers,
> you could collate the responses into a single entity.
>
> On Thu, Jul 28, 2011 at 12:28 PM, chang liu <li...@gmail.com> wrote:
>
> > Thank you very much!
> >
> > A little more, hadoop IPC has such a method:
> > public Writable[] call(Writable[] params, InetSocketAddress[]
> addresses)
> > in its Client class, I want to implement the same function using thrift,
> > where should I start from? do you have any suggestion?
> >
> >
> >
> > On Thu, Jul 28, 2011 at 2:36 PM, Chiku <ha...@gmail.com> wrote:
> >
> > > You will have to write your own wrapper for that.
> > >
> > > As far as I know, Thrift has no such aggregation service and in my
> > opinion
> > > there is a reason for that too:
> > >
> > > 1) What if a Server fails to respond to the request? How does the
> Service
> > > react? It will be very specific to what kind of behavior you want to
> > > achieve.
> > > Example: Ignore the request which is taking a lot of time? All the
> other
> > > requests become invalid or not.
> > >
> > >
> > > On Thu, Jul 28, 2011 at 10:34 AM, chang liu <li...@gmail.com>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > I have a cluster of servers running the same service but with
> different
> > > > data, an aggregator acting as a client in front of the cluster
> > > communicats
> > > > to all of the servers and collects data from each service, so the
> > client
> > > > gets a data set finally. I am going to use thrift as the RPC
> framework.
> > > >
> > > > the service defination is below:
> > > > service MyService {
> > > > Data call(1:Param param);
> > > > }
> > > >
> > > > But the generated code only has the method I defined, I wish some
> > method
> > > > like this:
> > > > List<Data> callMulti(1:Param param);
> > > > to be generated from the defination above. Maybe there is also
> > necessary
> > > to
> > > > have some TTransport implementation like TMultiServerTransport.
> > > >
> > > > Is there any way to archive this?
> > > >
> > > > Any Advice will be appropriated.
> > > >
> > >
> >
>
Re: Does thrift support one-client communicating with multi-servers?
Posted by Chiku <ha...@gmail.com>.
I guess you do not need to implement something like Hadoop IPC Client class.
You already have the logic for Client Server rpc mechanism through Thrift
API.
Are you looking into Async Clients or the Sync Clients?
It should be a simple producer consumer problem. The Servers are the
response generators or the producers. The responses from the various Servers
are put into the queue. When you have responses from the all the Servers,
you could collate the responses into a single entity.
On Thu, Jul 28, 2011 at 12:28 PM, chang liu <li...@gmail.com> wrote:
> Thank you very much!
>
> A little more, hadoop IPC has such a method:
> public Writable[] call(Writable[] params, InetSocketAddress[] addresses)
> in its Client class, I want to implement the same function using thrift,
> where should I start from? do you have any suggestion?
>
>
>
> On Thu, Jul 28, 2011 at 2:36 PM, Chiku <ha...@gmail.com> wrote:
>
> > You will have to write your own wrapper for that.
> >
> > As far as I know, Thrift has no such aggregation service and in my
> opinion
> > there is a reason for that too:
> >
> > 1) What if a Server fails to respond to the request? How does the Service
> > react? It will be very specific to what kind of behavior you want to
> > achieve.
> > Example: Ignore the request which is taking a lot of time? All the other
> > requests become invalid or not.
> >
> >
> > On Thu, Jul 28, 2011 at 10:34 AM, chang liu <li...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > I have a cluster of servers running the same service but with different
> > > data, an aggregator acting as a client in front of the cluster
> > communicats
> > > to all of the servers and collects data from each service, so the
> client
> > > gets a data set finally. I am going to use thrift as the RPC framework.
> > >
> > > the service defination is below:
> > > service MyService {
> > > Data call(1:Param param);
> > > }
> > >
> > > But the generated code only has the method I defined, I wish some
> method
> > > like this:
> > > List<Data> callMulti(1:Param param);
> > > to be generated from the defination above. Maybe there is also
> necessary
> > to
> > > have some TTransport implementation like TMultiServerTransport.
> > >
> > > Is there any way to archive this?
> > >
> > > Any Advice will be appropriated.
> > >
> >
>
Re: Does thrift support one-client communicating with multi-servers?
Posted by chang liu <li...@gmail.com>.
Thank you very much!
A little more, hadoop IPC has such a method:
public Writable[] call(Writable[] params, InetSocketAddress[] addresses)
in its Client class, I want to implement the same function using thrift,
where should I start from? do you have any suggestion?
On Thu, Jul 28, 2011 at 2:36 PM, Chiku <ha...@gmail.com> wrote:
> You will have to write your own wrapper for that.
>
> As far as I know, Thrift has no such aggregation service and in my opinion
> there is a reason for that too:
>
> 1) What if a Server fails to respond to the request? How does the Service
> react? It will be very specific to what kind of behavior you want to
> achieve.
> Example: Ignore the request which is taking a lot of time? All the other
> requests become invalid or not.
>
>
> On Thu, Jul 28, 2011 at 10:34 AM, chang liu <li...@gmail.com> wrote:
>
> > Hi,
> >
> > I have a cluster of servers running the same service but with different
> > data, an aggregator acting as a client in front of the cluster
> communicats
> > to all of the servers and collects data from each service, so the client
> > gets a data set finally. I am going to use thrift as the RPC framework.
> >
> > the service defination is below:
> > service MyService {
> > Data call(1:Param param);
> > }
> >
> > But the generated code only has the method I defined, I wish some method
> > like this:
> > List<Data> callMulti(1:Param param);
> > to be generated from the defination above. Maybe there is also necessary
> to
> > have some TTransport implementation like TMultiServerTransport.
> >
> > Is there any way to archive this?
> >
> > Any Advice will be appropriated.
> >
>
RE: thrift and JNI
Posted by Mike Margozzi <Mi...@quantum.com>.
Have you considered using JNA rather than JNI? No C++ code to write at
all.
-----Original Message-----
From: Steve Angelovich [mailto:SAngelovich@lgc.com]
Sent: Thursday, July 28, 2011 12:52 PM
To: user@thrift.apache.org
Subject: thrift and JNI
All,
Has anybody considered using thrift in conjunction with JNI?
I have a large java application where I want to be able to implement
some algorithms in either java or c++. Right now we do this by writing
a small amount of JNI code or using SWIG.
I'd like to generalize this a little so that the JNI code only has to be
written once instead of for each algorithm. I can do this purely using
JNI but then I end up with very tight coupling between my c++ code and
the java code.
Does it make any sense to use thrift to define my interface. Then when
moving between the java and c++ I wouldn't require a socket connection
but would use a memory based transport.
Does this very small code segment look correct for how to move the
contents of the TData java class into a memory buffer and then get it
back? This is all java code but I could do the same thing with the
first half java and the complementary code in c++.
byte[] buf = null ;
{
TData data = new TData() ;
...
TMemoryBuffer transportIn = new TMemoryBuffer(0) ;
TProtocol oprot = new TBinaryProtocol(transportIn) ;
data.write(oprot) ;
transportIn.flush() ;
buf = transportIn.getArray() ;
System.out.println("buf.length=" + buf.length) ;
}
{
TTransport transportOut = new TMemoryInputTransport(buf) ;
TProtocol iprot = new TBinaryProtocol(transportOut) ;
TData data = new TData() ;
data.read(iprot) ;
System.out.println(data.toString()) ;
}
Has anybody considered writing an implementation of TTransport based on
the NIO ByteBuffer class?
Thanks,
Steve
----------------------------------------------------------------------
This e-mail, including any attached files, may contain confidential and
privileged information for the sole use of the intended recipient. Any
review, use, distribution, or disclosure by others is strictly
prohibited. If you are not the intended recipient (or authorized to
receive information for the intended recipient), please contact the
sender by reply e-mail and delete all copies of this message.
----------------------------------------------------------------------
The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt.
thrift and JNI
Posted by Steve Angelovich <SA...@lgc.com>.
All,
Has anybody considered using thrift in conjunction with JNI?
I have a large java application where I want to be able to implement
some algorithms in either java or c++. Right now we do this by writing
a small amount of JNI code or using SWIG.
I'd like to generalize this a little so that the JNI code only has to be
written once instead of for each algorithm. I can do this purely using
JNI but then I end up with very tight coupling between my c++ code and
the java code.
Does it make any sense to use thrift to define my interface. Then when
moving between the java and c++ I wouldn't require a socket connection
but would use a memory based transport.
Does this very small code segment look correct for how to move the
contents of the TData java class into a memory buffer and then get it
back? This is all java code but I could do the same thing with the
first half java and the complementary code in c++.
byte[] buf = null ;
{
TData data = new TData() ;
...
TMemoryBuffer transportIn = new TMemoryBuffer(0) ;
TProtocol oprot = new TBinaryProtocol(transportIn) ;
data.write(oprot) ;
transportIn.flush() ;
buf = transportIn.getArray() ;
System.out.println("buf.length=" + buf.length) ;
}
{
TTransport transportOut = new TMemoryInputTransport(buf) ;
TProtocol iprot = new TBinaryProtocol(transportOut) ;
TData data = new TData() ;
data.read(iprot) ;
System.out.println(data.toString()) ;
}
Has anybody considered writing an implementation of TTransport based on
the NIO ByteBuffer class?
Thanks,
Steve
----------------------------------------------------------------------
This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.
RE: Does thrift support one-client communicating with multi-servers?
Posted by Pr...@cognizant.com.
You also can have Apache ZooKeeper on top of thrift to manage failover
Thanks and Regards
Prabakaran.N
Architect - Technology, GTO - SOA CoE
Cognizant Technology Solutions, BTP, Bangalore
Mobile:+91-9739386923; Vnet: 58157; Direct: +91-80-30748157
-----Original Message-----
From: Chiku [mailto:hakisenin@gmail.com]
Sent: Thursday, July 28, 2011 12:06 PM
To: user@thrift.apache.org
Subject: Re: Does thrift support one-client communicating with
multi-servers?
You will have to write your own wrapper for that.
As far as I know, Thrift has no such aggregation service and in my
opinion
there is a reason for that too:
1) What if a Server fails to respond to the request? How does the
Service
react? It will be very specific to what kind of behavior you want to
achieve.
Example: Ignore the request which is taking a lot of time? All the
other
requests become invalid or not.
On Thu, Jul 28, 2011 at 10:34 AM, chang liu <li...@gmail.com>
wrote:
> Hi,
>
> I have a cluster of servers running the same service but with
different
> data, an aggregator acting as a client in front of the cluster
communicats
> to all of the servers and collects data from each service, so the
client
> gets a data set finally. I am going to use thrift as the RPC
framework.
>
> the service defination is below:
> service MyService {
> Data call(1:Param param);
> }
>
> But the generated code only has the method I defined, I wish some
method
> like this:
> List<Data> callMulti(1:Param param);
> to be generated from the defination above. Maybe there is also
necessary to
> have some TTransport implementation like TMultiServerTransport.
>
> Is there any way to archive this?
>
> Any Advice will be appropriated.
>
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly prohibited and may be unlawful.
Re: Does thrift support one-client communicating with multi-servers?
Posted by Chiku <ha...@gmail.com>.
You will have to write your own wrapper for that.
As far as I know, Thrift has no such aggregation service and in my opinion
there is a reason for that too:
1) What if a Server fails to respond to the request? How does the Service
react? It will be very specific to what kind of behavior you want to
achieve.
Example: Ignore the request which is taking a lot of time? All the other
requests become invalid or not.
On Thu, Jul 28, 2011 at 10:34 AM, chang liu <li...@gmail.com> wrote:
> Hi,
>
> I have a cluster of servers running the same service but with different
> data, an aggregator acting as a client in front of the cluster communicats
> to all of the servers and collects data from each service, so the client
> gets a data set finally. I am going to use thrift as the RPC framework.
>
> the service defination is below:
> service MyService {
> Data call(1:Param param);
> }
>
> But the generated code only has the method I defined, I wish some method
> like this:
> List<Data> callMulti(1:Param param);
> to be generated from the defination above. Maybe there is also necessary to
> have some TTransport implementation like TMultiServerTransport.
>
> Is there any way to archive this?
>
> Any Advice will be appropriated.
>