You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Robert Grandl <rg...@yahoo.com.INVALID> on 2017/03/02 21:33:16 UTC
Non blocking RPC service
Hi all,
This might be a bit out of topic, but I don't know where to ask.
I am trying to create a non blocking RPC service in Yarn and Tez. However, I observed that all the communication is blocking.
I was trying to implement a non blocking service inheriting from .Interface instead of .BlockingInterface, however I have a really hard time to make it work.
Is there any example in Tez or Yarn codebase I can look at. Or do you have any other good pointers on how I can build such a service atop?
Otherwise, how can I do a non blocking service using protobuf in any of the codebases? Is any such mechanism in Tez or Yarn?
Thanks in advance for any advice,Robert
Re: Non blocking RPC service
Posted by Robert Grandl <rg...@yahoo.com.INVALID>.
Thanks a lot Sid. That was very helpful.
- Robert
On Friday, March 3, 2017 9:40 PM, Siddharth Seth <ss...@apache.org> wrote:
Hi Robert
Hadoop RPC has typically been blocking. There were some changes made to
make the client side asynchronous -
https://issues.apache.org/jira/browse/HADOOP-12909. I believe this is used
in some HDFS interfaces.
https://issues.apache.org/jira/browse/HADOOP-11552 added an option for
server side handoff. For this one, I'd look at the example. I don't know if
anyone uses this yet. There were some plans of using it for YARN-1503. Not
sure if that has happened yet though. Hopefully for a lot more. Planning
on using this in Tez as well at some point, for task to AM communication.
Otherwise there's alternates like grpc, thrift.
HTH
On Thu, Mar 2, 2017 at 1:33 PM, Robert Grandl <rg...@yahoo.com.invalid>
wrote:
> Hi all,
> This might be a bit out of topic, but I don't know where to ask.
>
> I am trying to create a non blocking RPC service in Yarn and Tez. However,
> I observed that all the communication is blocking.
>
> I was trying to implement a non blocking service inheriting from
> .Interface instead of .BlockingInterface, however I have a really hard time
> to make it work.
>
> Is there any example in Tez or Yarn codebase I can look at. Or do you have
> any other good pointers on how I can build such a service atop?
> Otherwise, how can I do a non blocking service using protobuf in any of
> the codebases? Is any such mechanism in Tez or Yarn?
>
> Thanks in advance for any advice,Robert
>
>
Re: Non blocking RPC service
Posted by Robert Grandl <rg...@yahoo.com.INVALID>.
Thanks a lot Sid. That was very helpful.
- Robert
On Friday, March 3, 2017 9:40 PM, Siddharth Seth <ss...@apache.org> wrote:
Hi Robert
Hadoop RPC has typically been blocking. There were some changes made to
make the client side asynchronous -
https://issues.apache.org/jira/browse/HADOOP-12909. I believe this is used
in some HDFS interfaces.
https://issues.apache.org/jira/browse/HADOOP-11552 added an option for
server side handoff. For this one, I'd look at the example. I don't know if
anyone uses this yet. There were some plans of using it for YARN-1503. Not
sure if that has happened yet though. Hopefully for a lot more. Planning
on using this in Tez as well at some point, for task to AM communication.
Otherwise there's alternates like grpc, thrift.
HTH
On Thu, Mar 2, 2017 at 1:33 PM, Robert Grandl <rg...@yahoo.com.invalid>
wrote:
> Hi all,
> This might be a bit out of topic, but I don't know where to ask.
>
> I am trying to create a non blocking RPC service in Yarn and Tez. However,
> I observed that all the communication is blocking.
>
> I was trying to implement a non blocking service inheriting from
> .Interface instead of .BlockingInterface, however I have a really hard time
> to make it work.
>
> Is there any example in Tez or Yarn codebase I can look at. Or do you have
> any other good pointers on how I can build such a service atop?
> Otherwise, how can I do a non blocking service using protobuf in any of
> the codebases? Is any such mechanism in Tez or Yarn?
>
> Thanks in advance for any advice,Robert
>
>
Re: Non blocking RPC service
Posted by Siddharth Seth <ss...@apache.org>.
Hi Robert
Hadoop RPC has typically been blocking. There were some changes made to
make the client side asynchronous -
https://issues.apache.org/jira/browse/HADOOP-12909. I believe this is used
in some HDFS interfaces.
https://issues.apache.org/jira/browse/HADOOP-11552 added an option for
server side handoff. For this one, I'd look at the example. I don't know if
anyone uses this yet. There were some plans of using it for YARN-1503. Not
sure if that has happened yet though. Hopefully for a lot more. Planning
on using this in Tez as well at some point, for task to AM communication.
Otherwise there's alternates like grpc, thrift.
HTH
On Thu, Mar 2, 2017 at 1:33 PM, Robert Grandl <rg...@yahoo.com.invalid>
wrote:
> Hi all,
> This might be a bit out of topic, but I don't know where to ask.
>
> I am trying to create a non blocking RPC service in Yarn and Tez. However,
> I observed that all the communication is blocking.
>
> I was trying to implement a non blocking service inheriting from
> .Interface instead of .BlockingInterface, however I have a really hard time
> to make it work.
>
> Is there any example in Tez or Yarn codebase I can look at. Or do you have
> any other good pointers on how I can build such a service atop?
> Otherwise, how can I do a non blocking service using protobuf in any of
> the codebases? Is any such mechanism in Tez or Yarn?
>
> Thanks in advance for any advice,Robert
>
>
Re: Non blocking RPC service
Posted by Siddharth Seth <ss...@apache.org>.
Hi Robert
Hadoop RPC has typically been blocking. There were some changes made to
make the client side asynchronous -
https://issues.apache.org/jira/browse/HADOOP-12909. I believe this is used
in some HDFS interfaces.
https://issues.apache.org/jira/browse/HADOOP-11552 added an option for
server side handoff. For this one, I'd look at the example. I don't know if
anyone uses this yet. There were some plans of using it for YARN-1503. Not
sure if that has happened yet though. Hopefully for a lot more. Planning
on using this in Tez as well at some point, for task to AM communication.
Otherwise there's alternates like grpc, thrift.
HTH
On Thu, Mar 2, 2017 at 1:33 PM, Robert Grandl <rg...@yahoo.com.invalid>
wrote:
> Hi all,
> This might be a bit out of topic, but I don't know where to ask.
>
> I am trying to create a non blocking RPC service in Yarn and Tez. However,
> I observed that all the communication is blocking.
>
> I was trying to implement a non blocking service inheriting from
> .Interface instead of .BlockingInterface, however I have a really hard time
> to make it work.
>
> Is there any example in Tez or Yarn codebase I can look at. Or do you have
> any other good pointers on how I can build such a service atop?
> Otherwise, how can I do a non blocking service using protobuf in any of
> the codebases? Is any such mechanism in Tez or Yarn?
>
> Thanks in advance for any advice,Robert
>
>