You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Ralph Castain <rh...@open-mpi.org> on 2012/03/25 01:35:50 UTC
Proto files
Hi folks
I notice that the .proto files are not present in the built tarball. This presents a problem to those of us working on 3rd party tools that need to talk to Hadoop tools such as the resource manager. It means that anyone wanting to build our tools has to install an svn checkout of the code as opposed to simply installing the tarball.
Is there any reason -not- to include the .proto files in the tarball for distribution? It would help a great deal.
Thanks
Ralph
Re: Proto files
Posted by Ralph Castain <rh...@gmail.com>.
Has to be in C to support MPI wireup. We wrote a little lib that provides the required messaging support using protobuf-c, which works fine but protobuf-c needs to compile the proto files into something C can understand.
Sent from my iPad
On Mar 26, 2012, at 6:00 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Is your AM written in Java or C?
>
> On Mar 26, 2012, at 3:55 PM, Ralph Castain wrote:
>
>> Perhaps it would help if I outline the use case. I have a Java client that needs to launch a non-Java application manager. Obviously, the client talks to the RM using the Java APIs. The AM is passed all the info it needs to launch the actual application in its cmd line and environ, so no client to AM comm is required. Each client will launch it's own dedicated AM, so no cross-talk is possible.
>>
>> The AM needs to talk to the RM and NM to get an allocation and launch. Hence the proto requirement. I have not encountered any issue in that comm so far, provided I can build the AM against the proto files,
>>
>> Hence the jira. Hope that helps clarify the request.
>>
>> Sent from my iPad
>>
>> On Mar 26, 2012, at 8:30 AM, Robert Evans <ev...@yahoo-inc.com> wrote:
>>
>>> I responded in the JIRA for this. Because we wrap proto in Hadoop RPC right now those .proto files are not going to do very many people a lot of good, unless they have a client that can also communicate over a simple form of Hadoop RPC. I think it would be good to move to a pure PB RPC implementation, but that involves security changes and a lot of other things so it is not a small undertaking.
>>>
>>> --Bobby Evans
>>>
>>> On 3/24/12 8:38 PM, "Eli Collins" <el...@cloudera.com> wrote:
>>>
>>> Good idea, no reason we shouldn't, the build probably wasn't updated to
>>> include the who we added then. File a jira?
>>>
>>> On Saturday, March 24, 2012, Ralph Castain <rh...@open-mpi.org> wrote:
>>>> Hi folks
>>>>
>>>> I notice that the .proto files are not present in the built tarball. This
>>> presents a problem to those of us working on 3rd party tools that need to
>>> talk to Hadoop tools such as the resource manager. It means that anyone
>>> wanting to build our tools has to install an svn checkout of the code as
>>> opposed to simply installing the tarball.
>>>>
>>>> Is there any reason -not- to include the .proto files in the tarball for
>>> distribution? It would help a great deal.
>>>>
>>>> Thanks
>>>> Ralph
>>>>
>>>>
>>>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
Re: Proto files
Posted by Arun C Murthy <ac...@hortonworks.com>.
Is your AM written in Java or C?
On Mar 26, 2012, at 3:55 PM, Ralph Castain wrote:
> Perhaps it would help if I outline the use case. I have a Java client that needs to launch a non-Java application manager. Obviously, the client talks to the RM using the Java APIs. The AM is passed all the info it needs to launch the actual application in its cmd line and environ, so no client to AM comm is required. Each client will launch it's own dedicated AM, so no cross-talk is possible.
>
> The AM needs to talk to the RM and NM to get an allocation and launch. Hence the proto requirement. I have not encountered any issue in that comm so far, provided I can build the AM against the proto files,
>
> Hence the jira. Hope that helps clarify the request.
>
> Sent from my iPad
>
> On Mar 26, 2012, at 8:30 AM, Robert Evans <ev...@yahoo-inc.com> wrote:
>
>> I responded in the JIRA for this. Because we wrap proto in Hadoop RPC right now those .proto files are not going to do very many people a lot of good, unless they have a client that can also communicate over a simple form of Hadoop RPC. I think it would be good to move to a pure PB RPC implementation, but that involves security changes and a lot of other things so it is not a small undertaking.
>>
>> --Bobby Evans
>>
>> On 3/24/12 8:38 PM, "Eli Collins" <el...@cloudera.com> wrote:
>>
>> Good idea, no reason we shouldn't, the build probably wasn't updated to
>> include the who we added then. File a jira?
>>
>> On Saturday, March 24, 2012, Ralph Castain <rh...@open-mpi.org> wrote:
>>> Hi folks
>>>
>>> I notice that the .proto files are not present in the built tarball. This
>> presents a problem to those of us working on 3rd party tools that need to
>> talk to Hadoop tools such as the resource manager. It means that anyone
>> wanting to build our tools has to install an svn checkout of the code as
>> opposed to simply installing the tarball.
>>>
>>> Is there any reason -not- to include the .proto files in the tarball for
>> distribution? It would help a great deal.
>>>
>>> Thanks
>>> Ralph
>>>
>>>
>>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Proto files
Posted by Ralph Castain <rh...@gmail.com>.
Perhaps it would help if I outline the use case. I have a Java client that needs to launch a non-Java application manager. Obviously, the client talks to the RM using the Java APIs. The AM is passed all the info it needs to launch the actual application in its cmd line and environ, so no client to AM comm is required. Each client will launch it's own dedicated AM, so no cross-talk is possible.
The AM needs to talk to the RM and NM to get an allocation and launch. Hence the proto requirement. I have not encountered any issue in that comm so far, provided I can build the AM against the proto files,
Hence the jira. Hope that helps clarify the request.
Sent from my iPad
On Mar 26, 2012, at 8:30 AM, Robert Evans <ev...@yahoo-inc.com> wrote:
> I responded in the JIRA for this. Because we wrap proto in Hadoop RPC right now those .proto files are not going to do very many people a lot of good, unless they have a client that can also communicate over a simple form of Hadoop RPC. I think it would be good to move to a pure PB RPC implementation, but that involves security changes and a lot of other things so it is not a small undertaking.
>
> --Bobby Evans
>
> On 3/24/12 8:38 PM, "Eli Collins" <el...@cloudera.com> wrote:
>
> Good idea, no reason we shouldn't, the build probably wasn't updated to
> include the who we added then. File a jira?
>
> On Saturday, March 24, 2012, Ralph Castain <rh...@open-mpi.org> wrote:
>> Hi folks
>>
>> I notice that the .proto files are not present in the built tarball. This
> presents a problem to those of us working on 3rd party tools that need to
> talk to Hadoop tools such as the resource manager. It means that anyone
> wanting to build our tools has to install an svn checkout of the code as
> opposed to simply installing the tarball.
>>
>> Is there any reason -not- to include the .proto files in the tarball for
> distribution? It would help a great deal.
>>
>> Thanks
>> Ralph
>>
>>
>
Re: Proto files
Posted by Robert Evans <ev...@yahoo-inc.com>.
I responded in the JIRA for this. Because we wrap proto in Hadoop RPC right now those .proto files are not going to do very many people a lot of good, unless they have a client that can also communicate over a simple form of Hadoop RPC. I think it would be good to move to a pure PB RPC implementation, but that involves security changes and a lot of other things so it is not a small undertaking.
--Bobby Evans
On 3/24/12 8:38 PM, "Eli Collins" <el...@cloudera.com> wrote:
Good idea, no reason we shouldn't, the build probably wasn't updated to
include the who we added then. File a jira?
On Saturday, March 24, 2012, Ralph Castain <rh...@open-mpi.org> wrote:
> Hi folks
>
> I notice that the .proto files are not present in the built tarball. This
presents a problem to those of us working on 3rd party tools that need to
talk to Hadoop tools such as the resource manager. It means that anyone
wanting to build our tools has to install an svn checkout of the code as
opposed to simply installing the tarball.
>
> Is there any reason -not- to include the .proto files in the tarball for
distribution? It would help a great deal.
>
> Thanks
> Ralph
>
>
Re: Proto files
Posted by Eli Collins <el...@cloudera.com>.
Good idea, no reason we shouldn't, the build probably wasn't updated to
include the who we added then. File a jira?
On Saturday, March 24, 2012, Ralph Castain <rh...@open-mpi.org> wrote:
> Hi folks
>
> I notice that the .proto files are not present in the built tarball. This
presents a problem to those of us working on 3rd party tools that need to
talk to Hadoop tools such as the resource manager. It means that anyone
wanting to build our tools has to install an svn checkout of the code as
opposed to simply installing the tarball.
>
> Is there any reason -not- to include the .proto files in the tarball for
distribution? It would help a great deal.
>
> Thanks
> Ralph
>
>