You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Dhaivat Pandya <dh...@gmail.com> on 2014/03/22 21:16:44 UTC
DataNodeRegistration Serialization
Hi everyone,
I'm currently working on an application that requires some important
details about the DataNode registration (w/ NameNode) procedure.
Specifically, I have understood (after Wireshark-ing and looking through
the Hadoop code) that the DataNode is registered with the NameNode using a
single packet which tells the NameNode where the DataNode is "located"
(i.e. host and port).
However, in this packet, I'm not clear as to what scheme is used to
serialize the DataNode information. I am running Hadoop 1.2.1. Any
information will be appreciated.
Thank you,
Dhaivat Pandya
Re: DataNodeRegistration Serialization
Posted by Andrew Wang <an...@cloudera.com>.
I think the code is the only documentation. A Writable is just something
that has a readFrom and a writeTo method, the contents of these methods can
be essentially arbitrary.
Best,
Andrew
On Sun, Mar 23, 2014 at 7:43 PM, Dhaivat Pandya <dh...@gmail.com>wrote:
> Another question: is there some kind of documentation (other than the code)
> that specifies the kind of serialization algorithm Writables use? I think I
> may have to port the concept over to another language.
>
> Thanks,
>
> Dhaivat
>
>
> On Sun, Mar 23, 2014 at 9:11 AM, Dhaivat Pandya <dhaivatpandya@gmail.com
> >wrote:
>
> > Thank you very much, Andrew.
> >
> >
> > On Sat, Mar 22, 2014 at 7:02 PM, Andrew Wang <andrew.wang@cloudera.com
> >wrote:
> >
> >> Hi Dhaivat,
> >>
> >> Take a look at DatanodeRegistration and it's parent class DatanodeID. DR
> >> is
> >> a Writable, meaning it's a custom serialization format. Hadoop 2 uses
> >> protobuf for the RPC serialization rather than writables (with the
> >> exception of DN data transfer).
> >>
> >>
> >>
> https://github.com/apache/hadoop-common/blob/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/server/protocol/DatanodeRegistration.java
> >>
> >>
> https://github.com/apache/hadoop-common/blob/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/protocol/DatanodeID.java
> >>
> >> Best,
> >> Andrew
> >>
> >>
> >> On Sat, Mar 22, 2014 at 1:16 PM, Dhaivat Pandya <
> dhaivatpandya@gmail.com
> >> >wrote:
> >>
> >> > Hi everyone,
> >> >
> >> > I'm currently working on an application that requires some important
> >> > details about the DataNode registration (w/ NameNode) procedure.
> >> >
> >> > Specifically, I have understood (after Wireshark-ing and looking
> through
> >> > the Hadoop code) that the DataNode is registered with the NameNode
> >> using a
> >> > single packet which tells the NameNode where the DataNode is "located"
> >> > (i.e. host and port).
> >> >
> >> > However, in this packet, I'm not clear as to what scheme is used to
> >> > serialize the DataNode information. I am running Hadoop 1.2.1. Any
> >> > information will be appreciated.
> >> >
> >> > Thank you,
> >> >
> >> > Dhaivat Pandya
> >> >
> >>
> >
> >
>
Re: DataNodeRegistration Serialization
Posted by Dhaivat Pandya <dh...@gmail.com>.
Another question: is there some kind of documentation (other than the code)
that specifies the kind of serialization algorithm Writables use? I think I
may have to port the concept over to another language.
Thanks,
Dhaivat
On Sun, Mar 23, 2014 at 9:11 AM, Dhaivat Pandya <dh...@gmail.com>wrote:
> Thank you very much, Andrew.
>
>
> On Sat, Mar 22, 2014 at 7:02 PM, Andrew Wang <an...@cloudera.com>wrote:
>
>> Hi Dhaivat,
>>
>> Take a look at DatanodeRegistration and it's parent class DatanodeID. DR
>> is
>> a Writable, meaning it's a custom serialization format. Hadoop 2 uses
>> protobuf for the RPC serialization rather than writables (with the
>> exception of DN data transfer).
>>
>>
>> https://github.com/apache/hadoop-common/blob/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/server/protocol/DatanodeRegistration.java
>>
>> https://github.com/apache/hadoop-common/blob/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/protocol/DatanodeID.java
>>
>> Best,
>> Andrew
>>
>>
>> On Sat, Mar 22, 2014 at 1:16 PM, Dhaivat Pandya <dhaivatpandya@gmail.com
>> >wrote:
>>
>> > Hi everyone,
>> >
>> > I'm currently working on an application that requires some important
>> > details about the DataNode registration (w/ NameNode) procedure.
>> >
>> > Specifically, I have understood (after Wireshark-ing and looking through
>> > the Hadoop code) that the DataNode is registered with the NameNode
>> using a
>> > single packet which tells the NameNode where the DataNode is "located"
>> > (i.e. host and port).
>> >
>> > However, in this packet, I'm not clear as to what scheme is used to
>> > serialize the DataNode information. I am running Hadoop 1.2.1. Any
>> > information will be appreciated.
>> >
>> > Thank you,
>> >
>> > Dhaivat Pandya
>> >
>>
>
>
Re: DataNodeRegistration Serialization
Posted by Dhaivat Pandya <dh...@gmail.com>.
Thank you very much, Andrew.
On Sat, Mar 22, 2014 at 7:02 PM, Andrew Wang <an...@cloudera.com>wrote:
> Hi Dhaivat,
>
> Take a look at DatanodeRegistration and it's parent class DatanodeID. DR is
> a Writable, meaning it's a custom serialization format. Hadoop 2 uses
> protobuf for the RPC serialization rather than writables (with the
> exception of DN data transfer).
>
>
> https://github.com/apache/hadoop-common/blob/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/server/protocol/DatanodeRegistration.java
>
> https://github.com/apache/hadoop-common/blob/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/protocol/DatanodeID.java
>
> Best,
> Andrew
>
>
> On Sat, Mar 22, 2014 at 1:16 PM, Dhaivat Pandya <dhaivatpandya@gmail.com
> >wrote:
>
> > Hi everyone,
> >
> > I'm currently working on an application that requires some important
> > details about the DataNode registration (w/ NameNode) procedure.
> >
> > Specifically, I have understood (after Wireshark-ing and looking through
> > the Hadoop code) that the DataNode is registered with the NameNode using
> a
> > single packet which tells the NameNode where the DataNode is "located"
> > (i.e. host and port).
> >
> > However, in this packet, I'm not clear as to what scheme is used to
> > serialize the DataNode information. I am running Hadoop 1.2.1. Any
> > information will be appreciated.
> >
> > Thank you,
> >
> > Dhaivat Pandya
> >
>
Re: DataNodeRegistration Serialization
Posted by Andrew Wang <an...@cloudera.com>.
Hi Dhaivat,
Take a look at DatanodeRegistration and it's parent class DatanodeID. DR is
a Writable, meaning it's a custom serialization format. Hadoop 2 uses
protobuf for the RPC serialization rather than writables (with the
exception of DN data transfer).
https://github.com/apache/hadoop-common/blob/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/server/protocol/DatanodeRegistration.java
https://github.com/apache/hadoop-common/blob/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/protocol/DatanodeID.java
Best,
Andrew
On Sat, Mar 22, 2014 at 1:16 PM, Dhaivat Pandya <dh...@gmail.com>wrote:
> Hi everyone,
>
> I'm currently working on an application that requires some important
> details about the DataNode registration (w/ NameNode) procedure.
>
> Specifically, I have understood (after Wireshark-ing and looking through
> the Hadoop code) that the DataNode is registered with the NameNode using a
> single packet which tells the NameNode where the DataNode is "located"
> (i.e. host and port).
>
> However, in this packet, I'm not clear as to what scheme is used to
> serialize the DataNode information. I am running Hadoop 1.2.1. Any
> information will be appreciated.
>
> Thank you,
>
> Dhaivat Pandya
>