You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Radhe Radhe <ra...@live.com> on 2014/03/27 08:59:35 UTC
Hadoop Serialization mechanisms
Hello All,
AFAIK Hadoop serialization comes into picture in the 2 areas:putting data on the wire i.e., for interprocess communication between nodes using RPCputting data on disk i.e. using the Map Reduce for persistent storage say HDFS.
I have a couple of questions regarding the Serialization mechanisms used in Hadoop:
Does Hadoop provides a pluggible feature for Serialization for both the above cases?Is Writable the default Serialization mechanism for both the above cases?Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop 2.x?Will there be a significant performance gain if the default Serialization i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map Reduce programming?
Thanks,-RR
RE: Hadoop Serialization mechanisms
Posted by Peter Marron <Pe...@trilliumsoftware.com>.
Hi,
Certainly not an expert on this but if you ask twice...
2. My understanding is that Writables are the only way for Map/Reduce jobs.
4. I don't think that switching from Writables would benefit Map/Reduce.
The point is that in the context of a Map/Reduce
job you know the version of the serialization and the actual classes involved.
So the serialization contains just the data required to instantiate the objects
and nothing else. Serialization formats like Avro and protobufs etc
have mechanisms to handle different versions of the classes and typing
of the fields etc. They are, to a certain extent, self-describing.
This means that they must include an overhead to tag fields and classes
etc. and so, of necessity, they will be larger than the Writable classes.
(Here I'm talking about the raw objects before compression.)
I would have thought that this means that the Writable classes
will always be the correct choice for Map/Reduce.
As pointed out by others, if you write your own custom Writable
classes then you can exploit your domain-specific knowledge of
the data to provide even more efficient and compact representations.
Regards,
Peter Marron
Senior Developer
Trillium Software, A Harte Hanks Company
Theale Court, 1st Floor, 11-13 High Street
Theale
RG7 5AH
+44 (0) 118 940 7609 office
+44 (0) 118 940 7699 fax
[https://4b2685446389bc779b46-5f66fbb59518cc4fcae8900db28267f5.ssl.cf2.rackcdn.com/trillium.png]<http://www.trilliumsoftware.com/>
trilliumsoftware.com<http://www.trilliumsoftware.com/> / linkedin<http://www.linkedin.com/company/17710> / twitter<https://twitter.com/trilliumsw> / facebook<http://www.facebook.com/HarteHanks>
From: Radhe Radhe [mailto:radhe.krishna.radhe@live.com]
Sent: 30 March 2014 14:27
To: user@hadoop.apache.org
Subject: RE: Hadoop Serialization mechanisms
Second try.
Please find some time to answer some of my queries.
Thanks,
-RR
________________________________
From: radhe.krishna.radhe@live.com<ma...@live.com>
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Hadoop Serialization mechanisms
Date: Thu, 27 Mar 2014 13:29:35 +0530
Hello All,
AFAIK Hadoop serialization comes into picture in the 2 areas:
1. putting data on the wire i.e., for interprocess communication between nodes using RPC
2. putting data on disk i.e. using the Map Reduce for persistent storage say HDFS.
I have a couple of questions regarding the Serialization mechanisms used in Hadoop:
1. Does Hadoop provides a pluggible feature for Serialization for both the above cases?
2. Is Writable the default Serialization mechanism for both the above cases?
3. Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop 2.x?
4. Will there be a significant performance gain if the default Serialization i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map Reduce programming?
Thanks,
-RR
RE: Hadoop Serialization mechanisms
Posted by Peter Marron <Pe...@trilliumsoftware.com>.
Hi,
Certainly not an expert on this but if you ask twice...
2. My understanding is that Writables are the only way for Map/Reduce jobs.
4. I don't think that switching from Writables would benefit Map/Reduce.
The point is that in the context of a Map/Reduce
job you know the version of the serialization and the actual classes involved.
So the serialization contains just the data required to instantiate the objects
and nothing else. Serialization formats like Avro and protobufs etc
have mechanisms to handle different versions of the classes and typing
of the fields etc. They are, to a certain extent, self-describing.
This means that they must include an overhead to tag fields and classes
etc. and so, of necessity, they will be larger than the Writable classes.
(Here I'm talking about the raw objects before compression.)
I would have thought that this means that the Writable classes
will always be the correct choice for Map/Reduce.
As pointed out by others, if you write your own custom Writable
classes then you can exploit your domain-specific knowledge of
the data to provide even more efficient and compact representations.
Regards,
Peter Marron
Senior Developer
Trillium Software, A Harte Hanks Company
Theale Court, 1st Floor, 11-13 High Street
Theale
RG7 5AH
+44 (0) 118 940 7609 office
+44 (0) 118 940 7699 fax
[https://4b2685446389bc779b46-5f66fbb59518cc4fcae8900db28267f5.ssl.cf2.rackcdn.com/trillium.png]<http://www.trilliumsoftware.com/>
trilliumsoftware.com<http://www.trilliumsoftware.com/> / linkedin<http://www.linkedin.com/company/17710> / twitter<https://twitter.com/trilliumsw> / facebook<http://www.facebook.com/HarteHanks>
From: Radhe Radhe [mailto:radhe.krishna.radhe@live.com]
Sent: 30 March 2014 14:27
To: user@hadoop.apache.org
Subject: RE: Hadoop Serialization mechanisms
Second try.
Please find some time to answer some of my queries.
Thanks,
-RR
________________________________
From: radhe.krishna.radhe@live.com<ma...@live.com>
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Hadoop Serialization mechanisms
Date: Thu, 27 Mar 2014 13:29:35 +0530
Hello All,
AFAIK Hadoop serialization comes into picture in the 2 areas:
1. putting data on the wire i.e., for interprocess communication between nodes using RPC
2. putting data on disk i.e. using the Map Reduce for persistent storage say HDFS.
I have a couple of questions regarding the Serialization mechanisms used in Hadoop:
1. Does Hadoop provides a pluggible feature for Serialization for both the above cases?
2. Is Writable the default Serialization mechanism for both the above cases?
3. Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop 2.x?
4. Will there be a significant performance gain if the default Serialization i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map Reduce programming?
Thanks,
-RR
RE: Hadoop Serialization mechanisms
Posted by Peter Marron <Pe...@trilliumsoftware.com>.
Hi,
Certainly not an expert on this but if you ask twice...
2. My understanding is that Writables are the only way for Map/Reduce jobs.
4. I don't think that switching from Writables would benefit Map/Reduce.
The point is that in the context of a Map/Reduce
job you know the version of the serialization and the actual classes involved.
So the serialization contains just the data required to instantiate the objects
and nothing else. Serialization formats like Avro and protobufs etc
have mechanisms to handle different versions of the classes and typing
of the fields etc. They are, to a certain extent, self-describing.
This means that they must include an overhead to tag fields and classes
etc. and so, of necessity, they will be larger than the Writable classes.
(Here I'm talking about the raw objects before compression.)
I would have thought that this means that the Writable classes
will always be the correct choice for Map/Reduce.
As pointed out by others, if you write your own custom Writable
classes then you can exploit your domain-specific knowledge of
the data to provide even more efficient and compact representations.
Regards,
Peter Marron
Senior Developer
Trillium Software, A Harte Hanks Company
Theale Court, 1st Floor, 11-13 High Street
Theale
RG7 5AH
+44 (0) 118 940 7609 office
+44 (0) 118 940 7699 fax
[https://4b2685446389bc779b46-5f66fbb59518cc4fcae8900db28267f5.ssl.cf2.rackcdn.com/trillium.png]<http://www.trilliumsoftware.com/>
trilliumsoftware.com<http://www.trilliumsoftware.com/> / linkedin<http://www.linkedin.com/company/17710> / twitter<https://twitter.com/trilliumsw> / facebook<http://www.facebook.com/HarteHanks>
From: Radhe Radhe [mailto:radhe.krishna.radhe@live.com]
Sent: 30 March 2014 14:27
To: user@hadoop.apache.org
Subject: RE: Hadoop Serialization mechanisms
Second try.
Please find some time to answer some of my queries.
Thanks,
-RR
________________________________
From: radhe.krishna.radhe@live.com<ma...@live.com>
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Hadoop Serialization mechanisms
Date: Thu, 27 Mar 2014 13:29:35 +0530
Hello All,
AFAIK Hadoop serialization comes into picture in the 2 areas:
1. putting data on the wire i.e., for interprocess communication between nodes using RPC
2. putting data on disk i.e. using the Map Reduce for persistent storage say HDFS.
I have a couple of questions regarding the Serialization mechanisms used in Hadoop:
1. Does Hadoop provides a pluggible feature for Serialization for both the above cases?
2. Is Writable the default Serialization mechanism for both the above cases?
3. Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop 2.x?
4. Will there be a significant performance gain if the default Serialization i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map Reduce programming?
Thanks,
-RR
RE: Hadoop Serialization mechanisms
Posted by Peter Marron <Pe...@trilliumsoftware.com>.
Hi,
Certainly not an expert on this but if you ask twice...
2. My understanding is that Writables are the only way for Map/Reduce jobs.
4. I don't think that switching from Writables would benefit Map/Reduce.
The point is that in the context of a Map/Reduce
job you know the version of the serialization and the actual classes involved.
So the serialization contains just the data required to instantiate the objects
and nothing else. Serialization formats like Avro and protobufs etc
have mechanisms to handle different versions of the classes and typing
of the fields etc. They are, to a certain extent, self-describing.
This means that they must include an overhead to tag fields and classes
etc. and so, of necessity, they will be larger than the Writable classes.
(Here I'm talking about the raw objects before compression.)
I would have thought that this means that the Writable classes
will always be the correct choice for Map/Reduce.
As pointed out by others, if you write your own custom Writable
classes then you can exploit your domain-specific knowledge of
the data to provide even more efficient and compact representations.
Regards,
Peter Marron
Senior Developer
Trillium Software, A Harte Hanks Company
Theale Court, 1st Floor, 11-13 High Street
Theale
RG7 5AH
+44 (0) 118 940 7609 office
+44 (0) 118 940 7699 fax
[https://4b2685446389bc779b46-5f66fbb59518cc4fcae8900db28267f5.ssl.cf2.rackcdn.com/trillium.png]<http://www.trilliumsoftware.com/>
trilliumsoftware.com<http://www.trilliumsoftware.com/> / linkedin<http://www.linkedin.com/company/17710> / twitter<https://twitter.com/trilliumsw> / facebook<http://www.facebook.com/HarteHanks>
From: Radhe Radhe [mailto:radhe.krishna.radhe@live.com]
Sent: 30 March 2014 14:27
To: user@hadoop.apache.org
Subject: RE: Hadoop Serialization mechanisms
Second try.
Please find some time to answer some of my queries.
Thanks,
-RR
________________________________
From: radhe.krishna.radhe@live.com<ma...@live.com>
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Hadoop Serialization mechanisms
Date: Thu, 27 Mar 2014 13:29:35 +0530
Hello All,
AFAIK Hadoop serialization comes into picture in the 2 areas:
1. putting data on the wire i.e., for interprocess communication between nodes using RPC
2. putting data on disk i.e. using the Map Reduce for persistent storage say HDFS.
I have a couple of questions regarding the Serialization mechanisms used in Hadoop:
1. Does Hadoop provides a pluggible feature for Serialization for both the above cases?
2. Is Writable the default Serialization mechanism for both the above cases?
3. Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop 2.x?
4. Will there be a significant performance gain if the default Serialization i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map Reduce programming?
Thanks,
-RR
RE: Hadoop Serialization mechanisms
Posted by Radhe Radhe <ra...@live.com>.
Second try.
Please find some time to answer some of my queries.
Thanks,-RR
From: radhe.krishna.radhe@live.com
To: user@hadoop.apache.org
Subject: Hadoop Serialization mechanisms
Date: Thu, 27 Mar 2014 13:29:35 +0530
Hello All,
AFAIK Hadoop serialization comes into picture in the 2 areas:putting data on the wire i.e., for interprocess communication between nodes using RPCputting data on disk i.e. using the Map Reduce for persistent storage say HDFS.
I have a couple of questions regarding the Serialization mechanisms used in Hadoop:
Does Hadoop provides a pluggible feature for Serialization for both the above cases?Is Writable the default Serialization mechanism for both the above cases?Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop 2.x?Will there be a significant performance gain if the default Serialization i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map Reduce programming?
Thanks,-RR
Re: Hadoop Serialization mechanisms
Posted by Jay Vyas <ja...@gmail.com>.
But I believe w.r.t. "will we see performance gains when using
avro/thrift/... over writables" -- it depends on the writable
implementation. For example, If I have a writable serialization which
can use a bit map to store an enum, but then read that enum as a string:
It will look the same to user, but my writable implementation would be
superior. We can obviously say that if you use avro/thrift/pbuffers in
an efficient way, then yes, you will see a performance gain then say
storing everything as Text Writable objects. But clever optimizations can
be done even within the Writable framework as well.
On Sun, Mar 30, 2014 at 4:08 PM, Harsh J <ha...@cloudera.com> wrote:
> > Does Hadoop provides a pluggible feature for Serialization for both the
> above cases?
>
> - You can override the RPC serialisation module and engine with a
> custom class if you wish to, but it would not be trivial task.
> - You can easily use custom data serialisation modules for I/O.
>
> > Is Writable the default Serialization mechanism for both the above cases?
>
> While MR's built-in examples in Apache Hadoop continue to use
> Writables, the RPCs have moved to using Protocol buffers for
> themselves in 2.x onwards.
>
> > Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop
> 2.x?
>
> Yes "partially", see above.
>
> > Will there be a significant performance gain if the default
> Serialization i.e. Writables is replaced with Avro, Protol Buffers or
> Thrift in Map Reduce programming?
>
> Yes, you should see a gain in using a more efficient data
> serialisation format for data files.
>
> On Sun, Mar 30, 2014 at 9:09 PM, Jay Vyas <ja...@gmail.com> wrote:
> > Those are all great questions, and mostly difficultto answer. I havent
> > played with serialization APIs in some time, but let me try to give some
> > guidance. WRT to your bulleted questions above:
> >
> > 1) Serialization is file system independant: The use of any hadoop
> > compatible file system should support any kind of serialization.
> >
> > 2) See (1). The "default serialization" is Writables: But you can easily
> > add your own by modifiying the io.serializations configuration parameter.
> >
> > 3) I doubt anything significant effecting the way serialization works:
> The
> > main thrust of 1->2 was in the way services are deployed, not changing
> the
> > internals of how data is serialized. After all, the serialization APIs
> need
> > to remain stability even as the arch. of hadoop changes.
> >
> > 4) It depends on the implementation. If you have a custom writable that
> is
> > really good at compressing your data, that will be better than using a
> > thrift auto generated API for serialization that is uncustomized out of
> the
> > box. Example: Say you are writing "strings" and you know the string is
> max
> > 3 characters. A "smart" Writable serializer with custom implementations
> > optimized for your data will beat a thrift serialization approach. But I
> > think in general, the advantage of thrift/avro is that its easier to get
> > really good compression natively out-of-the-box, due to the fact that
> many
> > different data types are strongly supported by the way they apply the
> > schemas (for example , a thrift struct can contain a "boolean", two
> > "strings" , and an "int" These types will all be optmiized for you by
> > thrift.... Where as in Writables, you cannot as easily create
> sophisticated
> > types with optimization of nested properties.
> >
> >
> >
> >
> > On Thu, Mar 27, 2014 at 2:59 AM, Radhe Radhe <
> radhe.krishna.radhe@live.com>
> > wrote:
> >>
> >> Hello All,
> >>
> >> AFAIK Hadoop serialization comes into picture in the 2 areas:
> >>
> >> putting data on the wire i.e., for interprocess communication between
> >> nodes using RPC
> >> putting data on disk i.e. using the Map Reduce for persistent storage
> say
> >> HDFS.
> >>
> >>
> >> I have a couple of questions regarding the Serialization mechanisms used
> >> in Hadoop:
> >>
> >> Does Hadoop provides a pluggible feature for Serialization for both the
> >> above cases?
> >> Is Writable the default Serialization mechanism for both the above
> cases?
> >> Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop
> >> 2.x?
> >> Will there be a significant performance gain if the default
> Serialization
> >> i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map
> Reduce
> >> programming?
> >>
> >>
> >> Thanks,
> >> -RR
> >
> >
> >
> >
> > --
> > Jay Vyas
> > http://jayunit100.blogspot.com
>
>
>
> --
> Harsh J
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Hadoop Serialization mechanisms
Posted by Jay Vyas <ja...@gmail.com>.
But I believe w.r.t. "will we see performance gains when using
avro/thrift/... over writables" -- it depends on the writable
implementation. For example, If I have a writable serialization which
can use a bit map to store an enum, but then read that enum as a string:
It will look the same to user, but my writable implementation would be
superior. We can obviously say that if you use avro/thrift/pbuffers in
an efficient way, then yes, you will see a performance gain then say
storing everything as Text Writable objects. But clever optimizations can
be done even within the Writable framework as well.
On Sun, Mar 30, 2014 at 4:08 PM, Harsh J <ha...@cloudera.com> wrote:
> > Does Hadoop provides a pluggible feature for Serialization for both the
> above cases?
>
> - You can override the RPC serialisation module and engine with a
> custom class if you wish to, but it would not be trivial task.
> - You can easily use custom data serialisation modules for I/O.
>
> > Is Writable the default Serialization mechanism for both the above cases?
>
> While MR's built-in examples in Apache Hadoop continue to use
> Writables, the RPCs have moved to using Protocol buffers for
> themselves in 2.x onwards.
>
> > Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop
> 2.x?
>
> Yes "partially", see above.
>
> > Will there be a significant performance gain if the default
> Serialization i.e. Writables is replaced with Avro, Protol Buffers or
> Thrift in Map Reduce programming?
>
> Yes, you should see a gain in using a more efficient data
> serialisation format for data files.
>
> On Sun, Mar 30, 2014 at 9:09 PM, Jay Vyas <ja...@gmail.com> wrote:
> > Those are all great questions, and mostly difficultto answer. I havent
> > played with serialization APIs in some time, but let me try to give some
> > guidance. WRT to your bulleted questions above:
> >
> > 1) Serialization is file system independant: The use of any hadoop
> > compatible file system should support any kind of serialization.
> >
> > 2) See (1). The "default serialization" is Writables: But you can easily
> > add your own by modifiying the io.serializations configuration parameter.
> >
> > 3) I doubt anything significant effecting the way serialization works:
> The
> > main thrust of 1->2 was in the way services are deployed, not changing
> the
> > internals of how data is serialized. After all, the serialization APIs
> need
> > to remain stability even as the arch. of hadoop changes.
> >
> > 4) It depends on the implementation. If you have a custom writable that
> is
> > really good at compressing your data, that will be better than using a
> > thrift auto generated API for serialization that is uncustomized out of
> the
> > box. Example: Say you are writing "strings" and you know the string is
> max
> > 3 characters. A "smart" Writable serializer with custom implementations
> > optimized for your data will beat a thrift serialization approach. But I
> > think in general, the advantage of thrift/avro is that its easier to get
> > really good compression natively out-of-the-box, due to the fact that
> many
> > different data types are strongly supported by the way they apply the
> > schemas (for example , a thrift struct can contain a "boolean", two
> > "strings" , and an "int" These types will all be optmiized for you by
> > thrift.... Where as in Writables, you cannot as easily create
> sophisticated
> > types with optimization of nested properties.
> >
> >
> >
> >
> > On Thu, Mar 27, 2014 at 2:59 AM, Radhe Radhe <
> radhe.krishna.radhe@live.com>
> > wrote:
> >>
> >> Hello All,
> >>
> >> AFAIK Hadoop serialization comes into picture in the 2 areas:
> >>
> >> putting data on the wire i.e., for interprocess communication between
> >> nodes using RPC
> >> putting data on disk i.e. using the Map Reduce for persistent storage
> say
> >> HDFS.
> >>
> >>
> >> I have a couple of questions regarding the Serialization mechanisms used
> >> in Hadoop:
> >>
> >> Does Hadoop provides a pluggible feature for Serialization for both the
> >> above cases?
> >> Is Writable the default Serialization mechanism for both the above
> cases?
> >> Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop
> >> 2.x?
> >> Will there be a significant performance gain if the default
> Serialization
> >> i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map
> Reduce
> >> programming?
> >>
> >>
> >> Thanks,
> >> -RR
> >
> >
> >
> >
> > --
> > Jay Vyas
> > http://jayunit100.blogspot.com
>
>
>
> --
> Harsh J
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Hadoop Serialization mechanisms
Posted by Jay Vyas <ja...@gmail.com>.
But I believe w.r.t. "will we see performance gains when using
avro/thrift/... over writables" -- it depends on the writable
implementation. For example, If I have a writable serialization which
can use a bit map to store an enum, but then read that enum as a string:
It will look the same to user, but my writable implementation would be
superior. We can obviously say that if you use avro/thrift/pbuffers in
an efficient way, then yes, you will see a performance gain then say
storing everything as Text Writable objects. But clever optimizations can
be done even within the Writable framework as well.
On Sun, Mar 30, 2014 at 4:08 PM, Harsh J <ha...@cloudera.com> wrote:
> > Does Hadoop provides a pluggible feature for Serialization for both the
> above cases?
>
> - You can override the RPC serialisation module and engine with a
> custom class if you wish to, but it would not be trivial task.
> - You can easily use custom data serialisation modules for I/O.
>
> > Is Writable the default Serialization mechanism for both the above cases?
>
> While MR's built-in examples in Apache Hadoop continue to use
> Writables, the RPCs have moved to using Protocol buffers for
> themselves in 2.x onwards.
>
> > Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop
> 2.x?
>
> Yes "partially", see above.
>
> > Will there be a significant performance gain if the default
> Serialization i.e. Writables is replaced with Avro, Protol Buffers or
> Thrift in Map Reduce programming?
>
> Yes, you should see a gain in using a more efficient data
> serialisation format for data files.
>
> On Sun, Mar 30, 2014 at 9:09 PM, Jay Vyas <ja...@gmail.com> wrote:
> > Those are all great questions, and mostly difficultto answer. I havent
> > played with serialization APIs in some time, but let me try to give some
> > guidance. WRT to your bulleted questions above:
> >
> > 1) Serialization is file system independant: The use of any hadoop
> > compatible file system should support any kind of serialization.
> >
> > 2) See (1). The "default serialization" is Writables: But you can easily
> > add your own by modifiying the io.serializations configuration parameter.
> >
> > 3) I doubt anything significant effecting the way serialization works:
> The
> > main thrust of 1->2 was in the way services are deployed, not changing
> the
> > internals of how data is serialized. After all, the serialization APIs
> need
> > to remain stability even as the arch. of hadoop changes.
> >
> > 4) It depends on the implementation. If you have a custom writable that
> is
> > really good at compressing your data, that will be better than using a
> > thrift auto generated API for serialization that is uncustomized out of
> the
> > box. Example: Say you are writing "strings" and you know the string is
> max
> > 3 characters. A "smart" Writable serializer with custom implementations
> > optimized for your data will beat a thrift serialization approach. But I
> > think in general, the advantage of thrift/avro is that its easier to get
> > really good compression natively out-of-the-box, due to the fact that
> many
> > different data types are strongly supported by the way they apply the
> > schemas (for example , a thrift struct can contain a "boolean", two
> > "strings" , and an "int" These types will all be optmiized for you by
> > thrift.... Where as in Writables, you cannot as easily create
> sophisticated
> > types with optimization of nested properties.
> >
> >
> >
> >
> > On Thu, Mar 27, 2014 at 2:59 AM, Radhe Radhe <
> radhe.krishna.radhe@live.com>
> > wrote:
> >>
> >> Hello All,
> >>
> >> AFAIK Hadoop serialization comes into picture in the 2 areas:
> >>
> >> putting data on the wire i.e., for interprocess communication between
> >> nodes using RPC
> >> putting data on disk i.e. using the Map Reduce for persistent storage
> say
> >> HDFS.
> >>
> >>
> >> I have a couple of questions regarding the Serialization mechanisms used
> >> in Hadoop:
> >>
> >> Does Hadoop provides a pluggible feature for Serialization for both the
> >> above cases?
> >> Is Writable the default Serialization mechanism for both the above
> cases?
> >> Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop
> >> 2.x?
> >> Will there be a significant performance gain if the default
> Serialization
> >> i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map
> Reduce
> >> programming?
> >>
> >>
> >> Thanks,
> >> -RR
> >
> >
> >
> >
> > --
> > Jay Vyas
> > http://jayunit100.blogspot.com
>
>
>
> --
> Harsh J
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Hadoop Serialization mechanisms
Posted by Jay Vyas <ja...@gmail.com>.
But I believe w.r.t. "will we see performance gains when using
avro/thrift/... over writables" -- it depends on the writable
implementation. For example, If I have a writable serialization which
can use a bit map to store an enum, but then read that enum as a string:
It will look the same to user, but my writable implementation would be
superior. We can obviously say that if you use avro/thrift/pbuffers in
an efficient way, then yes, you will see a performance gain then say
storing everything as Text Writable objects. But clever optimizations can
be done even within the Writable framework as well.
On Sun, Mar 30, 2014 at 4:08 PM, Harsh J <ha...@cloudera.com> wrote:
> > Does Hadoop provides a pluggible feature for Serialization for both the
> above cases?
>
> - You can override the RPC serialisation module and engine with a
> custom class if you wish to, but it would not be trivial task.
> - You can easily use custom data serialisation modules for I/O.
>
> > Is Writable the default Serialization mechanism for both the above cases?
>
> While MR's built-in examples in Apache Hadoop continue to use
> Writables, the RPCs have moved to using Protocol buffers for
> themselves in 2.x onwards.
>
> > Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop
> 2.x?
>
> Yes "partially", see above.
>
> > Will there be a significant performance gain if the default
> Serialization i.e. Writables is replaced with Avro, Protol Buffers or
> Thrift in Map Reduce programming?
>
> Yes, you should see a gain in using a more efficient data
> serialisation format for data files.
>
> On Sun, Mar 30, 2014 at 9:09 PM, Jay Vyas <ja...@gmail.com> wrote:
> > Those are all great questions, and mostly difficultto answer. I havent
> > played with serialization APIs in some time, but let me try to give some
> > guidance. WRT to your bulleted questions above:
> >
> > 1) Serialization is file system independant: The use of any hadoop
> > compatible file system should support any kind of serialization.
> >
> > 2) See (1). The "default serialization" is Writables: But you can easily
> > add your own by modifiying the io.serializations configuration parameter.
> >
> > 3) I doubt anything significant effecting the way serialization works:
> The
> > main thrust of 1->2 was in the way services are deployed, not changing
> the
> > internals of how data is serialized. After all, the serialization APIs
> need
> > to remain stability even as the arch. of hadoop changes.
> >
> > 4) It depends on the implementation. If you have a custom writable that
> is
> > really good at compressing your data, that will be better than using a
> > thrift auto generated API for serialization that is uncustomized out of
> the
> > box. Example: Say you are writing "strings" and you know the string is
> max
> > 3 characters. A "smart" Writable serializer with custom implementations
> > optimized for your data will beat a thrift serialization approach. But I
> > think in general, the advantage of thrift/avro is that its easier to get
> > really good compression natively out-of-the-box, due to the fact that
> many
> > different data types are strongly supported by the way they apply the
> > schemas (for example , a thrift struct can contain a "boolean", two
> > "strings" , and an "int" These types will all be optmiized for you by
> > thrift.... Where as in Writables, you cannot as easily create
> sophisticated
> > types with optimization of nested properties.
> >
> >
> >
> >
> > On Thu, Mar 27, 2014 at 2:59 AM, Radhe Radhe <
> radhe.krishna.radhe@live.com>
> > wrote:
> >>
> >> Hello All,
> >>
> >> AFAIK Hadoop serialization comes into picture in the 2 areas:
> >>
> >> putting data on the wire i.e., for interprocess communication between
> >> nodes using RPC
> >> putting data on disk i.e. using the Map Reduce for persistent storage
> say
> >> HDFS.
> >>
> >>
> >> I have a couple of questions regarding the Serialization mechanisms used
> >> in Hadoop:
> >>
> >> Does Hadoop provides a pluggible feature for Serialization for both the
> >> above cases?
> >> Is Writable the default Serialization mechanism for both the above
> cases?
> >> Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop
> >> 2.x?
> >> Will there be a significant performance gain if the default
> Serialization
> >> i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map
> Reduce
> >> programming?
> >>
> >>
> >> Thanks,
> >> -RR
> >
> >
> >
> >
> > --
> > Jay Vyas
> > http://jayunit100.blogspot.com
>
>
>
> --
> Harsh J
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Hadoop Serialization mechanisms
Posted by Harsh J <ha...@cloudera.com>.
> Does Hadoop provides a pluggible feature for Serialization for both the above cases?
- You can override the RPC serialisation module and engine with a
custom class if you wish to, but it would not be trivial task.
- You can easily use custom data serialisation modules for I/O.
> Is Writable the default Serialization mechanism for both the above cases?
While MR's built-in examples in Apache Hadoop continue to use
Writables, the RPCs have moved to using Protocol buffers for
themselves in 2.x onwards.
> Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop 2.x?
Yes "partially", see above.
> Will there be a significant performance gain if the default Serialization i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map Reduce programming?
Yes, you should see a gain in using a more efficient data
serialisation format for data files.
On Sun, Mar 30, 2014 at 9:09 PM, Jay Vyas <ja...@gmail.com> wrote:
> Those are all great questions, and mostly difficultto answer. I havent
> played with serialization APIs in some time, but let me try to give some
> guidance. WRT to your bulleted questions above:
>
> 1) Serialization is file system independant: The use of any hadoop
> compatible file system should support any kind of serialization.
>
> 2) See (1). The "default serialization" is Writables: But you can easily
> add your own by modifiying the io.serializations configuration parameter.
>
> 3) I doubt anything significant effecting the way serialization works: The
> main thrust of 1->2 was in the way services are deployed, not changing the
> internals of how data is serialized. After all, the serialization APIs need
> to remain stability even as the arch. of hadoop changes.
>
> 4) It depends on the implementation. If you have a custom writable that is
> really good at compressing your data, that will be better than using a
> thrift auto generated API for serialization that is uncustomized out of the
> box. Example: Say you are writing "strings" and you know the string is max
> 3 characters. A "smart" Writable serializer with custom implementations
> optimized for your data will beat a thrift serialization approach. But I
> think in general, the advantage of thrift/avro is that its easier to get
> really good compression natively out-of-the-box, due to the fact that many
> different data types are strongly supported by the way they apply the
> schemas (for example , a thrift struct can contain a "boolean", two
> "strings" , and an "int" These types will all be optmiized for you by
> thrift.... Where as in Writables, you cannot as easily create sophisticated
> types with optimization of nested properties.
>
>
>
>
> On Thu, Mar 27, 2014 at 2:59 AM, Radhe Radhe <ra...@live.com>
> wrote:
>>
>> Hello All,
>>
>> AFAIK Hadoop serialization comes into picture in the 2 areas:
>>
>> putting data on the wire i.e., for interprocess communication between
>> nodes using RPC
>> putting data on disk i.e. using the Map Reduce for persistent storage say
>> HDFS.
>>
>>
>> I have a couple of questions regarding the Serialization mechanisms used
>> in Hadoop:
>>
>> Does Hadoop provides a pluggible feature for Serialization for both the
>> above cases?
>> Is Writable the default Serialization mechanism for both the above cases?
>> Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop
>> 2.x?
>> Will there be a significant performance gain if the default Serialization
>> i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map Reduce
>> programming?
>>
>>
>> Thanks,
>> -RR
>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
--
Harsh J
Re: Hadoop Serialization mechanisms
Posted by Harsh J <ha...@cloudera.com>.
> Does Hadoop provides a pluggible feature for Serialization for both the above cases?
- You can override the RPC serialisation module and engine with a
custom class if you wish to, but it would not be trivial task.
- You can easily use custom data serialisation modules for I/O.
> Is Writable the default Serialization mechanism for both the above cases?
While MR's built-in examples in Apache Hadoop continue to use
Writables, the RPCs have moved to using Protocol buffers for
themselves in 2.x onwards.
> Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop 2.x?
Yes "partially", see above.
> Will there be a significant performance gain if the default Serialization i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map Reduce programming?
Yes, you should see a gain in using a more efficient data
serialisation format for data files.
On Sun, Mar 30, 2014 at 9:09 PM, Jay Vyas <ja...@gmail.com> wrote:
> Those are all great questions, and mostly difficultto answer. I havent
> played with serialization APIs in some time, but let me try to give some
> guidance. WRT to your bulleted questions above:
>
> 1) Serialization is file system independant: The use of any hadoop
> compatible file system should support any kind of serialization.
>
> 2) See (1). The "default serialization" is Writables: But you can easily
> add your own by modifiying the io.serializations configuration parameter.
>
> 3) I doubt anything significant effecting the way serialization works: The
> main thrust of 1->2 was in the way services are deployed, not changing the
> internals of how data is serialized. After all, the serialization APIs need
> to remain stability even as the arch. of hadoop changes.
>
> 4) It depends on the implementation. If you have a custom writable that is
> really good at compressing your data, that will be better than using a
> thrift auto generated API for serialization that is uncustomized out of the
> box. Example: Say you are writing "strings" and you know the string is max
> 3 characters. A "smart" Writable serializer with custom implementations
> optimized for your data will beat a thrift serialization approach. But I
> think in general, the advantage of thrift/avro is that its easier to get
> really good compression natively out-of-the-box, due to the fact that many
> different data types are strongly supported by the way they apply the
> schemas (for example , a thrift struct can contain a "boolean", two
> "strings" , and an "int" These types will all be optmiized for you by
> thrift.... Where as in Writables, you cannot as easily create sophisticated
> types with optimization of nested properties.
>
>
>
>
> On Thu, Mar 27, 2014 at 2:59 AM, Radhe Radhe <ra...@live.com>
> wrote:
>>
>> Hello All,
>>
>> AFAIK Hadoop serialization comes into picture in the 2 areas:
>>
>> putting data on the wire i.e., for interprocess communication between
>> nodes using RPC
>> putting data on disk i.e. using the Map Reduce for persistent storage say
>> HDFS.
>>
>>
>> I have a couple of questions regarding the Serialization mechanisms used
>> in Hadoop:
>>
>> Does Hadoop provides a pluggible feature for Serialization for both the
>> above cases?
>> Is Writable the default Serialization mechanism for both the above cases?
>> Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop
>> 2.x?
>> Will there be a significant performance gain if the default Serialization
>> i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map Reduce
>> programming?
>>
>>
>> Thanks,
>> -RR
>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
--
Harsh J
Re: Hadoop Serialization mechanisms
Posted by Harsh J <ha...@cloudera.com>.
> Does Hadoop provides a pluggible feature for Serialization for both the above cases?
- You can override the RPC serialisation module and engine with a
custom class if you wish to, but it would not be trivial task.
- You can easily use custom data serialisation modules for I/O.
> Is Writable the default Serialization mechanism for both the above cases?
While MR's built-in examples in Apache Hadoop continue to use
Writables, the RPCs have moved to using Protocol buffers for
themselves in 2.x onwards.
> Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop 2.x?
Yes "partially", see above.
> Will there be a significant performance gain if the default Serialization i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map Reduce programming?
Yes, you should see a gain in using a more efficient data
serialisation format for data files.
On Sun, Mar 30, 2014 at 9:09 PM, Jay Vyas <ja...@gmail.com> wrote:
> Those are all great questions, and mostly difficultto answer. I havent
> played with serialization APIs in some time, but let me try to give some
> guidance. WRT to your bulleted questions above:
>
> 1) Serialization is file system independant: The use of any hadoop
> compatible file system should support any kind of serialization.
>
> 2) See (1). The "default serialization" is Writables: But you can easily
> add your own by modifiying the io.serializations configuration parameter.
>
> 3) I doubt anything significant effecting the way serialization works: The
> main thrust of 1->2 was in the way services are deployed, not changing the
> internals of how data is serialized. After all, the serialization APIs need
> to remain stability even as the arch. of hadoop changes.
>
> 4) It depends on the implementation. If you have a custom writable that is
> really good at compressing your data, that will be better than using a
> thrift auto generated API for serialization that is uncustomized out of the
> box. Example: Say you are writing "strings" and you know the string is max
> 3 characters. A "smart" Writable serializer with custom implementations
> optimized for your data will beat a thrift serialization approach. But I
> think in general, the advantage of thrift/avro is that its easier to get
> really good compression natively out-of-the-box, due to the fact that many
> different data types are strongly supported by the way they apply the
> schemas (for example , a thrift struct can contain a "boolean", two
> "strings" , and an "int" These types will all be optmiized for you by
> thrift.... Where as in Writables, you cannot as easily create sophisticated
> types with optimization of nested properties.
>
>
>
>
> On Thu, Mar 27, 2014 at 2:59 AM, Radhe Radhe <ra...@live.com>
> wrote:
>>
>> Hello All,
>>
>> AFAIK Hadoop serialization comes into picture in the 2 areas:
>>
>> putting data on the wire i.e., for interprocess communication between
>> nodes using RPC
>> putting data on disk i.e. using the Map Reduce for persistent storage say
>> HDFS.
>>
>>
>> I have a couple of questions regarding the Serialization mechanisms used
>> in Hadoop:
>>
>> Does Hadoop provides a pluggible feature for Serialization for both the
>> above cases?
>> Is Writable the default Serialization mechanism for both the above cases?
>> Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop
>> 2.x?
>> Will there be a significant performance gain if the default Serialization
>> i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map Reduce
>> programming?
>>
>>
>> Thanks,
>> -RR
>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
--
Harsh J
Re: Hadoop Serialization mechanisms
Posted by Harsh J <ha...@cloudera.com>.
> Does Hadoop provides a pluggible feature for Serialization for both the above cases?
- You can override the RPC serialisation module and engine with a
custom class if you wish to, but it would not be trivial task.
- You can easily use custom data serialisation modules for I/O.
> Is Writable the default Serialization mechanism for both the above cases?
While MR's built-in examples in Apache Hadoop continue to use
Writables, the RPCs have moved to using Protocol buffers for
themselves in 2.x onwards.
> Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop 2.x?
Yes "partially", see above.
> Will there be a significant performance gain if the default Serialization i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map Reduce programming?
Yes, you should see a gain in using a more efficient data
serialisation format for data files.
On Sun, Mar 30, 2014 at 9:09 PM, Jay Vyas <ja...@gmail.com> wrote:
> Those are all great questions, and mostly difficultto answer. I havent
> played with serialization APIs in some time, but let me try to give some
> guidance. WRT to your bulleted questions above:
>
> 1) Serialization is file system independant: The use of any hadoop
> compatible file system should support any kind of serialization.
>
> 2) See (1). The "default serialization" is Writables: But you can easily
> add your own by modifiying the io.serializations configuration parameter.
>
> 3) I doubt anything significant effecting the way serialization works: The
> main thrust of 1->2 was in the way services are deployed, not changing the
> internals of how data is serialized. After all, the serialization APIs need
> to remain stability even as the arch. of hadoop changes.
>
> 4) It depends on the implementation. If you have a custom writable that is
> really good at compressing your data, that will be better than using a
> thrift auto generated API for serialization that is uncustomized out of the
> box. Example: Say you are writing "strings" and you know the string is max
> 3 characters. A "smart" Writable serializer with custom implementations
> optimized for your data will beat a thrift serialization approach. But I
> think in general, the advantage of thrift/avro is that its easier to get
> really good compression natively out-of-the-box, due to the fact that many
> different data types are strongly supported by the way they apply the
> schemas (for example , a thrift struct can contain a "boolean", two
> "strings" , and an "int" These types will all be optmiized for you by
> thrift.... Where as in Writables, you cannot as easily create sophisticated
> types with optimization of nested properties.
>
>
>
>
> On Thu, Mar 27, 2014 at 2:59 AM, Radhe Radhe <ra...@live.com>
> wrote:
>>
>> Hello All,
>>
>> AFAIK Hadoop serialization comes into picture in the 2 areas:
>>
>> putting data on the wire i.e., for interprocess communication between
>> nodes using RPC
>> putting data on disk i.e. using the Map Reduce for persistent storage say
>> HDFS.
>>
>>
>> I have a couple of questions regarding the Serialization mechanisms used
>> in Hadoop:
>>
>> Does Hadoop provides a pluggible feature for Serialization for both the
>> above cases?
>> Is Writable the default Serialization mechanism for both the above cases?
>> Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop
>> 2.x?
>> Will there be a significant performance gain if the default Serialization
>> i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map Reduce
>> programming?
>>
>>
>> Thanks,
>> -RR
>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
--
Harsh J
Re: Hadoop Serialization mechanisms
Posted by Jay Vyas <ja...@gmail.com>.
Those are all great questions, and mostly difficultto answer. I havent
played with serialization APIs in some time, but let me try to give some
guidance. WRT to your bulleted questions above:
1) Serialization is file system independant: The use of any hadoop
compatible file system should support any kind of serialization.
2) See (1). The "default serialization" is Writables: But you can easily
add your own by modifiying the io.serializations configuration parameter.
3) I doubt anything significant effecting the way serialization works: The
main thrust of 1->2 was in the way services are deployed, not changing the
internals of how data is serialized. After all, the serialization APIs
need to remain stability even as the arch. of hadoop changes.
4) It depends on the implementation. If you have a custom writable that is
really good at compressing your data, that will be better than using a
thrift auto generated API for serialization that is uncustomized out of the
box. Example: Say you are writing "strings" and you know the string is
max 3 characters. A "smart" Writable serializer with custom
implementations optimized for your data will beat a thrift serialization
approach. But I think in general, the advantage of thrift/avro is that its
easier to get really good compression natively out-of-the-box, due to the
fact that many different data types are strongly supported by the way they
apply the schemas (for example , a thrift struct can contain a "boolean",
two "strings" , and an "int" These types will all be optmiized for you by
thrift.... Where as in Writables, you cannot as easily create sophisticated
types with optimization of nested properties.
On Thu, Mar 27, 2014 at 2:59 AM, Radhe Radhe
<ra...@live.com>wrote:
> Hello All,
>
> AFAIK Hadoop serialization comes into picture in the 2 areas:
>
> 1. putting data on the wire i.e., for interprocess communication
> between nodes using RPC
> 2. putting data on disk i.e. using the Map Reduce for persistent
> storage say HDFS.
>
>
> I have a couple of questions regarding the Serialization mechanisms used
> in Hadoop:
>
>
> 1. Does Hadoop provides a pluggible feature for Serialization for both
> the above cases?
> 2. Is Writable the default Serialization mechanism for both the above
> cases?
> 3. Were there any changes w.r.t. to Serialization from Hadoop 1.x to
> Hadoop 2.x?
> 4. Will there be a significant performance gain if the default
> Serialization i.e. Writables is replaced with Avro, Protol Buffers or
> Thrift in Map Reduce programming?
>
>
> Thanks,
> -RR
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Hadoop Serialization mechanisms
Posted by Jay Vyas <ja...@gmail.com>.
Those are all great questions, and mostly difficultto answer. I havent
played with serialization APIs in some time, but let me try to give some
guidance. WRT to your bulleted questions above:
1) Serialization is file system independant: The use of any hadoop
compatible file system should support any kind of serialization.
2) See (1). The "default serialization" is Writables: But you can easily
add your own by modifiying the io.serializations configuration parameter.
3) I doubt anything significant effecting the way serialization works: The
main thrust of 1->2 was in the way services are deployed, not changing the
internals of how data is serialized. After all, the serialization APIs
need to remain stability even as the arch. of hadoop changes.
4) It depends on the implementation. If you have a custom writable that is
really good at compressing your data, that will be better than using a
thrift auto generated API for serialization that is uncustomized out of the
box. Example: Say you are writing "strings" and you know the string is
max 3 characters. A "smart" Writable serializer with custom
implementations optimized for your data will beat a thrift serialization
approach. But I think in general, the advantage of thrift/avro is that its
easier to get really good compression natively out-of-the-box, due to the
fact that many different data types are strongly supported by the way they
apply the schemas (for example , a thrift struct can contain a "boolean",
two "strings" , and an "int" These types will all be optmiized for you by
thrift.... Where as in Writables, you cannot as easily create sophisticated
types with optimization of nested properties.
On Thu, Mar 27, 2014 at 2:59 AM, Radhe Radhe
<ra...@live.com>wrote:
> Hello All,
>
> AFAIK Hadoop serialization comes into picture in the 2 areas:
>
> 1. putting data on the wire i.e., for interprocess communication
> between nodes using RPC
> 2. putting data on disk i.e. using the Map Reduce for persistent
> storage say HDFS.
>
>
> I have a couple of questions regarding the Serialization mechanisms used
> in Hadoop:
>
>
> 1. Does Hadoop provides a pluggible feature for Serialization for both
> the above cases?
> 2. Is Writable the default Serialization mechanism for both the above
> cases?
> 3. Were there any changes w.r.t. to Serialization from Hadoop 1.x to
> Hadoop 2.x?
> 4. Will there be a significant performance gain if the default
> Serialization i.e. Writables is replaced with Avro, Protol Buffers or
> Thrift in Map Reduce programming?
>
>
> Thanks,
> -RR
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Hadoop Serialization mechanisms
Posted by Jay Vyas <ja...@gmail.com>.
Those are all great questions, and mostly difficultto answer. I havent
played with serialization APIs in some time, but let me try to give some
guidance. WRT to your bulleted questions above:
1) Serialization is file system independant: The use of any hadoop
compatible file system should support any kind of serialization.
2) See (1). The "default serialization" is Writables: But you can easily
add your own by modifiying the io.serializations configuration parameter.
3) I doubt anything significant effecting the way serialization works: The
main thrust of 1->2 was in the way services are deployed, not changing the
internals of how data is serialized. After all, the serialization APIs
need to remain stability even as the arch. of hadoop changes.
4) It depends on the implementation. If you have a custom writable that is
really good at compressing your data, that will be better than using a
thrift auto generated API for serialization that is uncustomized out of the
box. Example: Say you are writing "strings" and you know the string is
max 3 characters. A "smart" Writable serializer with custom
implementations optimized for your data will beat a thrift serialization
approach. But I think in general, the advantage of thrift/avro is that its
easier to get really good compression natively out-of-the-box, due to the
fact that many different data types are strongly supported by the way they
apply the schemas (for example , a thrift struct can contain a "boolean",
two "strings" , and an "int" These types will all be optmiized for you by
thrift.... Where as in Writables, you cannot as easily create sophisticated
types with optimization of nested properties.
On Thu, Mar 27, 2014 at 2:59 AM, Radhe Radhe
<ra...@live.com>wrote:
> Hello All,
>
> AFAIK Hadoop serialization comes into picture in the 2 areas:
>
> 1. putting data on the wire i.e., for interprocess communication
> between nodes using RPC
> 2. putting data on disk i.e. using the Map Reduce for persistent
> storage say HDFS.
>
>
> I have a couple of questions regarding the Serialization mechanisms used
> in Hadoop:
>
>
> 1. Does Hadoop provides a pluggible feature for Serialization for both
> the above cases?
> 2. Is Writable the default Serialization mechanism for both the above
> cases?
> 3. Were there any changes w.r.t. to Serialization from Hadoop 1.x to
> Hadoop 2.x?
> 4. Will there be a significant performance gain if the default
> Serialization i.e. Writables is replaced with Avro, Protol Buffers or
> Thrift in Map Reduce programming?
>
>
> Thanks,
> -RR
>
--
Jay Vyas
http://jayunit100.blogspot.com
RE: Hadoop Serialization mechanisms
Posted by Radhe Radhe <ra...@live.com>.
Second try.
Please find some time to answer some of my queries.
Thanks,-RR
From: radhe.krishna.radhe@live.com
To: user@hadoop.apache.org
Subject: Hadoop Serialization mechanisms
Date: Thu, 27 Mar 2014 13:29:35 +0530
Hello All,
AFAIK Hadoop serialization comes into picture in the 2 areas:putting data on the wire i.e., for interprocess communication between nodes using RPCputting data on disk i.e. using the Map Reduce for persistent storage say HDFS.
I have a couple of questions regarding the Serialization mechanisms used in Hadoop:
Does Hadoop provides a pluggible feature for Serialization for both the above cases?Is Writable the default Serialization mechanism for both the above cases?Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop 2.x?Will there be a significant performance gain if the default Serialization i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map Reduce programming?
Thanks,-RR
RE: Hadoop Serialization mechanisms
Posted by Radhe Radhe <ra...@live.com>.
Second try.
Please find some time to answer some of my queries.
Thanks,-RR
From: radhe.krishna.radhe@live.com
To: user@hadoop.apache.org
Subject: Hadoop Serialization mechanisms
Date: Thu, 27 Mar 2014 13:29:35 +0530
Hello All,
AFAIK Hadoop serialization comes into picture in the 2 areas:putting data on the wire i.e., for interprocess communication between nodes using RPCputting data on disk i.e. using the Map Reduce for persistent storage say HDFS.
I have a couple of questions regarding the Serialization mechanisms used in Hadoop:
Does Hadoop provides a pluggible feature for Serialization for both the above cases?Is Writable the default Serialization mechanism for both the above cases?Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop 2.x?Will there be a significant performance gain if the default Serialization i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map Reduce programming?
Thanks,-RR
RE: Hadoop Serialization mechanisms
Posted by Radhe Radhe <ra...@live.com>.
Second try.
Please find some time to answer some of my queries.
Thanks,-RR
From: radhe.krishna.radhe@live.com
To: user@hadoop.apache.org
Subject: Hadoop Serialization mechanisms
Date: Thu, 27 Mar 2014 13:29:35 +0530
Hello All,
AFAIK Hadoop serialization comes into picture in the 2 areas:putting data on the wire i.e., for interprocess communication between nodes using RPCputting data on disk i.e. using the Map Reduce for persistent storage say HDFS.
I have a couple of questions regarding the Serialization mechanisms used in Hadoop:
Does Hadoop provides a pluggible feature for Serialization for both the above cases?Is Writable the default Serialization mechanism for both the above cases?Were there any changes w.r.t. to Serialization from Hadoop 1.x to Hadoop 2.x?Will there be a significant performance gain if the default Serialization i.e. Writables is replaced with Avro, Protol Buffers or Thrift in Map Reduce programming?
Thanks,-RR
Re: Hadoop Serialization mechanisms
Posted by Jay Vyas <ja...@gmail.com>.
Those are all great questions, and mostly difficultto answer. I havent
played with serialization APIs in some time, but let me try to give some
guidance. WRT to your bulleted questions above:
1) Serialization is file system independant: The use of any hadoop
compatible file system should support any kind of serialization.
2) See (1). The "default serialization" is Writables: But you can easily
add your own by modifiying the io.serializations configuration parameter.
3) I doubt anything significant effecting the way serialization works: The
main thrust of 1->2 was in the way services are deployed, not changing the
internals of how data is serialized. After all, the serialization APIs
need to remain stability even as the arch. of hadoop changes.
4) It depends on the implementation. If you have a custom writable that is
really good at compressing your data, that will be better than using a
thrift auto generated API for serialization that is uncustomized out of the
box. Example: Say you are writing "strings" and you know the string is
max 3 characters. A "smart" Writable serializer with custom
implementations optimized for your data will beat a thrift serialization
approach. But I think in general, the advantage of thrift/avro is that its
easier to get really good compression natively out-of-the-box, due to the
fact that many different data types are strongly supported by the way they
apply the schemas (for example , a thrift struct can contain a "boolean",
two "strings" , and an "int" These types will all be optmiized for you by
thrift.... Where as in Writables, you cannot as easily create sophisticated
types with optimization of nested properties.
On Thu, Mar 27, 2014 at 2:59 AM, Radhe Radhe
<ra...@live.com>wrote:
> Hello All,
>
> AFAIK Hadoop serialization comes into picture in the 2 areas:
>
> 1. putting data on the wire i.e., for interprocess communication
> between nodes using RPC
> 2. putting data on disk i.e. using the Map Reduce for persistent
> storage say HDFS.
>
>
> I have a couple of questions regarding the Serialization mechanisms used
> in Hadoop:
>
>
> 1. Does Hadoop provides a pluggible feature for Serialization for both
> the above cases?
> 2. Is Writable the default Serialization mechanism for both the above
> cases?
> 3. Were there any changes w.r.t. to Serialization from Hadoop 1.x to
> Hadoop 2.x?
> 4. Will there be a significant performance gain if the default
> Serialization i.e. Writables is replaced with Avro, Protol Buffers or
> Thrift in Map Reduce programming?
>
>
> Thanks,
> -RR
>
--
Jay Vyas
http://jayunit100.blogspot.com