You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Tom White (JIRA)" <ji...@apache.org> on 2008/07/18 13:02:32 UTC
[jira] Created: (HADOOP-3788) Add serialization for Protocol
Buffers
Add serialization for Protocol Buffers
--------------------------------------
Key: HADOOP-3788
URL: https://issues.apache.org/jira/browse/HADOOP-3788
Project: Hadoop Core
Issue Type: Wish
Components: examples, mapred
Reporter: Tom White
Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Created: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by eishay <ei...@gmail.com>.
Sorry to interrupt.
I had a similar problem and I used this wrapper class so solve it. The
implementation is a bit tricky, but the class usage is simple.
The implementation:
-----------------------------------------------------------------------------
/**
* Manually serializing Protobuf objects
* The serialize form first has an integer SIZE which is the size of the
test of the serialized protobuf.
* After the integer there are SIZE bites of the protobuf serialized object
* @author esmith
*
*/
class ProtobufSerializer<T extends GeneratedMessage> implements
Externalizable
{
/**
* Object to serialize
*/
private transient T _proto;
private transient String _className;
public ProtobufSerializer()
{
}
public ProtobufSerializer(T proto)
{
_proto = proto;
if(null != _proto)
{
_className = _proto.getClass().getName();
}
}
public T get()
{
return _proto;
}
/**
* If the first byte is the size of zero, the object is null
* @see java.io.Externalizable#readExternal(java.io.ObjectInput)
*/
@SuppressWarnings("unchecked")
public void readExternal (ObjectInput in)
throws IOException, ClassNotFoundException
{
int size = in.readInt();
if(0 == size)
{
return;
}
byte[] array = new byte[size];
in.readFully(array, 0, size);
_className = new String(array);
size = in.readInt();
array = new byte[size];
in.readFully(array, 0, size);
try
{
Class<?> clazz = getClass().getClassLoader().loadClass(_className);
Method parseMethod = clazz.getMethod("parseFrom", array.getClass());
_proto = (T)parseMethod.invoke(clazz, array);
}
catch (Exception e)
{
e.printStackTrace();
throw new IOException("could not load class " + _className);
}
}
/**
* If the the object is null then the int zero is written to the stream
* @see java.io.Externalizable#writeExternal(java.io.ObjectOutput)
*/
public void writeExternal (ObjectOutput out)
throws IOException
{
if(null == _proto)
{
out.writeInt(0);
return;
}
out.writeInt(_className.getBytes().length);
out.write(_className.getBytes());
ByteArrayOutputStream baos = new ByteArrayOutputStream();
_proto.writeTo(baos);
baos.close();
byte[] array = baos.toByteArray();
out.writeInt(array.length);
out.write(array);
}
}
-----------------------------------------------------------------------------
Using it:
-----------------------------------------------------------------------------
/**
* a wrapper that makes sure the media is java serializable
*/
private ProtobufSerializer<NewsMediaContent> _mediaHolder;
/**
* unwraps the object from the serializer
* @return object media
*/
public NewsMediaContent getNewsMediaContent ()
{
return null == _mediaHolder ? null : _mediaHolder.get();
}
/**
* encapsulates the media in a serializer
* @param media
*/
public void setNewsMediaContent (NewsMediaContent media)
{
_mediaHolder = new ProtobufSerializer<NewsMediaContent>(media);
}
-----------------------------------------------------------------------------
Regards, Eishay
JIRA jira@apache.org wrote:
>
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Reporter: Tom White
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of
> encoding data in a compact binary format. This issue is to write a
> ProtocolBuffersSerialization to support using Protocol Buffers types in
> MapReduce programs, including an example program. This should probably go
> into contrib.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
>
--
View this message in context: http://www.nabble.com/-jira--Created%3A-%28HADOOP-3788%29-Add-serialization-for-Protocol-Buffers-tp18526920p20488500.html
Sent from the Hadoop core-dev mailing list archive at Nabble.com.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Chris Dyer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649503#action_12649503 ]
Chris Dyer commented on HADOOP-3788:
------------------------------------
Other pieces of my system are using protocol buffers already, so I'm stuck with it for the pieces that have to interact with Hadoop. I am currently using HadoopStreaming, but the hoops I jump through are quite extensive- I serialize the protocol buffer to byte arrays and then encode them using base64 so that they can be put into Streaming's text format's key-value pair encoding, where the key is separated from the value by a tab and the record is terminated with a newline. These extra layers aren't really a problem since what i'm computing is computationally quite expensive (i could serialize to XML and it would be just a drop in the bucket in terms of running time). But, it does complicate my code in ways I think should be unnecessary. There is a paucity of information for how to use streaming with non-text data, so I haven't really been able to figure out of there's an easier way to do all of this.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/serialization, examples, mapred
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.20.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, hadoop-3788-v3.patch, protobuf-java-2.0.1.jar, protobuf-java-2.0.2.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Alex Loddengaard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Loddengaard updated HADOOP-3788:
-------------------------------------
Attachment: protobuf-java-2.0.1.jar
hadoop-3788-v1.patch
Attaching the patch and PB library. The patch is discussed in great detail in my previous comment.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3788) Add serialization for
Protocol Buffers
Posted by "Alex Loddengaard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630139#action_12630139 ]
alexlod edited comment on HADOOP-3788 at 9/11/08 2:30 AM:
-------------------------------------------------------------------
Attaching a new patch. Changes:
* Removed _*Tracker_ and _TestPBHadoopStreams_ because they aren't very useful now that we've established streams have trailing data
* Did not keep a single Builder instance in _PBDeserializer_, because Builders cannot be reused once _build()_ has been called. From the PB API: "[build()] Construct the final message. Once [build()] is called, the Builder is no longer valid, and calling any other method may throw a NullPointerException. If you need to continue working with the builder after calling build(), clone() it first." I made the decision to just re-instantiate instead of clone, because I thought the performance differences were negligible. Please argue with me if I'm wrong.
* Changed SequenceFile.Reader#next(Object)
* Changed _TestPBSerialization_ to just write and read a SequenceFile, respectively.
* Created a new test, _TestPBSerializationMapReduce_, that uses PBs in a MapReduce program
_TestPBSerialization_ passes, but _TestPBSerializationMapReduce_ does not, which means you're right, Tom, that other code will need to change, though I'm not familiar enough with Hadoop to say more than that. If we decide to move further along by changing Hadoop such that deserializers will never be given trailing data, then more guidance would be greatly appreciated :).
This patch breaks a few existing tests such as _org.apache.hadoop.fs.TestCopyFiles_ and _org.apache.hadoop.fs.TestFileSystem_. It's unclear if my change causes these or if my lack of change to others areas does. Regardless, I think this proves that creating the contract of not having extra data in the _Deserializer_'s _InputStream_ would probably be a large change.
There is a discussion going on in the PB Google Group about possibly making PBs self-delimiting. Take a look [here|http://groups.google.com/group/protobuf/browse_thread/thread/b0ce2c7d8b05896e?hl=en]. In summary, a few different people are trying to determine the best way to allow self-delimiting, though there hasn't been any talk about a schedule.
was (Author: alexlod):
Attaching a new patch. Changes:
* Removed _*Tracker_ and _TestPBHadoopStreams_ because they weren't very useful now that we've established streams have trailing data
* Did not keep a single Builder instance in _PBDeserializer_, because Builders need to be rebuilt once _build()_ has been called. From the PB API: "[build()] Construct the final message. Once [build()] is called, the Builder is no longer valid, and calling any other method may throw a NullPointerException. If you need to continue working with the builder after calling build(), clone() it first." I made the decision to just re-instantiate instead of clone, because I thought the performance differences were negligible. Please argue with me if I'm wrong.
* Changed SequenceFile.Reader#next(Object)
* Changed _TestPBSerialization_ to just write and read a SequenceFile, respectively.
* Created a new test, _TestPBSerializationMapReduce_, that uses PBs in a MapReduce program
_TestPBSerialization_ passes, but _TestPBSerializationMapReduce_ does not, which means you're right, Tom, that other code will need to change, though I'm not familiar enough with Hadoop to say more than that. If we decide to move further along by changing Hadoop such that deserializers will never be given trailing data, then more guidance would be greatly appreciated :).
This patch breaks a few existing tests such as _org.apache.hadoop.fs.TestCopyFiles_ and _org.apache.hadoop.fs.TestFileSystem_. It's unclear if my change causes these or if my lack of change to others areas does. Regardless, I think this proves that creating the contract of not having extra data in the _Deserializer_'s _InputStream_ would probably be a large change.
There is a discussion going on in the PB Google Group about possibly making PBs self-delimiting. Take a look [here|http://groups.google.com/group/protobuf/browse_thread/thread/b0ce2c7d8b05896e?hl=en]. In summary, a few different people are trying to determine the best way to allow self-delimiting, though there hasn't been any talk about a schedule.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Alex Loddengaard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644801#action_12644801 ]
Alex Loddengaard commented on HADOOP-3788:
------------------------------------------
Chris, while I stagger with Protocol Buffers, you should look in to Thrift, which is described in HADOOP-3787.
Similarly, you should learn about Hadoop Streaming ([http://wiki.apache.org/hadoop/HadoopStreaming]), which will let you run any executable (C++, Python, etc).
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/serialization, examples, mapred
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.20.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633818#action_12633818 ]
Doug Cutting commented on HADOOP-3788:
--------------------------------------
> Don't we already have one of those floating around Hadoop somewhere?
If not, there's one in TFile (HADOOP-3315) that should be shared.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/serialization, examples, mapred
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.20.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Chris Dyer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644794#action_12644794 ]
Chris Dyer commented on HADOOP-3788:
------------------------------------
Apologies for just jumping in on this and commenting without much context, but one of my great hopes for protocol buffer integration in Hadoop is that it will make it easier to integrate rich, structured data types with non-Java Hadoop clients (specifically c++, but python is also quite well supported in Hadoop). Specifically, it would be nice if there could be some kind of interface definition that would prescribe how non-Java clients could implement mappers and reducers that would return protocol buffer serializations. In most cases, I don't think the framework would even need to know anything about the data types (unless you were implementing some kind of custom comparator), but if I could easily mix and match languages I would be a much, much happier camper. Authors of this patch: can you estimate how difficult incorporating c++ support would be?
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/serialization, examples, mapred
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.20.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Alex Loddengaard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Loddengaard updated HADOOP-3788:
-------------------------------------
Attachment: hadoop-3788-v2.patch
Attaching a new patch. Changes:
* Removed _*Tracker_ and _TestPBHadoopStreams_ because they weren't very useful now that we've established streams have trailing data
* Did not keep a single Builder instance in _PBDeserializer_, because Builders need to be rebuilt once _build()_ has been called. From the PB API: "[build()] Construct the final message. Once [build()] is called, the Builder is no longer valid, and calling any other method may throw a NullPointerException. If you need to continue working with the builder after calling build(), clone() it first." I made the decision to just re-instantiate instead of clone, because I thought the performance differences were negligible. Please argue with me if I'm wrong.
* Changed SequenceFile.Reader#next(Object)
* Changed _TestPBSerialization_ to just write and read a SequenceFile, respectively.
* Created a new test, _TestPBSerializationMapReduce_, that uses PBs in a MapReduce program
_TestPBSerialization_ passes, but _TestPBSerializationMapReduce_ does not, which means you're right, Tom, that other code will need to change, though I'm not familiar enough with Hadoop to say more than that. If we decide to move further along by changing Hadoop such that deserializers will never be given trailing data, then more guidance would be greatly appreciated :).
This patch breaks a few existing tests such as _org.apache.hadoop.fs.TestCopyFiles_ and _org.apache.hadoop.fs.TestFileSystem_. It's unclear if my change causes these or if my lack of change to others areas does. Regardless, I think this proves that creating the contract of not having extra data in the _Deserializer_'s _InputStream_ would probably be a large change.
There is a discussion going on in the PB Google Group about possibly making PBs self-delimiting. Take a look [here|http://groups.google.com/group/protobuf/browse_thread/thread/b0ce2c7d8b05896e?hl=en]. In summary, a few different people are trying to determine the best way to allow self-delimiting, though there hasn't been any talk about a schedule.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-3788:
------------------------------
Affects Version/s: (was: 0.19.0)
0.20.0
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/serialization, examples, mapred
> Affects Versions: 0.20.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Alex Loddengaard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627948#action_12627948 ]
Alex Loddengaard commented on HADOOP-3788:
------------------------------------------
After fiddling with Protocol Buffers (PBs) and reading documentation around them, actually using PBs may not require the introduction of a new Serialization class.
PBs work in the following way:
First, the developer defines a .proto file, which is essentially a schema that describes the type of data the user wishes to deal with. Below is an example of an addressbook.proto file taken from the PB documentation, located here: http://code.google.com/apis/protocolbuffers/docs/javatutorial.html
{code:title=addressbook.proto|borderStyle=solid}
package tutorial;
option java_package = "com.example.tutorial";
option java_outer_classname = "AddressBookProtos";
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phone = 4;
}
message AddressBook {
repeated Person person = 1;
}
{code}
Once the user has defined their .proto file, they use PB's compiler, _protoc_, to generate an outer Java class, accompanied by a few subclasses. Amongst the generated code are methods to serialize and deserialize given an OutputStream and InputStream, respectively.
Refactoring .proto files is somewhat tricky given the way PBs work (read their documentation for more info), so Google recommends that PBs are wrapped inside of other classes and only used when serializing and deserializing. This structure fits in perfectly with Hadoop's Writable structure. That is, if a user wants to utilize PBs, they need to use _protoc_ to create a Java class, which is essentially a Bean. They can then define a new Writable implementation that uses the _protoc_-generated class to serialize and deserialize. This is all possible without creating a ProtocolBuffersSerialization class because the default Serialization, org.apache.hadoop.io.serializer.WritableSerialization, delegates its read and write methods to the Writable that is being serialized or deserialized.
A general ProtocolBuffersSerialization class would not fully utilize PBs to their fullest, because it would have to use a very primitive, generalized .proto file (for example a file with just one field: a large String).
With that said, a few things can be done with regard to this feature:
# I can create an example that extends Text and overwrites its serialization methods to use a _protoc_-generated class
# I can begin extending Hadoop's Writable implementations to use PBs instead
# I can begin replacing Hadoop's Writable implementations to use PBs instead
# I can try and create a general ProtocolBuffersSerialization and see how it performs, though this solution seems against the premise of using PBs
# You can tell me that my understanding of PBs is completely wrong (please follow-up with a more accurate description if this is the case :))
Before either option 2 or 3 is decided on, profiling should be done to ensure that PBs are in fact faster than Java's built in mechanism. If profiling proves PBs are faster in all cases, then it seems like option 3 would be the most desirable. However, perhaps more discussion should be made to determine if 2 or 3 or some other solution altogether is better.
Again, I'm totally new here, so please argue with me if I have misunderstood Hadoop's workings, PBs, or anything else. While I'm waiting for responses, I can begin working on option 1 to prove my understanding of PBs is correct.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Reporter: Tom White
> Assignee: Alex Loddengaard
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629813#action_12629813 ]
Hadoop QA commented on HADOOP-3788:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12389811/protobuf-java-2.0.1.jar
against trunk revision 693705.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3231/console
This message is automatically generated.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Alex Loddengaard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629739#action_12629739 ]
Alex Loddengaard commented on HADOOP-3788:
------------------------------------------
A quick update: I applied Tom's patch from HADOOP-3787 to a fresh trunk build and looked at the _InputStream_ given to _ThriftDeserialization_. This stream has trailing binary data as well, that very closely resembles the trailing binary I saw in my PB example. Here is the output from _ThriftSerializer_:
Again, the strange characters below are a result of this being copy-pasted from _less_.
{noformat}
^@^A^@^@^@^@^@^@^@^A^@
{noformat}
Here's the _InputStream_ given to _ThriftDeserializer_
{noformat}
^@^A^@^@^@^@^@^@^@^A^@x��f`d```NN,a^@^@^D3^AH
{noformat}
As far as I can tell, this find proves that PBs are not compatible with Hadoop's current implementation. Can someone verify this, please, and also recommend possible next steps towards compatibility? In the meantime I'll dig into Hadoop more. Thanks!
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633820#action_12633820 ]
Doug Cutting commented on HADOOP-3788:
--------------------------------------
Should this instead go in contrib/protocol-buffer-serialization? It seems to me that this won't share a lot with other serialization implementations (e.g, HADOOP-3787) and that each should be packaged in its own jar. Separate contrib modules would achieve this.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/serialization, examples, mapred
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.20.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley reassigned HADOOP-3788:
-------------------------------------
Assignee: Alex Loddengaard
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Reporter: Tom White
> Assignee: Alex Loddengaard
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Alex Loddengaard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Loddengaard updated HADOOP-3788:
-------------------------------------
Fix Version/s: 0.19.0
Affects Version/s: 0.19.0
Release Note:
The patch being submitted is an in-progress patch. It is being submitted with failing tests because I seek help and advice from others.
See my comment for more information.
Hadoop Flags: [Incompatible change]
Status: Patch Available (was: Open)
Per Tom's advice, I created a Protocol Buffer (PB) serialization framework and tests to show its usage. I used HADOOP-3787 as a guide while doing so.
I ran into a problem, though. My test, _TestPBSerialization_, is precisely the same as the test in HADOOP-3787 with the exception of using PBs instead of Thrift. My test threw PB exceptions due to issues with deserializing. I engaged in dialog with a Google employee via the the PB Google Group in hopes of diagnosing my problem. Our thread can be found [here|http://groups.google.com/group/protobuf/browse_thread/thread/19ab6bbb364fef35]. The key point to the thread is that the _InputStream_ passed to a PB _Message_ instance during deserialization cannot have trailing binary data. For example, if a _Message_ instance is serialized to "<binary>asdf", then giving "<binary>asdf<arbitrary binary>" to a PB deserializer will break by means of a PB Exception. This is due to serialized PB _Message_ instances not being self-delimiting, which was a design decision made by Google to guarantee small serialized size and speed.
I created a second test, _TestPBSerializationIsolated_, that demonstrates the correctness of _PBSerializer_ and _PBDeserializer_, the two classes that actually do the work to serialize and deserialize. This test passed, hinting that perhaps there is an incompatibility with Hadoop's current workings and PBs.
I then created a third test, _TestPBHadoopStreams_, which tries to understands the Hadoop stream that is given to _PBDeserializer_. Though this test is somewhat silly (it always passes with an assertTrue(true) -- read the class comment for an explanation), its System.err output shows the makeup of the serialized _StringMessage_ and the _InputStream_ given for deserialization. I discovered that when Hadoop gives _PBDeserializer_ an _InputStream_, the stream contains arbitrary binary data after the serialized PB _Message_ instance. This is problematic for reasons I have previously discussed. I am confident that this extra arbitrary data is not a result of using PBs but instead a Hadoop implementation decision.
To be very precise, below is a serialized _StringMessage_ with a value, "testKey". Note that the below was copy-pasted from _less_, which is why strange ASCII characters are displayed:
{noformat}
^GtestKey
{noformat}
Below is the stream given to _PBDeserializer_ to deserialize:
{noformat}
^GtestKeyx���,I-.K�)M^E^@^S�^C�
{noformat}
Again, take note to the trailing bits, starting with 'x'.
Can someone comment on this issue? Was having trailing binary information a design decision? Can it be avoided somehow easily? Is there a way around this issue?
In the meantime, I plan to dig deeper into Hadoop's inner workings to understand why the _InputStream_ might have extra binary data. Similarly. I plan to use Hadoop's default _Serialization_ to see if the _InputStream_ also has arbitrary trailing bytes.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Alex Loddengaard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630072#action_12630072 ]
Alex Loddengaard commented on HADOOP-3788:
------------------------------------------
Tom, thanks for correcting my JIRA misuse.
PBs do not provide a mechanism to limit the amount of data read from a stream, so your solution of breaking key, value pairs into two streams is the approach we should take. I'll make this change.
As for your other comments, I included the *Tracker classes and TestPBHadoopStreams to prove that streams have extra binary data. I was planning on removing these classes once it was widely understood that the streams had extra data, but I'll keep them around and modify them appropriately.
Good idea on keeping a single Builder instance per deserializer. That's a much better solution.
I'll get working on this and submit a patch. Thanks again, Tom!
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633723#action_12633723 ]
Owen O'Malley commented on HADOOP-3788:
---------------------------------------
What went wrong with a FilterInputStream that has an artificial limit on the data read? Even better would be one that was resettable to a new length for the next object. Don't we already have one of those floating around Hadoop somewhere?
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/serialization, examples, mapred
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.20.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630237#action_12630237 ]
Doug Cutting commented on HADOOP-3788:
--------------------------------------
> I worry about defining the contract for deserializers so that the end of the stream marks the end of the object being read
That's certainly not the contract in effect today . One possible advantage of such a contract is that, if the framework knows the length, it can communicate it in this way to the object, so that lengths need not be stored twice. For example, SequenceFile stores the lengths of keys and values, and, if keys and values are Text, we store their lenght again in the key and value. But reading until EOF seems a poor way of communicating this. Perhaps we could change the deserialize API to be optionally passed a length or somesuch.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-3788:
------------------------------
Release Note: (was: The patch being submitted is an in-progress patch. It is being submitted with failing tests because I seek help and advice from others.
See my comment for more information.)
Hadoop Flags: (was: [Incompatible change])
Cleared the incompatible change flag, since as it stands this patch does not introduce any incompatible changes.
Also cleared the release note, which is intended as documentation that goes into the release only after the change has been committed.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-3788:
------------------------------
Attachment: protobuf-java-2.0.2.jar
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/serialization, examples, mapred
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.20.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, hadoop-3788-v3.patch, protobuf-java-2.0.1.jar, protobuf-java-2.0.2.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630695#action_12630695 ]
Steve Loughran commented on HADOOP-3788:
----------------------------------------
1. I could imagine the deserializers being handed a special kind of stream that counts down the remaining bytes and then signals EOF.
2. On a personal note, I think excluding length from packets is risky. It implies a protocol that assumes that TCP connections never stop half way. If you've ever seen a web browser only render half a photo, you know that assumption isn't always valid.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-3788:
------------------------------
Status: Open (was: Patch Available)
Cancelling patch while discussion continues.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628035#action_12628035 ]
Tom White commented on HADOOP-3788:
-----------------------------------
Alex, Thanks for looking at this.
It shouldn't be necessary to create a new Writable implementation for each protoc-generated class (if that's what you are suggesting). By writing a ProtocolBuffersSerialization it should be possible to avoid having to use Writables at all.
I imagined that the implementation of ProtocolBuffersSerializer would create a CodedOutputStream in the open method, then call Message#writeTo with the CodedOutputStream in the serialize method. ProtocolBuffersDeserializer is a bit more tricky. It would find the com.google.protobuf.Descriptors.Descriptor for the message class being deserialized, then use DynamicMessage#parseFrom to construct a message from the descriptor and the input stream.
To test this you could write some PB types to a Hadoop sequence file, then write a MapReduce program to process it and write it out to another sequence file containing PB types. See HADOOP-3787.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Reporter: Tom White
> Assignee: Alex Loddengaard
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630684#action_12630684 ]
Doug Cutting commented on HADOOP-3788:
--------------------------------------
> T deserialize(T t, int length) throws IOException;
The serializer would have to know that it will be deserialized this way, so that it knows not to include the length. So we'd need a different serialize() method too.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-3788:
------------------------------
Component/s: contrib/serialization
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/serialization, examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-3788:
------------------------------
Attachment: hadoop-3788-v3.patch
A patch that correctly reads Protocol Buffers from a SequenceFile. MapReduce doesn't work yet since Protocol Buffers always create a new object on deserialization, so we need the new MapReduce interfaces from HADOOP-1230.
I also moved the code to protocol-buffers-serialization under contrib, and updated to the latest PB release.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/serialization, examples, mapred
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.20.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, hadoop-3788-v3.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630701#action_12630701 ]
Pete Wyckoff commented on HADOOP-3788:
--------------------------------------
bq. could imagine the deserializers being handed a special kind of stream that counts down the remaining bytes and then signals EOF.
One problem i ran into related to this is what if your deserializer does buffering? For EOF, one can return null from deserialize or in theory throw EOFException :(, but for getProgress, RecordReader's use getPos which is off if the deserializer has its own buffers. e.g., if you were to implement LineReaderDeserializer using LineRecordReader.LineReader.
-- pete
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632229#action_12632229 ]
Tom White commented on HADOOP-3788:
-----------------------------------
bq. the OutputStream when serializing would need meta data included in it
I don't think we want to invent a new format here - this issue is to make serialization work with existing formats, such as SequenceFile (or the new TFile, or HADOOP-4065).
As an experiment, I modified PBDeserializer to have a deserialize method that takes a length (+in+ is now a CodedInputStream):
{code}
public T deserialize(T t, int length) throws IOException {
t = (t == null) ? (T) newInstance() : t;
int limit = in.pushLimit(length);
Message result =
t.newBuilderForType().mergeFrom(in).build();
in.popLimit(limit);
return (T) result;
}
{code}
I then modified TestPBSerializationIsolated to serialize two strings to the stream. When using the deserialize method that doesn't take a length the test failed, but when I passed the length the test succeeded.
So, I think we can do this without modifying Protocol Buffers. The change needed is the new method on Deserializer (and Serializer?) that takes a length, and then changes in the framework to call the new method when appropriate.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-3788:
------------------------------
Affects Version/s: (was: 0.20.0)
Fix Version/s: (was: 0.19.0)
0.20.0
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: contrib/serialization, examples, mapred
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.20.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Alex Loddengaard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629358#action_12629358 ]
Alex Loddengaard commented on HADOOP-3788:
------------------------------------------
Tom, thanks for your feedback. I've created a thread on the Protocol Buffer discussion group; I'm having trouble deserializing. I wanted to link to the thread here for reference purposes and also in hopes of you giving your feedback :).
http://groups.google.com/group/protobuf/browse_thread/thread/19ab6bbb364fef35#
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Reporter: Tom White
> Assignee: Alex Loddengaard
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630531#action_12630531 ]
Tom White commented on HADOOP-3788:
-----------------------------------
bq. But reading until EOF seems a poor way of communicating this. Perhaps we could change the deserialize API to be optionally passed a length or somesuch.
+1
How about adding a method to Deserializer (an interface, sigh), which is called when the length of the serialized object is known:
{noformat}
T deserialize(T t, int length) throws IOException;
{noformat}
This still needs help from PB to work. I wonder if CodedInputStream#pushLimit() would do the trick?
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630130#action_12630130 ]
Tom White commented on HADOOP-3788:
-----------------------------------
bq. PBs do not provide a mechanism to limit the amount of data read from a stream, so your solution of breaking key, value pairs into two streams is the approach we should take.
Of course the other option is to propose changes to PB (which is open source) to limit the amount of data read. I think a change to CodedInputStream would be relatively simple.
As a quick experiment I modified a working MapReduce program so that the deserializer read to the end of the stream. It failed in ReduceValuesIterator. So to make this work would require changing more than just SequenceFile. Perhaps this reveals a bug in the MR system - one that has been masked because existing serializers only consume as much as they need. (So if they are given more than they need it's not a problem.) Either way I worry about defining the contract for deserializers so that the end of the stream marks the end of the object being read as it might limit optimizations we may make in the future. What do others think?
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629809#action_12629809 ]
Tom White commented on HADOOP-3788:
-----------------------------------
Looks like your on the right track with PBDeserializer and PBSerializer. I think one of the problems is with the sequence file format and how it interacts with protocol buffers.
SequenceFile.Reader#next(Object) reads the next key and value into a single buffer DataInputBuffer which is given to PBDeserializer to deserialize from in the mergeFrom call. As you point out, PB reads to the end of the stream, so when it tries to read the key it consumes the whole buffer, consuming the value as well, which causes the exception. This is not a problem with other serialization frameworks that we have seen so far (Writable, Java Serialization, Thrift), which know how much of the stream to consume.
We could fix this by having separate buffers for key and value, much like org.apache.hadoop.mapred.IFile does. Or perhaps we could change deserializer to take a length. The latter would only work if it is possible to restrict the number of bytes read from the stream in PB. Is this the case?
Does it work if you don't try to read from a sequence file? Is your use-case based on being able to read from sequence files?
A couple of other points:
* PBDeserializerTracker reads from the stream twice, which isn't going to work. You need to tee the stream to do debugging.
* Can we keep a Builder instance per deserializer rather than create a new one for each call to deserialize? We should call clear on it each time to reset its state.
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3788) Add serialization for Protocol
Buffers
Posted by "Alex Loddengaard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631239#action_12631239 ]
Alex Loddengaard commented on HADOOP-3788:
------------------------------------------
FYI: from the PB discussion group [here|http://groups.google.com/group/protobuf/browse_thread/thread/19ab6bbb364fef35]:
{noformat}
At Google, we have lots of various container formats, for streaming, record-based files, database tables,
etc., where each record is a protocol buffer. All of these formats store the size of the message before the
message itself. Our philosophy is that because we have protocol buffers, all of these *other* formats and
protocols can be designed to pass around arbitrary byte blobs, which greatly simplifies them. An arbitrary
byte blob is not necessarily self-delimiting, so it's up to these container formats to keep track of the
size separately.
{noformat}
A possible solution would be to change the interface between _Message_ instances and the _PBSerialization_ framework such that a wrapping class, call it _PBMessageWrapper_ contains the length and logic to delimit the stream. Instances of this interface could create a new stream for deserializing, though serializing would now become more tricky -- the _OutputStream_ when serializing would need meta data included in it. It might also be possible to create a general instance of _PBMessageWrapper_, instead of creating wrappers for each _Message_.
Thoughts?
> Add serialization for Protocol Buffers
> --------------------------------------
>
> Key: HADOOP-3788
> URL: https://issues.apache.org/jira/browse/HADOOP-3788
> Project: Hadoop Core
> Issue Type: Wish
> Components: examples, mapred
> Affects Versions: 0.19.0
> Reporter: Tom White
> Assignee: Alex Loddengaard
> Fix For: 0.19.0
>
> Attachments: hadoop-3788-v1.patch, hadoop-3788-v2.patch, protobuf-java-2.0.1.jar
>
>
> Protocol Buffers (http://code.google.com/p/protobuf/) are a way of encoding data in a compact binary format. This issue is to write a ProtocolBuffersSerialization to support using Protocol Buffers types in MapReduce programs, including an example program. This should probably go into contrib.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.