You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Matthew Willson (JIRA)" <ji...@apache.org> on 2015/03/05 16:32:39 UTC
[jira] [Updated] (HADOOP-11678) AvroSerializer buffers output in
violation of contract for Serializer
[ https://issues.apache.org/jira/browse/HADOOP-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthew Willson updated HADOOP-11678:
-------------------------------------
Description:
We've had issues with the deserializer running into EOFException when using Cascading's TupleSerialization (which delegates to other hadoop serializers to serialize entries within its tuples) in combination with AvroSerialization.
Eventually tracked it down to the fact that AvroSerialization#AvroSerializer is buffering output (since it uses a buffering BinaryEncoder rather than a non-buffering directBinaryEncoder):
https://github.com/apache/hadoop/blob/e1109fb65608a668cd53dc324dadc6f63a74eeb9/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/serializer/avro/AvroSerialization.java#L105
The contract for Serializer explicitly states "Serializers ... must not buffer the output since other producers may write to the output between calls to #serialize(Object)."
TupleSerialization does exactly that (write to the output between calls to #serialize), hence our problem.
Switching it to use the non-buffering `EncoderFactory#directBinaryEncoder` and `DecoderFactory#directBinaryDecoder` fixes the issue for us.
was:
We've had issues with the deserializer running into EOFException when using Cascading's TupleSerialization (which delegates to other hadoop serializers to serialize entries within its tuples) in combination with AvroSerialization.
Eventually tracked it down to the fact that AvroSerialization#AvroSerializer is buffering output (since it uses an avro BinaryEncoder):
https://github.com/apache/hadoop/blob/e1109fb65608a668cd53dc324dadc6f63a74eeb9/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/serializer/avro/AvroSerialization.java#L105
The contract for Serializer explicitly states "Serializers ... must not buffer the output since other producers may write to the output between calls to #serialize(Object)."
TupleSerialization does exactly that (write to the output between calls to #serialize), hence our problem.
It's not sufficient just to flush the encoder on close, it needs to be flushed after every write. Doing this fixes our issue.
> AvroSerializer buffers output in violation of contract for Serializer
> ---------------------------------------------------------------------
>
> Key: HADOOP-11678
> URL: https://issues.apache.org/jira/browse/HADOOP-11678
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 2.6.0
> Reporter: Matthew Willson
>
> We've had issues with the deserializer running into EOFException when using Cascading's TupleSerialization (which delegates to other hadoop serializers to serialize entries within its tuples) in combination with AvroSerialization.
> Eventually tracked it down to the fact that AvroSerialization#AvroSerializer is buffering output (since it uses a buffering BinaryEncoder rather than a non-buffering directBinaryEncoder):
> https://github.com/apache/hadoop/blob/e1109fb65608a668cd53dc324dadc6f63a74eeb9/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/serializer/avro/AvroSerialization.java#L105
> The contract for Serializer explicitly states "Serializers ... must not buffer the output since other producers may write to the output between calls to #serialize(Object)."
> TupleSerialization does exactly that (write to the output between calls to #serialize), hence our problem.
> Switching it to use the non-buffering `EncoderFactory#directBinaryEncoder` and `DecoderFactory#directBinaryDecoder` fixes the issue for us.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)