You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mrunit.apache.org by "Meloss Xeloss (Created) (JIRA)" <ji...@apache.org> on 2012/02/28 16:53:46 UTC
[jira] [Created] (MRUNIT-70) copy in Serialization shouldn't
require objects to have a no-args constructor, and
copy in Serialization shouldn't require objects to have a no-args constructor, and
-----------------------------------------------------------------------------------
Key: MRUNIT-70
URL: https://issues.apache.org/jira/browse/MRUNIT-70
Project: MRUnit
Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Meloss Xeloss
Priority: Minor
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-70) copy(orig, conf) in Serialization
shouldn't require objects to have a no-args constructor, and copy(orig,
copy, conf) seems to violate contract for deserializer.deserialize()
Posted by "Jim Donofrio (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-70?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226759#comment-13226759 ]
Jim Donofrio commented on MRUNIT-70:
------------------------------------
I dont consider the failure of ByteBuffer to be mrunit's problem, there should be a serialization class that you can set in io.serializations to get ByteBuffer to work
> copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize()
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-70
> URL: https://issues.apache.org/jira/browse/MRUNIT-70
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Meloss Xeloss
> Assignee: Jim Donofrio
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: MRUNIT-70-1.patch
>
>
> The copy(orig, conf) method requires objects to have a no-args constructor, which is non-ideal, since it passes on to Deserializer.deserialize, which can create a new object if passed in null. Additionally, the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not.
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/serializer/Deserializer.html#deserialize%28T%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MRUNIT-70) copy(orig, conf) in Serialization
shouldn't require objects to have a no-args constructor, and copy(orig,
copy, conf) seems to violate contract for deserializer.deserialize()
Posted by "Jim Donofrio (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-70?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Donofrio updated MRUNIT-70:
-------------------------------
Attachment: MRUNIT-70-1.patch
Is this what you had in mind Brock?
> copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize()
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-70
> URL: https://issues.apache.org/jira/browse/MRUNIT-70
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Meloss Xeloss
> Priority: Minor
> Attachments: MRUNIT-70-1.patch
>
>
> The copy(orig, conf) method requires objects to have a no-args constructor, which is non-ideal, since it passes on to Deserializer.deserialize, which can create a new object if passed in null. Additionally, the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not.
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/serializer/Deserializer.html#deserialize%28T%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MRUNIT-70) copy(orig, conf) in Serialization
shouldn't require objects to have a no-args constructor, and copy(orig,
copy, conf) seems to violate contract for deserializer.deserialize()
Posted by "Jim Donofrio (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-70?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Donofrio resolved MRUNIT-70.
--------------------------------
Resolution: Fixed
Fix Version/s: 1.0.0
Assignee: Jim Donofrio
added javadoc on Serialization class methods
change copy(orig, copy, conf) to allow null for copy which means to create a new object
change copy(orig, conf) to not use ReflectionUtils but to pass in null to copy(orig, copy, conf)
acknowledge in javadoc that depending on the serialization class, copy may or may not copy orig into copy
return the result of deserialize since depending on serialization class, copy may or may not copy orig into copy
committed in 1299150
> copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize()
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-70
> URL: https://issues.apache.org/jira/browse/MRUNIT-70
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Meloss Xeloss
> Assignee: Jim Donofrio
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: MRUNIT-70-1.patch
>
>
> The copy(orig, conf) method requires objects to have a no-args constructor, which is non-ideal, since it passes on to Deserializer.deserialize, which can create a new object if passed in null. Additionally, the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not.
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/serializer/Deserializer.html#deserialize%28T%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-70) copy(orig, conf) in Serialization
shouldn't require objects to have a no-args constructor, and copy(orig,
copy, conf) seems to violate contract for deserializer.deserialize()
Posted by "Brock Noland (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-70?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218302#comment-13218302 ]
Brock Noland commented on MRUNIT-70:
------------------------------------
Writables must have a default constructor because they are created via reflection. WritableSerialization, the class, does the same thing we do via ReflectionUtils. Are you not using Writables?
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1/src/core/org/apache/hadoop/io/serializer/
> copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize()
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-70
> URL: https://issues.apache.org/jira/browse/MRUNIT-70
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Meloss Xeloss
> Priority: Minor
>
> The copy(orig, conf) method requires objects to have a no-args constructor, which is non-ideal, since it passes on to Deserializer.deserialize, which can create a new object if passed in null. Additionally, the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not.
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/serializer/Deserializer.html#deserialize%28T%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (MRUNIT-70) copy(orig, conf) in
Serialization shouldn't require objects to have a no-args constructor, and
copy(orig, copy, conf) seems to violate contract for
deserializer.deserialize()
Posted by "Jim Donofrio (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-70?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226088#comment-13226088 ]
Jim Donofrio edited comment on MRUNIT-70 at 3/9/12 2:19 PM:
------------------------------------------------------------
I uploaded a patch, is this what you had in mind Brock?
If so I will commit this later
was (Author: jdonofrio):
Is this what you had in mind Brock?
> copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize()
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-70
> URL: https://issues.apache.org/jira/browse/MRUNIT-70
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Meloss Xeloss
> Priority: Minor
> Attachments: MRUNIT-70-1.patch
>
>
> The copy(orig, conf) method requires objects to have a no-args constructor, which is non-ideal, since it passes on to Deserializer.deserialize, which can create a new object if passed in null. Additionally, the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not.
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/serializer/Deserializer.html#deserialize%28T%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-70) copy(orig, conf) in Serialization
shouldn't require objects to have a no-args constructor, and copy(orig,
copy, conf) seems to violate contract for deserializer.deserialize()
Posted by "Meloss Xeloss (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-70?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218308#comment-13218308 ]
Meloss Xeloss commented on MRUNIT-70:
-------------------------------------
Implemented a Serialization and passing it it via "io.serializations" in a Configuration, and as you correctly inferred, I'm not using a Writable.
> copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize()
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-70
> URL: https://issues.apache.org/jira/browse/MRUNIT-70
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Meloss Xeloss
> Priority: Minor
>
> The copy(orig, conf) method requires objects to have a no-args constructor, which is non-ideal, since it passes on to Deserializer.deserialize, which can create a new object if passed in null. Additionally, the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not.
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/serializer/Deserializer.html#deserialize%28T%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-70) copy(orig, conf) in Serialization
shouldn't require objects to have a no-args constructor, and copy(orig,
copy, conf) seems to violate contract for deserializer.deserialize()
Posted by "Goir Riog (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-70?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226111#comment-13226111 ]
Goir Riog commented on MRUNIT-70:
---------------------------------
I'm getting
12/03/09 15:39:57 INFO junit.TestRunner: java.lang.NullPointerException
at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
at org.apache.hadoop.mrunit.Serialization.copy(Serialization.java:42)
at org.apache.hadoop.mrunit.Serialization.copy(Serialization.java:61)
at org.apache.hadoop.mrunit.mapreduce.mock.MockContextWrapper$4.answer(MockContextWrapper.java:74)
at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:29)
at org.mockito.internal.MockHandler.handle(MockHandler.java:95)
at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:47)
at org.apache.hadoop.mapreduce.Reducer$Context$$EnhancerByMockitoWithCGLIB$$2bef43d9.write(<generated>)
since copy is still "null" on line 51 in Serialization.java ?!
But i have no idea how to fix it.
> copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize()
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-70
> URL: https://issues.apache.org/jira/browse/MRUNIT-70
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Meloss Xeloss
> Priority: Minor
> Attachments: MRUNIT-70-1.patch
>
>
> The copy(orig, conf) method requires objects to have a no-args constructor, which is non-ideal, since it passes on to Deserializer.deserialize, which can create a new object if passed in null. Additionally, the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not.
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/serializer/Deserializer.html#deserialize%28T%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (MRUNIT-70) copy(orig, conf) in
Serialization shouldn't require objects to have a no-args constructor, and
copy(orig, copy, conf) seems to violate contract for
deserializer.deserialize()
Posted by "Jim Donofrio (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-70?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226621#comment-13226621 ]
Jim Donofrio edited comment on MRUNIT-70 at 3/10/12 3:56 AM:
-------------------------------------------------------------
I am not familiar with Cassandra at all but you need to specify a different serialization since ByteBuffer does not implement Serializable. I think the patch is correct but I dont understand the other comment "the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not"
was (Author: jdonofrio):
You need to specify a different serialization since ByteBuffer does not implement Serializable
> copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize()
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-70
> URL: https://issues.apache.org/jira/browse/MRUNIT-70
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Meloss Xeloss
> Priority: Minor
> Attachments: MRUNIT-70-1.patch
>
>
> The copy(orig, conf) method requires objects to have a no-args constructor, which is non-ideal, since it passes on to Deserializer.deserialize, which can create a new object if passed in null. Additionally, the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not.
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/serializer/Deserializer.html#deserialize%28T%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-70) copy(orig, conf) in Serialization
shouldn't require objects to have a no-args constructor, and copy(orig,
copy, conf) seems to violate contract for deserializer.deserialize()
Posted by "Goir Riog (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-70?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226044#comment-13226044 ]
Goir Riog commented on MRUNIT-70:
---------------------------------
We're using Hadoop with Cassandra.
This requires to use ByteBuffer objects for the key as return values from the Reducer. The ByteBuffer doesnt have a empty Constructor so we get the following exception.
java.lang.RuntimeException: java.lang.NoSuchMethodException: java.nio.HeapByteBuffer.<init>()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
at org.apache.hadoop.mrunit.mock.MockOutputCollector.getInstance(MockOutputCollector.java:62)
at org.apache.hadoop.mrunit.mock.MockOutputCollector.deepCopy(MockOutputCollector.java:73)
at org.apache.hadoop.mrunit.mock.MockOutputCollector.collect(MockOutputCollector.java:110)
at org.apache.hadoop.mrunit.mapreduce.mock.MockReduceContextWrapper$MockReduceContext.write(MockReduceContextWrapper.java:204)
[...]
is there any patch yet ?
> copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize()
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-70
> URL: https://issues.apache.org/jira/browse/MRUNIT-70
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Meloss Xeloss
> Priority: Minor
>
> The copy(orig, conf) method requires objects to have a no-args constructor, which is non-ideal, since it passes on to Deserializer.deserialize, which can create a new object if passed in null. Additionally, the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not.
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/serializer/Deserializer.html#deserialize%28T%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MRUNIT-70) copy(orig, conf) in Serialization
shouldn't require objects to have a no-args constructor, and copy(orig,
copy, conf) seems to violate contract for deserializer.deserialize()
Posted by "Meloss Xeloss (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-70?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Meloss Xeloss updated MRUNIT-70:
--------------------------------
Description:
The copy(orig, conf) method requires objects to have a no-args constructor, which is non-ideal, since it passes on to Deserializer.deserialize, which can create a new object if passed in null. Additionally, the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not.
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/serializer/Deserializer.html#deserialize%28T%29
Summary: copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize() (was: copy in Serialization shouldn't require objects to have a no-args constructor, and )
> copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize()
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-70
> URL: https://issues.apache.org/jira/browse/MRUNIT-70
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Meloss Xeloss
> Priority: Minor
>
> The copy(orig, conf) method requires objects to have a no-args constructor, which is non-ideal, since it passes on to Deserializer.deserialize, which can create a new object if passed in null. Additionally, the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not.
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/serializer/Deserializer.html#deserialize%28T%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-70) copy(orig, conf) in Serialization
shouldn't require objects to have a no-args constructor, and copy(orig,
copy, conf) seems to violate contract for deserializer.deserialize()
Posted by "Brock Noland (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-70?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226774#comment-13226774 ]
Brock Noland commented on MRUNIT-70:
------------------------------------
LGTM! Thank you Jim!
Also, yes I agree, I think to serialize a BB you'd need to set io.serializations.
> copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize()
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-70
> URL: https://issues.apache.org/jira/browse/MRUNIT-70
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Meloss Xeloss
> Assignee: Jim Donofrio
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: MRUNIT-70-1.patch
>
>
> The copy(orig, conf) method requires objects to have a no-args constructor, which is non-ideal, since it passes on to Deserializer.deserialize, which can create a new object if passed in null. Additionally, the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not.
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/serializer/Deserializer.html#deserialize%28T%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-70) copy(orig, conf) in Serialization
shouldn't require objects to have a no-args constructor, and copy(orig,
copy, conf) seems to violate contract for deserializer.deserialize()
Posted by "Jim Donofrio (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-70?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226756#comment-13226756 ]
Jim Donofrio commented on MRUNIT-70:
------------------------------------
Ok I get it now, the contract for deserialize states that "If the object t is non-null then this deserializer may set its internal state to the next object read from the input stream. " So the deserialize method may still return a new object instead of changing the state if the object passed in. WritableSerialization changes the state of the non null object passed in. JavaSerialization ignores the object passed in and always return a new object.
public T deserialize(T object) throws IOException {
try {
// ignore passed-in object
return (T) ois.readObject();
} catch (ClassNotFoundException e) {
throw new IOException(e.toString());
}
}
> copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize()
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-70
> URL: https://issues.apache.org/jira/browse/MRUNIT-70
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Meloss Xeloss
> Priority: Minor
> Attachments: MRUNIT-70-1.patch
>
>
> The copy(orig, conf) method requires objects to have a no-args constructor, which is non-ideal, since it passes on to Deserializer.deserialize, which can create a new object if passed in null. Additionally, the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not.
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/serializer/Deserializer.html#deserialize%28T%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-70) copy(orig, conf) in Serialization
shouldn't require objects to have a no-args constructor, and copy(orig,
copy, conf) seems to violate contract for deserializer.deserialize()
Posted by "Brock Noland (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-70?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218314#comment-13218314 ]
Brock Noland commented on MRUNIT-70:
------------------------------------
OK, looks like a fairly easy fix. We basically shouldn't be calling ReflectionUtils and should return whatever deserialize returns. If anyone creates a patch, please post it. Otherwise I should have time tomorrow.
> copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize()
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-70
> URL: https://issues.apache.org/jira/browse/MRUNIT-70
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Meloss Xeloss
> Priority: Minor
>
> The copy(orig, conf) method requires objects to have a no-args constructor, which is non-ideal, since it passes on to Deserializer.deserialize, which can create a new object if passed in null. Additionally, the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not.
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/serializer/Deserializer.html#deserialize%28T%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-70) copy(orig, conf) in Serialization
shouldn't require objects to have a no-args constructor, and copy(orig,
copy, conf) seems to violate contract for deserializer.deserialize()
Posted by "Jim Donofrio (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-70?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226621#comment-13226621 ]
Jim Donofrio commented on MRUNIT-70:
------------------------------------
You need to specify a different serialization since ByteBuffer does not implement Serializable
> copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize()
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-70
> URL: https://issues.apache.org/jira/browse/MRUNIT-70
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Meloss Xeloss
> Priority: Minor
> Attachments: MRUNIT-70-1.patch
>
>
> The copy(orig, conf) method requires objects to have a no-args constructor, which is non-ideal, since it passes on to Deserializer.deserialize, which can create a new object if passed in null. Additionally, the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not.
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/serializer/Deserializer.html#deserialize%28T%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-70) copy(orig, conf) in Serialization
shouldn't require objects to have a no-args constructor, and copy(orig,
copy, conf) seems to violate contract for deserializer.deserialize()
Posted by "Brock Noland (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-70?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218305#comment-13218305 ]
Brock Noland commented on MRUNIT-70:
------------------------------------
Yes I do think this is a bug. I am guessing you are using JavaSerialization?
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1/src/core/org/apache/hadoop/io/serializer/JavaSerialization.java
> copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize()
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-70
> URL: https://issues.apache.org/jira/browse/MRUNIT-70
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Meloss Xeloss
> Priority: Minor
>
> The copy(orig, conf) method requires objects to have a no-args constructor, which is non-ideal, since it passes on to Deserializer.deserialize, which can create a new object if passed in null. Additionally, the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not.
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/serializer/Deserializer.html#deserialize%28T%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-70) copy(orig, conf) in Serialization
shouldn't require objects to have a no-args constructor, and copy(orig,
copy, conf) seems to violate contract for deserializer.deserialize()
Posted by "Goir Riog (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-70?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226127#comment-13226127 ]
Goir Riog commented on MRUNIT-70:
---------------------------------
I've changes you test to use a ByteBuffer. Which throws the NullPointerException.
public void testClassWithoutNoArgConstructor() {
Configuration conf = new Configuration();
conf.setStrings("io.serializations", "org.apache.hadoop.io.serializer.JavaSerialization");
ByteBuffer bb1 = ByteBuffer.allocate(1);
bb1.put((byte)0xFF);
ByteBuffer bb2 = ByteBuffer.allocate(1);
bb2.put((byte)0xFF);
assertEquals(bb1, Serialization.copy(bb2, conf));
}
> copy(orig, conf) in Serialization shouldn't require objects to have a no-args constructor, and copy(orig, copy, conf) seems to violate contract for deserializer.deserialize()
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-70
> URL: https://issues.apache.org/jira/browse/MRUNIT-70
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Meloss Xeloss
> Priority: Minor
> Attachments: MRUNIT-70-1.patch
>
>
> The copy(orig, conf) method requires objects to have a no-args constructor, which is non-ideal, since it passes on to Deserializer.deserialize, which can create a new object if passed in null. Additionally, the contract for deserialize only _can_ fill in the object passed in, so the copy method will not perform as expected if it does not.
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/serializer/Deserializer.html#deserialize%28T%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira