[jira] [Comment Edited] (GORA-401) Serialization and deserialization of Persistent does not hold the entity dirty state from Map to Reduce

["Kevin Ratnasekera (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/GORA-401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15501312#comment-15501312 ] 

Kevin Ratnasekera edited comment on GORA-401 at 9/18/16 5:17 PM:
-----------------------------------------------------------------

[~alfonso.nishikawa] I don't have any background knowledge as you do regarding this issue and related issues in Apache Nutch. I had a look on the code, it seems like PersistentSerializer and PersistentDeserializer can be modified to preserve the dirty bytes. Since we are registering those Serializer/Deserializer for persistent data beans at Hadoop conf, all the serialization/de-serialization will be delegated to these classes. 

I have added simple check for dirty bytes comparison in TestIOUtils#testSerializeDeserialize, so that we can make whether proposed fix preserve the dirty bytes, after data bean is serialized and de-serializated, earlier checks only guarantees equality of fields between two persistent data beans.
I submitted a pull request on this, please do review and if you can provide some more directions, I can have detailed look into the issue. 

Regards
Kevin



was (Author: djkevincr):
[~alfonso.nishikawa] I don't have any background knowledge as you do regarding this issue and related issues in Apache Nutch. I had a look on the code, it seems like PersistentSerializer and PersistentDeserializer can be modified to preserve the dirty bytes. Since we are registering those Serializer/Deserializer for persistent data beans at Hadoop conf, all the serialization/de-serialization will be delegated to these classes. 

I have added simple check for dirty bytes comparison in TestIOUtils#testSerializeDeserialize, so that we can make whether proposed fix preserve the dirty bytes, after data bean is serialized and de-serializated, earlier checks only guarantees equality of fields between two persistent data beans.
Please do review and if you can provide some more directions, I can have detailed look into the issue.

Regards
Kevin


> Serialization and deserialization of Persistent does not hold the entity dirty state from Map to Reduce
> -------------------------------------------------------------------------------------------------------
>
>                 Key: GORA-401
>                 URL: https://issues.apache.org/jira/browse/GORA-401
>             Project: Apache Gora
>          Issue Type: Bug
>          Components: gora-core
>    Affects Versions: 0.4, 0.5
>         Environment: Tested on gora-0.4, but seems logically to hold on gora-0.5. HBase backend.
>            Reporter: Alfonso Nishikawa
>            Assignee: Alfonso Nishikawa
>            Priority: Critical
>              Labels: serialization
>             Fix For: 0.8
>
>         Attachments: GORA-401-tests.patch, GORA-401v1.patch, GORA-401v2.patch, GORA-401v3.patch, GORA-401v4.patch, GORA-401v5.patch
>
>   Original Estimate: 35h
>          Time Spent: 21h
>  Remaining Estimate: 14h
>
> After removing __g__dirty field in GORA-326, dirty field is not serialized. In GORA-321 {{[PersistentSerializer|https://github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/mapreduce/PersistentSerializer.java]}} went from using {{[PersistentDatumWriter|https://github.com/apache/gora/blob/apache-gora-0.3/gora-core/src/main/java/org/apache/gora/avro/PersistentDatumWriter.java](/Reader)}} to Avro's {{SpecificDatumWriter}}, delegating the serialization of the dirty field to Avro (but really not desirable to have that field as a main field in the entities).
> The proposal is to reintroduce the {{PersistentDatumWriter/Reader}} which will serialize the internal fields of the entities.
> This bug affects, for example, Nutch, which loads only some fields in it's phases, serializes entities (from Map to Reduce), and when deserializes finds all fields as "dirty", independently of what fields were modified in the Map, and overwrite all data in datastore (deleting much things: downloaded content, parsed content, etc).
> This effect can be seen in {{TestPersistentSerialization#testSerderEmployeeTwoFields}}, when debuging in {{TestIOUtils#testSerializeDeserialize}}. Proper breakpoints an inspections shows that, entities are "equal" when it's fields are equal. This is fine as "equal" definition, but another test must be added to check that serialization an deserialization keeps the dirty state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)