You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2006/09/26 00:23:50 UTC
[jira] Created: (HADOOP-558) Adding versioning to Writable classes
Adding versioning to Writable classes
-------------------------------------
Key: HADOOP-558
URL: http://issues.apache.org/jira/browse/HADOOP-558
Project: Hadoop
Issue Type: Improvement
Components: io
Affects Versions: 0.6.2
Reporter: Owen O'Malley
Assigned To: Owen O'Malley
We currently don't have a way to support versioned Writables, which among other problems means that it is very difficult to change the serialization of any types. (A recent example is that we can't change any of the Writables that currently use UTF8.writeString to use Text.writeString with out breaking everything.) Just changing the file version doesn't work because you can't read the old versions.
Therefore, I propose adding a new interface:
public interface VersionMapWritable extends Writable {
/**
* Destructively read into this object from in based on the version map.
* @param versionMap a map from each class to its version in the DataInput (version 0 classes are omitted)
*/
public void readFields(DataInput in, Map<Class<VersionMapWritable>, int> versionMap) throws IOException;
/**
* Classes with non-zero versions should register themselves in static blocks.
*/
public static void registerCurrentVersion(Class<VersionMapWritable> class, int version) {...}
/**
* If a version map includes the parent type, always include the child type as well.
*/
public static void addDependence(Class<VersionMapWritable> parent, Class<VersionMapWritable> child) { ... }
/**
* Build a version map for a given list of classes, including any dependent types.
*/
public static Map<Class<VersionMapWritable>, int>
buildVersionMap(Set<Class<VersionMapWritable>> classes) {...}
/**
* Add the types in the parameter/result types to the list of classes.
* Recursively adds the field types for any new types that are added to the set.
*/
public static void addMethodTypes(Set<Class<VersionMapWritable>> result,
Class<VersionedProtocol> protocol) {...}
/**
* Add the non-transient fields to the list of classes.
*/
public static void addFieldTypes(Set<Class<VersionMapWritable>> result, Class<Writable> writable) {...}
public static Map<Class<VersionMapWritable>, int> readVersionMap(DataInput in) throws IOException { ... }
public static void writeVersionMap(DataOutput out,
Map<Class<VersionMapWritable> versionMap) throws IOException {...}
}
VersionedWritable, which stored a version byte within each object, will be depriciated.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-558) Adding versioning to Writable
classes
Posted by "David Bowen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486690 ]
David Bowen commented on HADOOP-558:
------------------------------------
The suggested API is putting static methods on an interface, which isn't possible. We need a class to put them in, so how about:
class VersionMap extends HashMap<Class<VersionMapWritable>,Integer> {
...
}
In addDependence, are the terms "parent" and "child" are referring to the inheritance relationship or the "contains" relationship - between a class and the classes of its fields?
I don't understand addMethodTypes and addFieldTypes. Please can you explain them more fully?
> Adding versioning to Writable classes
> -------------------------------------
>
> Key: HADOOP-558
> URL: https://issues.apache.org/jira/browse/HADOOP-558
> Project: Hadoop
> Issue Type: Improvement
> Components: io
> Affects Versions: 0.6.2
> Reporter: Owen O'Malley
> Assigned To: David Bowen
>
> We currently don't have a way to support versioned Writables, which among other problems means that it is very difficult to change the serialization of any types. (A recent example is that we can't change any of the Writables that currently use UTF8.writeString to use Text.writeString with out breaking everything.) Just changing the file version doesn't work because you can't read the old versions.
> Therefore, I propose adding a new interface:
> public interface VersionMapWritable extends Writable {
> /**
> * Destructively read into this object from in based on the version map.
> * @param versionMap a map from each class to its version in the DataInput (version 0 classes are omitted)
> */
> public void readFields(DataInput in, Map<Class<VersionMapWritable>, int> versionMap) throws IOException;
> /**
> * Classes with non-zero versions should register themselves in static blocks.
> */
> public static void registerCurrentVersion(Class<VersionMapWritable> class, int version) {...}
> /**
> * If a version map includes the parent type, always include the child type as well.
> */
> public static void addDependence(Class<VersionMapWritable> parent, Class<VersionMapWritable> child) { ... }
> /**
> * Build a version map for a given list of classes, including any dependent types.
> */
> public static Map<Class<VersionMapWritable>, int>
> buildVersionMap(Set<Class<VersionMapWritable>> classes) {...}
> /**
> * Add the types in the parameter/result types to the list of classes.
> * Recursively adds the field types for any new types that are added to the set.
> */
> public static void addMethodTypes(Set<Class<VersionMapWritable>> result,
> Class<VersionedProtocol> protocol) {...}
> /**
> * Add the non-transient fields to the list of classes.
> */
> public static void addFieldTypes(Set<Class<VersionMapWritable>> result, Class<Writable> writable) {...}
> public static Map<Class<VersionMapWritable>, int> readVersionMap(DataInput in) throws IOException { ... }
> public static void writeVersionMap(DataOutput out,
> Map<Class<VersionMapWritable> versionMap) throws IOException {...}
> }
> VersionedWritable, which stored a version byte within each object, will be depriciated.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-558) Adding versioning to Writable
classes
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-558?page=comments#action_12437681 ]
Doug Cutting commented on HADOOP-558:
-------------------------------------
Part of this, I assume, will include:
1.In SequenceFile, adding to the header the version map for key and value classes (& all classes referenced by their non-transient fields), and passing this map to keys and values as they're read.
2. In RPC, Transmitting the version map for all parameter & return classes (& those refernced by their fields) from the client to the server as a handshake on connect.
> Adding versioning to Writable classes
> -------------------------------------
>
> Key: HADOOP-558
> URL: http://issues.apache.org/jira/browse/HADOOP-558
> Project: Hadoop
> Issue Type: Improvement
> Components: io
> Affects Versions: 0.6.2
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
>
> We currently don't have a way to support versioned Writables, which among other problems means that it is very difficult to change the serialization of any types. (A recent example is that we can't change any of the Writables that currently use UTF8.writeString to use Text.writeString with out breaking everything.) Just changing the file version doesn't work because you can't read the old versions.
> Therefore, I propose adding a new interface:
> public interface VersionMapWritable extends Writable {
> /**
> * Destructively read into this object from in based on the version map.
> * @param versionMap a map from each class to its version in the DataInput (version 0 classes are omitted)
> */
> public void readFields(DataInput in, Map<Class<VersionMapWritable>, int> versionMap) throws IOException;
> /**
> * Classes with non-zero versions should register themselves in static blocks.
> */
> public static void registerCurrentVersion(Class<VersionMapWritable> class, int version) {...}
> /**
> * If a version map includes the parent type, always include the child type as well.
> */
> public static void addDependence(Class<VersionMapWritable> parent, Class<VersionMapWritable> child) { ... }
> /**
> * Build a version map for a given list of classes, including any dependent types.
> */
> public static Map<Class<VersionMapWritable>, int>
> buildVersionMap(Set<Class<VersionMapWritable>> classes) {...}
> /**
> * Add the types in the parameter/result types to the list of classes.
> * Recursively adds the field types for any new types that are added to the set.
> */
> public static void addMethodTypes(Set<Class<VersionMapWritable>> result,
> Class<VersionedProtocol> protocol) {...}
> /**
> * Add the non-transient fields to the list of classes.
> */
> public static void addFieldTypes(Set<Class<VersionMapWritable>> result, Class<Writable> writable) {...}
> public static Map<Class<VersionMapWritable>, int> readVersionMap(DataInput in) throws IOException { ... }
> public static void writeVersionMap(DataOutput out,
> Map<Class<VersionMapWritable> versionMap) throws IOException {...}
> }
> VersionedWritable, which stored a version byte within each object, will be depriciated.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-558) Adding versioning to Writable
classes
Posted by "David Bowen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486705 ]
David Bowen commented on HADOOP-558:
------------------------------------
A couple more points:
(1) VersionMapWritable interface extends Writable, so implementing classes need to implement readFields without the VersionMap argument. A possible way to handle this is with a ThreadLocal. E.g. have a static method VersionMap.setThreadLocal(VersionMap v) which attaches a VersionMap to the current thread for use in subsequent deserialization of VersionMapWritables.
(2) The use of DataInput and DataOutput implies that the external format of VersionMaps is binary. It might be preferable to have it be human-readable. E.g. they could be Properties files with lines of the form <class name>=<int>.
> Adding versioning to Writable classes
> -------------------------------------
>
> Key: HADOOP-558
> URL: https://issues.apache.org/jira/browse/HADOOP-558
> Project: Hadoop
> Issue Type: Improvement
> Components: io
> Affects Versions: 0.6.2
> Reporter: Owen O'Malley
> Assigned To: David Bowen
>
> We currently don't have a way to support versioned Writables, which among other problems means that it is very difficult to change the serialization of any types. (A recent example is that we can't change any of the Writables that currently use UTF8.writeString to use Text.writeString with out breaking everything.) Just changing the file version doesn't work because you can't read the old versions.
> Therefore, I propose adding a new interface:
> public interface VersionMapWritable extends Writable {
> /**
> * Destructively read into this object from in based on the version map.
> * @param versionMap a map from each class to its version in the DataInput (version 0 classes are omitted)
> */
> public void readFields(DataInput in, Map<Class<VersionMapWritable>, int> versionMap) throws IOException;
> /**
> * Classes with non-zero versions should register themselves in static blocks.
> */
> public static void registerCurrentVersion(Class<VersionMapWritable> class, int version) {...}
> /**
> * If a version map includes the parent type, always include the child type as well.
> */
> public static void addDependence(Class<VersionMapWritable> parent, Class<VersionMapWritable> child) { ... }
> /**
> * Build a version map for a given list of classes, including any dependent types.
> */
> public static Map<Class<VersionMapWritable>, int>
> buildVersionMap(Set<Class<VersionMapWritable>> classes) {...}
> /**
> * Add the types in the parameter/result types to the list of classes.
> * Recursively adds the field types for any new types that are added to the set.
> */
> public static void addMethodTypes(Set<Class<VersionMapWritable>> result,
> Class<VersionedProtocol> protocol) {...}
> /**
> * Add the non-transient fields to the list of classes.
> */
> public static void addFieldTypes(Set<Class<VersionMapWritable>> result, Class<Writable> writable) {...}
> public static Map<Class<VersionMapWritable>, int> readVersionMap(DataInput in) throws IOException { ... }
> public static void writeVersionMap(DataOutput out,
> Map<Class<VersionMapWritable> versionMap) throws IOException {...}
> }
> VersionedWritable, which stored a version byte within each object, will be depriciated.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-558) Adding versioning to Writable classes
Posted by "David Bowen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Bowen reassigned HADOOP-558:
----------------------------------
Assignee: Owen O'Malley (was: David Bowen)
I'm told that we don't want to do this any more.
> Adding versioning to Writable classes
> -------------------------------------
>
> Key: HADOOP-558
> URL: https://issues.apache.org/jira/browse/HADOOP-558
> Project: Hadoop
> Issue Type: Improvement
> Components: io
> Affects Versions: 0.6.2
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
>
> We currently don't have a way to support versioned Writables, which among other problems means that it is very difficult to change the serialization of any types. (A recent example is that we can't change any of the Writables that currently use UTF8.writeString to use Text.writeString with out breaking everything.) Just changing the file version doesn't work because you can't read the old versions.
> Therefore, I propose adding a new interface:
> public interface VersionMapWritable extends Writable {
> /**
> * Destructively read into this object from in based on the version map.
> * @param versionMap a map from each class to its version in the DataInput (version 0 classes are omitted)
> */
> public void readFields(DataInput in, Map<Class<VersionMapWritable>, int> versionMap) throws IOException;
> /**
> * Classes with non-zero versions should register themselves in static blocks.
> */
> public static void registerCurrentVersion(Class<VersionMapWritable> class, int version) {...}
> /**
> * If a version map includes the parent type, always include the child type as well.
> */
> public static void addDependence(Class<VersionMapWritable> parent, Class<VersionMapWritable> child) { ... }
> /**
> * Build a version map for a given list of classes, including any dependent types.
> */
> public static Map<Class<VersionMapWritable>, int>
> buildVersionMap(Set<Class<VersionMapWritable>> classes) {...}
> /**
> * Add the types in the parameter/result types to the list of classes.
> * Recursively adds the field types for any new types that are added to the set.
> */
> public static void addMethodTypes(Set<Class<VersionMapWritable>> result,
> Class<VersionedProtocol> protocol) {...}
> /**
> * Add the non-transient fields to the list of classes.
> */
> public static void addFieldTypes(Set<Class<VersionMapWritable>> result, Class<Writable> writable) {...}
> public static Map<Class<VersionMapWritable>, int> readVersionMap(DataInput in) throws IOException { ... }
> public static void writeVersionMap(DataOutput out,
> Map<Class<VersionMapWritable> versionMap) throws IOException {...}
> }
> VersionedWritable, which stored a version byte within each object, will be depriciated.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-558) Adding versioning to Writable classes
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley reassigned HADOOP-558:
------------------------------------
Assignee: David Bowen (was: Owen O'Malley)
> Adding versioning to Writable classes
> -------------------------------------
>
> Key: HADOOP-558
> URL: https://issues.apache.org/jira/browse/HADOOP-558
> Project: Hadoop
> Issue Type: Improvement
> Components: io
> Affects Versions: 0.6.2
> Reporter: Owen O'Malley
> Assigned To: David Bowen
>
> We currently don't have a way to support versioned Writables, which among other problems means that it is very difficult to change the serialization of any types. (A recent example is that we can't change any of the Writables that currently use UTF8.writeString to use Text.writeString with out breaking everything.) Just changing the file version doesn't work because you can't read the old versions.
> Therefore, I propose adding a new interface:
> public interface VersionMapWritable extends Writable {
> /**
> * Destructively read into this object from in based on the version map.
> * @param versionMap a map from each class to its version in the DataInput (version 0 classes are omitted)
> */
> public void readFields(DataInput in, Map<Class<VersionMapWritable>, int> versionMap) throws IOException;
> /**
> * Classes with non-zero versions should register themselves in static blocks.
> */
> public static void registerCurrentVersion(Class<VersionMapWritable> class, int version) {...}
> /**
> * If a version map includes the parent type, always include the child type as well.
> */
> public static void addDependence(Class<VersionMapWritable> parent, Class<VersionMapWritable> child) { ... }
> /**
> * Build a version map for a given list of classes, including any dependent types.
> */
> public static Map<Class<VersionMapWritable>, int>
> buildVersionMap(Set<Class<VersionMapWritable>> classes) {...}
> /**
> * Add the types in the parameter/result types to the list of classes.
> * Recursively adds the field types for any new types that are added to the set.
> */
> public static void addMethodTypes(Set<Class<VersionMapWritable>> result,
> Class<VersionedProtocol> protocol) {...}
> /**
> * Add the non-transient fields to the list of classes.
> */
> public static void addFieldTypes(Set<Class<VersionMapWritable>> result, Class<Writable> writable) {...}
> public static Map<Class<VersionMapWritable>, int> readVersionMap(DataInput in) throws IOException { ... }
> public static void writeVersionMap(DataOutput out,
> Map<Class<VersionMapWritable> versionMap) throws IOException {...}
> }
> VersionedWritable, which stored a version byte within each object, will be depriciated.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.