You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/01/16 20:09:25 UTC
[GitHub] [hudi] satishkotha commented on a change in pull request #2453: [HUDI-1533] Make SerializableSchema work for large schemas
satishkotha commented on a change in pull request #2453:
URL: https://github.com/apache/hudi/pull/2453#discussion_r559030870
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/config/SerializableSchema.java
##########
@@ -62,12 +62,17 @@ private void readObject(ObjectInputStream in) throws IOException {
// create a public write method for unit test
public void writeObjectTo(ObjectOutputStream out) throws IOException {
- out.writeUTF(schema.toString());
+ // Note: writeUTF cannot support string length > 64K. So use writeObject which has small overhead (relatively).
+ out.writeObject(schema.toString());
}
// create a public read method for unit test
public void readObjectFrom(ObjectInputStream in) throws IOException {
- schema = new Schema.Parser().parse(in.readUTF());
+ try {
+ schema = new Schema.Parser().parse(in.readObject().toString());
Review comment:
This was initial implementation. Unfortunately, it doesnt well with java serialization because we have 'out.defaultWriteObject();' in front. Per Vinoth's suggestion I used writeUTF, see https://issues.apache.org/jira/browse/HUDI-1443 for some context.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/config/SerializableSchema.java
##########
@@ -62,12 +62,17 @@ private void readObject(ObjectInputStream in) throws IOException {
// create a public write method for unit test
public void writeObjectTo(ObjectOutputStream out) throws IOException {
- out.writeUTF(schema.toString());
+ // Note: writeUTF cannot support string length > 64K. So use writeObject which has small overhead (relatively).
+ out.writeObject(schema.toString());
Review comment:
This was initial implementation. Unfortunately, it doesnt well with java serialization because we have 'out.defaultWriteObject();' in front. Per Vinoth's suggestion I used writeUTF, see https://issues.apache.org/jira/browse/HUDI-1443 for some context.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org