You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Alex Holmes (JIRA)" <ji...@apache.org> on 2011/09/20 12:43:08 UTC
[jira] [Created] (AVRO-891) Exception when attempting to read
modified schema containing deleted field
Exception when attempting to read modified schema containing deleted field
--------------------------------------------------------------------------
Key: AVRO-891
URL: https://issues.apache.org/jira/browse/AVRO-891
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.5.4
Environment: OSX 10.7
Reporter: Alex Holmes
An AvroRuntimeException exception is thrown when attempting to read an Avro file serialized with an older version of a schema containing a field which has been subsequently removed in the newer schema.
*Exception*
{code}
Exception in thread "main" org.apache.avro.AvroRuntimeException: Bad index
at Record.put(Unknown Source)
at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220)
at Read.readFromAvro(Unknown Source)
at Read.main(Unknown Source)
{code}
*Steps to reproduce*
# Generate code for schema v1 and v2
# Write an Avro file with the v1 code-generated Record class using the DataFileWriter and SpecificDatumWriter
# (informational only) Read the Avro file using the v1 code-generated Record class using DataFileStream and SpecificDatumReader (output follows)
{code}
Record@2ec791b9[name=r1,id=1]
Record@bd86fd3[name=r2,id=2]
{code}
# Read the Avro file using the v2 code-generated Record class using DataFileStream and SpecificDatumReader
*Schema details*
v1 schema:
{code}
{"name": "Record", "type": "record",
"fields": [
{"name": "name", "type": "string"},
{"name": "id", "type": "int"}
]
}
{code}
v2 schema:
{code}
{"name": "Record", "type": "record",
"fields": [
{"name": "name", "type": "string"}
]
}
{code}
*Write code*
{code}
public static Record createRecord(String name, int id) {
Record record = new Record();
record.name = name;
record.id = id;
return record;
}
public static void writeToAvro(OutputStream outputStream)
throws IOException {
DataFileWriter<Record> writer =
new DataFileWriter<Record>(new SpecificDatumWriter<Record>());
writer.create(Record.SCHEMA$, outputStream);
writer.append(createRecord("r1", 1));
writer.append(createRecord("r2", 2));
writer.close();
outputStream.close();
}
{code}
*Read code*
{code}
public static void readFromAvro(InputStream is) throws IOException {
DataFileStream<Record> reader = new DataFileStream<Record>(
is, new SpecificDatumReader<Record>());
for (Record a : reader) {
System.out.println(ToStringBuilder.reflectionToString(a));
}
IOUtils.cleanup(null, is);
IOUtils.cleanup(null, reader);
}
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-891) Change SpecificDatumReader to default
reader schema from loaded class
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-891:
------------------------------
Summary: Change SpecificDatumReader to default reader schema from loaded class (was: Change SpecificDatum)
> Change SpecificDatumReader to default reader schema from loaded class
> ---------------------------------------------------------------------
>
> Key: AVRO-891
> URL: https://issues.apache.org/jira/browse/AVRO-891
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.5.4
> Environment: OSX 10.7
> Reporter: Alex Holmes
> Assignee: Doug Cutting
> Fix For: 1.6.0
>
> Attachments: AVRO-891.patch, AVRO-891.patch
>
>
> An AvroRuntimeException exception is thrown when attempting to read an Avro file serialized with an older version of a schema containing a field which has been subsequently removed in the newer schema.
> *Exception*
> {code}
> Exception in thread "main" org.apache.avro.AvroRuntimeException: Bad index
> at Record.put(Unknown Source)
> at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
> at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220)
> at Read.readFromAvro(Unknown Source)
> at Read.main(Unknown Source)
> {code}
> *Steps to reproduce*
> # Generate code for schema v1 and v2
> # Write an Avro file with the v1 code-generated Record class using the DataFileWriter and SpecificDatumWriter
> # (informational only) Read the Avro file using the v1 code-generated Record class using DataFileStream and SpecificDatumReader (output follows)
> {code}
> Record@2ec791b9[name=r1,id=1]
> Record@bd86fd3[name=r2,id=2]
> {code}
> # Read the Avro file using the v2 code-generated Record class using DataFileStream and SpecificDatumReader
> *Schema details*
> v1 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"},
> {"name": "id", "type": "int"}
> ]
> }
> {code}
> v2 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"}
> ]
> }
> {code}
> *Write code*
> {code}
> public static Record createRecord(String name, int id) {
> Record record = new Record();
> record.name = name;
> record.id = id;
> return record;
> }
> public static void writeToAvro(OutputStream outputStream)
> throws IOException {
> DataFileWriter<Record> writer =
> new DataFileWriter<Record>(new SpecificDatumWriter<Record>());
> writer.create(Record.SCHEMA$, outputStream);
> writer.append(createRecord("r1", 1));
> writer.append(createRecord("r2", 2));
> writer.close();
> outputStream.close();
> }
> {code}
> *Read code*
> {code}
> public static void readFromAvro(InputStream is) throws IOException {
> DataFileStream<Record> reader = new DataFileStream<Record>(
> is, new SpecificDatumReader<Record>());
> for (Record a : reader) {
> System.out.println(ToStringBuilder.reflectionToString(a));
> }
> IOUtils.cleanup(null, is);
> IOUtils.cleanup(null, reader);
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-891) Exception when attempting to read
modified schema containing deleted field
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108960#comment-13108960 ]
Doug Cutting commented on AVRO-891:
-----------------------------------
Here are two ways we might avoid this:
# Make the default constructor for SpecificDatumReader protected instead of public. That would force folks to always provide a schema and would be incompatible.
# Change SpecificDatumReader#setSchema() to, when the expected schema is null, set it to getSpecificData().getClass(actual).getSchema(). That would get the schema with the same name as the one in the file from the currently loaded class.
I prefer the second option.
> Exception when attempting to read modified schema containing deleted field
> --------------------------------------------------------------------------
>
> Key: AVRO-891
> URL: https://issues.apache.org/jira/browse/AVRO-891
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.5.4
> Environment: OSX 10.7
> Reporter: Alex Holmes
>
> An AvroRuntimeException exception is thrown when attempting to read an Avro file serialized with an older version of a schema containing a field which has been subsequently removed in the newer schema.
> *Exception*
> {code}
> Exception in thread "main" org.apache.avro.AvroRuntimeException: Bad index
> at Record.put(Unknown Source)
> at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
> at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220)
> at Read.readFromAvro(Unknown Source)
> at Read.main(Unknown Source)
> {code}
> *Steps to reproduce*
> # Generate code for schema v1 and v2
> # Write an Avro file with the v1 code-generated Record class using the DataFileWriter and SpecificDatumWriter
> # (informational only) Read the Avro file using the v1 code-generated Record class using DataFileStream and SpecificDatumReader (output follows)
> {code}
> Record@2ec791b9[name=r1,id=1]
> Record@bd86fd3[name=r2,id=2]
> {code}
> # Read the Avro file using the v2 code-generated Record class using DataFileStream and SpecificDatumReader
> *Schema details*
> v1 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"},
> {"name": "id", "type": "int"}
> ]
> }
> {code}
> v2 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"}
> ]
> }
> {code}
> *Write code*
> {code}
> public static Record createRecord(String name, int id) {
> Record record = new Record();
> record.name = name;
> record.id = id;
> return record;
> }
> public static void writeToAvro(OutputStream outputStream)
> throws IOException {
> DataFileWriter<Record> writer =
> new DataFileWriter<Record>(new SpecificDatumWriter<Record>());
> writer.create(Record.SCHEMA$, outputStream);
> writer.append(createRecord("r1", 1));
> writer.append(createRecord("r2", 2));
> writer.close();
> outputStream.close();
> }
> {code}
> *Read code*
> {code}
> public static void readFromAvro(InputStream is) throws IOException {
> DataFileStream<Record> reader = new DataFileStream<Record>(
> is, new SpecificDatumReader<Record>());
> for (Record a : reader) {
> System.out.println(ToStringBuilder.reflectionToString(a));
> }
> IOUtils.cleanup(null, is);
> IOUtils.cleanup(null, reader);
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-891) Exception when attempting to read
modified schema containing deleted field
Posted by "Scott Carey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108925#comment-13108925 ]
Scott Carey commented on AVRO-891:
----------------------------------
I should have caught that before.
We may need better Javadoc and some example uses in the doc, but this definitely works for me and has worked for a long time --- but I use the constructor above.
> Exception when attempting to read modified schema containing deleted field
> --------------------------------------------------------------------------
>
> Key: AVRO-891
> URL: https://issues.apache.org/jira/browse/AVRO-891
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.5.4
> Environment: OSX 10.7
> Reporter: Alex Holmes
>
> An AvroRuntimeException exception is thrown when attempting to read an Avro file serialized with an older version of a schema containing a field which has been subsequently removed in the newer schema.
> *Exception*
> {code}
> Exception in thread "main" org.apache.avro.AvroRuntimeException: Bad index
> at Record.put(Unknown Source)
> at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
> at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220)
> at Read.readFromAvro(Unknown Source)
> at Read.main(Unknown Source)
> {code}
> *Steps to reproduce*
> # Generate code for schema v1 and v2
> # Write an Avro file with the v1 code-generated Record class using the DataFileWriter and SpecificDatumWriter
> # (informational only) Read the Avro file using the v1 code-generated Record class using DataFileStream and SpecificDatumReader (output follows)
> {code}
> Record@2ec791b9[name=r1,id=1]
> Record@bd86fd3[name=r2,id=2]
> {code}
> # Read the Avro file using the v2 code-generated Record class using DataFileStream and SpecificDatumReader
> *Schema details*
> v1 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"},
> {"name": "id", "type": "int"}
> ]
> }
> {code}
> v2 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"}
> ]
> }
> {code}
> *Write code*
> {code}
> public static Record createRecord(String name, int id) {
> Record record = new Record();
> record.name = name;
> record.id = id;
> return record;
> }
> public static void writeToAvro(OutputStream outputStream)
> throws IOException {
> DataFileWriter<Record> writer =
> new DataFileWriter<Record>(new SpecificDatumWriter<Record>());
> writer.create(Record.SCHEMA$, outputStream);
> writer.append(createRecord("r1", 1));
> writer.append(createRecord("r2", 2));
> writer.close();
> outputStream.close();
> }
> {code}
> *Read code*
> {code}
> public static void readFromAvro(InputStream is) throws IOException {
> DataFileStream<Record> reader = new DataFileStream<Record>(
> is, new SpecificDatumReader<Record>());
> for (Record a : reader) {
> System.out.println(ToStringBuilder.reflectionToString(a));
> }
> IOUtils.cleanup(null, is);
> IOUtils.cleanup(null, reader);
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-891) Exception when attempting to read
modified schema containing deleted field
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108920#comment-13108920 ]
Doug Cutting commented on AVRO-891:
-----------------------------------
When you read you should use 'new SpecificDatumReader<Record>(Record.class)' to pass the new schema to the reader. Otherwise it assumes the same schema that's used to write should be used to read. Does that help?
> Exception when attempting to read modified schema containing deleted field
> --------------------------------------------------------------------------
>
> Key: AVRO-891
> URL: https://issues.apache.org/jira/browse/AVRO-891
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.5.4
> Environment: OSX 10.7
> Reporter: Alex Holmes
>
> An AvroRuntimeException exception is thrown when attempting to read an Avro file serialized with an older version of a schema containing a field which has been subsequently removed in the newer schema.
> *Exception*
> {code}
> Exception in thread "main" org.apache.avro.AvroRuntimeException: Bad index
> at Record.put(Unknown Source)
> at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
> at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220)
> at Read.readFromAvro(Unknown Source)
> at Read.main(Unknown Source)
> {code}
> *Steps to reproduce*
> # Generate code for schema v1 and v2
> # Write an Avro file with the v1 code-generated Record class using the DataFileWriter and SpecificDatumWriter
> # (informational only) Read the Avro file using the v1 code-generated Record class using DataFileStream and SpecificDatumReader (output follows)
> {code}
> Record@2ec791b9[name=r1,id=1]
> Record@bd86fd3[name=r2,id=2]
> {code}
> # Read the Avro file using the v2 code-generated Record class using DataFileStream and SpecificDatumReader
> *Schema details*
> v1 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"},
> {"name": "id", "type": "int"}
> ]
> }
> {code}
> v2 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"}
> ]
> }
> {code}
> *Write code*
> {code}
> public static Record createRecord(String name, int id) {
> Record record = new Record();
> record.name = name;
> record.id = id;
> return record;
> }
> public static void writeToAvro(OutputStream outputStream)
> throws IOException {
> DataFileWriter<Record> writer =
> new DataFileWriter<Record>(new SpecificDatumWriter<Record>());
> writer.create(Record.SCHEMA$, outputStream);
> writer.append(createRecord("r1", 1));
> writer.append(createRecord("r2", 2));
> writer.close();
> outputStream.close();
> }
> {code}
> *Read code*
> {code}
> public static void readFromAvro(InputStream is) throws IOException {
> DataFileStream<Record> reader = new DataFileStream<Record>(
> is, new SpecificDatumReader<Record>());
> for (Record a : reader) {
> System.out.println(ToStringBuilder.reflectionToString(a));
> }
> IOUtils.cleanup(null, is);
> IOUtils.cleanup(null, reader);
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (AVRO-891) Exception when attempting
to read modified schema containing deleted field
Posted by "Scott Carey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108925#comment-13108925 ]
Scott Carey edited comment on AVRO-891 at 9/20/11 6:50 PM:
-----------------------------------------------------------
I should have caught that before.
We may need better Javadoc and some example uses in the doc, but this definitely works for me and has worked for a long time --- but I use the constructor that passes in the class.
was (Author: scott_carey):
I should have caught that before.
We may need better Javadoc and some example uses in the doc, but this definitely works for me and has worked for a long time --- but I use the constructor above.
> Exception when attempting to read modified schema containing deleted field
> --------------------------------------------------------------------------
>
> Key: AVRO-891
> URL: https://issues.apache.org/jira/browse/AVRO-891
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.5.4
> Environment: OSX 10.7
> Reporter: Alex Holmes
>
> An AvroRuntimeException exception is thrown when attempting to read an Avro file serialized with an older version of a schema containing a field which has been subsequently removed in the newer schema.
> *Exception*
> {code}
> Exception in thread "main" org.apache.avro.AvroRuntimeException: Bad index
> at Record.put(Unknown Source)
> at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
> at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220)
> at Read.readFromAvro(Unknown Source)
> at Read.main(Unknown Source)
> {code}
> *Steps to reproduce*
> # Generate code for schema v1 and v2
> # Write an Avro file with the v1 code-generated Record class using the DataFileWriter and SpecificDatumWriter
> # (informational only) Read the Avro file using the v1 code-generated Record class using DataFileStream and SpecificDatumReader (output follows)
> {code}
> Record@2ec791b9[name=r1,id=1]
> Record@bd86fd3[name=r2,id=2]
> {code}
> # Read the Avro file using the v2 code-generated Record class using DataFileStream and SpecificDatumReader
> *Schema details*
> v1 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"},
> {"name": "id", "type": "int"}
> ]
> }
> {code}
> v2 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"}
> ]
> }
> {code}
> *Write code*
> {code}
> public static Record createRecord(String name, int id) {
> Record record = new Record();
> record.name = name;
> record.id = id;
> return record;
> }
> public static void writeToAvro(OutputStream outputStream)
> throws IOException {
> DataFileWriter<Record> writer =
> new DataFileWriter<Record>(new SpecificDatumWriter<Record>());
> writer.create(Record.SCHEMA$, outputStream);
> writer.append(createRecord("r1", 1));
> writer.append(createRecord("r2", 2));
> writer.close();
> outputStream.close();
> }
> {code}
> *Read code*
> {code}
> public static void readFromAvro(InputStream is) throws IOException {
> DataFileStream<Record> reader = new DataFileStream<Record>(
> is, new SpecificDatumReader<Record>());
> for (Record a : reader) {
> System.out.println(ToStringBuilder.reflectionToString(a));
> }
> IOUtils.cleanup(null, is);
> IOUtils.cleanup(null, reader);
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-891) Exception when attempting to read
modified schema containing deleted field
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-891:
------------------------------
Fix Version/s: 1.6.0
Assignee: Doug Cutting
Status: Patch Available (was: Open)
> Exception when attempting to read modified schema containing deleted field
> --------------------------------------------------------------------------
>
> Key: AVRO-891
> URL: https://issues.apache.org/jira/browse/AVRO-891
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.5.4
> Environment: OSX 10.7
> Reporter: Alex Holmes
> Assignee: Doug Cutting
> Fix For: 1.6.0
>
> Attachments: AVRO-891.patch
>
>
> An AvroRuntimeException exception is thrown when attempting to read an Avro file serialized with an older version of a schema containing a field which has been subsequently removed in the newer schema.
> *Exception*
> {code}
> Exception in thread "main" org.apache.avro.AvroRuntimeException: Bad index
> at Record.put(Unknown Source)
> at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
> at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220)
> at Read.readFromAvro(Unknown Source)
> at Read.main(Unknown Source)
> {code}
> *Steps to reproduce*
> # Generate code for schema v1 and v2
> # Write an Avro file with the v1 code-generated Record class using the DataFileWriter and SpecificDatumWriter
> # (informational only) Read the Avro file using the v1 code-generated Record class using DataFileStream and SpecificDatumReader (output follows)
> {code}
> Record@2ec791b9[name=r1,id=1]
> Record@bd86fd3[name=r2,id=2]
> {code}
> # Read the Avro file using the v2 code-generated Record class using DataFileStream and SpecificDatumReader
> *Schema details*
> v1 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"},
> {"name": "id", "type": "int"}
> ]
> }
> {code}
> v2 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"}
> ]
> }
> {code}
> *Write code*
> {code}
> public static Record createRecord(String name, int id) {
> Record record = new Record();
> record.name = name;
> record.id = id;
> return record;
> }
> public static void writeToAvro(OutputStream outputStream)
> throws IOException {
> DataFileWriter<Record> writer =
> new DataFileWriter<Record>(new SpecificDatumWriter<Record>());
> writer.create(Record.SCHEMA$, outputStream);
> writer.append(createRecord("r1", 1));
> writer.append(createRecord("r2", 2));
> writer.close();
> outputStream.close();
> }
> {code}
> *Read code*
> {code}
> public static void readFromAvro(InputStream is) throws IOException {
> DataFileStream<Record> reader = new DataFileStream<Record>(
> is, new SpecificDatumReader<Record>());
> for (Record a : reader) {
> System.out.println(ToStringBuilder.reflectionToString(a));
> }
> IOUtils.cleanup(null, is);
> IOUtils.cleanup(null, reader);
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-891) Change SpecificDatum
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-891:
------------------------------
Issue Type: Improvement (was: Bug)
Summary: Change SpecificDatum (was: Exception when attempting to read modified schema containing deleted field)
> Change SpecificDatum
> --------------------
>
> Key: AVRO-891
> URL: https://issues.apache.org/jira/browse/AVRO-891
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.5.4
> Environment: OSX 10.7
> Reporter: Alex Holmes
> Assignee: Doug Cutting
> Fix For: 1.6.0
>
> Attachments: AVRO-891.patch, AVRO-891.patch
>
>
> An AvroRuntimeException exception is thrown when attempting to read an Avro file serialized with an older version of a schema containing a field which has been subsequently removed in the newer schema.
> *Exception*
> {code}
> Exception in thread "main" org.apache.avro.AvroRuntimeException: Bad index
> at Record.put(Unknown Source)
> at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
> at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220)
> at Read.readFromAvro(Unknown Source)
> at Read.main(Unknown Source)
> {code}
> *Steps to reproduce*
> # Generate code for schema v1 and v2
> # Write an Avro file with the v1 code-generated Record class using the DataFileWriter and SpecificDatumWriter
> # (informational only) Read the Avro file using the v1 code-generated Record class using DataFileStream and SpecificDatumReader (output follows)
> {code}
> Record@2ec791b9[name=r1,id=1]
> Record@bd86fd3[name=r2,id=2]
> {code}
> # Read the Avro file using the v2 code-generated Record class using DataFileStream and SpecificDatumReader
> *Schema details*
> v1 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"},
> {"name": "id", "type": "int"}
> ]
> }
> {code}
> v2 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"}
> ]
> }
> {code}
> *Write code*
> {code}
> public static Record createRecord(String name, int id) {
> Record record = new Record();
> record.name = name;
> record.id = id;
> return record;
> }
> public static void writeToAvro(OutputStream outputStream)
> throws IOException {
> DataFileWriter<Record> writer =
> new DataFileWriter<Record>(new SpecificDatumWriter<Record>());
> writer.create(Record.SCHEMA$, outputStream);
> writer.append(createRecord("r1", 1));
> writer.append(createRecord("r2", 2));
> writer.close();
> outputStream.close();
> }
> {code}
> *Read code*
> {code}
> public static void readFromAvro(InputStream is) throws IOException {
> DataFileStream<Record> reader = new DataFileStream<Record>(
> is, new SpecificDatumReader<Record>());
> for (Record a : reader) {
> System.out.println(ToStringBuilder.reflectionToString(a));
> }
> IOUtils.cleanup(null, is);
> IOUtils.cleanup(null, reader);
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-891) Exception when attempting to read
modified schema containing deleted field
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-891:
------------------------------
Attachment: AVRO-891.patch
Here's a patch that implements (2) above. This should make things like the code in the description of this issue just work.
> Exception when attempting to read modified schema containing deleted field
> --------------------------------------------------------------------------
>
> Key: AVRO-891
> URL: https://issues.apache.org/jira/browse/AVRO-891
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.5.4
> Environment: OSX 10.7
> Reporter: Alex Holmes
> Attachments: AVRO-891.patch
>
>
> An AvroRuntimeException exception is thrown when attempting to read an Avro file serialized with an older version of a schema containing a field which has been subsequently removed in the newer schema.
> *Exception*
> {code}
> Exception in thread "main" org.apache.avro.AvroRuntimeException: Bad index
> at Record.put(Unknown Source)
> at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
> at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220)
> at Read.readFromAvro(Unknown Source)
> at Read.main(Unknown Source)
> {code}
> *Steps to reproduce*
> # Generate code for schema v1 and v2
> # Write an Avro file with the v1 code-generated Record class using the DataFileWriter and SpecificDatumWriter
> # (informational only) Read the Avro file using the v1 code-generated Record class using DataFileStream and SpecificDatumReader (output follows)
> {code}
> Record@2ec791b9[name=r1,id=1]
> Record@bd86fd3[name=r2,id=2]
> {code}
> # Read the Avro file using the v2 code-generated Record class using DataFileStream and SpecificDatumReader
> *Schema details*
> v1 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"},
> {"name": "id", "type": "int"}
> ]
> }
> {code}
> v2 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"}
> ]
> }
> {code}
> *Write code*
> {code}
> public static Record createRecord(String name, int id) {
> Record record = new Record();
> record.name = name;
> record.id = id;
> return record;
> }
> public static void writeToAvro(OutputStream outputStream)
> throws IOException {
> DataFileWriter<Record> writer =
> new DataFileWriter<Record>(new SpecificDatumWriter<Record>());
> writer.create(Record.SCHEMA$, outputStream);
> writer.append(createRecord("r1", 1));
> writer.append(createRecord("r2", 2));
> writer.close();
> outputStream.close();
> }
> {code}
> *Read code*
> {code}
> public static void readFromAvro(InputStream is) throws IOException {
> DataFileStream<Record> reader = new DataFileStream<Record>(
> is, new SpecificDatumReader<Record>());
> for (Record a : reader) {
> System.out.println(ToStringBuilder.reflectionToString(a));
> }
> IOUtils.cleanup(null, is);
> IOUtils.cleanup(null, reader);
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-891) Exception when attempting to read
modified schema containing deleted field
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112163#comment-13112163 ]
Doug Cutting commented on AVRO-891:
-----------------------------------
I'll commit this tomorrow if no one objects.
> Exception when attempting to read modified schema containing deleted field
> --------------------------------------------------------------------------
>
> Key: AVRO-891
> URL: https://issues.apache.org/jira/browse/AVRO-891
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.5.4
> Environment: OSX 10.7
> Reporter: Alex Holmes
> Assignee: Doug Cutting
> Fix For: 1.6.0
>
> Attachments: AVRO-891.patch, AVRO-891.patch
>
>
> An AvroRuntimeException exception is thrown when attempting to read an Avro file serialized with an older version of a schema containing a field which has been subsequently removed in the newer schema.
> *Exception*
> {code}
> Exception in thread "main" org.apache.avro.AvroRuntimeException: Bad index
> at Record.put(Unknown Source)
> at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
> at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220)
> at Read.readFromAvro(Unknown Source)
> at Read.main(Unknown Source)
> {code}
> *Steps to reproduce*
> # Generate code for schema v1 and v2
> # Write an Avro file with the v1 code-generated Record class using the DataFileWriter and SpecificDatumWriter
> # (informational only) Read the Avro file using the v1 code-generated Record class using DataFileStream and SpecificDatumReader (output follows)
> {code}
> Record@2ec791b9[name=r1,id=1]
> Record@bd86fd3[name=r2,id=2]
> {code}
> # Read the Avro file using the v2 code-generated Record class using DataFileStream and SpecificDatumReader
> *Schema details*
> v1 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"},
> {"name": "id", "type": "int"}
> ]
> }
> {code}
> v2 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"}
> ]
> }
> {code}
> *Write code*
> {code}
> public static Record createRecord(String name, int id) {
> Record record = new Record();
> record.name = name;
> record.id = id;
> return record;
> }
> public static void writeToAvro(OutputStream outputStream)
> throws IOException {
> DataFileWriter<Record> writer =
> new DataFileWriter<Record>(new SpecificDatumWriter<Record>());
> writer.create(Record.SCHEMA$, outputStream);
> writer.append(createRecord("r1", 1));
> writer.append(createRecord("r2", 2));
> writer.close();
> outputStream.close();
> }
> {code}
> *Read code*
> {code}
> public static void readFromAvro(InputStream is) throws IOException {
> DataFileStream<Record> reader = new DataFileStream<Record>(
> is, new SpecificDatumReader<Record>());
> for (Record a : reader) {
> System.out.println(ToStringBuilder.reflectionToString(a));
> }
> IOUtils.cleanup(null, is);
> IOUtils.cleanup(null, reader);
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-891) Change SpecificDatumReader to default
reader schema from loaded class
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-891:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I committed this.
> Change SpecificDatumReader to default reader schema from loaded class
> ---------------------------------------------------------------------
>
> Key: AVRO-891
> URL: https://issues.apache.org/jira/browse/AVRO-891
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.5.4
> Environment: OSX 10.7
> Reporter: Alex Holmes
> Assignee: Doug Cutting
> Fix For: 1.6.0
>
> Attachments: AVRO-891.patch, AVRO-891.patch
>
>
> An AvroRuntimeException exception is thrown when attempting to read an Avro file serialized with an older version of a schema containing a field which has been subsequently removed in the newer schema.
> *Exception*
> {code}
> Exception in thread "main" org.apache.avro.AvroRuntimeException: Bad index
> at Record.put(Unknown Source)
> at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
> at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220)
> at Read.readFromAvro(Unknown Source)
> at Read.main(Unknown Source)
> {code}
> *Steps to reproduce*
> # Generate code for schema v1 and v2
> # Write an Avro file with the v1 code-generated Record class using the DataFileWriter and SpecificDatumWriter
> # (informational only) Read the Avro file using the v1 code-generated Record class using DataFileStream and SpecificDatumReader (output follows)
> {code}
> Record@2ec791b9[name=r1,id=1]
> Record@bd86fd3[name=r2,id=2]
> {code}
> # Read the Avro file using the v2 code-generated Record class using DataFileStream and SpecificDatumReader
> *Schema details*
> v1 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"},
> {"name": "id", "type": "int"}
> ]
> }
> {code}
> v2 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"}
> ]
> }
> {code}
> *Write code*
> {code}
> public static Record createRecord(String name, int id) {
> Record record = new Record();
> record.name = name;
> record.id = id;
> return record;
> }
> public static void writeToAvro(OutputStream outputStream)
> throws IOException {
> DataFileWriter<Record> writer =
> new DataFileWriter<Record>(new SpecificDatumWriter<Record>());
> writer.create(Record.SCHEMA$, outputStream);
> writer.append(createRecord("r1", 1));
> writer.append(createRecord("r2", 2));
> writer.close();
> outputStream.close();
> }
> {code}
> *Read code*
> {code}
> public static void readFromAvro(InputStream is) throws IOException {
> DataFileStream<Record> reader = new DataFileStream<Record>(
> is, new SpecificDatumReader<Record>());
> for (Record a : reader) {
> System.out.println(ToStringBuilder.reflectionToString(a));
> }
> IOUtils.cleanup(null, is);
> IOUtils.cleanup(null, reader);
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-891) Exception when attempting to read
modified schema containing deleted field
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-891:
------------------------------
Attachment: AVRO-891.patch
Here's an improved patch that includes a test.
> Exception when attempting to read modified schema containing deleted field
> --------------------------------------------------------------------------
>
> Key: AVRO-891
> URL: https://issues.apache.org/jira/browse/AVRO-891
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.5.4
> Environment: OSX 10.7
> Reporter: Alex Holmes
> Assignee: Doug Cutting
> Fix For: 1.6.0
>
> Attachments: AVRO-891.patch, AVRO-891.patch
>
>
> An AvroRuntimeException exception is thrown when attempting to read an Avro file serialized with an older version of a schema containing a field which has been subsequently removed in the newer schema.
> *Exception*
> {code}
> Exception in thread "main" org.apache.avro.AvroRuntimeException: Bad index
> at Record.put(Unknown Source)
> at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
> at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220)
> at Read.readFromAvro(Unknown Source)
> at Read.main(Unknown Source)
> {code}
> *Steps to reproduce*
> # Generate code for schema v1 and v2
> # Write an Avro file with the v1 code-generated Record class using the DataFileWriter and SpecificDatumWriter
> # (informational only) Read the Avro file using the v1 code-generated Record class using DataFileStream and SpecificDatumReader (output follows)
> {code}
> Record@2ec791b9[name=r1,id=1]
> Record@bd86fd3[name=r2,id=2]
> {code}
> # Read the Avro file using the v2 code-generated Record class using DataFileStream and SpecificDatumReader
> *Schema details*
> v1 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"},
> {"name": "id", "type": "int"}
> ]
> }
> {code}
> v2 schema:
> {code}
> {"name": "Record", "type": "record",
> "fields": [
> {"name": "name", "type": "string"}
> ]
> }
> {code}
> *Write code*
> {code}
> public static Record createRecord(String name, int id) {
> Record record = new Record();
> record.name = name;
> record.id = id;
> return record;
> }
> public static void writeToAvro(OutputStream outputStream)
> throws IOException {
> DataFileWriter<Record> writer =
> new DataFileWriter<Record>(new SpecificDatumWriter<Record>());
> writer.create(Record.SCHEMA$, outputStream);
> writer.append(createRecord("r1", 1));
> writer.append(createRecord("r2", 2));
> writer.close();
> outputStream.close();
> }
> {code}
> *Read code*
> {code}
> public static void readFromAvro(InputStream is) throws IOException {
> DataFileStream<Record> reader = new DataFileStream<Record>(
> is, new SpecificDatumReader<Record>());
> for (Record a : reader) {
> System.out.println(ToStringBuilder.reflectionToString(a));
> }
> IOUtils.cleanup(null, is);
> IOUtils.cleanup(null, reader);
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira