You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Jeffrey Theobald (JIRA)" <ji...@apache.org> on 2012/10/24 07:56:11 UTC
[jira] [Created] (AVRO-1183) Avro Pair fails to initialize when
either they key or value is a list.
Jeffrey Theobald created AVRO-1183:
--------------------------------------
Summary: Avro Pair fails to initialize when either they key or value is a list.
Key: AVRO-1183
URL: https://issues.apache.org/jira/browse/AVRO-1183
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.7.2
Reporter: Jeffrey Theobald
Priority: Minor
Hello,
It seems that a change somewhere between 1.5.7 and 1.7.2 (maybe AVRO-966 ?) has broken the Avro Pair object, so that you can no longer instantiate a pair when either the key or the value is a list.
To replicate this bug, try the following:
{code}
Integer key = new Integer(0);
List<Integer> list = new ArrayList<Integer>();
Pair<Integer, List<Integer>> pair = new Pair<Integer, List<Integer>>(key, list);
{code}
It will produce something like the following exception:
{noformat}org.apache.avro.AvroRuntimeException: Can't find element type of Collection
at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:281)
at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:151)
at org.apache.avro.mapred.Pair.<init>(Pair.java:312)
.....SNIP.....
{noformat}
I've used Integer here for illustration but it also affects SpecificRecord objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1183) Avro Pair fails to initialize when
either they key or value is a list.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-1183:
-------------------------------
Attachment: AVRO-1183.patch
Here's a patch that changes the message to:
Cannot infer Schema for: class Collection. Must create Pair with explicit key and value schemas.
> Avro Pair fails to initialize when either they key or value is a list.
> ----------------------------------------------------------------------
>
> Key: AVRO-1183
> URL: https://issues.apache.org/jira/browse/AVRO-1183
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.7.2
> Reporter: Jeffrey Theobald
> Priority: Minor
> Fix For: 1.7.3
>
> Attachments: AVRO-1183.patch
>
>
> Hello,
> It seems that a change somewhere between 1.5.7 and 1.7.2 (maybe AVRO-966 ?) has broken the Avro Pair object, so that you can no longer instantiate a pair when either the key or the value is a list.
> To replicate this bug, try the following:
> {code}
> Integer key = new Integer(0);
> List<Integer> list = new ArrayList<Integer>();
> Pair<Integer, List<Integer>> pair = new Pair<Integer, List<Integer>>(key, list);
> {code}
> It will produce something like the following exception:
> {noformat}org.apache.avro.AvroRuntimeException: Can't find element type of Collection
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:281)
> at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:151)
> at org.apache.avro.mapred.Pair.<init>(Pair.java:312)
> .....SNIP.....
> {noformat}
> I've used Integer here for illustration but it also affects SpecificRecord objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1183) Avro Pair fails to initialize when
either they key or value is a list.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-1183:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I added a test and committed this.
> Avro Pair fails to initialize when either they key or value is a list.
> ----------------------------------------------------------------------
>
> Key: AVRO-1183
> URL: https://issues.apache.org/jira/browse/AVRO-1183
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.7.2
> Reporter: Jeffrey Theobald
> Assignee: Doug Cutting
> Priority: Minor
> Fix For: 1.7.3
>
> Attachments: AVRO-1183.patch
>
>
> Hello,
> It seems that a change somewhere between 1.5.7 and 1.7.2 (maybe AVRO-966 ?) has broken the Avro Pair object, so that you can no longer instantiate a pair when either the key or the value is a list.
> To replicate this bug, try the following:
> {code}
> Integer key = new Integer(0);
> List<Integer> list = new ArrayList<Integer>();
> Pair<Integer, List<Integer>> pair = new Pair<Integer, List<Integer>>(key, list);
> {code}
> It will produce something like the following exception:
> {noformat}org.apache.avro.AvroRuntimeException: Can't find element type of Collection
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:281)
> at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:151)
> at org.apache.avro.mapred.Pair.<init>(Pair.java:312)
> .....SNIP.....
> {noformat}
> I've used Integer here for illustration but it also affects SpecificRecord objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1183) Avro Pair fails to initialize when
either they key or value is a list.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-1183:
-------------------------------
Fix Version/s: 1.7.3
Assignee: Doug Cutting
Status: Patch Available (was: Open)
I'll commit this soon unless someone objects.
> Avro Pair fails to initialize when either they key or value is a list.
> ----------------------------------------------------------------------
>
> Key: AVRO-1183
> URL: https://issues.apache.org/jira/browse/AVRO-1183
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.7.2
> Reporter: Jeffrey Theobald
> Assignee: Doug Cutting
> Priority: Minor
> Fix For: 1.7.3
>
> Attachments: AVRO-1183.patch
>
>
> Hello,
> It seems that a change somewhere between 1.5.7 and 1.7.2 (maybe AVRO-966 ?) has broken the Avro Pair object, so that you can no longer instantiate a pair when either the key or the value is a list.
> To replicate this bug, try the following:
> {code}
> Integer key = new Integer(0);
> List<Integer> list = new ArrayList<Integer>();
> Pair<Integer, List<Integer>> pair = new Pair<Integer, List<Integer>>(key, list);
> {code}
> It will produce something like the following exception:
> {noformat}org.apache.avro.AvroRuntimeException: Can't find element type of Collection
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:281)
> at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:151)
> at org.apache.avro.mapred.Pair.<init>(Pair.java:312)
> .....SNIP.....
> {noformat}
> I've used Integer here for illustration but it also affects SpecificRecord objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1183) Avro Pair fails to initialize when
either they key or value is a list.
Posted by "Jeffrey Theobald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490452#comment-13490452 ]
Jeffrey Theobald commented on AVRO-1183:
----------------------------------------
Ahh, of course. We've been using that pair constructor to get around the exception but now that you've pointed it out why it won't work, it makes sense. Thank you for clearing that up.
Is it possible that the error message could be a little more descriptive? I'm not sure what the convention is for these kinds of exceptions within the Avro project but I'm sure I'm not the only one who's bumped into this issue. The current exception message is technically correct but it's very minimal which is slightly confusing.
> Avro Pair fails to initialize when either they key or value is a list.
> ----------------------------------------------------------------------
>
> Key: AVRO-1183
> URL: https://issues.apache.org/jira/browse/AVRO-1183
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.7.2
> Reporter: Jeffrey Theobald
> Priority: Minor
>
> Hello,
> It seems that a change somewhere between 1.5.7 and 1.7.2 (maybe AVRO-966 ?) has broken the Avro Pair object, so that you can no longer instantiate a pair when either the key or the value is a list.
> To replicate this bug, try the following:
> {code}
> Integer key = new Integer(0);
> List<Integer> list = new ArrayList<Integer>();
> Pair<Integer, List<Integer>> pair = new Pair<Integer, List<Integer>>(key, list);
> {code}
> It will produce something like the following exception:
> {noformat}org.apache.avro.AvroRuntimeException: Can't find element type of Collection
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:281)
> at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:151)
> at org.apache.avro.mapred.Pair.<init>(Pair.java:312)
> .....SNIP.....
> {noformat}
> I've used Integer here for illustration but it also affects SpecificRecord objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1183) Avro Pair fails to initialize when
either they key or value is a list.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486264#comment-13486264 ]
Doug Cutting commented on AVRO-1183:
------------------------------------
This used to yield a schema like:
{code}
{
"type" : "record",
"name" : "Pair",
"namespace" : "org.apache.avro.mapred",
"fields" : [ {
"name" : "key",
"type" : "int",
}, {
"name" : "value",
"type" : {
"type" : "record",
"name" : "ArrayList",
"namespace" : "java.util",
"fields" : [ ]
},
"order" : "ignore"
} ]
}
{code}
Such a schema will not correctly write values, since the ArrayList record has no fields. This is a limitation of Java reflection: one cannot find the element type of an ArrayList at runtime. So to create a schema for a Pair you'd need to do something like:
{code}
Schema keySchema = Schema.create(Type.INT);
Schema valueSchema = Schema.createArray(Schema.create(Type.INT));
new Pair(key, keySchema, value, valueSchema);
{code}
> Avro Pair fails to initialize when either they key or value is a list.
> ----------------------------------------------------------------------
>
> Key: AVRO-1183
> URL: https://issues.apache.org/jira/browse/AVRO-1183
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.7.2
> Reporter: Jeffrey Theobald
> Priority: Minor
>
> Hello,
> It seems that a change somewhere between 1.5.7 and 1.7.2 (maybe AVRO-966 ?) has broken the Avro Pair object, so that you can no longer instantiate a pair when either the key or the value is a list.
> To replicate this bug, try the following:
> {code}
> Integer key = new Integer(0);
> List<Integer> list = new ArrayList<Integer>();
> Pair<Integer, List<Integer>> pair = new Pair<Integer, List<Integer>>(key, list);
> {code}
> It will produce something like the following exception:
> {noformat}org.apache.avro.AvroRuntimeException: Can't find element type of Collection
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:281)
> at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:151)
> at org.apache.avro.mapred.Pair.<init>(Pair.java:312)
> .....SNIP.....
> {noformat}
> I've used Integer here for illustration but it also affects SpecificRecord objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1183) Avro Pair fails to initialize when
either they key or value is a list.
Posted by "Jeffrey Theobald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496723#comment-13496723 ]
Jeffrey Theobald commented on AVRO-1183:
----------------------------------------
Sounds good to me!
> Avro Pair fails to initialize when either they key or value is a list.
> ----------------------------------------------------------------------
>
> Key: AVRO-1183
> URL: https://issues.apache.org/jira/browse/AVRO-1183
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.7.2
> Reporter: Jeffrey Theobald
> Assignee: Doug Cutting
> Priority: Minor
> Fix For: 1.7.3
>
> Attachments: AVRO-1183.patch
>
>
> Hello,
> It seems that a change somewhere between 1.5.7 and 1.7.2 (maybe AVRO-966 ?) has broken the Avro Pair object, so that you can no longer instantiate a pair when either the key or the value is a list.
> To replicate this bug, try the following:
> {code}
> Integer key = new Integer(0);
> List<Integer> list = new ArrayList<Integer>();
> Pair<Integer, List<Integer>> pair = new Pair<Integer, List<Integer>>(key, list);
> {code}
> It will produce something like the following exception:
> {noformat}org.apache.avro.AvroRuntimeException: Can't find element type of Collection
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:281)
> at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:151)
> at org.apache.avro.mapred.Pair.<init>(Pair.java:312)
> .....SNIP.....
> {noformat}
> I've used Integer here for illustration but it also affects SpecificRecord objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1183) Avro Pair fails to initialize when
either they key or value is a list.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490794#comment-13490794 ]
Doug Cutting commented on AVRO-1183:
------------------------------------
Improving the error message is a good idea. Would something like, "Cannot infer schema for Collection" be much better?
> Avro Pair fails to initialize when either they key or value is a list.
> ----------------------------------------------------------------------
>
> Key: AVRO-1183
> URL: https://issues.apache.org/jira/browse/AVRO-1183
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.7.2
> Reporter: Jeffrey Theobald
> Priority: Minor
>
> Hello,
> It seems that a change somewhere between 1.5.7 and 1.7.2 (maybe AVRO-966 ?) has broken the Avro Pair object, so that you can no longer instantiate a pair when either the key or the value is a list.
> To replicate this bug, try the following:
> {code}
> Integer key = new Integer(0);
> List<Integer> list = new ArrayList<Integer>();
> Pair<Integer, List<Integer>> pair = new Pair<Integer, List<Integer>>(key, list);
> {code}
> It will produce something like the following exception:
> {noformat}org.apache.avro.AvroRuntimeException: Can't find element type of Collection
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:281)
> at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:151)
> at org.apache.avro.mapred.Pair.<init>(Pair.java:312)
> .....SNIP.....
> {noformat}
> I've used Integer here for illustration but it also affects SpecificRecord objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1183) Avro Pair fails to initialize when
either they key or value is a list.
Posted by "Jeffrey Theobald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495080#comment-13495080 ]
Jeffrey Theobald commented on AVRO-1183:
----------------------------------------
Yeah, that seems more helpful. Would it be too verbose to go even further and suggest the remedy in the exception message? i.e.,_ "Cannot infer schema for Collection. Specify the schema for Pairs containing Collections to avoid this error"_. Again, I'm not sure of the Avro convention is, but either way works for me.
> Avro Pair fails to initialize when either they key or value is a list.
> ----------------------------------------------------------------------
>
> Key: AVRO-1183
> URL: https://issues.apache.org/jira/browse/AVRO-1183
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.7.2
> Reporter: Jeffrey Theobald
> Priority: Minor
>
> Hello,
> It seems that a change somewhere between 1.5.7 and 1.7.2 (maybe AVRO-966 ?) has broken the Avro Pair object, so that you can no longer instantiate a pair when either the key or the value is a list.
> To replicate this bug, try the following:
> {code}
> Integer key = new Integer(0);
> List<Integer> list = new ArrayList<Integer>();
> Pair<Integer, List<Integer>> pair = new Pair<Integer, List<Integer>>(key, list);
> {code}
> It will produce something like the following exception:
> {noformat}org.apache.avro.AvroRuntimeException: Can't find element type of Collection
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:281)
> at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:151)
> at org.apache.avro.mapred.Pair.<init>(Pair.java:312)
> .....SNIP.....
> {noformat}
> I've used Integer here for illustration but it also affects SpecificRecord objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira