You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Jeremy Lewi <je...@lewi.us> on 2012/03/11 21:40:46 UTC
Make a copy of an avro record
Hi,
In java, I'd like to make a deep copy of an avro record.
Looking at the code that the avro compiler generates it looks like the way
to do this for records of type GraphNodeData
would be
GraphNodeData copy = GraphNodeData.newBuilder(existing_value).build();
where GraphNodeData is the name of my avro record.
Unfortunately, this generates a compile time error because
the method GraphNodeData.build is decorated with "@Override" but it
apparently does not override any method.
Deleting "@Override" makes the code work. Unfortunately, I would have to do
this every time I regenerated my avro classes from the schema.
Am I doing something wrong or should I file a bug?
Thanks
J
Re: Make a copy of an avro record
Posted by Doug Cutting <cu...@apache.org>.
On 03/12/2012 11:09 AM, Jeremy Lewi wrote:
> I was using 1.6.0_27 and the @override decorator was a problem, maybe
> newer releases in the 1.6 set will work.
Re-reading the thread I now see the problem is compiling code generated
by Avro. The builder's generated build() method has an @Override
declaration. It implements a method in the RecordBuilder interface.
Using @Override for interface methods was not permitted in Java 5 but is
permitted in Java 6 and Java 7. So I'm now confused how this fails in
any version of Java 6. Regardless, if its really a problem for even
some versions of Java 6 then we should probably just remove it from the
generated code. It doesn't add much value there.
Doug
Re: Make a copy of an avro record
Posted by Scott Carey <sc...@apache.org>.
I am able to use Avro 1.6.2 with the Specific builder API with no issue with
Java 1.6.0_29.
I also double checked the build, it sets the source and target versions to
1.6 when compiling. The avro-service-archetype does as well.
On 3/12/12 11:09 AM, "Jeremy Lewi" <je...@lewi.us> wrote:
> I was using 1.6.0_27 and the @override decorator was a problem, maybe newer
> releases in the 1.6 set will work.
>
> J
>
> On Mon, Mar 12, 2012 at 10:42 AM, Doug Cutting <cu...@apache.org> wrote:
>> On 03/12/2012 10:20 AM, Scott Carey wrote:
>>> > We should be generating Java 1.6 compatible code.
>>
>> I may have inadvertantly used Java 1.7 to compile recent releases.
>>
>> Does this create problems? If so, we might consider re-publising
>> binaries for these releases. The sources should not change, but
>> re-generating jar files seems reasonable.
>>
>> Doug
>
Re: Make a copy of an avro record
Posted by Jeremy Lewi <je...@lewi.us>.
I was using 1.6.0_27 and the @override decorator was a problem, maybe newer
releases in the 1.6 set will work.
J
On Mon, Mar 12, 2012 at 10:42 AM, Doug Cutting <cu...@apache.org> wrote:
> On 03/12/2012 10:20 AM, Scott Carey wrote:
> > We should be generating Java 1.6 compatible code.
>
> I may have inadvertantly used Java 1.7 to compile recent releases.
>
> Does this create problems? If so, we might consider re-publising
> binaries for these releases. The sources should not change, but
> re-generating jar files seems reasonable.
>
> Doug
>
Re: Make a copy of an avro record
Posted by Doug Cutting <cu...@apache.org>.
On 03/12/2012 10:20 AM, Scott Carey wrote:
> We should be generating Java 1.6 compatible code.
I may have inadvertantly used Java 1.7 to compile recent releases.
Does this create problems? If so, we might consider re-publising
binaries for these releases. The sources should not change, but
re-generating jar files seems reasonable.
Doug
Re: Make a copy of an avro record
Posted by Scott Carey <sc...@apache.org>.
We should be generating Java 1.6 compatible code.
What version were you testing?
1.6.3 is near release, the RC is available here:
http://mail-archives.apache.org/mod_mbox/avro-dev/201203.mbox/%3C4F514F22.8
=070700@apache.org%3E
Does it have the same problem?
On 3/12/12 9:27 AM, "Jeremy Lewi" <je...@lewi.us> wrote:
>Thanks James and Doug. I was able to simply cast the output of
>SpecificData...deepCopy to my type and it seems to bypass the problematic
>methods decorated with @override.
>What about the potential incompatibility with earlier versions of java
>due to the change in semantics of @override? If this is really an issue
>this seems like it would affect a lot of users particularly people using
>Avro MapReduce on a cluster where upgrading java is not a trivial
>proposition. In my particular case, the reduce processing requires
>loading all values associated with the key into memory, which
>necessitates a deep copy because the iterable object passed to the
>reducer seems to be reusing the same instance.
>
>Using SpecificData.get().deepCopy(record) seems like a viable workaround.
>Nonetheless, it does seem a bit problematic if the compiler is generating
>code that is incompatible with earlier versions of java.
>
>J
>
>On Mon, Mar 12, 2012 at 9:05 AM, Doug Cutting <cu...@apache.org> wrote:
>
>On 03/11/2012 10:22 PM, James Baldassari wrote:
>> If you want to make a deep copy of a specific record, the easiest way is
>> probably to use the Builder API,
>> e.g. GraphNodeData.newBuilder(recordToCopy).build().
>
>
>SpecificData.get().deepCopy(record) should work too.
>
>Doug
>
>
>
Re: Make a copy of an avro record
Posted by Jeremy Lewi <je...@lewi.us>.
Thanks James and Doug. I was able to simply cast the output of
SpecificData...deepCopy to my type and it seems to bypass the problematic
methods decorated with @override.
What about the potential incompatibility with earlier versions of java due
to the change in semantics of @override? If this is really an issue this
seems like it would affect a lot of users particularly people using Avro
MapReduce on a cluster where upgrading java is not a trivial proposition.
In my particular case, the reduce processing requires loading all values
associated with the key into memory, which necessitates a deep copy because
the iterable object passed to the reducer seems to be reusing the same
instance.
Using SpecificData.get().deepCopy(record) seems like a viable workaround.
Nonetheless, it does seem a bit problematic if the compiler is generating
code that is incompatible with earlier versions of java.
J
On Mon, Mar 12, 2012 at 9:05 AM, Doug Cutting <cu...@apache.org> wrote:
> On 03/11/2012 10:22 PM, James Baldassari wrote:
> > If you want to make a deep copy of a specific record, the easiest way is
> > probably to use the Builder API,
> > e.g. GraphNodeData.newBuilder(recordToCopy).build().
>
> SpecificData.get().deepCopy(record) should work too.
>
> Doug
>
Re: Make a copy of an avro record
Posted by Doug Cutting <cu...@apache.org>.
On 03/11/2012 10:22 PM, James Baldassari wrote:
> If you want to make a deep copy of a specific record, the easiest way is
> probably to use the Builder API,
> e.g. GraphNodeData.newBuilder(recordToCopy).build().
SpecificData.get().deepCopy(record) should work too.
Doug
Re: Make a copy of an avro record
Posted by James Baldassari <jb...@gmail.com>.
If you want to make a deep copy of a specific record, the easiest way is
probably to use the Builder API,
e.g. GraphNodeData.newBuilder(recordToCopy).build(). This will create a
deep copy of 'recordToBuild' and then return it as a new GraphNodeData
object. Keep in mind that the performance of the Builder API is not great
at the moment. There are a couple of issues out there for improving this:
https://issues.apache.org/jira/browse/AVRO-985
https://issues.apache.org/jira/browse/AVRO-989
-James
On Mon, Mar 12, 2012 at 12:26 AM, Jeremy Lewi <je...@lewi.us> wrote:
> Looks like when I upgraded to Java 7 it ran fine.
>
>
> On Sun, Mar 11, 2012 at 9:07 PM, Jeremy Lewi <je...@lewi.us> wrote:
>
>> GenericData.deepCopy gives me an instance of GenericData.Record. How do I
>> convert this to an instance of my specific data (e.g GraphNodeData).
>>
>> It looks like the issue with the @override statement for the build might
>> have something to do with the fact that the specification for @Override
>> changed in jdk1.6 (see this post)
>>
>> http://stackoverflow.com/questions/2335655/why-is-javac-failing-on-override-annotation.
>> However, I'm using java version 1.6.0_27 so I'm not sure why its not
>> working for me.
>>
>> To clarify, the statement
>> GraphNodeData copy = GraphNodeData.newBuilder(this.data).build()
>> Compiles fine, but generates a runtime warning
>> The method build() of type GraphNodeData.Builder must override a
>> superclass method
>>
>> J
>>
>>
>> On Sun, Mar 11, 2012 at 7:37 PM, Jeremy Lewi <je...@lewi.us> wrote:
>>
>>> Thanks.
>>>
>>> J
>>>
>>>
>>> On Sun, Mar 11, 2012 at 6:20 PM, James Baldassari <jbaldassari@gmail.com
>>> > wrote:
>>>
>>>> GenericData is a singleton. You can use the static method
>>>> GenericData#get() to obtain the singleton instance:
>>>>
>>>>
>>>> http://avro.apache.org/docs/current/api/java/org/apache/avro/generic/GenericData.html#get()
>>>>
>>>> -James
>>>>
>>>>
>>>> On Sun, Mar 11, 2012 at 5:22 PM, Jeremy Lewi <je...@lewi.us> wrote:
>>>>
>>>>> Yes. How do I instantiate an instance of GenericData though? It looks
>>>>> like its constructors are protected.
>>>>>
>>>>> Thanks
>>>>> J
>>>>>
>>>>>
>>>>> On Sun, Mar 11, 2012 at 1:52 PM, <ea...@gmail.com> wrote:
>>>>>
>>>>>> There is deepcopy method on genericdata I think
>>>>>>
>>>>>> I used it to create deep copies of the avro data
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>> On Mar 11, 2012, at 3:40 PM, Jeremy Lewi <je...@lewi.us> wrote:
>>>>>>
>>>>>> > Hi,
>>>>>> >
>>>>>> > In java, I'd like to make a deep copy of an avro record.
>>>>>> >
>>>>>> > Looking at the code that the avro compiler generates it looks like
>>>>>> the way to do this for records of type GraphNodeData
>>>>>> > would be
>>>>>> > GraphNodeData copy =
>>>>>> GraphNodeData.newBuilder(existing_value).build();
>>>>>> >
>>>>>> > where GraphNodeData is the name of my avro record.
>>>>>> >
>>>>>> > Unfortunately, this generates a compile time error because
>>>>>> > the method GraphNodeData.build is decorated with "@Override" but
>>>>>> it apparently does not override any method.
>>>>>> > Deleting "@Override" makes the code work. Unfortunately, I would
>>>>>> have to do this every time I regenerated my avro classes from the schema.
>>>>>> >
>>>>>> > Am I doing something wrong or should I file a bug?
>>>>>> >
>>>>>> > Thanks
>>>>>> > J
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Make a copy of an avro record
Posted by Jeremy Lewi <je...@lewi.us>.
Looks like when I upgraded to Java 7 it ran fine.
On Sun, Mar 11, 2012 at 9:07 PM, Jeremy Lewi <je...@lewi.us> wrote:
> GenericData.deepCopy gives me an instance of GenericData.Record. How do I
> convert this to an instance of my specific data (e.g GraphNodeData).
>
> It looks like the issue with the @override statement for the build might
> have something to do with the fact that the specification for @Override
> changed in jdk1.6 (see this post)
>
> http://stackoverflow.com/questions/2335655/why-is-javac-failing-on-override-annotation.
> However, I'm using java version 1.6.0_27 so I'm not sure why its not
> working for me.
>
> To clarify, the statement
> GraphNodeData copy = GraphNodeData.newBuilder(this.data).build()
> Compiles fine, but generates a runtime warning
> The method build() of type GraphNodeData.Builder must override a
> superclass method
>
> J
>
>
> On Sun, Mar 11, 2012 at 7:37 PM, Jeremy Lewi <je...@lewi.us> wrote:
>
>> Thanks.
>>
>> J
>>
>>
>> On Sun, Mar 11, 2012 at 6:20 PM, James Baldassari <jb...@gmail.com>wrote:
>>
>>> GenericData is a singleton. You can use the static method
>>> GenericData#get() to obtain the singleton instance:
>>>
>>>
>>> http://avro.apache.org/docs/current/api/java/org/apache/avro/generic/GenericData.html#get()
>>>
>>> -James
>>>
>>>
>>> On Sun, Mar 11, 2012 at 5:22 PM, Jeremy Lewi <je...@lewi.us> wrote:
>>>
>>>> Yes. How do I instantiate an instance of GenericData though? It looks
>>>> like its constructors are protected.
>>>>
>>>> Thanks
>>>> J
>>>>
>>>>
>>>> On Sun, Mar 11, 2012 at 1:52 PM, <ea...@gmail.com> wrote:
>>>>
>>>>> There is deepcopy method on genericdata I think
>>>>>
>>>>> I used it to create deep copies of the avro data
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>> On Mar 11, 2012, at 3:40 PM, Jeremy Lewi <je...@lewi.us> wrote:
>>>>>
>>>>> > Hi,
>>>>> >
>>>>> > In java, I'd like to make a deep copy of an avro record.
>>>>> >
>>>>> > Looking at the code that the avro compiler generates it looks like
>>>>> the way to do this for records of type GraphNodeData
>>>>> > would be
>>>>> > GraphNodeData copy =
>>>>> GraphNodeData.newBuilder(existing_value).build();
>>>>> >
>>>>> > where GraphNodeData is the name of my avro record.
>>>>> >
>>>>> > Unfortunately, this generates a compile time error because
>>>>> > the method GraphNodeData.build is decorated with "@Override" but it
>>>>> apparently does not override any method.
>>>>> > Deleting "@Override" makes the code work. Unfortunately, I would
>>>>> have to do this every time I regenerated my avro classes from the schema.
>>>>> >
>>>>> > Am I doing something wrong or should I file a bug?
>>>>> >
>>>>> > Thanks
>>>>> > J
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Make a copy of an avro record
Posted by Jeremy Lewi <je...@lewi.us>.
GenericData.deepCopy gives me an instance of GenericData.Record. How do I
convert this to an instance of my specific data (e.g GraphNodeData).
It looks like the issue with the @override statement for the build might
have something to do with the fact that the specification for @Override
changed in jdk1.6 (see this post)
http://stackoverflow.com/questions/2335655/why-is-javac-failing-on-override-annotation.
However, I'm using java version 1.6.0_27 so I'm not sure why its not
working for me.
To clarify, the statement
GraphNodeData copy = GraphNodeData.newBuilder(this.data).build()
Compiles fine, but generates a runtime warning
The method build() of type GraphNodeData.Builder must override a superclass
method
J
On Sun, Mar 11, 2012 at 7:37 PM, Jeremy Lewi <je...@lewi.us> wrote:
> Thanks.
>
> J
>
>
> On Sun, Mar 11, 2012 at 6:20 PM, James Baldassari <jb...@gmail.com>wrote:
>
>> GenericData is a singleton. You can use the static method
>> GenericData#get() to obtain the singleton instance:
>>
>>
>> http://avro.apache.org/docs/current/api/java/org/apache/avro/generic/GenericData.html#get()
>>
>> -James
>>
>>
>> On Sun, Mar 11, 2012 at 5:22 PM, Jeremy Lewi <je...@lewi.us> wrote:
>>
>>> Yes. How do I instantiate an instance of GenericData though? It looks
>>> like its constructors are protected.
>>>
>>> Thanks
>>> J
>>>
>>>
>>> On Sun, Mar 11, 2012 at 1:52 PM, <ea...@gmail.com> wrote:
>>>
>>>> There is deepcopy method on genericdata I think
>>>>
>>>> I used it to create deep copies of the avro data
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Mar 11, 2012, at 3:40 PM, Jeremy Lewi <je...@lewi.us> wrote:
>>>>
>>>> > Hi,
>>>> >
>>>> > In java, I'd like to make a deep copy of an avro record.
>>>> >
>>>> > Looking at the code that the avro compiler generates it looks like
>>>> the way to do this for records of type GraphNodeData
>>>> > would be
>>>> > GraphNodeData copy = GraphNodeData.newBuilder(existing_value).build();
>>>> >
>>>> > where GraphNodeData is the name of my avro record.
>>>> >
>>>> > Unfortunately, this generates a compile time error because
>>>> > the method GraphNodeData.build is decorated with "@Override" but it
>>>> apparently does not override any method.
>>>> > Deleting "@Override" makes the code work. Unfortunately, I would have
>>>> to do this every time I regenerated my avro classes from the schema.
>>>> >
>>>> > Am I doing something wrong or should I file a bug?
>>>> >
>>>> > Thanks
>>>> > J
>>>>
>>>
>>>
>>
>
Re: Make a copy of an avro record
Posted by Jeremy Lewi <je...@lewi.us>.
Thanks.
J
On Sun, Mar 11, 2012 at 6:20 PM, James Baldassari <jb...@gmail.com>wrote:
> GenericData is a singleton. You can use the static method
> GenericData#get() to obtain the singleton instance:
>
>
> http://avro.apache.org/docs/current/api/java/org/apache/avro/generic/GenericData.html#get()
>
> -James
>
>
> On Sun, Mar 11, 2012 at 5:22 PM, Jeremy Lewi <je...@lewi.us> wrote:
>
>> Yes. How do I instantiate an instance of GenericData though? It looks
>> like its constructors are protected.
>>
>> Thanks
>> J
>>
>>
>> On Sun, Mar 11, 2012 at 1:52 PM, <ea...@gmail.com> wrote:
>>
>>> There is deepcopy method on genericdata I think
>>>
>>> I used it to create deep copies of the avro data
>>>
>>> Sent from my iPhone
>>>
>>> On Mar 11, 2012, at 3:40 PM, Jeremy Lewi <je...@lewi.us> wrote:
>>>
>>> > Hi,
>>> >
>>> > In java, I'd like to make a deep copy of an avro record.
>>> >
>>> > Looking at the code that the avro compiler generates it looks like the
>>> way to do this for records of type GraphNodeData
>>> > would be
>>> > GraphNodeData copy = GraphNodeData.newBuilder(existing_value).build();
>>> >
>>> > where GraphNodeData is the name of my avro record.
>>> >
>>> > Unfortunately, this generates a compile time error because
>>> > the method GraphNodeData.build is decorated with "@Override" but it
>>> apparently does not override any method.
>>> > Deleting "@Override" makes the code work. Unfortunately, I would have
>>> to do this every time I regenerated my avro classes from the schema.
>>> >
>>> > Am I doing something wrong or should I file a bug?
>>> >
>>> > Thanks
>>> > J
>>>
>>
>>
>
Re: Make a copy of an avro record
Posted by James Baldassari <jb...@gmail.com>.
GenericData is a singleton. You can use the static method
GenericData#get() to obtain the singleton instance:
http://avro.apache.org/docs/current/api/java/org/apache/avro/generic/GenericData.html#get()
-James
On Sun, Mar 11, 2012 at 5:22 PM, Jeremy Lewi <je...@lewi.us> wrote:
> Yes. How do I instantiate an instance of GenericData though? It looks like
> its constructors are protected.
>
> Thanks
> J
>
>
> On Sun, Mar 11, 2012 at 1:52 PM, <ea...@gmail.com> wrote:
>
>> There is deepcopy method on genericdata I think
>>
>> I used it to create deep copies of the avro data
>>
>> Sent from my iPhone
>>
>> On Mar 11, 2012, at 3:40 PM, Jeremy Lewi <je...@lewi.us> wrote:
>>
>> > Hi,
>> >
>> > In java, I'd like to make a deep copy of an avro record.
>> >
>> > Looking at the code that the avro compiler generates it looks like the
>> way to do this for records of type GraphNodeData
>> > would be
>> > GraphNodeData copy = GraphNodeData.newBuilder(existing_value).build();
>> >
>> > where GraphNodeData is the name of my avro record.
>> >
>> > Unfortunately, this generates a compile time error because
>> > the method GraphNodeData.build is decorated with "@Override" but it
>> apparently does not override any method.
>> > Deleting "@Override" makes the code work. Unfortunately, I would have
>> to do this every time I regenerated my avro classes from the schema.
>> >
>> > Am I doing something wrong or should I file a bug?
>> >
>> > Thanks
>> > J
>>
>
>
Re: Make a copy of an avro record
Posted by Jeremy Lewi <je...@lewi.us>.
Yes. How do I instantiate an instance of GenericData though? It looks like
its constructors are protected.
Thanks
J
On Sun, Mar 11, 2012 at 1:52 PM, <ea...@gmail.com> wrote:
> There is deepcopy method on genericdata I think
>
> I used it to create deep copies of the avro data
>
> Sent from my iPhone
>
> On Mar 11, 2012, at 3:40 PM, Jeremy Lewi <je...@lewi.us> wrote:
>
> > Hi,
> >
> > In java, I'd like to make a deep copy of an avro record.
> >
> > Looking at the code that the avro compiler generates it looks like the
> way to do this for records of type GraphNodeData
> > would be
> > GraphNodeData copy = GraphNodeData.newBuilder(existing_value).build();
> >
> > where GraphNodeData is the name of my avro record.
> >
> > Unfortunately, this generates a compile time error because
> > the method GraphNodeData.build is decorated with "@Override" but it
> apparently does not override any method.
> > Deleting "@Override" makes the code work. Unfortunately, I would have to
> do this every time I regenerated my avro classes from the schema.
> >
> > Am I doing something wrong or should I file a bug?
> >
> > Thanks
> > J
>
Re: Make a copy of an avro record
Posted by ea...@gmail.com.
There is deepcopy method on genericdata I think
I used it to create deep copies of the avro data
Sent from my iPhone
On Mar 11, 2012, at 3:40 PM, Jeremy Lewi <je...@lewi.us> wrote:
> Hi,
>
> In java, I'd like to make a deep copy of an avro record.
>
> Looking at the code that the avro compiler generates it looks like the way to do this for records of type GraphNodeData
> would be
> GraphNodeData copy = GraphNodeData.newBuilder(existing_value).build();
>
> where GraphNodeData is the name of my avro record.
>
> Unfortunately, this generates a compile time error because
> the method GraphNodeData.build is decorated with "@Override" but it apparently does not override any method.
> Deleting "@Override" makes the code work. Unfortunately, I would have to do this every time I regenerated my avro classes from the schema.
>
> Am I doing something wrong or should I file a bug?
>
> Thanks
> J