Tests for compatibility with Avro 1.8 and 1.9

[Piotr Szuberski <pi...@polidea.com>]

I'd like to add tests verifying that Beam is compatible with both Avro 1.8 and 1.9 similar to what has been done to Hadoop and Kafka.

Probably all Avro dependencies would have to be changed from compile to provided - won't it be problematic for users? They will be broken after the update unless they add Avro dependency. On the other hand they'll be able to choose which version do they prefer.

At the moment Beam doesn't work with Avro 1.10 so users will be resticted to use either 1.8 or 1.9.

Does changing Avro dependencies to provided sounds reasonable? Are there particular modules that should not be changed? Or is there a better approach?

Re: Tests for compatibility with Avro 1.8 and 1.9

[Ismaël Mejía <ie...@gmail.com>]

After some offline discussion with Piotr we discovered two issues:

1. The gradle avro plugin we use needs a specific version of Avro in
each of his versions, so we would need to use different versions of
the plugin to generate the Avro objects for our tests with each
version, because the generated objects are not compatible if they have
dates on it (because of the removal of joda-time).

2. There is an Avro test class we depend on to generate random data in
our tests that it is in a different package in recent versions of Avro
so we will probably need to detect this or move the class into Beam
(maintenance should not be hard for that class).

Both annoyances happen also with Avro 1.9, at least it seems that once
we have compatibility with 1.9 we will get 1.10 'for free'.

On Thu, Dec 3, 2020 at 6:29 PM Brian Hulette <bh...@google.com> wrote:
>
>
>
> On Thu, Dec 3, 2020 at 1:02 AM Piotr Szuberski <pi...@polidea.com> wrote:
>>
>> > A softer approach would be to let it as it is (1.8) and document
>> > explicitly that we check upwards compatibility with 1.9 and suggest
>> > users to explicitly override the version if required.
>>
>> Ok, thanks! I think it's the better option.
>>
>>
>> > I have not followed your work on the compatibility tests but I am
>> > curious what is the issue with Avro 1.10?
>>
>> AFAIR Avro 1.10 completely removes the support for joda time and for now Beam makes use of both Avro time interfaces (one of which is removed in 1.10)
>
>
> Could you file a jira for this (if we don't have one already)?

Re: Tests for compatibility with Avro 1.8 and 1.9

[Brian Hulette <bh...@google.com>]

On Thu, Dec 3, 2020 at 1:02 AM Piotr Szuberski <pi...@polidea.com>
wrote:

> > A softer approach would be to let it as it is (1.8) and document
> > explicitly that we check upwards compatibility with 1.9 and suggest
> > users to explicitly override the version if required.
>
> Ok, thanks! I think it's the better option.
>
>
> > I have not followed your work on the compatibility tests but I am
> > curious what is the issue with Avro 1.10?
>
> AFAIR Avro 1.10 completely removes the support for joda time and for now
> Beam makes use of both Avro time interfaces (one of which is removed in
> 1.10)
>

Could you file a jira for this (if we don't have one already)?

Re: Tests for compatibility with Avro 1.8 and 1.9

[Piotr Szuberski <pi...@polidea.com>]

> A softer approach would be to let it as it is (1.8) and document
> explicitly that we check upwards compatibility with 1.9 and suggest
> users to explicitly override the version if required.

Ok, thanks! I think it's the better option.


> I have not followed your work on the compatibility tests but I am
> curious what is the issue with Avro 1.10?

AFAIR Avro 1.10 completely removes the support for joda time and for now Beam makes use of both Avro time interfaces (one of which is removed in 1.10)

Re: Tests for compatibility with Avro 1.8 and 1.9

[Ismaël Mejía <ie...@gmail.com>]

Changing dependency to provided is a breaking change for
sdks-java-core because all users of Beam would have to explicitly pass
their Avro version once they upgrade their Beam version. That's not so
nice but maybe is the intended behavior.

A softer approach would be to let it as it is (1.8) and document
explicitly that we check upwards compatibility with 1.9 and suggest
users to explicitly override the version if required.

I have not followed your work on the compatibility tests but I am
curious what is the issue with Avro 1.10?


On Tue, Dec 1, 2020 at 5:23 PM Piotr Szuberski
<pi...@polidea.com> wrote:
>
> I'd like to add tests verifying that Beam is compatible with both Avro 1.8 and 1.9 similar to what has been done to Hadoop and Kafka.
>
> Probably all Avro dependencies would have to be changed from compile to provided - won't it be problematic for users? They will be broken after the update unless they add Avro dependency. On the other hand they'll be able to choose which version do they prefer.
>
> At the moment Beam doesn't work with Avro 1.10 so users will be resticted to use either 1.8 or 1.9.
>
> Does changing Avro dependencies to provided sounds reasonable? Are there particular modules that should not be changed? Or is there a better approach?