You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by Christophe Le Saëc <ch...@gmail.com> on 2022/11/08 08:52:48 UTC
Programming languages for Avro
Avro is currently implemented in 10 languages, this leads to some issue :
- For a new feature, it's difficult to add it everywhere.
- Difficult to ensure that each implementations have same possibilities.
- Bug fix can be hard (This JIRA
<https://issues.apache.org/jira/browse/AVRO-3591> try to solve by adding
common test system).
On the other hand, it can be hard to decide which implementation to remove;
and each developer & user would will have different advice (*Mine would be
to remove perl, php and ruby*).
May be a doodle like system would be a solution (for each language, get pro
& cons; and remove an implementation when there are more than 80% cons).
WDYT ?
Re: Programming languages for Avro
Posted by Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>.
Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>
On thu 10 nov. 2022, 19:32 Ryan Skraba <ry...@skraba.com> wrote:
> I actually met some people from the French perl Mongueurs yesterday!
> They're still going strong (http://mongueurs.pm/), and I talked to
> some engineers from a big music streaming platform still based on PHP.
> I tried to sell them on coming and contributing, and perhaps there's a
> chance we can get some help :D I would rather not release an SDK
> occasionally than consider it "dropped", which would be a discouraging
> signal.
>
Now that's unexpected (for me at least)... thanks for sharing!
Oscar
>
Re: Programming languages for Avro
Posted by Ryan Skraba <ry...@skraba.com>.
There's a couple of things we COULD talk about -- we've brought up
versioning language SDKs separately from Avro core, allowing us to
release them at a pace that corresponds to the bugs/contributions they
receive.
This adds quite a bit of complexity to the release, and we might have
difficulties reaching the requirements for Apache releases, but it
might be a fairer way to judge interest in an SDK.
I actually met some people from the French perl Mongueurs yesterday!
They're still going strong (http://mongueurs.pm/), and I talked to
some engineers from a big music streaming platform still based on PHP.
I tried to sell them on coming and contributing, and perhaps there's a
chance we can get some help :D I would rather not release an SDK
occasionally than consider it "dropped", which would be a discouraging
signal.
Abandoning Ruby isn't even on the table for discussion :D
https://rubygems.org/gems/avro/versions/1.11.1
The compatibility kit is a strong +1 from me too! At the very least,
we could get into a state where we know where behaviours diverge
across SDKs, so a new feature wouldn't necessarily be "broken" in
perl, just "not yet available".
All my best, Ryan
On Wed, Nov 9, 2022 at 10:06 AM Martin Grigorov <mg...@apache.org> wrote:
>
> Hello,
>
> I don't see a need to remove any SDK for now.
> As with any other open source project the development is driven by users'
> needs.
> IMO it is not necessary a feature to be added to all SDKs at the same time,
> i.e. the SDKs don't need to evolve in the same pace.
> Whenever a user needs some functionality (s)he can help to implement it.
>
> The bigger problem we have now is that some of the SDKs don't have active
> maintainers to review and merge users' contributions.
> For example the C# SDK received many PRs but there is no knowledgeable
> committer to approve them.
> My current approach is to merge such PRs if they are approved by at least
> two C# contributors.
>
> A TCK would be definitely helpful but again if there is no one to merge PRs
> I don't see who will implement it for the respective SDK.
>
> Martin
>
> On Tue, Nov 8, 2022 at 3:34 PM Christophe Le Saëc <ch...@gmail.com>
> wrote:
>
> > test compatibility kit : Yes, that the purpose of AVRO-3591
> > <https://issues.apache.org/jira/browse/AVRO-3591>
> > +1 for Required vs Optional.
> >
> > For choice of PHP, Perl and Ruby : it's only a "personnal subjective
> > advice" on languages i don't like; but, as i said before, with 100
> > developers, you'll get 100 different advises (that's why this kind of
> > decision is hard to take).
> >
> > Le mar. 8 nov. 2022 à 13:38, Tim Perkins <tj...@gmail.com> a écrit :
> >
> > > Hi everyone,
> > >
> > > Can I ask why you would target PHP, Perl, and Ruby specifically for
> > > removal?
> > >
> > > Thanks,
> > >
> > > -Tim
> > >
> > >
> > > On Tue, Nov 8, 2022 at 6:50 AM Oscar Westra van Holthe - Kind <
> > > oscar@westravanholthe.nl> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Feature parity is indeed very difficult, and IMHO practically
> > > impossible. I
> > > > do think however, we'll need some requirements on implementations to
> > > ensure
> > > > maintainability and a minimum level of features.
> > > >
> > > > Prerequisite for all this is a TCK (test compatibility kit) with common
> > > > schemata and a dataset (both 'readable' and encoded). This proves
> > > > interoperability, and supported features. A good test set also makes
> > > > implementing features in different languages easier.
> > > >
> > > > For maintainability we should at least require a knowledgeable
> > committer,
> > > > and perhaps also a minimum amount of changes to keep knowledge current.
> > > For
> > > > committers I happily defer to the PMCs. On changes I agree with
> > > Christophe
> > > > to remove support for PHP, Perl and Ruby.
> > > >
> > > > Below is my opinion on features (the required features are perhaps too
> > > > minimal):
> > > >
> > > > Required:
> > > >
> > > > - Parse JSON formatted schemata, supporting only the features of
> > > > the *parsing
> > > > canonical form*, plus aliases
> > > > - Schema resolution/evolution on all supported schema features
> > > > - Read/write Avro files using the binary or JSON encoding
> > > > - Encode/decode records using the single-object, (raw) binary or
> > JSON
> > > > encoding
> > > >
> > > > Optional:
> > > >
> > > > - Construct schemata using an API and/or builder pattern
> > > > - Parse JSON formatted schemata, supporting all features
> > > > - Format schemata into all formats supported for parsing
> > > > - RPC using Avro protocols
> > > > - Parse schemata / protocols from IDL or other formats
> > > > - Generate code from schemata (like SpecificRecord subclasses in
> > Java
> > > > using the Maven or Gradle plugins)
> > > > - Schema resolution/evolution on logical types with different
> > > underlying
> > > > presentations (e.g. decimals from fixed or string to bytes)
> > > > - ... all other features (like generating random data)
> > > >
> > > >
> > > >
> > > > Kind regards,
> > > > Oscar
> > > >
> > > >
> > > > On Tue, 8 Nov 2022 at 09:52, Christophe Le Saëc <ch...@gmail.com>
> > > > wrote:
> > > >
> > > > > Avro is currently implemented in 10 languages, this leads to some
> > > issue :
> > > > >
> > > > > - For a new feature, it's difficult to add it everywhere.
> > > > > - Difficult to ensure that each implementations have same
> > > > possibilities.
> > > > > - Bug fix can be hard (This JIRA
> > > > > <https://issues.apache.org/jira/browse/AVRO-3591> try to solve by
> > > > > adding
> > > > > common test system).
> > > > >
> > > > > On the other hand, it can be hard to decide which implementation to
> > > > remove;
> > > > > and each developer & user would will have different advice (*Mine
> > would
> > > > be
> > > > > to remove perl, php and ruby*).
> > > > > May be a doodle like system would be a solution (for each language,
> > get
> > > > pro
> > > > > & cons; and remove an implementation when there are more than 80%
> > > cons).
> > > > >
> > > > > WDYT ?
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > ✉️ Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>
> > > >
> > >
> >
Re: Programming languages for Avro
Posted by Martin Grigorov <mg...@apache.org>.
Hello,
I don't see a need to remove any SDK for now.
As with any other open source project the development is driven by users'
needs.
IMO it is not necessary a feature to be added to all SDKs at the same time,
i.e. the SDKs don't need to evolve in the same pace.
Whenever a user needs some functionality (s)he can help to implement it.
The bigger problem we have now is that some of the SDKs don't have active
maintainers to review and merge users' contributions.
For example the C# SDK received many PRs but there is no knowledgeable
committer to approve them.
My current approach is to merge such PRs if they are approved by at least
two C# contributors.
A TCK would be definitely helpful but again if there is no one to merge PRs
I don't see who will implement it for the respective SDK.
Martin
On Tue, Nov 8, 2022 at 3:34 PM Christophe Le Saëc <ch...@gmail.com>
wrote:
> test compatibility kit : Yes, that the purpose of AVRO-3591
> <https://issues.apache.org/jira/browse/AVRO-3591>
> +1 for Required vs Optional.
>
> For choice of PHP, Perl and Ruby : it's only a "personnal subjective
> advice" on languages i don't like; but, as i said before, with 100
> developers, you'll get 100 different advises (that's why this kind of
> decision is hard to take).
>
> Le mar. 8 nov. 2022 à 13:38, Tim Perkins <tj...@gmail.com> a écrit :
>
> > Hi everyone,
> >
> > Can I ask why you would target PHP, Perl, and Ruby specifically for
> > removal?
> >
> > Thanks,
> >
> > -Tim
> >
> >
> > On Tue, Nov 8, 2022 at 6:50 AM Oscar Westra van Holthe - Kind <
> > oscar@westravanholthe.nl> wrote:
> >
> > > Hi everyone,
> > >
> > > Feature parity is indeed very difficult, and IMHO practically
> > impossible. I
> > > do think however, we'll need some requirements on implementations to
> > ensure
> > > maintainability and a minimum level of features.
> > >
> > > Prerequisite for all this is a TCK (test compatibility kit) with common
> > > schemata and a dataset (both 'readable' and encoded). This proves
> > > interoperability, and supported features. A good test set also makes
> > > implementing features in different languages easier.
> > >
> > > For maintainability we should at least require a knowledgeable
> committer,
> > > and perhaps also a minimum amount of changes to keep knowledge current.
> > For
> > > committers I happily defer to the PMCs. On changes I agree with
> > Christophe
> > > to remove support for PHP, Perl and Ruby.
> > >
> > > Below is my opinion on features (the required features are perhaps too
> > > minimal):
> > >
> > > Required:
> > >
> > > - Parse JSON formatted schemata, supporting only the features of
> > > the *parsing
> > > canonical form*, plus aliases
> > > - Schema resolution/evolution on all supported schema features
> > > - Read/write Avro files using the binary or JSON encoding
> > > - Encode/decode records using the single-object, (raw) binary or
> JSON
> > > encoding
> > >
> > > Optional:
> > >
> > > - Construct schemata using an API and/or builder pattern
> > > - Parse JSON formatted schemata, supporting all features
> > > - Format schemata into all formats supported for parsing
> > > - RPC using Avro protocols
> > > - Parse schemata / protocols from IDL or other formats
> > > - Generate code from schemata (like SpecificRecord subclasses in
> Java
> > > using the Maven or Gradle plugins)
> > > - Schema resolution/evolution on logical types with different
> > underlying
> > > presentations (e.g. decimals from fixed or string to bytes)
> > > - ... all other features (like generating random data)
> > >
> > >
> > >
> > > Kind regards,
> > > Oscar
> > >
> > >
> > > On Tue, 8 Nov 2022 at 09:52, Christophe Le Saëc <ch...@gmail.com>
> > > wrote:
> > >
> > > > Avro is currently implemented in 10 languages, this leads to some
> > issue :
> > > >
> > > > - For a new feature, it's difficult to add it everywhere.
> > > > - Difficult to ensure that each implementations have same
> > > possibilities.
> > > > - Bug fix can be hard (This JIRA
> > > > <https://issues.apache.org/jira/browse/AVRO-3591> try to solve by
> > > > adding
> > > > common test system).
> > > >
> > > > On the other hand, it can be hard to decide which implementation to
> > > remove;
> > > > and each developer & user would will have different advice (*Mine
> would
> > > be
> > > > to remove perl, php and ruby*).
> > > > May be a doodle like system would be a solution (for each language,
> get
> > > pro
> > > > & cons; and remove an implementation when there are more than 80%
> > cons).
> > > >
> > > > WDYT ?
> > > >
> > >
> > >
> > > --
> > >
> > > ✉️ Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>
> > >
> >
>
Re: Programming languages for Avro
Posted by Christophe Le Saëc <ch...@gmail.com>.
test compatibility kit : Yes, that the purpose of AVRO-3591
<https://issues.apache.org/jira/browse/AVRO-3591>
+1 for Required vs Optional.
For choice of PHP, Perl and Ruby : it's only a "personnal subjective
advice" on languages i don't like; but, as i said before, with 100
developers, you'll get 100 different advises (that's why this kind of
decision is hard to take).
Le mar. 8 nov. 2022 à 13:38, Tim Perkins <tj...@gmail.com> a écrit :
> Hi everyone,
>
> Can I ask why you would target PHP, Perl, and Ruby specifically for
> removal?
>
> Thanks,
>
> -Tim
>
>
> On Tue, Nov 8, 2022 at 6:50 AM Oscar Westra van Holthe - Kind <
> oscar@westravanholthe.nl> wrote:
>
> > Hi everyone,
> >
> > Feature parity is indeed very difficult, and IMHO practically
> impossible. I
> > do think however, we'll need some requirements on implementations to
> ensure
> > maintainability and a minimum level of features.
> >
> > Prerequisite for all this is a TCK (test compatibility kit) with common
> > schemata and a dataset (both 'readable' and encoded). This proves
> > interoperability, and supported features. A good test set also makes
> > implementing features in different languages easier.
> >
> > For maintainability we should at least require a knowledgeable committer,
> > and perhaps also a minimum amount of changes to keep knowledge current.
> For
> > committers I happily defer to the PMCs. On changes I agree with
> Christophe
> > to remove support for PHP, Perl and Ruby.
> >
> > Below is my opinion on features (the required features are perhaps too
> > minimal):
> >
> > Required:
> >
> > - Parse JSON formatted schemata, supporting only the features of
> > the *parsing
> > canonical form*, plus aliases
> > - Schema resolution/evolution on all supported schema features
> > - Read/write Avro files using the binary or JSON encoding
> > - Encode/decode records using the single-object, (raw) binary or JSON
> > encoding
> >
> > Optional:
> >
> > - Construct schemata using an API and/or builder pattern
> > - Parse JSON formatted schemata, supporting all features
> > - Format schemata into all formats supported for parsing
> > - RPC using Avro protocols
> > - Parse schemata / protocols from IDL or other formats
> > - Generate code from schemata (like SpecificRecord subclasses in Java
> > using the Maven or Gradle plugins)
> > - Schema resolution/evolution on logical types with different
> underlying
> > presentations (e.g. decimals from fixed or string to bytes)
> > - ... all other features (like generating random data)
> >
> >
> >
> > Kind regards,
> > Oscar
> >
> >
> > On Tue, 8 Nov 2022 at 09:52, Christophe Le Saëc <ch...@gmail.com>
> > wrote:
> >
> > > Avro is currently implemented in 10 languages, this leads to some
> issue :
> > >
> > > - For a new feature, it's difficult to add it everywhere.
> > > - Difficult to ensure that each implementations have same
> > possibilities.
> > > - Bug fix can be hard (This JIRA
> > > <https://issues.apache.org/jira/browse/AVRO-3591> try to solve by
> > > adding
> > > common test system).
> > >
> > > On the other hand, it can be hard to decide which implementation to
> > remove;
> > > and each developer & user would will have different advice (*Mine would
> > be
> > > to remove perl, php and ruby*).
> > > May be a doodle like system would be a solution (for each language, get
> > pro
> > > & cons; and remove an implementation when there are more than 80%
> cons).
> > >
> > > WDYT ?
> > >
> >
> >
> > --
> >
> > ✉️ Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>
> >
>
Re: Programming languages for Avro
Posted by Tim Perkins <tj...@gmail.com>.
Hi everyone,
Can I ask why you would target PHP, Perl, and Ruby specifically for removal?
Thanks,
-Tim
On Tue, Nov 8, 2022 at 6:50 AM Oscar Westra van Holthe - Kind <
oscar@westravanholthe.nl> wrote:
> Hi everyone,
>
> Feature parity is indeed very difficult, and IMHO practically impossible. I
> do think however, we'll need some requirements on implementations to ensure
> maintainability and a minimum level of features.
>
> Prerequisite for all this is a TCK (test compatibility kit) with common
> schemata and a dataset (both 'readable' and encoded). This proves
> interoperability, and supported features. A good test set also makes
> implementing features in different languages easier.
>
> For maintainability we should at least require a knowledgeable committer,
> and perhaps also a minimum amount of changes to keep knowledge current. For
> committers I happily defer to the PMCs. On changes I agree with Christophe
> to remove support for PHP, Perl and Ruby.
>
> Below is my opinion on features (the required features are perhaps too
> minimal):
>
> Required:
>
> - Parse JSON formatted schemata, supporting only the features of
> the *parsing
> canonical form*, plus aliases
> - Schema resolution/evolution on all supported schema features
> - Read/write Avro files using the binary or JSON encoding
> - Encode/decode records using the single-object, (raw) binary or JSON
> encoding
>
> Optional:
>
> - Construct schemata using an API and/or builder pattern
> - Parse JSON formatted schemata, supporting all features
> - Format schemata into all formats supported for parsing
> - RPC using Avro protocols
> - Parse schemata / protocols from IDL or other formats
> - Generate code from schemata (like SpecificRecord subclasses in Java
> using the Maven or Gradle plugins)
> - Schema resolution/evolution on logical types with different underlying
> presentations (e.g. decimals from fixed or string to bytes)
> - ... all other features (like generating random data)
>
>
>
> Kind regards,
> Oscar
>
>
> On Tue, 8 Nov 2022 at 09:52, Christophe Le Saëc <ch...@gmail.com>
> wrote:
>
> > Avro is currently implemented in 10 languages, this leads to some issue :
> >
> > - For a new feature, it's difficult to add it everywhere.
> > - Difficult to ensure that each implementations have same
> possibilities.
> > - Bug fix can be hard (This JIRA
> > <https://issues.apache.org/jira/browse/AVRO-3591> try to solve by
> > adding
> > common test system).
> >
> > On the other hand, it can be hard to decide which implementation to
> remove;
> > and each developer & user would will have different advice (*Mine would
> be
> > to remove perl, php and ruby*).
> > May be a doodle like system would be a solution (for each language, get
> pro
> > & cons; and remove an implementation when there are more than 80% cons).
> >
> > WDYT ?
> >
>
>
> --
>
> ✉️ Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>
>
Re: Programming languages for Avro
Posted by Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>.
Hi everyone,
Feature parity is indeed very difficult, and IMHO practically impossible. I
do think however, we'll need some requirements on implementations to ensure
maintainability and a minimum level of features.
Prerequisite for all this is a TCK (test compatibility kit) with common
schemata and a dataset (both 'readable' and encoded). This proves
interoperability, and supported features. A good test set also makes
implementing features in different languages easier.
For maintainability we should at least require a knowledgeable committer,
and perhaps also a minimum amount of changes to keep knowledge current. For
committers I happily defer to the PMCs. On changes I agree with Christophe
to remove support for PHP, Perl and Ruby.
Below is my opinion on features (the required features are perhaps too
minimal):
Required:
- Parse JSON formatted schemata, supporting only the features of
the *parsing
canonical form*, plus aliases
- Schema resolution/evolution on all supported schema features
- Read/write Avro files using the binary or JSON encoding
- Encode/decode records using the single-object, (raw) binary or JSON
encoding
Optional:
- Construct schemata using an API and/or builder pattern
- Parse JSON formatted schemata, supporting all features
- Format schemata into all formats supported for parsing
- RPC using Avro protocols
- Parse schemata / protocols from IDL or other formats
- Generate code from schemata (like SpecificRecord subclasses in Java
using the Maven or Gradle plugins)
- Schema resolution/evolution on logical types with different underlying
presentations (e.g. decimals from fixed or string to bytes)
- ... all other features (like generating random data)
Kind regards,
Oscar
On Tue, 8 Nov 2022 at 09:52, Christophe Le Saëc <ch...@gmail.com> wrote:
> Avro is currently implemented in 10 languages, this leads to some issue :
>
> - For a new feature, it's difficult to add it everywhere.
> - Difficult to ensure that each implementations have same possibilities.
> - Bug fix can be hard (This JIRA
> <https://issues.apache.org/jira/browse/AVRO-3591> try to solve by
> adding
> common test system).
>
> On the other hand, it can be hard to decide which implementation to remove;
> and each developer & user would will have different advice (*Mine would be
> to remove perl, php and ruby*).
> May be a doodle like system would be a solution (for each language, get pro
> & cons; and remove an implementation when there are more than 80% cons).
>
> WDYT ?
>
--
✉️ Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>