You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@xtable.apache.org by Jesus Camacho Rodriguez <jc...@apache.org> on 2024/09/13 03:47:25 UTC
[DISCUSS] Excluding xtable-utilities from the release process
Hello everyone,
First, I'd like to thank Vinish for driving our first release as an ASF
project, which is often the most challenging one.
Unfortunately, during the recent release vote on the incubator mailing
list, licensing issues in the xtable-utilities module were found [1].
Honestly, I did not check licensing that carefully in our first release
because I assumed the WIP disclaimer [2] would allow us more flexibility to
address these issues in future releases. However, it appears that was not
the case, as discussed in the mailing list.
We need to address these licensing concerns for the xtable-utilities
artifact for our release, but this requires considerable effort; Vinish has
outlined the plan in this GitHub issue [3].
One suggestion provided in the thread was to exclude the xtable-utilities
bundle, at least from the initial releases, to simplify the process.
I'm +1 on excluding it from the first releases, but I would also like to
discuss more broadly whether we should consider including it in future
releases in its current form or whether we should explore other options.
The bundle is quite large (over 600MB) and includes many artifacts, which
means that any changes in dependency versions in the future will require
close monitoring for licensing issues. I'd appreciate input from others,
especially if they have experience in similar situations, on how they think
we should proceed.
Thanks,
Jesús
[1] https://lists.apache.org/thread/pcs3c7dq6bljzxmwlcyrwns9ndbfd23g
[2] https://github.com/apache/incubator-xtable/blob/main/DISCLAIMER-WIP
[3] https://github.com/apache/incubator-xtable/issues/536
Re: [DISCUSS] Excluding xtable-utilities from the release process
Posted by Vinish Reddy <vi...@apache.org>.
Hey everyone.
We have fixed all the release blockers for 0.2.0 and excluded the utilities bundle from the release command for mvn deploy and source artifacts. I am planning to start rc-1 for 0.2.0 tomorrow, any other release blockers or ASF naming conventions/practices the community thinks we need to address ?
-Vinish
On 2024/09/17 14:03:28 Vinish Reddy wrote:
> Thanks for the feedback. I have pushed few changes for removing xtable-utilities and updating LICENSE file for xtable.
> https://github.com/apache/incubator-xtable/pull/540
> https://github.com/apache/incubator-xtable/pull/541
>
> I have also compiled the list of dependencies for each module (Dependency, Dependency Type Dependency Scope, License, Category, ASF Compliant) in this sheet, please review if anything still needs to be added in LICENSE file.
> https://docs.google.com/spreadsheets/d/1XBCZBeWqF2D5d1L4H9fRQ1QLarJq4tLgIEgnGiNdoV8/edit?usp=sharing
>
> Next steps,
> 1. Release 0.2.0 after addressing feedback and merging PR's mentioned above.
> 2. Exclude category X dependencies from xtable-utilities and reduce the size of bundled jar.
> 3. Release 0.3.0 with the goal of moving from DISCLAIMER-WIP to DISCLAIMER.
>
> Thanks
> Vinish
>
> On 2024/09/13 16:27:58 Vinoth Chandar wrote:
> > Thanks everyone for jumping on this. Truly appreciate the complexity in
> > getting these things right, given how Apache processes/instructions can be
> > vaguely defined at best, in many places.
> >
> > +1 on removing the bundled jar and pushing out another release right away.
> >
> > On the bundled jar etc, many projects including Iceberg provide these
> > convenience jars for users. Asking users to build it themselves for XTable,
> > where we expect it be added to data pipeline runtimes, will be hard.
> >
> > From hudi experience, we moved to a model of specifically "whitelisting"
> > jars (vs bundling every dependency), carefully understanding what are
> > already provided by different compute engines for e.g .
> > That reduced the overall jar size, that seems to have caught everyone's
> > imagination, as well as reduce the overall exposure to licenses, ensure we
> > never whitelist a non-permissible licence etc.
> > I suggest we work towards that approach..
> >
> > Cheers,
> > Vinoth
> >
> >
> > On Fri, Sep 13, 2024 at 8:02 AM Stamatis Zampetakis <za...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > From a maven release perspective, I don't think there is significant
> > > benefit in publishing a fat/shaded jar of xtable-utilities. If we drop
> > > shading, the xtable-utilities jars should be rather minimal without
> > > special requirements for licensing. XTable consumers that want to use
> > > the xtable-utilities via maven will automatically get all transitive
> > > dependencies via maven's dependency resolution mechanism.
> > >
> > > From the user's perspective, a fat jar makes sense since it greatly
> > > simplifies execution using a simple java command. An alternative way
> > > to provide simplicity to the users is to gather dependencies in a lib
> > > directory during the build (using maven-assembly-plugin,
> > > maven-dependency-plugin, etc.) and instruct the users to run a script
> > > (usually in a bin directory) which is mostly there to manage the
> > > classpath. The main advantage of this approach is that 3-party
> > > binaries will not be published/released as Apache XTable code.
> > >
> > > In all cases we need to verify the licenses of our dependencies but if
> > > we don't mingle 3rd party binaries in Apache binaries things are
> > > significantly easier to document.
> > >
> > > Lastly there are various maven plugins (such as
> > > https://www.mojohaus.org/license-maven-plugin/) that can aid the
> > > LICENSE documentation and checks but from the moment that the project
> > > does not perform binary releases this is not strictly necessary.
> > >
> > > Best,
> > > Stamatis
> > >
> > >
> > > On Fri, Sep 13, 2024 at 5:47 AM Jesus Camacho Rodriguez
> > > <jc...@apache.org> wrote:
> > > >
> > > > Hello everyone,
> > > >
> > > > First, I'd like to thank Vinish for driving our first release as an ASF
> > > > project, which is often the most challenging one.
> > > >
> > > > Unfortunately, during the recent release vote on the incubator mailing
> > > > list, licensing issues in the xtable-utilities module were found [1].
> > > > Honestly, I did not check licensing that carefully in our first release
> > > > because I assumed the WIP disclaimer [2] would allow us more flexibility
> > > to
> > > > address these issues in future releases. However, it appears that was not
> > > > the case, as discussed in the mailing list.
> > > >
> > > > We need to address these licensing concerns for the xtable-utilities
> > > > artifact for our release, but this requires considerable effort; Vinish
> > > has
> > > > outlined the plan in this GitHub issue [3].
> > > > One suggestion provided in the thread was to exclude the xtable-utilities
> > > > bundle, at least from the initial releases, to simplify the process.
> > > >
> > > > I'm +1 on excluding it from the first releases, but I would also like to
> > > > discuss more broadly whether we should consider including it in future
> > > > releases in its current form or whether we should explore other options.
> > > > The bundle is quite large (over 600MB) and includes many artifacts, which
> > > > means that any changes in dependency versions in the future will require
> > > > close monitoring for licensing issues. I'd appreciate input from others,
> > > > especially if they have experience in similar situations, on how they
> > > think
> > > > we should proceed.
> > > >
> > > > Thanks,
> > > > Jesús
> > > >
> > > > [1] https://lists.apache.org/thread/pcs3c7dq6bljzxmwlcyrwns9ndbfd23g
> > > > [2] https://github.com/apache/incubator-xtable/blob/main/DISCLAIMER-WIP
> > > > [3] https://github.com/apache/incubator-xtable/issues/536
> > >
> >
>
Re: [DISCUSS] Excluding xtable-utilities from the release process
Posted by Vinish Reddy <vi...@apache.org>.
Thanks for the feedback. I have pushed few changes for removing xtable-utilities and updating LICENSE file for xtable.
https://github.com/apache/incubator-xtable/pull/540
https://github.com/apache/incubator-xtable/pull/541
I have also compiled the list of dependencies for each module (Dependency, Dependency Type Dependency Scope, License, Category, ASF Compliant) in this sheet, please review if anything still needs to be added in LICENSE file.
https://docs.google.com/spreadsheets/d/1XBCZBeWqF2D5d1L4H9fRQ1QLarJq4tLgIEgnGiNdoV8/edit?usp=sharing
Next steps,
1. Release 0.2.0 after addressing feedback and merging PR's mentioned above.
2. Exclude category X dependencies from xtable-utilities and reduce the size of bundled jar.
3. Release 0.3.0 with the goal of moving from DISCLAIMER-WIP to DISCLAIMER.
Thanks
Vinish
On 2024/09/13 16:27:58 Vinoth Chandar wrote:
> Thanks everyone for jumping on this. Truly appreciate the complexity in
> getting these things right, given how Apache processes/instructions can be
> vaguely defined at best, in many places.
>
> +1 on removing the bundled jar and pushing out another release right away.
>
> On the bundled jar etc, many projects including Iceberg provide these
> convenience jars for users. Asking users to build it themselves for XTable,
> where we expect it be added to data pipeline runtimes, will be hard.
>
> From hudi experience, we moved to a model of specifically "whitelisting"
> jars (vs bundling every dependency), carefully understanding what are
> already provided by different compute engines for e.g .
> That reduced the overall jar size, that seems to have caught everyone's
> imagination, as well as reduce the overall exposure to licenses, ensure we
> never whitelist a non-permissible licence etc.
> I suggest we work towards that approach..
>
> Cheers,
> Vinoth
>
>
> On Fri, Sep 13, 2024 at 8:02 AM Stamatis Zampetakis <za...@gmail.com>
> wrote:
>
> > Hello,
> >
> > From a maven release perspective, I don't think there is significant
> > benefit in publishing a fat/shaded jar of xtable-utilities. If we drop
> > shading, the xtable-utilities jars should be rather minimal without
> > special requirements for licensing. XTable consumers that want to use
> > the xtable-utilities via maven will automatically get all transitive
> > dependencies via maven's dependency resolution mechanism.
> >
> > From the user's perspective, a fat jar makes sense since it greatly
> > simplifies execution using a simple java command. An alternative way
> > to provide simplicity to the users is to gather dependencies in a lib
> > directory during the build (using maven-assembly-plugin,
> > maven-dependency-plugin, etc.) and instruct the users to run a script
> > (usually in a bin directory) which is mostly there to manage the
> > classpath. The main advantage of this approach is that 3-party
> > binaries will not be published/released as Apache XTable code.
> >
> > In all cases we need to verify the licenses of our dependencies but if
> > we don't mingle 3rd party binaries in Apache binaries things are
> > significantly easier to document.
> >
> > Lastly there are various maven plugins (such as
> > https://www.mojohaus.org/license-maven-plugin/) that can aid the
> > LICENSE documentation and checks but from the moment that the project
> > does not perform binary releases this is not strictly necessary.
> >
> > Best,
> > Stamatis
> >
> >
> > On Fri, Sep 13, 2024 at 5:47 AM Jesus Camacho Rodriguez
> > <jc...@apache.org> wrote:
> > >
> > > Hello everyone,
> > >
> > > First, I'd like to thank Vinish for driving our first release as an ASF
> > > project, which is often the most challenging one.
> > >
> > > Unfortunately, during the recent release vote on the incubator mailing
> > > list, licensing issues in the xtable-utilities module were found [1].
> > > Honestly, I did not check licensing that carefully in our first release
> > > because I assumed the WIP disclaimer [2] would allow us more flexibility
> > to
> > > address these issues in future releases. However, it appears that was not
> > > the case, as discussed in the mailing list.
> > >
> > > We need to address these licensing concerns for the xtable-utilities
> > > artifact for our release, but this requires considerable effort; Vinish
> > has
> > > outlined the plan in this GitHub issue [3].
> > > One suggestion provided in the thread was to exclude the xtable-utilities
> > > bundle, at least from the initial releases, to simplify the process.
> > >
> > > I'm +1 on excluding it from the first releases, but I would also like to
> > > discuss more broadly whether we should consider including it in future
> > > releases in its current form or whether we should explore other options.
> > > The bundle is quite large (over 600MB) and includes many artifacts, which
> > > means that any changes in dependency versions in the future will require
> > > close monitoring for licensing issues. I'd appreciate input from others,
> > > especially if they have experience in similar situations, on how they
> > think
> > > we should proceed.
> > >
> > > Thanks,
> > > Jesús
> > >
> > > [1] https://lists.apache.org/thread/pcs3c7dq6bljzxmwlcyrwns9ndbfd23g
> > > [2] https://github.com/apache/incubator-xtable/blob/main/DISCLAIMER-WIP
> > > [3] https://github.com/apache/incubator-xtable/issues/536
> >
>
Re: [DISCUSS] Excluding xtable-utilities from the release process
Posted by Vinoth Chandar <vi...@apache.org>.
Thanks everyone for jumping on this. Truly appreciate the complexity in
getting these things right, given how Apache processes/instructions can be
vaguely defined at best, in many places.
+1 on removing the bundled jar and pushing out another release right away.
On the bundled jar etc, many projects including Iceberg provide these
convenience jars for users. Asking users to build it themselves for XTable,
where we expect it be added to data pipeline runtimes, will be hard.
From hudi experience, we moved to a model of specifically "whitelisting"
jars (vs bundling every dependency), carefully understanding what are
already provided by different compute engines for e.g .
That reduced the overall jar size, that seems to have caught everyone's
imagination, as well as reduce the overall exposure to licenses, ensure we
never whitelist a non-permissible licence etc.
I suggest we work towards that approach..
Cheers,
Vinoth
On Fri, Sep 13, 2024 at 8:02 AM Stamatis Zampetakis <za...@gmail.com>
wrote:
> Hello,
>
> From a maven release perspective, I don't think there is significant
> benefit in publishing a fat/shaded jar of xtable-utilities. If we drop
> shading, the xtable-utilities jars should be rather minimal without
> special requirements for licensing. XTable consumers that want to use
> the xtable-utilities via maven will automatically get all transitive
> dependencies via maven's dependency resolution mechanism.
>
> From the user's perspective, a fat jar makes sense since it greatly
> simplifies execution using a simple java command. An alternative way
> to provide simplicity to the users is to gather dependencies in a lib
> directory during the build (using maven-assembly-plugin,
> maven-dependency-plugin, etc.) and instruct the users to run a script
> (usually in a bin directory) which is mostly there to manage the
> classpath. The main advantage of this approach is that 3-party
> binaries will not be published/released as Apache XTable code.
>
> In all cases we need to verify the licenses of our dependencies but if
> we don't mingle 3rd party binaries in Apache binaries things are
> significantly easier to document.
>
> Lastly there are various maven plugins (such as
> https://www.mojohaus.org/license-maven-plugin/) that can aid the
> LICENSE documentation and checks but from the moment that the project
> does not perform binary releases this is not strictly necessary.
>
> Best,
> Stamatis
>
>
> On Fri, Sep 13, 2024 at 5:47 AM Jesus Camacho Rodriguez
> <jc...@apache.org> wrote:
> >
> > Hello everyone,
> >
> > First, I'd like to thank Vinish for driving our first release as an ASF
> > project, which is often the most challenging one.
> >
> > Unfortunately, during the recent release vote on the incubator mailing
> > list, licensing issues in the xtable-utilities module were found [1].
> > Honestly, I did not check licensing that carefully in our first release
> > because I assumed the WIP disclaimer [2] would allow us more flexibility
> to
> > address these issues in future releases. However, it appears that was not
> > the case, as discussed in the mailing list.
> >
> > We need to address these licensing concerns for the xtable-utilities
> > artifact for our release, but this requires considerable effort; Vinish
> has
> > outlined the plan in this GitHub issue [3].
> > One suggestion provided in the thread was to exclude the xtable-utilities
> > bundle, at least from the initial releases, to simplify the process.
> >
> > I'm +1 on excluding it from the first releases, but I would also like to
> > discuss more broadly whether we should consider including it in future
> > releases in its current form or whether we should explore other options.
> > The bundle is quite large (over 600MB) and includes many artifacts, which
> > means that any changes in dependency versions in the future will require
> > close monitoring for licensing issues. I'd appreciate input from others,
> > especially if they have experience in similar situations, on how they
> think
> > we should proceed.
> >
> > Thanks,
> > Jesús
> >
> > [1] https://lists.apache.org/thread/pcs3c7dq6bljzxmwlcyrwns9ndbfd23g
> > [2] https://github.com/apache/incubator-xtable/blob/main/DISCLAIMER-WIP
> > [3] https://github.com/apache/incubator-xtable/issues/536
>
Re: [DISCUSS] Excluding xtable-utilities from the release process
Posted by Stamatis Zampetakis <za...@gmail.com>.
Hello,
From a maven release perspective, I don't think there is significant
benefit in publishing a fat/shaded jar of xtable-utilities. If we drop
shading, the xtable-utilities jars should be rather minimal without
special requirements for licensing. XTable consumers that want to use
the xtable-utilities via maven will automatically get all transitive
dependencies via maven's dependency resolution mechanism.
From the user's perspective, a fat jar makes sense since it greatly
simplifies execution using a simple java command. An alternative way
to provide simplicity to the users is to gather dependencies in a lib
directory during the build (using maven-assembly-plugin,
maven-dependency-plugin, etc.) and instruct the users to run a script
(usually in a bin directory) which is mostly there to manage the
classpath. The main advantage of this approach is that 3-party
binaries will not be published/released as Apache XTable code.
In all cases we need to verify the licenses of our dependencies but if
we don't mingle 3rd party binaries in Apache binaries things are
significantly easier to document.
Lastly there are various maven plugins (such as
https://www.mojohaus.org/license-maven-plugin/) that can aid the
LICENSE documentation and checks but from the moment that the project
does not perform binary releases this is not strictly necessary.
Best,
Stamatis
On Fri, Sep 13, 2024 at 5:47 AM Jesus Camacho Rodriguez
<jc...@apache.org> wrote:
>
> Hello everyone,
>
> First, I'd like to thank Vinish for driving our first release as an ASF
> project, which is often the most challenging one.
>
> Unfortunately, during the recent release vote on the incubator mailing
> list, licensing issues in the xtable-utilities module were found [1].
> Honestly, I did not check licensing that carefully in our first release
> because I assumed the WIP disclaimer [2] would allow us more flexibility to
> address these issues in future releases. However, it appears that was not
> the case, as discussed in the mailing list.
>
> We need to address these licensing concerns for the xtable-utilities
> artifact for our release, but this requires considerable effort; Vinish has
> outlined the plan in this GitHub issue [3].
> One suggestion provided in the thread was to exclude the xtable-utilities
> bundle, at least from the initial releases, to simplify the process.
>
> I'm +1 on excluding it from the first releases, but I would also like to
> discuss more broadly whether we should consider including it in future
> releases in its current form or whether we should explore other options.
> The bundle is quite large (over 600MB) and includes many artifacts, which
> means that any changes in dependency versions in the future will require
> close monitoring for licensing issues. I'd appreciate input from others,
> especially if they have experience in similar situations, on how they think
> we should proceed.
>
> Thanks,
> Jesús
>
> [1] https://lists.apache.org/thread/pcs3c7dq6bljzxmwlcyrwns9ndbfd23g
> [2] https://github.com/apache/incubator-xtable/blob/main/DISCLAIMER-WIP
> [3] https://github.com/apache/incubator-xtable/issues/536