[Rust] Proposal to move Ballista to a top-level arrow-ballista repository

[Andy Grove <an...@gmail.com>]

I would like to propose that we move the Ballista project to a new
top-level *arrow-ballista* repository.

The rationale for this (copied from the GitHub issue [1]) is:

   - Decouple release process for DataFusion and Ballista
   - Allow each project to have top-level documentation and user guides
   that are targeting the appropriate audience
   - Reduce issue tracking and PR review burden for DataFusion maintainers
   who are not as interested in Ballista
   - Help avoid accidental circular dependencies being introduced between
   the projects (such as
   - datafusion-cli crate has circular dependency #2433
   <https://github.com/apache/arrow-datafusion/issues/2433>)
   - Helps formalize the public API for DataFusion that other query engines
   should be using

There is also a design document [2] where we will be discussing the finer
details of this and coordinating on the plan to implement.

I do not recall if a change of this nature requires a formal vote or not
but I will plan on holding one before we create the new repo unless anyone
tells me this is not required.

Thanks,

Andy.

[1] https://github.com/apache/arrow-datafusion/issues/2502
[2]
https://docs.google.com/document/d/1jNRbadyStSrV5kifwn0khufAwq6OnzGczG4z8oTQJP4/edit?usp=sharing

Re: [Rust] Proposal to move Ballista to a top-level arrow-ballista repository

[Andy Grove <an...@gmail.com>]

I have updated the proposal, especially around how we use CI to avoid
breaking compatibility between DataFusion and Ballista, based on the
earlier discussions in the document.

I think that it would be ideal to put this plan into action as soon as
possible after the DataFusion 8.0.0 release so would like to see if there
is support for the latest proposal.

Thanks,

Andy.

On Wed, May 11, 2022 at 8:17 AM Andy Grove <an...@gmail.com> wrote:

> I would like to propose that we move the Ballista project to a new
> top-level *arrow-ballista* repository.
>
> The rationale for this (copied from the GitHub issue [1]) is:
>
>    - Decouple release process for DataFusion and Ballista
>    - Allow each project to have top-level documentation and user guides
>    that are targeting the appropriate audience
>    - Reduce issue tracking and PR review burden for DataFusion
>    maintainers who are not as interested in Ballista
>    - Help avoid accidental circular dependencies being introduced between
>    the projects (such as
>    - datafusion-cli crate has circular dependency #2433
>    <https://github.com/apache/arrow-datafusion/issues/2433>)
>    - Helps formalize the public API for DataFusion that other query
>    engines should be using
>
> There is also a design document [2] where we will be discussing the finer
> details of this and coordinating on the plan to implement.
>
> I do not recall if a change of this nature requires a formal vote or not
> but I will plan on holding one before we create the new repo unless anyone
> tells me this is not required.
>
> Thanks,
>
> Andy.
>
> [1] https://github.com/apache/arrow-datafusion/issues/2502
> [2]
> https://docs.google.com/document/d/1jNRbadyStSrV5kifwn0khufAwq6OnzGczG4z8oTQJP4/edit?usp=sharing
>