Re: Status of Arrow Julia implementation?

[Jacob Quinn <qu...@gmail.com>]

Hi Kou,

Sorry for the slow response here, but it's been great to see how the new
Rust process has shaken out and I think it working well. I'd like to move
forward with transferring the JuliaData/Arrow.jl repository to
apache/arrow-julia and following a similar process to Rust in terms of
development/release. I can start on a Julia-specific proposal with
specifics.

Thanks for all the help!

-Jacob

On Sun, Apr 25, 2021 at 11:56 PM Sutou Kouhei <ko...@clear-code.com> wrote:

> Hi,
>
> I think that we can say the Rust migration is complete once
> we merge https://github.com/apache/arrow/pull/10096. But
> it's a good time to think about the Julia migration.
>
> Jacob, here is the Rust's new development process:
>
>
> https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit#
>
> (It seems that an anonymous user deleted a part of it
> accidentally.)
>
> Do you want to use the same development process as the
> Rust's one? Do you have any item you want to change?
>
>
> Thanks,
> --
> kou
>
> In <CA...@mail.gmail.com>
>   "Re: Status of Arrow Julia implementation?" on Sun, 25 Apr 2021 13:34:04
> -0700,
>   Micah Kornfield <em...@gmail.com> wrote:
>
> > It seems the Rust migration is now complete.  Do we want to wait to iron
> > out the other potential issues?
> >
> > I think the outstanding ones might be:
> > 1.  Issue management
> > 2.  Integration testing
> >
> > -Micah
> >
> > On Wed, Apr 14, 2021 at 11:01 PM Sutou Kouhei <ko...@clear-code.com>
> wrote:
> >
> >> Hi Jacob,
> >>
> >> OK. Here is my plan:
> >>
> >>   1. We wait for the Rust's move to complete
> >>   2. We use a process similar to the Rust's move
> >>
> >>
> >> Thanks,
> >> --
> >> kou
> >>
> >> In <CA...@mail.gmail.com>
> >>   "Re: Status of Arrow Julia implementation?" on Wed, 14 Apr 2021
> 08:37:41
> >> -0600,
> >>   Jacob Quinn <qu...@gmail.com> wrote:
> >>
> >> > Thank you kou! I appreciate the help. I'm happy to do whatever is
> >> required
> >> > to facilitate the moving/donating process from JuliaData/Arrow.jl to
> >> > apache/arrow-julia.
> >> >
> >> > -Jacob
> >> >
> >> > On Mon, Apr 12, 2021 at 7:53 PM Sutou Kouhei <ko...@clear-code.com>
> wrote:
> >> >
> >> >> Hi Jacob,
> >> >>
> >> >> I, a PMC member, talked to Kenta Murata, a commiter and a
> >> >> Julia user, about this.
> >> >>
> >> >> We support that you and Julia folks work on
> >> >> arrow/arrow-julia until we have enough PMC members from
> >> >> Julia folks. For example, we'll help IP clearance process to
> >> >> import the latest JuliaData/Arrow.js changes to apache/ and
> >> >> we'll start voting on Julia package release.
> >> >>
> >> >>
> >> >> Thanks,
> >> >> --
> >> >> kou
> >> >>
> >> >> In <CAKyXBQo2=+NfS=hX6nEDPmMno-bSuXMR+=
> d-hEsKgN2mm574iQ@mail.gmail.com>
> >> >>   "Re: Status of Arrow Julia implementation?" on Sun, 11 Apr 2021
> >> 23:06:27
> >> >> -0600,
> >> >>   Jacob Quinn <qu...@gmail.com> wrote:
> >> >>
> >> >> > Micah/Wes,
> >> >> >
> >> >> > Yes, I've been following the rust proposal thread with great
> >> interest. I
> >> >> do
> >> >> > think that provides a great path forward: transferring the
> >> >> > JuliaData/Arrow.jl repo to apache/arrow-julia would help to solve
> the
> >> >> > "package history" technical challenges that in part led to the
> current
> >> >> > setup and concerns. I think being able to utilize github issues
> would
> >> >> also
> >> >> > be great; as I've mentioned elsewhere, it's much more
> >> >> traditional/expected
> >> >> > in the Julia ecosystem.
> >> >> >
> >> >> > I think the package could retain an independent versioning scheme.
> The
> >> >> >> additional process would be voting on release candidates. If the
> >> Julia
> >> >> >> folks want to try again and move development to a new,
> Julia-specific
> >> >> >> apache/* repository and apply the ASF governance to the project,
> the
> >> >> >> Arrow PMC could probably fast-track making Jacob a committer. In
> some
> >> >> >> code donations / IP clearance, the contributors for the donated
> code
> >> >> >> become committers as part of the transaction.
> >> >> >>
> >> >> >
> >> >> > These all sound great and would greatly facilitate a better
> >> integration
> >> >> > under ASF governance. These points definitely resolve my main
> >> concerns.
> >> >> >
> >> >> > As I commented on the rust thread, I'm mostly interested in the
> >> future of
> >> >> > integration testing for rust/julia if they are split out into
> separate
> >> >> > repos. In the current Julia implementation, we have all the code to
> >> read
> >> >> > arrow json, and I just hand-generated the integration test data and
> >> >> > committed them in the repo itself, but it doesn't interface with
> other
> >> >> > languages (just reads arrow json, produces arrow file, reads arrow
> >> file,
> >> >> > compares w/ original arrow json). I'm happy to help work on the
> >> details
> >> >> of
> >> >> > what that looks like and pilot some solutions. I think with a solid
> >> >> > inter-repo integration testing framework, we can keep a strong sync
> >> >> between
> >> >> > projects.
> >> >> >
> >> >> > -Jacob
> >> >> >
> >> >> >
> >> >> > On Sun, Apr 11, 2021 at 5:08 PM Wes McKinney <we...@gmail.com>
> >> >> wrote:
> >> >> >
> >> >> >> On Sat, Apr 10, 2021 at 4:07 PM Micah Kornfield <
> >> emkornfield@gmail.com>
> >> >> >> wrote:
> >> >> >> >
> >> >> >> > >
> >> >> >> > > Ok, I've had a chance to discuss with a few other Julia
> >> developers
> >> >> and
> >> >> >> > > review various options. I think it's best to drop the Julia
> code
> >> >> from
> >> >> >> the
> >> >> >> > > physical apache/arrow repo. The extra overhead on development,
> >> >> release
> >> >> >> > > process, and user issue reporting and PR contributing are too
> >> much
> >> >> in
> >> >> >> > > addition to the technical challenges that we never resolved
> >> >> involving
> >> >> >> > > including the past Arrow.jl release version git trees in the
> >> >> >> apache/arrow
> >> >> >> > > repo.
> >> >> >> >
> >> >> >> >
> >> >> >> > Hi Jacob,
> >> >> >> > It seems you are on the new thread discussing a proposal for
> >> changing
> >> >> >> > Rust's development model.   Would the proposal [1] address most
> of
> >> >> these
> >> >> >> > concerns if Julia was set up in the same way?
> >> >> >> >
> >> >> >> >  It seems in the short term the stickiest point would be
> committer
> >> >> access
> >> >> >> > to the new repos, and I suppose the release mechanics still
> might
> >> be
> >> >> >> > challenging?
> >> >> >>
> >> >> >> I think the package could retain an independent versioning scheme.
> >> The
> >> >> >> additional process would be voting on release candidates. If the
> >> Julia
> >> >> >> folks want to try again and move development to a new,
> Julia-specific
> >> >> >> apache/* repository and apply the ASF governance to the project,
> the
> >> >> >> Arrow PMC could probably fast-track making Jacob a committer. In
> some
> >> >> >> code donations / IP clearance, the contributors for the donated
> code
> >> >> >> become committers as part of the transaction.
> >> >> >>
> >> >> >> >
> >> >> >> > Thanks,
> >> >> >> > Micah
> >> >> >> >
> >> >> >> > [1]
> >> >> >> >
> >> >> >>
> >> >>
> >>
> https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit
> >> >> >> >
> >> >> >> > On Wed, Apr 7, 2021 at 4:17 AM Wes McKinney <
> wesmckinn@gmail.com>
> >> >> wrote:
> >> >> >> >
> >> >> >> > > I went back and read the mailing list discussions from
> September
> >> >> about
> >> >> >> > > the donation and I would say there was not a clear enough
> >> statement
> >> >> >> > > from us about what the donation and IP clearance meant as far
> as
> >> the
> >> >> >> > > future of the Julia codebase. This is partly our fault — we
> have
> >> >> taken
> >> >> >> > > in 9 other code donations over the last 5 years, and in all
> cases
> >> >> the
> >> >> >> > > developers understood that they were to move their process to
> the
> >> >> >> > > Arrow repositories and communications channels.
> >> >> >> > >
> >> >> >> > > It did not occur to me at all that the code that you were
> >> putting in
> >> >> >> > > the Arrow repository would get treated like a read-only fork
> that
> >> >> you
> >> >> >> > > update periodically. If I had realized that, we wouldn't be in
> >> this
> >> >> >> > > situation.
> >> >> >> > >
> >> >> >> > > As a reminder about what Arrow and the ASF are all about:
> >> Community
> >> >> >> > > over Code. We think that building a collaborative, open
> community
> >> >> that
> >> >> >> > > works and plans together in public, makes decisions based on
> >> >> consensus
> >> >> >> > > with clear meritocratic ("doers decide") governance is the
> best
> >> way
> >> >> to
> >> >> >> > > build this project. The concerns that you have around the
> timing
> >> and
> >> >> >> > > frequency of releases for the Julia codebase are in my mind
> easy
> >> to
> >> >> >> > > resolve, and if you had indicated that having a customized
> >> process
> >> >> for
> >> >> >> > > Julia releases was a condition for your joining the community
> >> >> >> > > wholeheartedly, we would have been happy to help. I think that
> >> the
> >> >> >> > > benefits of common CI/CD infrastructure and opportunities to
> >> build
> >> >> >> > > deeper integrations between the Julia implementation and the
> >> other
> >> >> >> > > implementations (imagine... Julia kernels running in
> DataFusion?)
> >> >> >> > > would outweigh the sense of "loss of control" from developing
> >> >> within a
> >> >> >> > > larger project.
> >> >> >> > >
> >> >> >> > > On Wed, Apr 7, 2021 at 12:16 AM Jacob Quinn <
> >> quinn.jacobd@gmail.com
> >> >> >
> >> >> >> > > wrote:
> >> >> >> > > >
> >> >> >> > > > Responses inline below:
> >> >> >> > > >
> >> >> >> > > > On Tue, Apr 6, 2021 at 9:46 PM Jorge Cardoso Leitão <
> >> >> >> > > > jorgecarleitao@gmail.com> wrote:
> >> >> >> > > >
> >> >> >> > > > > Hi,
> >> >> >> > > > >
> >> >> >> > > > > > you all did not attempt to work in the community for any
> >> >> >> meaningful
> >> >> >> > > > > amount of time and
> >> >> >> > > > > are choosing not to try based on the perception that it
> will
> >> >> create
> >> >> >> > > > > unacceptable overhead for you
> >> >> >> > > > >
> >> >> >> > > > > It is not self-evident to me that Julia's community was
> >> >> >> sufficiently
> >> >> >> > > > > informed about what they
> >> >> >> > > > > had to give in in terms of process and release management
> >> when
> >> >> >> merging
> >> >> >> > > /
> >> >> >> > > > > donating.
> >> >> >> > > > >
> >> >> >> > > >
> >> >> >> > > > Yes, it was pretty unclear what the process was if we needed
> >> to do
> >> >> >> any
> >> >> >> > > kind
> >> >> >> > > > of patch release. I know that has been sorted out better
> >> recently,
> >> >> >> but
> >> >> >> > > back
> >> >> >> > > > in November, it didn't really seem like an option (i.e.
> >> >> independent
> >> >> >> > > > language patch releases).
> >> >> >> > > >
> >> >> >> > > >
> >> >> >> > > > > IMO this is a plausible explanation as to why the donation
> >> was
> >> >> >> made and
> >> >> >> > > > > then later abandoned.
> >> >> >> > > > >
> >> >> >> > > > >
> >> >> >> > > > I'll just note that the "abandonment" can only be a
> perception
> >> >> from
> >> >> >> the
> >> >> >> > > > apache/arrow side of things, but as I mentioned above, I
> also
> >> >> tried
> >> >> >> to
> >> >> >> > > > clearly state in the julia/Arrow/README that the development
> >> >> process
> >> >> >> > > would
> >> >> >> > > > continue with the JuliaData/Arrow.jl repo as the main "dev"
> >> >> branch,
> >> >> >> with
> >> >> >> > > > changes being upstreamed to the apache/arrow repo, which was
> >> >> followed
> >> >> >> > > > through, having an upstream of commits right before the
> 3.0.0
> >> >> >> release,
> >> >> >> > > and
> >> >> >> > > > I was planning on doing the same soon for the 4.0.0 release.
> >> That
> >> >> is
> >> >> >> to
> >> >> >> > > > say, the Julia implementation has continued progressing
> forward
> >> >> quite
> >> >> >> > > > rapidly, IMO, but I can see that perhaps apache/arrow repo
> >> members
> >> >> >> may
> >> >> >> > > have
> >> >> >> > > > viewed it as "abandoned".
> >> >> >> > > >
> >> >> >> > > >
> >> >> >> > > > > I do not fully understand why the pain points Jacob
> mentioned
> >> >> were
> >> >> >> not
> >> >> >> > > > > brought up to the mailing list sooner, though.
> >> >> >> > > > >
> >> >> >> > > >
> >> >> >> > > > To be honest and frank, I didn't have pain points with the
> >> >> >> development
> >> >> >> > > > process I outlined when the code was donated and as stated
> in
> >> the
> >> >> >> README.
> >> >> >> > > > That was the process that made the donation possible and I
> >> >> imagined
> >> >> >> would
> >> >> >> > > > work well going forward, and has, until this thread started
> >> and it
> >> >> >> was
> >> >> >> > > > pointed out that this process isn't viable. The pain points
> >> were
> >> >> >> > > discussed
> >> >> >> > > > with the initial code donation, but in my mind were resolved
> >> with
> >> >> the
> >> >> >> > > > development process that was decided upon.
> >> >> >> > > >
> >> >> >> > > >
> >> >> >> > > > > This made us unable to potentially take corrective
> measures.
> >> I
> >> >> >> think
> >> >> >> > > that
> >> >> >> > > > > this is why everyone was taken a bit by surprise with
> this.
> >> >> >> > > > >
> >> >> >> > > > > Best,
> >> >> >> > > > > Jorge
> >> >> >> > > > >
> >> >> >> > > > >
> >> >> >> > > > > On Fri, Apr 2, 2021 at 10:18 PM Wes McKinney <
> >> >> wesmckinn@gmail.com>
> >> >> >> > > wrote:
> >> >> >> > > > >
> >> >> >> > > > > > hi Jacob — sorry to hear that. It's a bummer that you
> all
> >> did
> >> >> not
> >> >> >> > > > > > attempt to work in the community for any meaningful
> amount
> >> of
> >> >> >> time
> >> >> >> > > and
> >> >> >> > > > > > are choosing not to try based on the perception that it
> >> will
> >> >> >> create
> >> >> >> > > > > > unacceptable overhead for you. I believe the benefits
> would
> >> >> >> outweigh
> >> >> >> > > > > > the costs, but I suppose we will have to agree to
> disagree.
> >> >> >> > > > > >
> >> >> >> > > > > > Can you prepare a pull request to do the requisite
> >> repository
> >> >> >> > > surgery?
> >> >> >> > > > > > I hope the development goes well in the future and look
> >> >> forward
> >> >> >> to
> >> >> >> > > > > > seeing folks from the Julia ecosystem engaged here on
> >> growing
> >> >> the
> >> >> >> > > > > > Arrow ecosystem.
> >> >> >> > > > > >
> >> >> >> > > > > > Thanks,
> >> >> >> > > > > > Wes
> >> >> >> > > > > >
> >> >> >> > > > > > On Fri, Apr 2, 2021 at 3:03 PM Jacob Quinn <
> >> >> >> quinn.jacobd@gmail.com>
> >> >> >> > > > > wrote:
> >> >> >> > > > > > >
> >> >> >> > > > > > > Ok, I've had a chance to discuss with a few other
> Julia
> >> >> >> developers
> >> >> >> > > and
> >> >> >> > > > > > > review various options. I think it's best to drop the
> >> Julia
> >> >> >> code
> >> >> >> > > from
> >> >> >> > > > > the
> >> >> >> > > > > > > physical apache/arrow repo. The extra overhead on
> >> >> development,
> >> >> >> > > release
> >> >> >> > > > > > > process, and user issue reporting and PR contributing
> are
> >> >> too
> >> >> >> much
> >> >> >> > > in
> >> >> >> > > > > > > addition to the technical challenges that we never
> >> resolved
> >> >> >> > > involving
> >> >> >> > > > > > > including the past Arrow.jl release version git trees
> in
> >> the
> >> >> >> > > > > apache/arrow
> >> >> >> > > > > > > repo.
> >> >> >> > > > > > >
> >> >> >> > > > > > > We're still very much committed to working on the
> Julia
> >> >> >> > > implementation
> >> >> >> > > > > > and
> >> >> >> > > > > > > participating in the broader arrow community. I've
> >> enjoyed
> >> >> >> > > following
> >> >> >> > > > > the
> >> >> >> > > > > > > user/dev mailing lists and will continue to do so. We
> >> >> monitor
> >> >> >> > > format
> >> >> >> > > > > > > proposals and try to implement new functionality as
> >> quickly
> >> >> as
> >> >> >> > > > > possible.
> >> >> >> > > > > > We
> >> >> >> > > > > > > got the initial arrow flight proto code generated just
> >> last
> >> >> >> night
> >> >> >> > > in
> >> >> >> > > > > > fact.
> >> >> >> > > > > > > I'd still like to explore official integration with
> the
> >> >> archery
> >> >> >> > > test
> >> >> >> > > > > > suite
> >> >> >> > > > > > > to solidify the Julia implementation with integration
> >> >> tests; I
> >> >> >> > > think
> >> >> >> > > > > that
> >> >> >> > > > > > > would be very valuable for long-term confidence in the
> >> >> >> > > cross-language
> >> >> >> > > > > > > support of the Julia implementation.
> >> >> >> > > > > > >
> >> >> >> > > > > > > We realize one of the main implications will probably
> be
> >> >> >> dropping
> >> >> >> > > Julia
> >> >> >> > > > > > > from the list of "official implementations". We're
> >> >> encouraged
> >> >> >> by
> >> >> >> > > the
> >> >> >> > > > > many
> >> >> >> > > > > > > users who have already started using the Julia
> >> >> implementation
> >> >> >> and
> >> >> >> > > will
> >> >> >> > > > > > > strive to maintain a high rate of issue responsiveness
> >> and
> >> >> >> feature
> >> >> >> > > > > > > development to maintain project confidence. If
> there's a
> >> >> >> > > possibility of
> >> >> >> > > > > > > being included somewhere as an "unofficial" or
> >> >> "semi-official"
> >> >> >> > > > > > > implementation, we'd love to still be bundled with the
> >> >> broader
> >> >> >> > > arrow
> >> >> >> > > > > > > project somehow, like, for example, showing how Julia
> >> >> >> integrates
> >> >> >> > > with
> >> >> >> > > > > the
> >> >> >> > > > > > > archery test suite, once the work there is done.
> >> >> >> > > > > > >
> >> >> >> > > > > > > Best,
> >> >> >> > > > > > >
> >> >> >> > > > > > > -Jacob
> >> >> >> > > > > > >
> >> >> >> > > > > > >
> >> >> >> > > > > > >
> >> >> >> > > > > > > On Tue, Mar 30, 2021 at 4:10 PM Wes McKinney <
> >> >> >> wesmckinn@gmail.com>
> >> >> >> > > > > > wrote:
> >> >> >> > > > > > >
> >> >> >> > > > > > > > Also, on the issue that there are no Julia-focused
> PMC
> >> >> >> members —
> >> >> >> > > note
> >> >> >> > > > > > > > that I helped the JavaScript folks make their own
> >> >> independent
> >> >> >> > > > > releases
> >> >> >> > > > > > > > for quite a while: called the votes (e.g. [1]),
> helped
> >> get
> >> >> >> > > people to
> >> >> >> > > > > > > > verify and vote on the releases. After a time, it
> was
> >> >> >> decided to
> >> >> >> > > stop
> >> >> >> > > > > > > > releasing independently because there wasn't enough
> >> >> >> development
> >> >> >> > > > > > > > activity to justify it.
> >> >> >> > > > > > > >
> >> >> >> > > > > > > > [1]:
> >> >> >> > >
> https://www.mail-archive.com/dev@arrow.apache.org/msg05971.html
> >> >> >> > > > > > > >
> >> >> >> > > > > > > > On Tue, Mar 30, 2021 at 4:54 PM Wes McKinney <
> >> >> >> > > wesmckinn@gmail.com>
> >> >> >> > > > > > wrote:
> >> >> >> > > > > > > > >
> >> >> >> > > > > > > > > hi Jacob,
> >> >> >> > > > > > > > >
> >> >> >> > > > > > > > > On Tue, Mar 30, 2021 at 4:18 PM Jacob Quinn <
> >> >> >> > > > > quinn.jacobd@gmail.com>
> >> >> >> > > > > > > > wrote:
> >> >> >> > > > > > > > > >
> >> >> >> > > > > > > > > > I can comment as the primary apache arrow
> liaison
> >> for
> >> >> the
> >> >> >> > > > > Arrow.jl
> >> >> >> > > > > > > > > > repository and original code donator.
> >> >> >> > > > > > > > > >
> >> >> >> > > > > > > > > > I apologize for the "surprise", but I commented
> a
> >> few
> >> >> >> times
> >> >> >> > > in
> >> >> >> > > > > > various
> >> >> >> > > > > > > > > > places and put a snippet in the README
> >> >> >> > > > > > > > > > <
> >> >> >> > > > > > > >
> >> >> >> > > > > >
> >> >> >> > > > >
> >> >> >> > >
> >> >> >>
> >> >>
> >>
> https://github.com/apache/arrow/tree/master/julia/Arrow#difference-between-this-code-and-the-juliadataarrowjl-repository
> >> >> >> > > > > > > > >
> >> >> >> > > > > > > > > > about
> >> >> >> > > > > > > > > > the approach I wanted to take w/ the Julia
> >> >> >> implementation in
> >> >> >> > > > > terms
> >> >> >> > > > > > of
> >> >> >> > > > > > > > > > keeping the JuliaData/Arrow.jl repository as a
> "dev
> >> >> >> branch"
> >> >> >> > > of
> >> >> >> > > > > > sorts
> >> >> >> > > > > > > > of the
> >> >> >> > > > > > > > > > apache/arrow code, upstreaming changes
> >> periodically.
> >> >> >> There's
> >> >> >> > > > > even a
> >> >> >> > > > > > > > script
> >> >> >> > > > > > > > > > <
> >> >> >> > > > > > > >
> >> >> >> > > > > >
> >> >> >> > > > >
> >> >> >> > >
> >> >> >>
> >> >>
> >>
> https://github.com/JuliaData/Arrow.jl/blob/main/scripts/update_apache_arrow_code.jl
> >> >> >> > > > > > > > >
> >> >> >> > > > > > > > > > I wrote to mostly automate this upstreaming. I
> >> realize
> >> >> >> now
> >> >> >> > > that I
> >> >> >> > > > > > > > didn't
> >> >> >> > > > > > > > > > consider the "Arrow PMC" position on this kind
> of
> >> >> setup
> >> >> >> or
> >> >> >> > > seek
> >> >> >> > > > > to
> >> >> >> > > > > > > > affirm
> >> >> >> > > > > > > > > > that it would be ok to approach things like
> this.
> >> >> >> > > > > > > > > >
> >> >> >> > > > > > > > > > The reality is that Julia users are very
> engrained
> >> to
> >> >> >> expect
> >> >> >> > > > > Julia
> >> >> >> > > > > > > > packages
> >> >> >> > > > > > > > > > to live in a single stand-alone github repo,
> where
> >> >> issues
> >> >> >> > > can be
> >> >> >> > > > > > > > opened,
> >> >> >> > > > > > > > > > and pull requests are welcome. It was hard and
> >> still
> >> >> is
> >> >> >> hard
> >> >> >> > > to
> >> >> >> > > > > > imagine
> >> >> >> > > > > > > > > > "turning that off", since I believe we would
> lose a
> >> >> lot
> >> >> >> of
> >> >> >> > > > > > valuable bug
> >> >> >> > > > > > > > > > reports and first-time contributions. This isn't
> >> >> >> necessarily
> >> >> >> > > any
> >> >> >> > > > > > fault
> >> >> >> > > > > > > > of
> >> >> >> > > > > > > > > > how the bug report/contribution process is
> handled
> >> for
> >> >> >> the
> >> >> >> > > arrow
> >> >> >> > > > > > > > project
> >> >> >> > > > > > > > > > overall, though I'm also aware that there's a
> >> desire
> >> >> to
> >> >> >> make
> >> >> >> > > it
> >> >> >> > > > > > easier
> >> >> >> > > > > > > > > >
> >> >> >> > > > > > > > > >
> >> >> >> > > > > > > > > <
> >> >> >> > > > > > > >
> >> >> >> > > > > >
> >> >> >> > > > >
> >> >> >> > >
> >> >> >>
> >> >>
> >>
> https://lists.apache.org/x/thread.html/r8817dfba08ef8daa210956db69d513fd27b7a751d28fb8f27e39cc7e@%3Cdev.arrow.apache.org%3E
> >> >> >> > > > > > > > >
> >> >> >> > > > > > > > > > and
> >> >> >> > > > > > > > > > it currently requires more and different effort
> >> than
> >> >> >> Julia
> >> >> >> > > users
> >> >> >> > > > > > are
> >> >> >> > > > > > > > used
> >> >> >> > > > > > > > > > to. I think it's more from how open, welcoming,
> and
> >> >> how
> >> >> >> > > strong
> >> >> >> > > > > the
> >> >> >> > > > > > > > culture
> >> >> >> > > > > > > > > > is in Julia around encouraging community
> >> contributions
> >> >> >> and
> >> >> >> > > the
> >> >> >> > > > > > tight
> >> >> >> > > > > > > > > > integration with github and its open-source
> project
> >> >> >> > > management
> >> >> >> > > > > > tools.
> >> >> >> > > > > > > > > >
> >> >> >> > > > > > > > >
> >> >> >> > > > > > > > > Well, we are on track to having 1000 different
> people
> >> >> >> > > contribute to
> >> >> >> > > > > > > > > the project and have over 12,000 issues, so I
> don't
> >> >> think
> >> >> >> > > there is
> >> >> >> > > > > > > > > evidence that we are failing to attract new
> >> >> contributors or
> >> >> >> > > that
> >> >> >> > > > > > > > > feature requests / bugs aren't being reported. The
> >> way
> >> >> >> that we
> >> >> >> > > work
> >> >> >> > > > > > is
> >> >> >> > > > > > > > > _different_, so adapting to the Apache process
> will
> >> >> require
> >> >> >> > > change.
> >> >> >> > > > > > > > >
> >> >> >> > > > > > > > > > Additionally, I was and still am concerned about
> >> the
> >> >> >> overall
> >> >> >> > > > > > release
> >> >> >> > > > > > > > > > process of the apache/arrow project. I know
> there
> >> have
> >> >> >> been
> >> >> >> > > > > efforts
> >> >> >> > > > > > > > there
> >> >> >> > > > > > > > > > as well to make it easier for individual
> languages
> >> to
> >> >> >> > > release on
> >> >> >> > > > > > their
> >> >> >> > > > > > > > own
> >> >> >> > > > > > > > > > cadence, but just anecdotally, the
> >> JuliaData/Arrow.jl
> >> >> has
> >> >> >> > > > > > > > had/needed/wanted
> >> >> >> > > > > > > > > > 10 patch and minor releases since the original
> code
> >> >> >> donation,
> >> >> >> > > > > > whereas
> >> >> >> > > > > > > > the
> >> >> >> > > > > > > > > > apache/arrow project has had one (3.0.0). This
> >> leads
> >> >> to
> >> >> >> some
> >> >> >> > > of
> >> >> >> > > > > the
> >> >> >> > > > > > > > > > concerns I have with restricting development to
> >> just
> >> >> the
> >> >> >> > > > > > apache/arrow
> >> >> >> > > > > > > > > > repository: how exactly does the release process
> >> work
> >> >> for
> >> >> >> > > > > > individual
> >> >> >> > > > > > > > > > languages who may desire independent releases
> apart
> >> >> from
> >> >> >> the
> >> >> >> > > > > > quarterly
> >> >> >> > > > > > > > > > overall project releases? I think from the Rust
> >> >> thread I
> >> >> >> > > remember
> >> >> >> > > > > > that
> >> >> >> > > > > > > > you
> >> >> >> > > > > > > > > > just need a group of language contributors to
> all
> >> >> agree,
> >> >> >> but
> >> >> >> > > what
> >> >> >> > > > > > if
> >> >> >> > > > > > > > I'm
> >> >> >> > > > > > > > > > the only "active" Julia contributor? It's also
> >> unclear
> >> >> >> what
> >> >> >> > > the
> >> >> >> > > > > > > > > > expectations are for actual development: with
> the
> >> >> >> original
> >> >> >> > > code
> >> >> >> > > > > > > > donation
> >> >> >> > > > > > > > > > PRs, I know Neal "reviewed" the PRs, but perhaps
> >> >> missed
> >> >> >> the
> >> >> >> > > > > details
> >> >> >> > > > > > > > around
> >> >> >> > > > > > > > > > how I proposed development continue going
> forward.
> >> Is
> >> >> it
> >> >> >> > > required
> >> >> >> > > > > > to
> >> >> >> > > > > > > > have a
> >> >> >> > > > > > > > > > certain number of reviews before merging? On the
> >> Julia
> >> >> >> side,
> >> >> >> > > I
> >> >> >> > > > > can
> >> >> >> > > > > > try
> >> >> >> > > > > > > > to
> >> >> >> > > > > > > > > > encourage/push for those who have contributed to
> >> the
> >> >> >> > > > > > JuliaData/Arrow.jl
> >> >> >> > > > > > > > > > repository to help review PRs to apache/arrow,
> but
> >> I
> >> >> also
> >> >> >> > > can't
> >> >> >> > > > > > > > guarantee
> >> >> >> > > > > > > > > > we would always have someone to review. It just
> >> feels
> >> >> >> pretty
> >> >> >> > > > > > awkward
> >> >> >> > > > > > > > if I
> >> >> >> > > > > > > > > > keep needing to ping non-Julia people to
> "review" a
> >> >> PR to
> >> >> >> > > merge
> >> >> >> > > > > it.
> >> >> >> > > > > > > > Perhaps
> >> >> >> > > > > > > > > > this is just a problem of the overall Julia
> >> >> >> implementation
> >> >> >> > > > > > "smallness"
> >> >> >> > > > > > > > in
> >> >> >> > > > > > > > > > terms of contributors, but I'm not sure on the
> best
> >> >> >> answer
> >> >> >> > > here.
> >> >> >> > > > > > > > > >
> >> >> >> > > > > > > > >
> >> >> >> > > > > > > > > Several things here:
> >> >> >> > > > > > > > >
> >> >> >> > > > > > > > > * If you want to do separate Julia releases, you
> are
> >> >> free
> >> >> >> to do
> >> >> >> > > > > that,
> >> >> >> > > > > > > > > but you have to follow the process (voting on the
> >> >> mailing
> >> >> >> list,
> >> >> >> > > > > > > > > publishing GPG-signed source artifacts)
> >> >> >> > > > > > > > > * If you had been working "in the community" since
> >> >> >> November,
> >> >> >> > > you
> >> >> >> > > > > > would
> >> >> >> > > > > > > > > probably already be a committer, so there is a
> >> >> >> bootstrapping
> >> >> >> > > here
> >> >> >> > > > > > that
> >> >> >> > > > > > > > > has failed to take place. In the meantime, we are
> >> more
> >> >> than
> >> >> >> > > happy
> >> >> >> > > > > to
> >> >> >> > > > > > > > > help you "earn your wings" (as a committer) as
> >> quickly
> >> >> as
> >> >> >> > > possible.
> >> >> >> > > > > > > > > But from my perspective, I see a code donation and
> >> two
> >> >> >> other
> >> >> >> > > > > commits,
> >> >> >> > > > > > > > > which isn't enough to make a case for
> committership.
> >> >> >> > > > > > > > >
> >> >> >> > > > > > > > > > So in short, I'm not sure on the best path
> >> forward. I
> >> >> >> think
> >> >> >> > > > > > strictly
> >> >> >> > > > > > > > > > restricting development to the apache/arrow
> >> physical
> >> >> >> > > repository
> >> >> >> > > > > > would
> >> >> >> > > > > > > > > > actively hurt the progress of the Julia
> >> >> implementation,
> >> >> >> > > whereas
> >> >> >> > > > > it
> >> >> >> > > > > > > > *has*
> >> >> >> > > > > > > > > > been progressing with increasing momentum since
> >> first
> >> >> >> > > released.
> >> >> >> > > > > > There
> >> >> >> > > > > > > > are
> >> >> >> > > > > > > > > > posts on the Julia discourse forum, in the Julia
> >> slack
> >> >> >> and
> >> >> >> > > zulip
> >> >> >> > > > > > > > > > communities, and quite a few issues/PRs being
> >> opened
> >> >> at
> >> >> >> the
> >> >> >> > > > > > > > > > JuliaData/Arrow.jl repository. There have been
> >> several
> >> >> >> calls
> >> >> >> > > for
> >> >> >> > > > > > arrow
> >> >> >> > > > > > > > > > flight support, with a member from Julia
> Computing
> >> >> >> actually
> >> >> >> > > close
> >> >> >> > > > > > to
> >> >> >> > > > > > > > > > releasing a gRPC client
> >> >> >> > > > > > > > > > <
> https://github.com/JuliaComputing/gRPCClient.jl>
> >> >> >> > > specifically
> >> >> >> > > > > > > > > > to help with flight support. But in terms of
> actual
> >> >> >> > > committers,
> >> >> >> > > > > > it's
> >> >> >> > > > > > > > been
> >> >> >> > > > > > > > > > primarily just myself, with a few minor
> >> contributions
> >> >> by
> >> >> >> > > others.
> >> >> >> > > > > > > > > >
> >> >> >> > > > > > > > > > I guess the big question that comes to mind is
> what
> >> >> are
> >> >> >> the
> >> >> >> > > hard
> >> >> >> > > > > > > > > > requirements to be considered an "official
> >> >> >> implementation"?
> >> >> >> > > Does
> >> >> >> > > > > > the
> >> >> >> > > > > > > > code
> >> >> >> > > > > > > > > > *have* to live in the same physical repo? Or if
> it
> >> >> >> passed the
> >> >> >> > > > > > series of
> >> >> >> > > > > > > > > > archery integration tests, would that be
> enough? I
> >> >> >> apologize
> >> >> >> > > for
> >> >> >> > > > > my
> >> >> >> > > > > > > > > > naivete/inexperience on all things "apache",
> but I
> >> >> >> imagine
> >> >> >> > > that's
> >> >> >> > > > > > a big
> >> >> >> > > > > > > > > > part of it: having official development/releases
> >> >> through
> >> >> >> the
> >> >> >> > > > > > > > apache/arrow
> >> >> >> > > > > > > > > > community, though again I'm not exactly sure on
> the
> >> >> >> formal
> >> >> >> > > > > > processes
> >> >> >> > > > > > > > here?
> >> >> >> > > > > > > > > > I would like to keep Julia as an official
> >> >> implementation,
> >> >> >> > > but I'm
> >> >> >> > > > > > also
> >> >> >> > > > > > > > > > mostly carrying the maintainership alone at the
> >> moment
> >> >> >> and
> >> >> >> > > want
> >> >> >> > > > > to
> >> >> >> > > > > > be
> >> >> >> > > > > > > > > > realistic with the future of the project.
> >> >> >> > > > > > > > > >
> >> >> >> > > > > > > > >
> >> >> >> > > > > > > > > The critical matter is whether the
> >> >> development/maintenance
> >> >> >> > > work is
> >> >> >> > > > > > > > > conducted by the "Arrow community" in accordance
> with
> >> >> the
> >> >> >> > > Apache
> >> >> >> > > > > Way,
> >> >> >> > > > > > > > > which is to say individuals collaborating with
> each
> >> >> other
> >> >> >> on
> >> >> >> > > Apache
> >> >> >> > > > > > > > > channels (for communication and development) and
> >> >> avoiding
> >> >> >> the
> >> >> >> > > bad
> >> >> >> > > > > > > > > patterns you see sometimes in other communities
> (e.g.
> >> >> >> > > inconsistent
> >> >> >> > > > > > > > > openness).
> >> >> >> > > > > > > > >
> >> >> >> > > > > > > > > It's fine — really, no pressure — if you want to
> be
> >> >> >> > > independent and
> >> >> >> > > > > > do
> >> >> >> > > > > > > > > things your own way, you just have to be clear
> that
> >> you
> >> >> are
> >> >> >> > > > > > > > > independent and not operating as part of the
> Apache
> >> >> Arrow
> >> >> >> > > > > community.
> >> >> >> > > > > > > > > You can't have it both ways, though. No hard
> feelings
> >> >> >> whatever
> >> >> >> > > you
> >> >> >> > > > > > > > > decide, but the current "dump code over the wall
> >> >> >> occasionally"
> >> >> >> > > > > > > > > approach but work on independent channels is not
> >> >> >> compatible.
> >> >> >> > > > > Building
> >> >> >> > > > > > > > > healthy open source communities is hard, but this
> way
> >> >> has
> >> >> >> been
> >> >> >> > > > > shown
> >> >> >> > > > > > > > > to work well, which is why I've spent the last 6
> >> years
> >> >> >> working
> >> >> >> > > hard
> >> >> >> > > > > > to
> >> >> >> > > > > > > > > bring people together to build this project and
> >> >> ecosystem!
> >> >> >> > > > > > > > >
> >> >> >> > > > > > > > > If you want to maintain a test harness here to
> >> verify an
> >> >> >> > > > > independent
> >> >> >> > > > > > > > > Julia implementation, that's fine, too. I'm
> >> disappointed
> >> >> >> that
> >> >> >> > > > > things
> >> >> >> > > > > > > > > failed to bootstrap after the code donation, so I
> >> want
> >> >> to
> >> >> >> see
> >> >> >> > > if we
> >> >> >> > > > > > > > > can course correct quickly or if not decide to go
> our
> >> >> >> separate
> >> >> >> > > > > ways.
> >> >> >> > > > > > > > >
> >> >> >> > > > > > > > > Thanks,
> >> >> >> > > > > > > > > Wes
> >> >> >> > > > > > > > >
> >> >> >> > > > > > > > > > I'm open to discussion and ideas on the best way
> >> >> forward.
> >> >> >> > > > > > > > > >
> >> >> >> > > > > > > > > > -Jacob
> >> >> >> > > > > > > > > >
> >> >> >> > > > > > > > > > On Tue, Mar 30, 2021 at 2:03 PM Wes McKinney <
> >> >> >> > > > > wesmckinn@gmail.com>
> >> >> >> > > > > > > > wrote:
> >> >> >> > > > > > > > > >
> >> >> >> > > > > > > > > > > hi folks,
> >> >> >> > > > > > > > > > >
> >> >> >> > > > > > > > > > > I was very surprised today to learn that the
> >> Julia
> >> >> >> Arrow
> >> >> >> > > > > > > > > > > implementation has continued operating more or
> >> less
> >> >> >> like an
> >> >> >> > > > > > > > > > > independent open source project since the code
> >> >> donation
> >> >> >> > > last
> >> >> >> > > > > > > > November:
> >> >> >> > > > > > > > > > >
> >> >> >> > > > > > > > > > >
> >> https://github.com/JuliaData/Arrow.jl/commits/main
> >> >> >> > > > > > > > > > >
> >> >> >> > > > > > > > > > > There may have been a misunderstanding about
> what
> >> >> was
> >> >> >> > > expected
> >> >> >> > > > > to
> >> >> >> > > > > > > > > > > occur after the code donation, but it's
> >> problematic
> >> >> >> for a
> >> >> >> > > bunch
> >> >> >> > > > > > of
> >> >> >> > > > > > > > > > > reasons (IP lineage / governance / community
> >> >> >> development)
> >> >> >> > > to
> >> >> >> > > > > have
> >> >> >> > > > > > > > work
> >> >> >> > > > > > > > > > > happening on the implementation "outside the
> >> >> >> community".
> >> >> >> > > > > > > > > > >
> >> >> >> > > > > > > > > > > In any case, what is done is done, so the
> Arrow
> >> >> PMC's
> >> >> >> > > position
> >> >> >> > > > > on
> >> >> >> > > > > > > > this
> >> >> >> > > > > > > > > > > would be roughly to regard the work as a hard
> >> fork
> >> >> of
> >> >> >> > > what's in
> >> >> >> > > > > > > > Apache
> >> >> >> > > > > > > > > > > Arrow, which given its development activity is
> >> more
> >> >> or
> >> >> >> less
> >> >> >> > > > > > inactive
> >> >> >> > > > > > > > > > > [1]. (I had actually thought the project was
> >> simply
> >> >> >> > > inactive
> >> >> >> > > > > > after
> >> >> >> > > > > > > > the
> >> >> >> > > > > > > > > > > code donation)
> >> >> >> > > > > > > > > > >
> >> >> >> > > > > > > > > > > The critical question now is, is there
> interest
> >> from
> >> >> >> Julia
> >> >> >> > > > > > developers
> >> >> >> > > > > > > > > > > in working "in the community", which is to
> say:
> >> >> >> > > > > > > > > > >
> >> >> >> > > > > > > > > > > * Having development discussions on ASF
> channels
> >> >> >> (mailing
> >> >> >> > > list,
> >> >> >> > > > > > > > > > > GitHub, JIRA), planning and communicating in
> the
> >> >> open
> >> >> >> > > > > > > > > > > * Doing all development in ASF GitHub
> >> repositories
> >> >> >> > > > > > > > > > >
> >> >> >> > > > > > > > > > > The answer to the question may be "no" (which
> is
> >> >> okay),
> >> >> >> > > but if
> >> >> >> > > > > > that's
> >> >> >> > > > > > > > > > > the case, I don't think we should be giving
> the
> >> >> >> impression
> >> >> >> > > that
> >> >> >> > > > > > we
> >> >> >> > > > > > > > > > > have an official Julia implementation that is
> >> >> >> developed and
> >> >> >> > > > > > > > maintained
> >> >> >> > > > > > > > > > > by the community (and so my argument would be
> >> >> >> > > unfortunately to
> >> >> >> > > > > > drop
> >> >> >> > > > > > > > > > > the donated code from the project).
> >> >> >> > > > > > > > > > >
> >> >> >> > > > > > > > > > > If the answer is "yes", there needs to be a
> hard
> >> >> >> > > commitment to
> >> >> >> > > > > > move
> >> >> >> > > > > > > > > > > development to Apache channels and not look
> >> back. We
> >> >> >> would
> >> >> >> > > also
> >> >> >> > > > > > need
> >> >> >> > > > > > > > > > > to figure out what to do to document and
> >> synchronize
> >> >> >> the
> >> >> >> > > new IP
> >> >> >> > > > > > > > that's
> >> >> >> > > > > > > > > > > been created since the code donation.
> >> >> >> > > > > > > > > > >
> >> >> >> > > > > > > > > > > Thanks,
> >> >> >> > > > > > > > > > > Wes
> >> >> >> > > > > > > > > > >
> >> >> >> > > > > > > > > > > [1]:
> >> >> >> > > > >
> https://github.com/apache/arrow/commits/master/julia/Arrow
> >> >> >> > > > > > > > > > >
> >> >> >> > > > > > > >
> >> >> >> > > > > >
> >> >> >> > > > >
> >> >> >> > >
> >> >> >>
> >> >>
> >>
>

Re: Status of Arrow Julia implementation?

[Wes McKinney <we...@gmail.com>]

hi Jacob — that's great to hear. We're standing by to help you out with this.

On Fri, Jun 25, 2021 at 4:57 PM Jacob Quinn <qu...@gmail.com> wrote:
>
> Hi Kou,
>
> Sorry for the slow response here, but it's been great to see how the new
> Rust process has shaken out and I think it working well. I'd like to move
> forward with transferring the JuliaData/Arrow.jl repository to
> apache/arrow-julia and following a similar process to Rust in terms of
> development/release. I can start on a Julia-specific proposal with
> specifics.
>
> Thanks for all the help!
>
> -Jacob
>
> On Sun, Apr 25, 2021 at 11:56 PM Sutou Kouhei <ko...@clear-code.com> wrote:
>
> > Hi,
> >
> > I think that we can say the Rust migration is complete once
> > we merge https://github.com/apache/arrow/pull/10096. But
> > it's a good time to think about the Julia migration.
> >
> > Jacob, here is the Rust's new development process:
> >
> >
> > https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit#
> >
> > (It seems that an anonymous user deleted a part of it
> > accidentally.)
> >
> > Do you want to use the same development process as the
> > Rust's one? Do you have any item you want to change?
> >
> >
> > Thanks,
> > --
> > kou
> >
> > In <CA...@mail.gmail.com>
> >   "Re: Status of Arrow Julia implementation?" on Sun, 25 Apr 2021 13:34:04
> > -0700,
> >   Micah Kornfield <em...@gmail.com> wrote:
> >
> > > It seems the Rust migration is now complete.  Do we want to wait to iron
> > > out the other potential issues?
> > >
> > > I think the outstanding ones might be:
> > > 1.  Issue management
> > > 2.  Integration testing
> > >
> > > -Micah
> > >
> > > On Wed, Apr 14, 2021 at 11:01 PM Sutou Kouhei <ko...@clear-code.com>
> > wrote:
> > >
> > >> Hi Jacob,
> > >>
> > >> OK. Here is my plan:
> > >>
> > >>   1. We wait for the Rust's move to complete
> > >>   2. We use a process similar to the Rust's move
> > >>
> > >>
> > >> Thanks,
> > >> --
> > >> kou
> > >>
> > >> In <CA...@mail.gmail.com>
> > >>   "Re: Status of Arrow Julia implementation?" on Wed, 14 Apr 2021
> > 08:37:41
> > >> -0600,
> > >>   Jacob Quinn <qu...@gmail.com> wrote:
> > >>
> > >> > Thank you kou! I appreciate the help. I'm happy to do whatever is
> > >> required
> > >> > to facilitate the moving/donating process from JuliaData/Arrow.jl to
> > >> > apache/arrow-julia.
> > >> >
> > >> > -Jacob
> > >> >
> > >> > On Mon, Apr 12, 2021 at 7:53 PM Sutou Kouhei <ko...@clear-code.com>
> > wrote:
> > >> >
> > >> >> Hi Jacob,
> > >> >>
> > >> >> I, a PMC member, talked to Kenta Murata, a commiter and a
> > >> >> Julia user, about this.
> > >> >>
> > >> >> We support that you and Julia folks work on
> > >> >> arrow/arrow-julia until we have enough PMC members from
> > >> >> Julia folks. For example, we'll help IP clearance process to
> > >> >> import the latest JuliaData/Arrow.js changes to apache/ and
> > >> >> we'll start voting on Julia package release.
> > >> >>
> > >> >>
> > >> >> Thanks,
> > >> >> --
> > >> >> kou
> > >> >>
> > >> >> In <CAKyXBQo2=+NfS=hX6nEDPmMno-bSuXMR+=
> > d-hEsKgN2mm574iQ@mail.gmail.com>
> > >> >>   "Re: Status of Arrow Julia implementation?" on Sun, 11 Apr 2021
> > >> 23:06:27
> > >> >> -0600,
> > >> >>   Jacob Quinn <qu...@gmail.com> wrote:
> > >> >>
> > >> >> > Micah/Wes,
> > >> >> >
> > >> >> > Yes, I've been following the rust proposal thread with great
> > >> interest. I
> > >> >> do
> > >> >> > think that provides a great path forward: transferring the
> > >> >> > JuliaData/Arrow.jl repo to apache/arrow-julia would help to solve
> > the
> > >> >> > "package history" technical challenges that in part led to the
> > current
> > >> >> > setup and concerns. I think being able to utilize github issues
> > would
> > >> >> also
> > >> >> > be great; as I've mentioned elsewhere, it's much more
> > >> >> traditional/expected
> > >> >> > in the Julia ecosystem.
> > >> >> >
> > >> >> > I think the package could retain an independent versioning scheme.
> > The
> > >> >> >> additional process would be voting on release candidates. If the
> > >> Julia
> > >> >> >> folks want to try again and move development to a new,
> > Julia-specific
> > >> >> >> apache/* repository and apply the ASF governance to the project,
> > the
> > >> >> >> Arrow PMC could probably fast-track making Jacob a committer. In
> > some
> > >> >> >> code donations / IP clearance, the contributors for the donated
> > code
> > >> >> >> become committers as part of the transaction.
> > >> >> >>
> > >> >> >
> > >> >> > These all sound great and would greatly facilitate a better
> > >> integration
> > >> >> > under ASF governance. These points definitely resolve my main
> > >> concerns.
> > >> >> >
> > >> >> > As I commented on the rust thread, I'm mostly interested in the
> > >> future of
> > >> >> > integration testing for rust/julia if they are split out into
> > separate
> > >> >> > repos. In the current Julia implementation, we have all the code to
> > >> read
> > >> >> > arrow json, and I just hand-generated the integration test data and
> > >> >> > committed them in the repo itself, but it doesn't interface with
> > other
> > >> >> > languages (just reads arrow json, produces arrow file, reads arrow
> > >> file,
> > >> >> > compares w/ original arrow json). I'm happy to help work on the
> > >> details
> > >> >> of
> > >> >> > what that looks like and pilot some solutions. I think with a solid
> > >> >> > inter-repo integration testing framework, we can keep a strong sync
> > >> >> between
> > >> >> > projects.
> > >> >> >
> > >> >> > -Jacob
> > >> >> >
> > >> >> >
> > >> >> > On Sun, Apr 11, 2021 at 5:08 PM Wes McKinney <we...@gmail.com>
> > >> >> wrote:
> > >> >> >
> > >> >> >> On Sat, Apr 10, 2021 at 4:07 PM Micah Kornfield <
> > >> emkornfield@gmail.com>
> > >> >> >> wrote:
> > >> >> >> >
> > >> >> >> > >
> > >> >> >> > > Ok, I've had a chance to discuss with a few other Julia
> > >> developers
> > >> >> and
> > >> >> >> > > review various options. I think it's best to drop the Julia
> > code
> > >> >> from
> > >> >> >> the
> > >> >> >> > > physical apache/arrow repo. The extra overhead on development,
> > >> >> release
> > >> >> >> > > process, and user issue reporting and PR contributing are too
> > >> much
> > >> >> in
> > >> >> >> > > addition to the technical challenges that we never resolved
> > >> >> involving
> > >> >> >> > > including the past Arrow.jl release version git trees in the
> > >> >> >> apache/arrow
> > >> >> >> > > repo.
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > Hi Jacob,
> > >> >> >> > It seems you are on the new thread discussing a proposal for
> > >> changing
> > >> >> >> > Rust's development model.   Would the proposal [1] address most
> > of
> > >> >> these
> > >> >> >> > concerns if Julia was set up in the same way?
> > >> >> >> >
> > >> >> >> >  It seems in the short term the stickiest point would be
> > committer
> > >> >> access
> > >> >> >> > to the new repos, and I suppose the release mechanics still
> > might
> > >> be
> > >> >> >> > challenging?
> > >> >> >>
> > >> >> >> I think the package could retain an independent versioning scheme.
> > >> The
> > >> >> >> additional process would be voting on release candidates. If the
> > >> Julia
> > >> >> >> folks want to try again and move development to a new,
> > Julia-specific
> > >> >> >> apache/* repository and apply the ASF governance to the project,
> > the
> > >> >> >> Arrow PMC could probably fast-track making Jacob a committer. In
> > some
> > >> >> >> code donations / IP clearance, the contributors for the donated
> > code
> > >> >> >> become committers as part of the transaction.
> > >> >> >>
> > >> >> >> >
> > >> >> >> > Thanks,
> > >> >> >> > Micah
> > >> >> >> >
> > >> >> >> > [1]
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> > https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit
> > >> >> >> >
> > >> >> >> > On Wed, Apr 7, 2021 at 4:17 AM Wes McKinney <
> > wesmckinn@gmail.com>
> > >> >> wrote:
> > >> >> >> >
> > >> >> >> > > I went back and read the mailing list discussions from
> > September
> > >> >> about
> > >> >> >> > > the donation and I would say there was not a clear enough
> > >> statement
> > >> >> >> > > from us about what the donation and IP clearance meant as far
> > as
> > >> the
> > >> >> >> > > future of the Julia codebase. This is partly our fault — we
> > have
> > >> >> taken
> > >> >> >> > > in 9 other code donations over the last 5 years, and in all
> > cases
> > >> >> the
> > >> >> >> > > developers understood that they were to move their process to
> > the
> > >> >> >> > > Arrow repositories and communications channels.
> > >> >> >> > >
> > >> >> >> > > It did not occur to me at all that the code that you were
> > >> putting in
> > >> >> >> > > the Arrow repository would get treated like a read-only fork
> > that
> > >> >> you
> > >> >> >> > > update periodically. If I had realized that, we wouldn't be in
> > >> this
> > >> >> >> > > situation.
> > >> >> >> > >
> > >> >> >> > > As a reminder about what Arrow and the ASF are all about:
> > >> Community
> > >> >> >> > > over Code. We think that building a collaborative, open
> > community
> > >> >> that
> > >> >> >> > > works and plans together in public, makes decisions based on
> > >> >> consensus
> > >> >> >> > > with clear meritocratic ("doers decide") governance is the
> > best
> > >> way
> > >> >> to
> > >> >> >> > > build this project. The concerns that you have around the
> > timing
> > >> and
> > >> >> >> > > frequency of releases for the Julia codebase are in my mind
> > easy
> > >> to
> > >> >> >> > > resolve, and if you had indicated that having a customized
> > >> process
> > >> >> for
> > >> >> >> > > Julia releases was a condition for your joining the community
> > >> >> >> > > wholeheartedly, we would have been happy to help. I think that
> > >> the
> > >> >> >> > > benefits of common CI/CD infrastructure and opportunities to
> > >> build
> > >> >> >> > > deeper integrations between the Julia implementation and the
> > >> other
> > >> >> >> > > implementations (imagine... Julia kernels running in
> > DataFusion?)
> > >> >> >> > > would outweigh the sense of "loss of control" from developing
> > >> >> within a
> > >> >> >> > > larger project.
> > >> >> >> > >
> > >> >> >> > > On Wed, Apr 7, 2021 at 12:16 AM Jacob Quinn <
> > >> quinn.jacobd@gmail.com
> > >> >> >
> > >> >> >> > > wrote:
> > >> >> >> > > >
> > >> >> >> > > > Responses inline below:
> > >> >> >> > > >
> > >> >> >> > > > On Tue, Apr 6, 2021 at 9:46 PM Jorge Cardoso Leitão <
> > >> >> >> > > > jorgecarleitao@gmail.com> wrote:
> > >> >> >> > > >
> > >> >> >> > > > > Hi,
> > >> >> >> > > > >
> > >> >> >> > > > > > you all did not attempt to work in the community for any
> > >> >> >> meaningful
> > >> >> >> > > > > amount of time and
> > >> >> >> > > > > are choosing not to try based on the perception that it
> > will
> > >> >> create
> > >> >> >> > > > > unacceptable overhead for you
> > >> >> >> > > > >
> > >> >> >> > > > > It is not self-evident to me that Julia's community was
> > >> >> >> sufficiently
> > >> >> >> > > > > informed about what they
> > >> >> >> > > > > had to give in in terms of process and release management
> > >> when
> > >> >> >> merging
> > >> >> >> > > /
> > >> >> >> > > > > donating.
> > >> >> >> > > > >
> > >> >> >> > > >
> > >> >> >> > > > Yes, it was pretty unclear what the process was if we needed
> > >> to do
> > >> >> >> any
> > >> >> >> > > kind
> > >> >> >> > > > of patch release. I know that has been sorted out better
> > >> recently,
> > >> >> >> but
> > >> >> >> > > back
> > >> >> >> > > > in November, it didn't really seem like an option (i.e.
> > >> >> independent
> > >> >> >> > > > language patch releases).
> > >> >> >> > > >
> > >> >> >> > > >
> > >> >> >> > > > > IMO this is a plausible explanation as to why the donation
> > >> was
> > >> >> >> made and
> > >> >> >> > > > > then later abandoned.
> > >> >> >> > > > >
> > >> >> >> > > > >
> > >> >> >> > > > I'll just note that the "abandonment" can only be a
> > perception
> > >> >> from
> > >> >> >> the
> > >> >> >> > > > apache/arrow side of things, but as I mentioned above, I
> > also
> > >> >> tried
> > >> >> >> to
> > >> >> >> > > > clearly state in the julia/Arrow/README that the development
> > >> >> process
> > >> >> >> > > would
> > >> >> >> > > > continue with the JuliaData/Arrow.jl repo as the main "dev"
> > >> >> branch,
> > >> >> >> with
> > >> >> >> > > > changes being upstreamed to the apache/arrow repo, which was
> > >> >> followed
> > >> >> >> > > > through, having an upstream of commits right before the
> > 3.0.0
> > >> >> >> release,
> > >> >> >> > > and
> > >> >> >> > > > I was planning on doing the same soon for the 4.0.0 release.
> > >> That
> > >> >> is
> > >> >> >> to
> > >> >> >> > > > say, the Julia implementation has continued progressing
> > forward
> > >> >> quite
> > >> >> >> > > > rapidly, IMO, but I can see that perhaps apache/arrow repo
> > >> members
> > >> >> >> may
> > >> >> >> > > have
> > >> >> >> > > > viewed it as "abandoned".
> > >> >> >> > > >
> > >> >> >> > > >
> > >> >> >> > > > > I do not fully understand why the pain points Jacob
> > mentioned
> > >> >> were
> > >> >> >> not
> > >> >> >> > > > > brought up to the mailing list sooner, though.
> > >> >> >> > > > >
> > >> >> >> > > >
> > >> >> >> > > > To be honest and frank, I didn't have pain points with the
> > >> >> >> development
> > >> >> >> > > > process I outlined when the code was donated and as stated
> > in
> > >> the
> > >> >> >> README.
> > >> >> >> > > > That was the process that made the donation possible and I
> > >> >> imagined
> > >> >> >> would
> > >> >> >> > > > work well going forward, and has, until this thread started
> > >> and it
> > >> >> >> was
> > >> >> >> > > > pointed out that this process isn't viable. The pain points
> > >> were
> > >> >> >> > > discussed
> > >> >> >> > > > with the initial code donation, but in my mind were resolved
> > >> with
> > >> >> the
> > >> >> >> > > > development process that was decided upon.
> > >> >> >> > > >
> > >> >> >> > > >
> > >> >> >> > > > > This made us unable to potentially take corrective
> > measures.
> > >> I
> > >> >> >> think
> > >> >> >> > > that
> > >> >> >> > > > > this is why everyone was taken a bit by surprise with
> > this.
> > >> >> >> > > > >
> > >> >> >> > > > > Best,
> > >> >> >> > > > > Jorge
> > >> >> >> > > > >
> > >> >> >> > > > >
> > >> >> >> > > > > On Fri, Apr 2, 2021 at 10:18 PM Wes McKinney <
> > >> >> wesmckinn@gmail.com>
> > >> >> >> > > wrote:
> > >> >> >> > > > >
> > >> >> >> > > > > > hi Jacob — sorry to hear that. It's a bummer that you
> > all
> > >> did
> > >> >> not
> > >> >> >> > > > > > attempt to work in the community for any meaningful
> > amount
> > >> of
> > >> >> >> time
> > >> >> >> > > and
> > >> >> >> > > > > > are choosing not to try based on the perception that it
> > >> will
> > >> >> >> create
> > >> >> >> > > > > > unacceptable overhead for you. I believe the benefits
> > would
> > >> >> >> outweigh
> > >> >> >> > > > > > the costs, but I suppose we will have to agree to
> > disagree.
> > >> >> >> > > > > >
> > >> >> >> > > > > > Can you prepare a pull request to do the requisite
> > >> repository
> > >> >> >> > > surgery?
> > >> >> >> > > > > > I hope the development goes well in the future and look
> > >> >> forward
> > >> >> >> to
> > >> >> >> > > > > > seeing folks from the Julia ecosystem engaged here on
> > >> growing
> > >> >> the
> > >> >> >> > > > > > Arrow ecosystem.
> > >> >> >> > > > > >
> > >> >> >> > > > > > Thanks,
> > >> >> >> > > > > > Wes
> > >> >> >> > > > > >
> > >> >> >> > > > > > On Fri, Apr 2, 2021 at 3:03 PM Jacob Quinn <
> > >> >> >> quinn.jacobd@gmail.com>
> > >> >> >> > > > > wrote:
> > >> >> >> > > > > > >
> > >> >> >> > > > > > > Ok, I've had a chance to discuss with a few other
> > Julia
> > >> >> >> developers
> > >> >> >> > > and
> > >> >> >> > > > > > > review various options. I think it's best to drop the
> > >> Julia
> > >> >> >> code
> > >> >> >> > > from
> > >> >> >> > > > > the
> > >> >> >> > > > > > > physical apache/arrow repo. The extra overhead on
> > >> >> development,
> > >> >> >> > > release
> > >> >> >> > > > > > > process, and user issue reporting and PR contributing
> > are
> > >> >> too
> > >> >> >> much
> > >> >> >> > > in
> > >> >> >> > > > > > > addition to the technical challenges that we never
> > >> resolved
> > >> >> >> > > involving
> > >> >> >> > > > > > > including the past Arrow.jl release version git trees
> > in
> > >> the
> > >> >> >> > > > > apache/arrow
> > >> >> >> > > > > > > repo.
> > >> >> >> > > > > > >
> > >> >> >> > > > > > > We're still very much committed to working on the
> > Julia
> > >> >> >> > > implementation
> > >> >> >> > > > > > and
> > >> >> >> > > > > > > participating in the broader arrow community. I've
> > >> enjoyed
> > >> >> >> > > following
> > >> >> >> > > > > the
> > >> >> >> > > > > > > user/dev mailing lists and will continue to do so. We
> > >> >> monitor
> > >> >> >> > > format
> > >> >> >> > > > > > > proposals and try to implement new functionality as
> > >> quickly
> > >> >> as
> > >> >> >> > > > > possible.
> > >> >> >> > > > > > We
> > >> >> >> > > > > > > got the initial arrow flight proto code generated just
> > >> last
> > >> >> >> night
> > >> >> >> > > in
> > >> >> >> > > > > > fact.
> > >> >> >> > > > > > > I'd still like to explore official integration with
> > the
> > >> >> archery
> > >> >> >> > > test
> > >> >> >> > > > > > suite
> > >> >> >> > > > > > > to solidify the Julia implementation with integration
> > >> >> tests; I
> > >> >> >> > > think
> > >> >> >> > > > > that
> > >> >> >> > > > > > > would be very valuable for long-term confidence in the
> > >> >> >> > > cross-language
> > >> >> >> > > > > > > support of the Julia implementation.
> > >> >> >> > > > > > >
> > >> >> >> > > > > > > We realize one of the main implications will probably
> > be
> > >> >> >> dropping
> > >> >> >> > > Julia
> > >> >> >> > > > > > > from the list of "official implementations". We're
> > >> >> encouraged
> > >> >> >> by
> > >> >> >> > > the
> > >> >> >> > > > > many
> > >> >> >> > > > > > > users who have already started using the Julia
> > >> >> implementation
> > >> >> >> and
> > >> >> >> > > will
> > >> >> >> > > > > > > strive to maintain a high rate of issue responsiveness
> > >> and
> > >> >> >> feature
> > >> >> >> > > > > > > development to maintain project confidence. If
> > there's a
> > >> >> >> > > possibility of
> > >> >> >> > > > > > > being included somewhere as an "unofficial" or
> > >> >> "semi-official"
> > >> >> >> > > > > > > implementation, we'd love to still be bundled with the
> > >> >> broader
> > >> >> >> > > arrow
> > >> >> >> > > > > > > project somehow, like, for example, showing how Julia
> > >> >> >> integrates
> > >> >> >> > > with
> > >> >> >> > > > > the
> > >> >> >> > > > > > > archery test suite, once the work there is done.
> > >> >> >> > > > > > >
> > >> >> >> > > > > > > Best,
> > >> >> >> > > > > > >
> > >> >> >> > > > > > > -Jacob
> > >> >> >> > > > > > >
> > >> >> >> > > > > > >
> > >> >> >> > > > > > >
> > >> >> >> > > > > > > On Tue, Mar 30, 2021 at 4:10 PM Wes McKinney <
> > >> >> >> wesmckinn@gmail.com>
> > >> >> >> > > > > > wrote:
> > >> >> >> > > > > > >
> > >> >> >> > > > > > > > Also, on the issue that there are no Julia-focused
> > PMC
> > >> >> >> members —
> > >> >> >> > > note
> > >> >> >> > > > > > > > that I helped the JavaScript folks make their own
> > >> >> independent
> > >> >> >> > > > > releases
> > >> >> >> > > > > > > > for quite a while: called the votes (e.g. [1]),
> > helped
> > >> get
> > >> >> >> > > people to
> > >> >> >> > > > > > > > verify and vote on the releases. After a time, it
> > was
> > >> >> >> decided to
> > >> >> >> > > stop
> > >> >> >> > > > > > > > releasing independently because there wasn't enough
> > >> >> >> development
> > >> >> >> > > > > > > > activity to justify it.
> > >> >> >> > > > > > > >
> > >> >> >> > > > > > > > [1]:
> > >> >> >> > >
> > https://www.mail-archive.com/dev@arrow.apache.org/msg05971.html
> > >> >> >> > > > > > > >
> > >> >> >> > > > > > > > On Tue, Mar 30, 2021 at 4:54 PM Wes McKinney <
> > >> >> >> > > wesmckinn@gmail.com>
> > >> >> >> > > > > > wrote:
> > >> >> >> > > > > > > > >
> > >> >> >> > > > > > > > > hi Jacob,
> > >> >> >> > > > > > > > >
> > >> >> >> > > > > > > > > On Tue, Mar 30, 2021 at 4:18 PM Jacob Quinn <
> > >> >> >> > > > > quinn.jacobd@gmail.com>
> > >> >> >> > > > > > > > wrote:
> > >> >> >> > > > > > > > > >
> > >> >> >> > > > > > > > > > I can comment as the primary apache arrow
> > liaison
> > >> for
> > >> >> the
> > >> >> >> > > > > Arrow.jl
> > >> >> >> > > > > > > > > > repository and original code donator.
> > >> >> >> > > > > > > > > >
> > >> >> >> > > > > > > > > > I apologize for the "surprise", but I commented
> > a
> > >> few
> > >> >> >> times
> > >> >> >> > > in
> > >> >> >> > > > > > various
> > >> >> >> > > > > > > > > > places and put a snippet in the README
> > >> >> >> > > > > > > > > > <
> > >> >> >> > > > > > > >
> > >> >> >> > > > > >
> > >> >> >> > > > >
> > >> >> >> > >
> > >> >> >>
> > >> >>
> > >>
> > https://github.com/apache/arrow/tree/master/julia/Arrow#difference-between-this-code-and-the-juliadataarrowjl-repository
> > >> >> >> > > > > > > > >
> > >> >> >> > > > > > > > > > about
> > >> >> >> > > > > > > > > > the approach I wanted to take w/ the Julia
> > >> >> >> implementation in
> > >> >> >> > > > > terms
> > >> >> >> > > > > > of
> > >> >> >> > > > > > > > > > keeping the JuliaData/Arrow.jl repository as a
> > "dev
> > >> >> >> branch"
> > >> >> >> > > of
> > >> >> >> > > > > > sorts
> > >> >> >> > > > > > > > of the
> > >> >> >> > > > > > > > > > apache/arrow code, upstreaming changes
> > >> periodically.
> > >> >> >> There's
> > >> >> >> > > > > even a
> > >> >> >> > > > > > > > script
> > >> >> >> > > > > > > > > > <
> > >> >> >> > > > > > > >
> > >> >> >> > > > > >
> > >> >> >> > > > >
> > >> >> >> > >
> > >> >> >>
> > >> >>
> > >>
> > https://github.com/JuliaData/Arrow.jl/blob/main/scripts/update_apache_arrow_code.jl
> > >> >> >> > > > > > > > >
> > >> >> >> > > > > > > > > > I wrote to mostly automate this upstreaming. I
> > >> realize
> > >> >> >> now
> > >> >> >> > > that I
> > >> >> >> > > > > > > > didn't
> > >> >> >> > > > > > > > > > consider the "Arrow PMC" position on this kind
> > of
> > >> >> setup
> > >> >> >> or
> > >> >> >> > > seek
> > >> >> >> > > > > to
> > >> >> >> > > > > > > > affirm
> > >> >> >> > > > > > > > > > that it would be ok to approach things like
> > this.
> > >> >> >> > > > > > > > > >
> > >> >> >> > > > > > > > > > The reality is that Julia users are very
> > engrained
> > >> to
> > >> >> >> expect
> > >> >> >> > > > > Julia
> > >> >> >> > > > > > > > packages
> > >> >> >> > > > > > > > > > to live in a single stand-alone github repo,
> > where
> > >> >> issues
> > >> >> >> > > can be
> > >> >> >> > > > > > > > opened,
> > >> >> >> > > > > > > > > > and pull requests are welcome. It was hard and
> > >> still
> > >> >> is
> > >> >> >> hard
> > >> >> >> > > to
> > >> >> >> > > > > > imagine
> > >> >> >> > > > > > > > > > "turning that off", since I believe we would
> > lose a
> > >> >> lot
> > >> >> >> of
> > >> >> >> > > > > > valuable bug
> > >> >> >> > > > > > > > > > reports and first-time contributions. This isn't
> > >> >> >> necessarily
> > >> >> >> > > any
> > >> >> >> > > > > > fault
> > >> >> >> > > > > > > > of
> > >> >> >> > > > > > > > > > how the bug report/contribution process is
> > handled
> > >> for
> > >> >> >> the
> > >> >> >> > > arrow
> > >> >> >> > > > > > > > project
> > >> >> >> > > > > > > > > > overall, though I'm also aware that there's a
> > >> desire
> > >> >> to
> > >> >> >> make
> > >> >> >> > > it
> > >> >> >> > > > > > easier
> > >> >> >> > > > > > > > > >
> > >> >> >> > > > > > > > > >
> > >> >> >> > > > > > > > > <
> > >> >> >> > > > > > > >
> > >> >> >> > > > > >
> > >> >> >> > > > >
> > >> >> >> > >
> > >> >> >>
> > >> >>
> > >>
> > https://lists.apache.org/x/thread.html/r8817dfba08ef8daa210956db69d513fd27b7a751d28fb8f27e39cc7e@%3Cdev.arrow.apache.org%3E
> > >> >> >> > > > > > > > >
> > >> >> >> > > > > > > > > > and
> > >> >> >> > > > > > > > > > it currently requires more and different effort
> > >> than
> > >> >> >> Julia
> > >> >> >> > > users
> > >> >> >> > > > > > are
> > >> >> >> > > > > > > > used
> > >> >> >> > > > > > > > > > to. I think it's more from how open, welcoming,
> > and
> > >> >> how
> > >> >> >> > > strong
> > >> >> >> > > > > the
> > >> >> >> > > > > > > > culture
> > >> >> >> > > > > > > > > > is in Julia around encouraging community
> > >> contributions
> > >> >> >> and
> > >> >> >> > > the
> > >> >> >> > > > > > tight
> > >> >> >> > > > > > > > > > integration with github and its open-source
> > project
> > >> >> >> > > management
> > >> >> >> > > > > > tools.
> > >> >> >> > > > > > > > > >
> > >> >> >> > > > > > > > >
> > >> >> >> > > > > > > > > Well, we are on track to having 1000 different
> > people
> > >> >> >> > > contribute to
> > >> >> >> > > > > > > > > the project and have over 12,000 issues, so I
> > don't
> > >> >> think
> > >> >> >> > > there is
> > >> >> >> > > > > > > > > evidence that we are failing to attract new
> > >> >> contributors or
> > >> >> >> > > that
> > >> >> >> > > > > > > > > feature requests / bugs aren't being reported. The
> > >> way
> > >> >> >> that we
> > >> >> >> > > work
> > >> >> >> > > > > > is
> > >> >> >> > > > > > > > > _different_, so adapting to the Apache process
> > will
> > >> >> require
> > >> >> >> > > change.
> > >> >> >> > > > > > > > >
> > >> >> >> > > > > > > > > > Additionally, I was and still am concerned about
> > >> the
> > >> >> >> overall
> > >> >> >> > > > > > release
> > >> >> >> > > > > > > > > > process of the apache/arrow project. I know
> > there
> > >> have
> > >> >> >> been
> > >> >> >> > > > > efforts
> > >> >> >> > > > > > > > there
> > >> >> >> > > > > > > > > > as well to make it easier for individual
> > languages
> > >> to
> > >> >> >> > > release on
> > >> >> >> > > > > > their
> > >> >> >> > > > > > > > own
> > >> >> >> > > > > > > > > > cadence, but just anecdotally, the
> > >> JuliaData/Arrow.jl
> > >> >> has
> > >> >> >> > > > > > > > had/needed/wanted
> > >> >> >> > > > > > > > > > 10 patch and minor releases since the original
> > code
> > >> >> >> donation,
> > >> >> >> > > > > > whereas
> > >> >> >> > > > > > > > the
> > >> >> >> > > > > > > > > > apache/arrow project has had one (3.0.0). This
> > >> leads
> > >> >> to
> > >> >> >> some
> > >> >> >> > > of
> > >> >> >> > > > > the
> > >> >> >> > > > > > > > > > concerns I have with restricting development to
> > >> just
> > >> >> the
> > >> >> >> > > > > > apache/arrow
> > >> >> >> > > > > > > > > > repository: how exactly does the release process
> > >> work
> > >> >> for
> > >> >> >> > > > > > individual
> > >> >> >> > > > > > > > > > languages who may desire independent releases
> > apart
> > >> >> from
> > >> >> >> the
> > >> >> >> > > > > > quarterly
> > >> >> >> > > > > > > > > > overall project releases? I think from the Rust
> > >> >> thread I
> > >> >> >> > > remember
> > >> >> >> > > > > > that
> > >> >> >> > > > > > > > you
> > >> >> >> > > > > > > > > > just need a group of language contributors to
> > all
> > >> >> agree,
> > >> >> >> but
> > >> >> >> > > what
> > >> >> >> > > > > > if
> > >> >> >> > > > > > > > I'm
> > >> >> >> > > > > > > > > > the only "active" Julia contributor? It's also
> > >> unclear
> > >> >> >> what
> > >> >> >> > > the
> > >> >> >> > > > > > > > > > expectations are for actual development: with
> > the
> > >> >> >> original
> > >> >> >> > > code
> > >> >> >> > > > > > > > donation
> > >> >> >> > > > > > > > > > PRs, I know Neal "reviewed" the PRs, but perhaps
> > >> >> missed
> > >> >> >> the
> > >> >> >> > > > > details
> > >> >> >> > > > > > > > around
> > >> >> >> > > > > > > > > > how I proposed development continue going
> > forward.
> > >> Is
> > >> >> it
> > >> >> >> > > required
> > >> >> >> > > > > > to
> > >> >> >> > > > > > > > have a
> > >> >> >> > > > > > > > > > certain number of reviews before merging? On the
> > >> Julia
> > >> >> >> side,
> > >> >> >> > > I
> > >> >> >> > > > > can
> > >> >> >> > > > > > try
> > >> >> >> > > > > > > > to
> > >> >> >> > > > > > > > > > encourage/push for those who have contributed to
> > >> the
> > >> >> >> > > > > > JuliaData/Arrow.jl
> > >> >> >> > > > > > > > > > repository to help review PRs to apache/arrow,
> > but
> > >> I
> > >> >> also
> > >> >> >> > > can't
> > >> >> >> > > > > > > > guarantee
> > >> >> >> > > > > > > > > > we would always have someone to review. It just
> > >> feels
> > >> >> >> pretty
> > >> >> >> > > > > > awkward
> > >> >> >> > > > > > > > if I
> > >> >> >> > > > > > > > > > keep needing to ping non-Julia people to
> > "review" a
> > >> >> PR to
> > >> >> >> > > merge
> > >> >> >> > > > > it.
> > >> >> >> > > > > > > > Perhaps
> > >> >> >> > > > > > > > > > this is just a problem of the overall Julia
> > >> >> >> implementation
> > >> >> >> > > > > > "smallness"
> > >> >> >> > > > > > > > in
> > >> >> >> > > > > > > > > > terms of contributors, but I'm not sure on the
> > best
> > >> >> >> answer
> > >> >> >> > > here.
> > >> >> >> > > > > > > > > >
> > >> >> >> > > > > > > > >
> > >> >> >> > > > > > > > > Several things here:
> > >> >> >> > > > > > > > >
> > >> >> >> > > > > > > > > * If you want to do separate Julia releases, you
> > are
> > >> >> free
> > >> >> >> to do
> > >> >> >> > > > > that,
> > >> >> >> > > > > > > > > but you have to follow the process (voting on the
> > >> >> mailing
> > >> >> >> list,
> > >> >> >> > > > > > > > > publishing GPG-signed source artifacts)
> > >> >> >> > > > > > > > > * If you had been working "in the community" since
> > >> >> >> November,
> > >> >> >> > > you
> > >> >> >> > > > > > would
> > >> >> >> > > > > > > > > probably already be a committer, so there is a
> > >> >> >> bootstrapping
> > >> >> >> > > here
> > >> >> >> > > > > > that
> > >> >> >> > > > > > > > > has failed to take place. In the meantime, we are
> > >> more
> > >> >> than
> > >> >> >> > > happy
> > >> >> >> > > > > to
> > >> >> >> > > > > > > > > help you "earn your wings" (as a committer) as
> > >> quickly
> > >> >> as
> > >> >> >> > > possible.
> > >> >> >> > > > > > > > > But from my perspective, I see a code donation and
> > >> two
> > >> >> >> other
> > >> >> >> > > > > commits,
> > >> >> >> > > > > > > > > which isn't enough to make a case for
> > committership.
> > >> >> >> > > > > > > > >
> > >> >> >> > > > > > > > > > So in short, I'm not sure on the best path
> > >> forward. I
> > >> >> >> think
> > >> >> >> > > > > > strictly
> > >> >> >> > > > > > > > > > restricting development to the apache/arrow
> > >> physical
> > >> >> >> > > repository
> > >> >> >> > > > > > would
> > >> >> >> > > > > > > > > > actively hurt the progress of the Julia
> > >> >> implementation,
> > >> >> >> > > whereas
> > >> >> >> > > > > it
> > >> >> >> > > > > > > > *has*
> > >> >> >> > > > > > > > > > been progressing with increasing momentum since
> > >> first
> > >> >> >> > > released.
> > >> >> >> > > > > > There
> > >> >> >> > > > > > > > are
> > >> >> >> > > > > > > > > > posts on the Julia discourse forum, in the Julia
> > >> slack
> > >> >> >> and
> > >> >> >> > > zulip
> > >> >> >> > > > > > > > > > communities, and quite a few issues/PRs being
> > >> opened
> > >> >> at
> > >> >> >> the
> > >> >> >> > > > > > > > > > JuliaData/Arrow.jl repository. There have been
> > >> several
> > >> >> >> calls
> > >> >> >> > > for
> > >> >> >> > > > > > arrow
> > >> >> >> > > > > > > > > > flight support, with a member from Julia
> > Computing
> > >> >> >> actually
> > >> >> >> > > close
> > >> >> >> > > > > > to
> > >> >> >> > > > > > > > > > releasing a gRPC client
> > >> >> >> > > > > > > > > > <
> > https://github.com/JuliaComputing/gRPCClient.jl>
> > >> >> >> > > specifically
> > >> >> >> > > > > > > > > > to help with flight support. But in terms of
> > actual
> > >> >> >> > > committers,
> > >> >> >> > > > > > it's
> > >> >> >> > > > > > > > been
> > >> >> >> > > > > > > > > > primarily just myself, with a few minor
> > >> contributions
> > >> >> by
> > >> >> >> > > others.
> > >> >> >> > > > > > > > > >
> > >> >> >> > > > > > > > > > I guess the big question that comes to mind is
> > what
> > >> >> are
> > >> >> >> the
> > >> >> >> > > hard
> > >> >> >> > > > > > > > > > requirements to be considered an "official
> > >> >> >> implementation"?
> > >> >> >> > > Does
> > >> >> >> > > > > > the
> > >> >> >> > > > > > > > code
> > >> >> >> > > > > > > > > > *have* to live in the same physical repo? Or if
> > it
> > >> >> >> passed the
> > >> >> >> > > > > > series of
> > >> >> >> > > > > > > > > > archery integration tests, would that be
> > enough? I
> > >> >> >> apologize
> > >> >> >> > > for
> > >> >> >> > > > > my
> > >> >> >> > > > > > > > > > naivete/inexperience on all things "apache",
> > but I
> > >> >> >> imagine
> > >> >> >> > > that's
> > >> >> >> > > > > > a big
> > >> >> >> > > > > > > > > > part of it: having official development/releases
> > >> >> through
> > >> >> >> the
> > >> >> >> > > > > > > > apache/arrow
> > >> >> >> > > > > > > > > > community, though again I'm not exactly sure on
> > the
> > >> >> >> formal
> > >> >> >> > > > > > processes
> > >> >> >> > > > > > > > here?
> > >> >> >> > > > > > > > > > I would like to keep Julia as an official
> > >> >> implementation,
> > >> >> >> > > but I'm
> > >> >> >> > > > > > also
> > >> >> >> > > > > > > > > > mostly carrying the maintainership alone at the
> > >> moment
> > >> >> >> and
> > >> >> >> > > want
> > >> >> >> > > > > to
> > >> >> >> > > > > > be
> > >> >> >> > > > > > > > > > realistic with the future of the project.
> > >> >> >> > > > > > > > > >
> > >> >> >> > > > > > > > >
> > >> >> >> > > > > > > > > The critical matter is whether the
> > >> >> development/maintenance
> > >> >> >> > > work is
> > >> >> >> > > > > > > > > conducted by the "Arrow community" in accordance
> > with
> > >> >> the
> > >> >> >> > > Apache
> > >> >> >> > > > > Way,
> > >> >> >> > > > > > > > > which is to say individuals collaborating with
> > each
> > >> >> other
> > >> >> >> on
> > >> >> >> > > Apache
> > >> >> >> > > > > > > > > channels (for communication and development) and
> > >> >> avoiding
> > >> >> >> the
> > >> >> >> > > bad
> > >> >> >> > > > > > > > > patterns you see sometimes in other communities
> > (e.g.
> > >> >> >> > > inconsistent
> > >> >> >> > > > > > > > > openness).
> > >> >> >> > > > > > > > >
> > >> >> >> > > > > > > > > It's fine — really, no pressure — if you want to
> > be
> > >> >> >> > > independent and
> > >> >> >> > > > > > do
> > >> >> >> > > > > > > > > things your own way, you just have to be clear
> > that
> > >> you
> > >> >> are
> > >> >> >> > > > > > > > > independent and not operating as part of the
> > Apache
> > >> >> Arrow
> > >> >> >> > > > > community.
> > >> >> >> > > > > > > > > You can't have it both ways, though. No hard
> > feelings
> > >> >> >> whatever
> > >> >> >> > > you
> > >> >> >> > > > > > > > > decide, but the current "dump code over the wall
> > >> >> >> occasionally"
> > >> >> >> > > > > > > > > approach but work on independent channels is not
> > >> >> >> compatible.
> > >> >> >> > > > > Building
> > >> >> >> > > > > > > > > healthy open source communities is hard, but this
> > way
> > >> >> has
> > >> >> >> been
> > >> >> >> > > > > shown
> > >> >> >> > > > > > > > > to work well, which is why I've spent the last 6
> > >> years
> > >> >> >> working
> > >> >> >> > > hard
> > >> >> >> > > > > > to
> > >> >> >> > > > > > > > > bring people together to build this project and
> > >> >> ecosystem!
> > >> >> >> > > > > > > > >
> > >> >> >> > > > > > > > > If you want to maintain a test harness here to
> > >> verify an
> > >> >> >> > > > > independent
> > >> >> >> > > > > > > > > Julia implementation, that's fine, too. I'm
> > >> disappointed
> > >> >> >> that
> > >> >> >> > > > > things
> > >> >> >> > > > > > > > > failed to bootstrap after the code donation, so I
> > >> want
> > >> >> to
> > >> >> >> see
> > >> >> >> > > if we
> > >> >> >> > > > > > > > > can course correct quickly or if not decide to go
> > our
> > >> >> >> separate
> > >> >> >> > > > > ways.
> > >> >> >> > > > > > > > >
> > >> >> >> > > > > > > > > Thanks,
> > >> >> >> > > > > > > > > Wes
> > >> >> >> > > > > > > > >
> > >> >> >> > > > > > > > > > I'm open to discussion and ideas on the best way
> > >> >> forward.
> > >> >> >> > > > > > > > > >
> > >> >> >> > > > > > > > > > -Jacob
> > >> >> >> > > > > > > > > >
> > >> >> >> > > > > > > > > > On Tue, Mar 30, 2021 at 2:03 PM Wes McKinney <
> > >> >> >> > > > > wesmckinn@gmail.com>
> > >> >> >> > > > > > > > wrote:
> > >> >> >> > > > > > > > > >
> > >> >> >> > > > > > > > > > > hi folks,
> > >> >> >> > > > > > > > > > >
> > >> >> >> > > > > > > > > > > I was very surprised today to learn that the
> > >> Julia
> > >> >> >> Arrow
> > >> >> >> > > > > > > > > > > implementation has continued operating more or
> > >> less
> > >> >> >> like an
> > >> >> >> > > > > > > > > > > independent open source project since the code
> > >> >> donation
> > >> >> >> > > last
> > >> >> >> > > > > > > > November:
> > >> >> >> > > > > > > > > > >
> > >> >> >> > > > > > > > > > >
> > >> https://github.com/JuliaData/Arrow.jl/commits/main
> > >> >> >> > > > > > > > > > >
> > >> >> >> > > > > > > > > > > There may have been a misunderstanding about
> > what
> > >> >> was
> > >> >> >> > > expected
> > >> >> >> > > > > to
> > >> >> >> > > > > > > > > > > occur after the code donation, but it's
> > >> problematic
> > >> >> >> for a
> > >> >> >> > > bunch
> > >> >> >> > > > > > of
> > >> >> >> > > > > > > > > > > reasons (IP lineage / governance / community
> > >> >> >> development)
> > >> >> >> > > to
> > >> >> >> > > > > have
> > >> >> >> > > > > > > > work
> > >> >> >> > > > > > > > > > > happening on the implementation "outside the
> > >> >> >> community".
> > >> >> >> > > > > > > > > > >
> > >> >> >> > > > > > > > > > > In any case, what is done is done, so the
> > Arrow
> > >> >> PMC's
> > >> >> >> > > position
> > >> >> >> > > > > on
> > >> >> >> > > > > > > > this
> > >> >> >> > > > > > > > > > > would be roughly to regard the work as a hard
> > >> fork
> > >> >> of
> > >> >> >> > > what's in
> > >> >> >> > > > > > > > Apache
> > >> >> >> > > > > > > > > > > Arrow, which given its development activity is
> > >> more
> > >> >> or
> > >> >> >> less
> > >> >> >> > > > > > inactive
> > >> >> >> > > > > > > > > > > [1]. (I had actually thought the project was
> > >> simply
> > >> >> >> > > inactive
> > >> >> >> > > > > > after
> > >> >> >> > > > > > > > the
> > >> >> >> > > > > > > > > > > code donation)
> > >> >> >> > > > > > > > > > >
> > >> >> >> > > > > > > > > > > The critical question now is, is there
> > interest
> > >> from
> > >> >> >> Julia
> > >> >> >> > > > > > developers
> > >> >> >> > > > > > > > > > > in working "in the community", which is to
> > say:
> > >> >> >> > > > > > > > > > >
> > >> >> >> > > > > > > > > > > * Having development discussions on ASF
> > channels
> > >> >> >> (mailing
> > >> >> >> > > list,
> > >> >> >> > > > > > > > > > > GitHub, JIRA), planning and communicating in
> > the
> > >> >> open
> > >> >> >> > > > > > > > > > > * Doing all development in ASF GitHub
> > >> repositories
> > >> >> >> > > > > > > > > > >
> > >> >> >> > > > > > > > > > > The answer to the question may be "no" (which
> > is
> > >> >> okay),
> > >> >> >> > > but if
> > >> >> >> > > > > > that's
> > >> >> >> > > > > > > > > > > the case, I don't think we should be giving
> > the
> > >> >> >> impression
> > >> >> >> > > that
> > >> >> >> > > > > > we
> > >> >> >> > > > > > > > > > > have an official Julia implementation that is
> > >> >> >> developed and
> > >> >> >> > > > > > > > maintained
> > >> >> >> > > > > > > > > > > by the community (and so my argument would be
> > >> >> >> > > unfortunately to
> > >> >> >> > > > > > drop
> > >> >> >> > > > > > > > > > > the donated code from the project).
> > >> >> >> > > > > > > > > > >
> > >> >> >> > > > > > > > > > > If the answer is "yes", there needs to be a
> > hard
> > >> >> >> > > commitment to
> > >> >> >> > > > > > move
> > >> >> >> > > > > > > > > > > development to Apache channels and not look
> > >> back. We
> > >> >> >> would
> > >> >> >> > > also
> > >> >> >> > > > > > need
> > >> >> >> > > > > > > > > > > to figure out what to do to document and
> > >> synchronize
> > >> >> >> the
> > >> >> >> > > new IP
> > >> >> >> > > > > > > > that's
> > >> >> >> > > > > > > > > > > been created since the code donation.
> > >> >> >> > > > > > > > > > >
> > >> >> >> > > > > > > > > > > Thanks,
> > >> >> >> > > > > > > > > > > Wes
> > >> >> >> > > > > > > > > > >
> > >> >> >> > > > > > > > > > > [1]:
> > >> >> >> > > > >
> > https://github.com/apache/arrow/commits/master/julia/Arrow
> > >> >> >> > > > > > > > > > >
> > >> >> >> > > > > > > >
> > >> >> >> > > > > >
> > >> >> >> > > > >
> > >> >> >> > >
> > >> >> >>
> > >> >>
> > >>
> >