You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Li Jin <ic...@gmail.com> on 2022/07/07 14:39:47 UTC

Do we have nightly source tar ball

Hello,

I wonder if we have nightly source tarball published somewhere?

Li

Re: Do we have nightly source tar ball

Posted by Li Jin <ic...@gmail.com>.
Thanks! This is very helpful.


On Thu, Jul 7, 2022 at 11:33 AM Jacob Wujciak <ja...@voltrondata.com> wrote:

> The crossbow tarballs do not contain the arrow source, they only contain
> the crossbow source (aka a few yaml files).
>
> On Thu, Jul 7, 2022 at 5:29 PM Li Jin <ic...@gmail.com> wrote:
>
> > Thanks both!
> >
> > This looks useful - I wonder if the version between the crossbow tarball
> > and pypi dist matches (i.e. are they published together)? i.e. the latest
> > crossbow source is "Nightly-test-2022-07-07" and the pypi fury is
> > "pyarrow-9.0.0.dev341.tar.gz"
> >
> >
> >
> > On Thu, Jul 7, 2022 at 11:19 AM Joris Van den Bossche <
> > jorisvandenbossche@gmail.com> wrote:
> >
> > > We do upload nightly wheels to an alternative PyPI index (see
> > >
> > >
> >
> https://arrow.apache.org/docs/python/install.html#installing-nightly-packages
> > > ),
> > > at https://pypi.fury.io/arrow-nightlies/pyarrow, and it seems we
> > actually
> > > also upload an sdist there.
> > >
> > > (it could still be more reliable to used HEAD, though, if you want to
> be
> > > sure to always have the latest. If our nightly release CI is failing,
> the
> > > index might be outdated for some days)
> > >
> > > Joris
> > >
> > > On Thu, 7 Jul 2022 at 17:11, Li Jin <ic...@gmail.com> wrote:
> > >
> > > > Thanks David - I am currently using [1]. A slight issue is that I
> don't
> > > > have the corresponding sdist for "pyarrow" so I'm wondering if there
> is
> > > an
> > > > easy way for me to grab matching [1] source and pyarrow source
> tarball.
> > > >
> > > > Not a big issue, I can create the pyarrow dist from [1]
> > > >
> > > > On Thu, Jul 7, 2022 at 11:03 AM David Li <li...@apache.org>
> wrote:
> > > >
> > > > > Does it work to just use the HEAD [1]?
> > > > >
> > > > > Otherwise, the nightly CI jobs do generate releases, but I don't
> > think
> > > > > there's much of a guarantee around the naming/format being stable
> > [2].
> > > > >
> > > > > I'm not sure if there's an explicit/dedicated "nightly source
> > release"
> > > > > (I'm not aware of one), or how that would differ significantly from
> > > just
> > > > > using HEAD (unless there's particular breakage you're concerned
> won't
> > > be
> > > > > caught by the regular CI?)
> > > > >
> > > > > [1]:
> > https://github.com/apache/arrow/archive/refs/heads/master.tar.gz
> > > > > [2]: https://github.com/ursacomputing/crossbow/releases
> > > > >
> > > > > On Thu, Jul 7, 2022, at 10:39, Li Jin wrote:
> > > > > > Hello,
> > > > > >
> > > > > > I wonder if we have nightly source tarball published somewhere?
> > > > > >
> > > > > > Li
> > > > >
> > > >
> > >
> >
>

Re: Do we have nightly source tar ball

Posted by Jacob Wujciak <ja...@voltrondata.com>.
The crossbow tarballs do not contain the arrow source, they only contain
the crossbow source (aka a few yaml files).

On Thu, Jul 7, 2022 at 5:29 PM Li Jin <ic...@gmail.com> wrote:

> Thanks both!
>
> This looks useful - I wonder if the version between the crossbow tarball
> and pypi dist matches (i.e. are they published together)? i.e. the latest
> crossbow source is "Nightly-test-2022-07-07" and the pypi fury is
> "pyarrow-9.0.0.dev341.tar.gz"
>
>
>
> On Thu, Jul 7, 2022 at 11:19 AM Joris Van den Bossche <
> jorisvandenbossche@gmail.com> wrote:
>
> > We do upload nightly wheels to an alternative PyPI index (see
> >
> >
> https://arrow.apache.org/docs/python/install.html#installing-nightly-packages
> > ),
> > at https://pypi.fury.io/arrow-nightlies/pyarrow, and it seems we
> actually
> > also upload an sdist there.
> >
> > (it could still be more reliable to used HEAD, though, if you want to be
> > sure to always have the latest. If our nightly release CI is failing, the
> > index might be outdated for some days)
> >
> > Joris
> >
> > On Thu, 7 Jul 2022 at 17:11, Li Jin <ic...@gmail.com> wrote:
> >
> > > Thanks David - I am currently using [1]. A slight issue is that I don't
> > > have the corresponding sdist for "pyarrow" so I'm wondering if there is
> > an
> > > easy way for me to grab matching [1] source and pyarrow source tarball.
> > >
> > > Not a big issue, I can create the pyarrow dist from [1]
> > >
> > > On Thu, Jul 7, 2022 at 11:03 AM David Li <li...@apache.org> wrote:
> > >
> > > > Does it work to just use the HEAD [1]?
> > > >
> > > > Otherwise, the nightly CI jobs do generate releases, but I don't
> think
> > > > there's much of a guarantee around the naming/format being stable
> [2].
> > > >
> > > > I'm not sure if there's an explicit/dedicated "nightly source
> release"
> > > > (I'm not aware of one), or how that would differ significantly from
> > just
> > > > using HEAD (unless there's particular breakage you're concerned won't
> > be
> > > > caught by the regular CI?)
> > > >
> > > > [1]:
> https://github.com/apache/arrow/archive/refs/heads/master.tar.gz
> > > > [2]: https://github.com/ursacomputing/crossbow/releases
> > > >
> > > > On Thu, Jul 7, 2022, at 10:39, Li Jin wrote:
> > > > > Hello,
> > > > >
> > > > > I wonder if we have nightly source tarball published somewhere?
> > > > >
> > > > > Li
> > > >
> > >
> >
>

Re: Do we have nightly source tar ball

Posted by Li Jin <ic...@gmail.com>.
Thanks both!

This looks useful - I wonder if the version between the crossbow tarball
and pypi dist matches (i.e. are they published together)? i.e. the latest
crossbow source is "Nightly-test-2022-07-07" and the pypi fury is
"pyarrow-9.0.0.dev341.tar.gz"



On Thu, Jul 7, 2022 at 11:19 AM Joris Van den Bossche <
jorisvandenbossche@gmail.com> wrote:

> We do upload nightly wheels to an alternative PyPI index (see
>
> https://arrow.apache.org/docs/python/install.html#installing-nightly-packages
> ),
> at https://pypi.fury.io/arrow-nightlies/pyarrow, and it seems we actually
> also upload an sdist there.
>
> (it could still be more reliable to used HEAD, though, if you want to be
> sure to always have the latest. If our nightly release CI is failing, the
> index might be outdated for some days)
>
> Joris
>
> On Thu, 7 Jul 2022 at 17:11, Li Jin <ic...@gmail.com> wrote:
>
> > Thanks David - I am currently using [1]. A slight issue is that I don't
> > have the corresponding sdist for "pyarrow" so I'm wondering if there is
> an
> > easy way for me to grab matching [1] source and pyarrow source tarball.
> >
> > Not a big issue, I can create the pyarrow dist from [1]
> >
> > On Thu, Jul 7, 2022 at 11:03 AM David Li <li...@apache.org> wrote:
> >
> > > Does it work to just use the HEAD [1]?
> > >
> > > Otherwise, the nightly CI jobs do generate releases, but I don't think
> > > there's much of a guarantee around the naming/format being stable [2].
> > >
> > > I'm not sure if there's an explicit/dedicated "nightly source release"
> > > (I'm not aware of one), or how that would differ significantly from
> just
> > > using HEAD (unless there's particular breakage you're concerned won't
> be
> > > caught by the regular CI?)
> > >
> > > [1]: https://github.com/apache/arrow/archive/refs/heads/master.tar.gz
> > > [2]: https://github.com/ursacomputing/crossbow/releases
> > >
> > > On Thu, Jul 7, 2022, at 10:39, Li Jin wrote:
> > > > Hello,
> > > >
> > > > I wonder if we have nightly source tarball published somewhere?
> > > >
> > > > Li
> > >
> >
>

Re: Do we have nightly source tar ball

Posted by Jacob Wujciak <ja...@voltrondata.com>.
Hello Li,

The jobs that create the wheels and sdist a part of the nightly packaging
builds [1].
Within the log [2] you can check the exact sha the wheels/sdist are built
with and use https://github.com/apache/arrow/tarball/<sha> to get the
matching source tarball.

Best
Jacob

[1]:
https://github.com/ursacomputing/crossbow/actions/workflows/nightly_packaging_submit.yml
[2]:
https://github.com/ursacomputing/crossbow/runs/7229755096?check_suite_focus=true#step:7:12
On Thu, Jul 7, 2022 at 5:19 PM Joris Van den Bossche <
jorisvandenbossche@gmail.com> wrote:

> We do upload nightly wheels to an alternative PyPI index (see
>
> https://arrow.apache.org/docs/python/install.html#installing-nightly-packages
> ),
> at https://pypi.fury.io/arrow-nightlies/pyarrow, and it seems we actually
> also upload an sdist there.
>
> (it could still be more reliable to used HEAD, though, if you want to be
> sure to always have the latest. If our nightly release CI is failing, the
> index might be outdated for some days)
>
> Joris
>
> On Thu, 7 Jul 2022 at 17:11, Li Jin <ic...@gmail.com> wrote:
>
> > Thanks David - I am currently using [1]. A slight issue is that I don't
> > have the corresponding sdist for "pyarrow" so I'm wondering if there is
> an
> > easy way for me to grab matching [1] source and pyarrow source tarball.
> >
> > Not a big issue, I can create the pyarrow dist from [1]
> >
> > On Thu, Jul 7, 2022 at 11:03 AM David Li <li...@apache.org> wrote:
> >
> > > Does it work to just use the HEAD [1]?
> > >
> > > Otherwise, the nightly CI jobs do generate releases, but I don't think
> > > there's much of a guarantee around the naming/format being stable [2].
> > >
> > > I'm not sure if there's an explicit/dedicated "nightly source release"
> > > (I'm not aware of one), or how that would differ significantly from
> just
> > > using HEAD (unless there's particular breakage you're concerned won't
> be
> > > caught by the regular CI?)
> > >
> > > [1]: https://github.com/apache/arrow/archive/refs/heads/master.tar.gz
> > > [2]: https://github.com/ursacomputing/crossbow/releases
> > >
> > > On Thu, Jul 7, 2022, at 10:39, Li Jin wrote:
> > > > Hello,
> > > >
> > > > I wonder if we have nightly source tarball published somewhere?
> > > >
> > > > Li
> > >
> >
>

Re: Do we have nightly source tar ball

Posted by Joris Van den Bossche <jo...@gmail.com>.
We do upload nightly wheels to an alternative PyPI index (see
https://arrow.apache.org/docs/python/install.html#installing-nightly-packages),
at https://pypi.fury.io/arrow-nightlies/pyarrow, and it seems we actually
also upload an sdist there.

(it could still be more reliable to used HEAD, though, if you want to be
sure to always have the latest. If our nightly release CI is failing, the
index might be outdated for some days)

Joris

On Thu, 7 Jul 2022 at 17:11, Li Jin <ic...@gmail.com> wrote:

> Thanks David - I am currently using [1]. A slight issue is that I don't
> have the corresponding sdist for "pyarrow" so I'm wondering if there is an
> easy way for me to grab matching [1] source and pyarrow source tarball.
>
> Not a big issue, I can create the pyarrow dist from [1]
>
> On Thu, Jul 7, 2022 at 11:03 AM David Li <li...@apache.org> wrote:
>
> > Does it work to just use the HEAD [1]?
> >
> > Otherwise, the nightly CI jobs do generate releases, but I don't think
> > there's much of a guarantee around the naming/format being stable [2].
> >
> > I'm not sure if there's an explicit/dedicated "nightly source release"
> > (I'm not aware of one), or how that would differ significantly from just
> > using HEAD (unless there's particular breakage you're concerned won't be
> > caught by the regular CI?)
> >
> > [1]: https://github.com/apache/arrow/archive/refs/heads/master.tar.gz
> > [2]: https://github.com/ursacomputing/crossbow/releases
> >
> > On Thu, Jul 7, 2022, at 10:39, Li Jin wrote:
> > > Hello,
> > >
> > > I wonder if we have nightly source tarball published somewhere?
> > >
> > > Li
> >
>

Re: Do we have nightly source tar ball

Posted by David Li <li...@apache.org>.
Ah. 

Hmm, while the Crossbow releases do include wheels, they don't include the source (the source link there is for the CI scripts, not Arrow itself). I'm not sure if there's anything else that packages the Python sources.

On Thu, Jul 7, 2022, at 11:10, Li Jin wrote:
> Thanks David - I am currently using [1]. A slight issue is that I don't
> have the corresponding sdist for "pyarrow" so I'm wondering if there is an
> easy way for me to grab matching [1] source and pyarrow source tarball.
>
> Not a big issue, I can create the pyarrow dist from [1]
>
> On Thu, Jul 7, 2022 at 11:03 AM David Li <li...@apache.org> wrote:
>
>> Does it work to just use the HEAD [1]?
>>
>> Otherwise, the nightly CI jobs do generate releases, but I don't think
>> there's much of a guarantee around the naming/format being stable [2].
>>
>> I'm not sure if there's an explicit/dedicated "nightly source release"
>> (I'm not aware of one), or how that would differ significantly from just
>> using HEAD (unless there's particular breakage you're concerned won't be
>> caught by the regular CI?)
>>
>> [1]: https://github.com/apache/arrow/archive/refs/heads/master.tar.gz
>> [2]: https://github.com/ursacomputing/crossbow/releases
>>
>> On Thu, Jul 7, 2022, at 10:39, Li Jin wrote:
>> > Hello,
>> >
>> > I wonder if we have nightly source tarball published somewhere?
>> >
>> > Li
>>

Re: Do we have nightly source tar ball

Posted by Li Jin <ic...@gmail.com>.
Thanks David - I am currently using [1]. A slight issue is that I don't
have the corresponding sdist for "pyarrow" so I'm wondering if there is an
easy way for me to grab matching [1] source and pyarrow source tarball.

Not a big issue, I can create the pyarrow dist from [1]

On Thu, Jul 7, 2022 at 11:03 AM David Li <li...@apache.org> wrote:

> Does it work to just use the HEAD [1]?
>
> Otherwise, the nightly CI jobs do generate releases, but I don't think
> there's much of a guarantee around the naming/format being stable [2].
>
> I'm not sure if there's an explicit/dedicated "nightly source release"
> (I'm not aware of one), or how that would differ significantly from just
> using HEAD (unless there's particular breakage you're concerned won't be
> caught by the regular CI?)
>
> [1]: https://github.com/apache/arrow/archive/refs/heads/master.tar.gz
> [2]: https://github.com/ursacomputing/crossbow/releases
>
> On Thu, Jul 7, 2022, at 10:39, Li Jin wrote:
> > Hello,
> >
> > I wonder if we have nightly source tarball published somewhere?
> >
> > Li
>

Re: Do we have nightly source tar ball

Posted by David Li <li...@apache.org>.
Does it work to just use the HEAD [1]?

Otherwise, the nightly CI jobs do generate releases, but I don't think there's much of a guarantee around the naming/format being stable [2].

I'm not sure if there's an explicit/dedicated "nightly source release" (I'm not aware of one), or how that would differ significantly from just using HEAD (unless there's particular breakage you're concerned won't be caught by the regular CI?)

[1]: https://github.com/apache/arrow/archive/refs/heads/master.tar.gz
[2]: https://github.com/ursacomputing/crossbow/releases

On Thu, Jul 7, 2022, at 10:39, Li Jin wrote:
> Hello,
>
> I wonder if we have nightly source tarball published somewhere?
>
> Li