You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by cj...@gmail.com,
cj...@gmail.com on 2018/12/27 21:59:31 UTC
Deploy parquet-format snapshot to maven repo
Re: Deploy parquet-format snapshot to maven repo
Posted by 俊杰陈 <cj...@gmail.com>.
Thanks and +1.
Once it gets merged, I will update PR to use it.
Zoltan Ivanfi <zi...@cloudera.com> 于2019年1月10日周四 下午10:58写道：
> Hi,
>
> I have created a pull request for the solution I suggested:
> https://github.com/apache/parquet-mr/pull/590
>
> With this solution, you could build parquet-format from
> dev/travis-before_install-bloom-filter.sh instead of directly from
> .travis.yml like you do in
> https://github.com/apache/parquet-mr/pull/587. The advantage is that
> even if that file gets merged into master accidentally, it will not
> have any effect there.
>
> Additionally, the dev/travis-before_install-master.sh script makes
> sure that SNAPSHOT dependencies are not merged into master
> accidentally. This is how it looks like when it catches a SNAPSHOT
> reference: https://travis-ci.org/apache/parquet-mr/jobs/477429867
>
> If there are no objections, I will merge this commit tomorrow.
>
> Br,
>
> Zoltan
>
>
> On Tue, Jan 8, 2019 at 6:37 AM 俊杰陈 <cj...@gmail.com> wrote:
> >
> > Many thanks Zoltan,
> >
> > The approach works for me, after adding 'git clone' and' mvn install'
> to before_install section in .travis.yml the build move forward to build.
> Also I agree with you about never depend on SNAPSHOT jar.
> >
> >
> > Zoltan Ivanfi <zi...@cloudera.com> 于2019年1月7日周一 下午10:38写道：
> >>
> >> Hi Junjie,
> >>
> >> There seems to be some problem with your email setup as your mail
> >> arrived to the list empty (you can see it in the online archive here:
> >> https://goo.gl/oZKz5P). I could only read it when you forwarded it to
> >> me. Other list members who could not see the original either can find
> >> the email at the end.
> >>
> >> My answer depends on whether the pull request is for the master branch
> >> or a feature branch. (You didn't link the PR, but I guess it is for
> >> the bloom filter feature branch.)
> >>
> >> I think we should never depend on the SNAPSHOT version of
> >> parquet-format on the master branch of parquet-mr. On a feature
> >> branch, on the other hand, it makes sense to depend on an unreleased
> >> SNAPSHOT version of parquet-format from the same branch, so that you
> >> can validate the design in parquet-format before releasing it by
> >> developing an actual implementation in parquet-mr. Having such a
> >> dependency, however, is very error-prone, because when merging the
> >> feature-branch, the SNAPSHOT dependecy can very easily get into the
> >> master branch accidentally.
> >>
> >> I think we should agree with the community on an approach and my
> >> suggestion would be to add branch-specific before_install scripts to
> >> .travis.yml:
> >> - The script for the master branch would check that the pom.xml file
> >> does not contain SNAPSHOT dependencies or fail the build otherwise.
> >> This would make sure that features branches can not be merged into
> >> master until the dependencies are finalized.
> >> - If needed, the script for a feature branch could clone a SNAPSHOT
> >> version of the parquet-format source code from the same branch (or
> >> even from master) and "mvn install" it, thereby making it available
> >> for the parquet-mr build.
> >>
> >> What do you think?
> >>
> >> Br,
> >>
> >> Zoltan
> >>
> >> On Fri, Jan 4, 2019 at 2:23 AM 俊杰陈 <cj...@gmail.com> wrote:
> >> >
> >> >
> >> > ---------- Forwarded message ---------
> >> > From: cjjnjust@gmail.com <cj...@gmail.com>
> >> > Date: 2018年12月28日周五 上午5:59
> >> > Subject: Deploy parquet-format snapshot to maven repo
> >> > To: dev@parquet.apache.org <de...@parquet.apache.org>
> >> >
> >> >
> >> > Hi devs
> >> >
> >> > Do we know how to handle build issue of PR that depends on new
> parquet format? It needs to download format jar that we haven't release. I
> tried to use jitpack.io to build a jar and change pom.xml in PR but the
> format repo needs thrift which jitpack.io build env doesn't have.
> >> >
> >> > Can someone with snapshot repo's permission help to deploy
> parquet-format snapshot (master branch current 2.7.0-SNAPSHOT) for a quick
> test? Many thanks.
> >> >
> >> > Happy New Year.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Thanks & Best Regards
> >
> >
> >
> > --
> > Thanks & Best Regards
>
--
Thanks & Best Regards
Re: Deploy parquet-format snapshot to maven repo
Posted by Zoltan Ivanfi <zi...@cloudera.com.INVALID>.
Hi,
I have created a pull request for the solution I suggested:
https://github.com/apache/parquet-mr/pull/590
With this solution, you could build parquet-format from
dev/travis-before_install-bloom-filter.sh instead of directly from
.travis.yml like you do in
https://github.com/apache/parquet-mr/pull/587. The advantage is that
even if that file gets merged into master accidentally, it will not
have any effect there.
Additionally, the dev/travis-before_install-master.sh script makes
sure that SNAPSHOT dependencies are not merged into master
accidentally. This is how it looks like when it catches a SNAPSHOT
reference: https://travis-ci.org/apache/parquet-mr/jobs/477429867
If there are no objections, I will merge this commit tomorrow.
Br,
Zoltan
On Tue, Jan 8, 2019 at 6:37 AM 俊杰陈 <cj...@gmail.com> wrote:
>
> Many thanks Zoltan,
>
> The approach works for me, after adding 'git clone' and' mvn install' to before_install section in .travis.yml the build move forward to build. Also I agree with you about never depend on SNAPSHOT jar.
>
>
> Zoltan Ivanfi <zi...@cloudera.com> 于2019年1月7日周一 下午10:38写道：
>>
>> Hi Junjie,
>>
>> There seems to be some problem with your email setup as your mail
>> arrived to the list empty (you can see it in the online archive here:
>> https://goo.gl/oZKz5P). I could only read it when you forwarded it to
>> me. Other list members who could not see the original either can find
>> the email at the end.
>>
>> My answer depends on whether the pull request is for the master branch
>> or a feature branch. (You didn't link the PR, but I guess it is for
>> the bloom filter feature branch.)
>>
>> I think we should never depend on the SNAPSHOT version of
>> parquet-format on the master branch of parquet-mr. On a feature
>> branch, on the other hand, it makes sense to depend on an unreleased
>> SNAPSHOT version of parquet-format from the same branch, so that you
>> can validate the design in parquet-format before releasing it by
>> developing an actual implementation in parquet-mr. Having such a
>> dependency, however, is very error-prone, because when merging the
>> feature-branch, the SNAPSHOT dependecy can very easily get into the
>> master branch accidentally.
>>
>> I think we should agree with the community on an approach and my
>> suggestion would be to add branch-specific before_install scripts to
>> .travis.yml:
>> - The script for the master branch would check that the pom.xml file
>> does not contain SNAPSHOT dependencies or fail the build otherwise.
>> This would make sure that features branches can not be merged into
>> master until the dependencies are finalized.
>> - If needed, the script for a feature branch could clone a SNAPSHOT
>> version of the parquet-format source code from the same branch (or
>> even from master) and "mvn install" it, thereby making it available
>> for the parquet-mr build.
>>
>> What do you think?
>>
>> Br,
>>
>> Zoltan
>>
>> On Fri, Jan 4, 2019 at 2:23 AM 俊杰陈 <cj...@gmail.com> wrote:
>> >
>> >
>> > ---------- Forwarded message ---------
>> > From: cjjnjust@gmail.com <cj...@gmail.com>
>> > Date: 2018年12月28日周五 上午5:59
>> > Subject: Deploy parquet-format snapshot to maven repo
>> > To: dev@parquet.apache.org <de...@parquet.apache.org>
>> >
>> >
>> > Hi devs
>> >
>> > Do we know how to handle build issue of PR that depends on new parquet format? It needs to download format jar that we haven't release. I tried to use jitpack.io to build a jar and change pom.xml in PR but the format repo needs thrift which jitpack.io build env doesn't have.
>> >
>> > Can someone with snapshot repo's permission help to deploy parquet-format snapshot (master branch current 2.7.0-SNAPSHOT) for a quick test? Many thanks.
>> >
>> > Happy New Year.
>> >
>> >
>> >
>> >
>> >
>> >
>> > --
>> > Thanks & Best Regards
>
>
>
> --
> Thanks & Best Regards
Re: Deploy parquet-format snapshot to maven repo
Posted by 俊杰陈 <cj...@gmail.com>.
Many thanks Zoltan,
The approach works for me, after adding 'git clone' and' mvn install' to
before_install section in .travis.yml the build move forward to build.
Also I agree with you about never depend on SNAPSHOT jar.
Zoltan Ivanfi <zi...@cloudera.com> 于2019年1月7日周一 下午10:38写道：
> Hi Junjie,
>
> There seems to be some problem with your email setup as your mail
> arrived to the list empty (you can see it in the online archive here:
> https://goo.gl/oZKz5P). I could only read it when you forwarded it to
> me. Other list members who could not see the original either can find
> the email at the end.
>
> My answer depends on whether the pull request is for the master branch
> or a feature branch. (You didn't link the PR, but I guess it is for
> the bloom filter feature branch.)
>
> I think we should never depend on the SNAPSHOT version of
> parquet-format on the master branch of parquet-mr. On a feature
> branch, on the other hand, it makes sense to depend on an unreleased
> SNAPSHOT version of parquet-format from the same branch, so that you
> can validate the design in parquet-format before releasing it by
> developing an actual implementation in parquet-mr. Having such a
> dependency, however, is very error-prone, because when merging the
> feature-branch, the SNAPSHOT dependecy can very easily get into the
> master branch accidentally.
>
> I think we should agree with the community on an approach and my
> suggestion would be to add branch-specific before_install scripts to
> .travis.yml:
> - The script for the master branch would check that the pom.xml file
> does not contain SNAPSHOT dependencies or fail the build otherwise.
> This would make sure that features branches can not be merged into
> master until the dependencies are finalized.
> - If needed, the script for a feature branch could clone a SNAPSHOT
> version of the parquet-format source code from the same branch (or
> even from master) and "mvn install" it, thereby making it available
> for the parquet-mr build.
>
> What do you think?
>
> Br,
>
> Zoltan
>
> On Fri, Jan 4, 2019 at 2:23 AM 俊杰陈 <cj...@gmail.com> wrote:
> >
> >
> > ---------- Forwarded message ---------
> > From: cjjnjust@gmail.com <cj...@gmail.com>
> > Date: 2018年12月28日周五 上午5:59
> > Subject: Deploy parquet-format snapshot to maven repo
> > To: dev@parquet.apache.org <de...@parquet.apache.org>
> >
> >
> > Hi devs
> >
> > Do we know how to handle build issue of PR that depends on new parquet
> format? It needs to download format jar that we haven't release. I tried
> to use jitpack.io to build a jar and change pom.xml in PR but the format
> repo needs thrift which jitpack.io build env doesn't have.
> >
> > Can someone with snapshot repo's permission help to deploy
> parquet-format snapshot (master branch current 2.7.0-SNAPSHOT) for a quick
> test? Many thanks.
> >
> > Happy New Year.
> >
> >
> >
> >
> >
> >
> > --
> > Thanks & Best Regards
>
--
Thanks & Best Regards
Re: Deploy parquet-format snapshot to maven repo
Posted by Zoltan Ivanfi <zi...@cloudera.com.INVALID>.
Hi Junjie,
There seems to be some problem with your email setup as your mail
arrived to the list empty (you can see it in the online archive here:
https://goo.gl/oZKz5P). I could only read it when you forwarded it to
me. Other list members who could not see the original either can find
the email at the end.
My answer depends on whether the pull request is for the master branch
or a feature branch. (You didn't link the PR, but I guess it is for
the bloom filter feature branch.)
I think we should never depend on the SNAPSHOT version of
parquet-format on the master branch of parquet-mr. On a feature
branch, on the other hand, it makes sense to depend on an unreleased
SNAPSHOT version of parquet-format from the same branch, so that you
can validate the design in parquet-format before releasing it by
developing an actual implementation in parquet-mr. Having such a
dependency, however, is very error-prone, because when merging the
feature-branch, the SNAPSHOT dependecy can very easily get into the
master branch accidentally.
I think we should agree with the community on an approach and my
suggestion would be to add branch-specific before_install scripts to
.travis.yml:
- The script for the master branch would check that the pom.xml file
does not contain SNAPSHOT dependencies or fail the build otherwise.
This would make sure that features branches can not be merged into
master until the dependencies are finalized.
- If needed, the script for a feature branch could clone a SNAPSHOT
version of the parquet-format source code from the same branch (or
even from master) and "mvn install" it, thereby making it available
for the parquet-mr build.
What do you think?
Br,
Zoltan
On Fri, Jan 4, 2019 at 2:23 AM 俊杰陈 <cj...@gmail.com> wrote:
>
>
> ---------- Forwarded message ---------
> From: cjjnjust@gmail.com <cj...@gmail.com>
> Date: 2018年12月28日周五 上午5:59
> Subject: Deploy parquet-format snapshot to maven repo
> To: dev@parquet.apache.org <de...@parquet.apache.org>
>
>
> Hi devs
>
> Do we know how to handle build issue of PR that depends on new parquet format? It needs to download format jar that we haven't release. I tried to use jitpack.io to build a jar and change pom.xml in PR but the format repo needs thrift which jitpack.io build env doesn't have.
>
> Can someone with snapshot repo's permission help to deploy parquet-format snapshot (master branch current 2.7.0-SNAPSHOT) for a quick test? Many thanks.
>
> Happy New Year.
>
>
>
>
>
>
> --
> Thanks & Best Regards