Sqoop build infrastructure improvements

[Szabolcs Vasas <va...@cloudera.com.INVALID>]

Dear Sqoop community,

We have been working on quite a few exciting build infrastructure
improvements recently, I am sending this email to summarize them.

*Gradle can now execute all the Sqoop tests in a single JVM*
This improvement makes the Gradle test tasks significantly faster since we
do not have to start up a new JVM for every test class. It also made
possible to introduce fine grained test categories which were essential to
be able to parallelize the test execution in our CI systems. For more
information please refer to COMPILING.txt
<https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.

*Apache Sqoop Jenkins job
<https://builds.apache.org/job/Sqoop-hadoop200/> now builds and tests with
Gradle*
Since our Gradle build became much more stable and faster it made sense to
reconfigure our Jenkins job to benefit from these improvements. The job is
faster now (~30 minutes instead of ~40) and it executes all of the tests
which can be run without external RDBMS or cloud systems (while the old Ant
based job executed the unit test suite only).

*Travis CI is enabled for Apache Sqoop*
The new Travis CI job <https://travis-ci.org/apache/sqoop> now runs for
every commit and every pull request on Apache Sqoop GitHub repository and
it executes all of the tests except the Oracle third party test cases. One
of the biggest benefit of Travis CI is that it can be really easily
configured for the individual forks as well so contributors get a well
configured CI job for their own feature branches for free. For more
information please refer to COMPILING.txt
<https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.


Since we have a CI job now which integrates very well with GitHub pull
requests I suggest deprecating the old Review Board and patch file based
contribution process and use pull requests in the future. We had a mail
chain about the same proposal last year and it seemed that the community
was happy about the idea so I think we can evaluate it for some time and if
everything goes well we can update our how to contribute wiki.

Feel free to reply to this chain with your questions and suggestions on the
above!

Regards,
Szabolcs

Re: Sqoop build infrastructure improvements

[Boglarka Egyed <bo...@apache.org>]

Thank you Szabolcs for driving these efforts!

+1 for using pull requests from now.

Thanks,
Bogi

On Wed, Nov 28, 2018 at 4:54 PM Szabolcs Vasas <va...@cloudera.com.invalid>
wrote:

> Dear Sqoop community,
>
> We have been working on quite a few exciting build infrastructure
> improvements recently, I am sending this email to summarize them.
>
> *Gradle can now execute all the Sqoop tests in a single JVM*
> This improvement makes the Gradle test tasks significantly faster since we
> do not have to start up a new JVM for every test class. It also made
> possible to introduce fine grained test categories which were essential to
> be able to parallelize the test execution in our CI systems. For more
> information please refer to COMPILING.txt
> <https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.
>
> *Apache Sqoop Jenkins job
> <https://builds.apache.org/job/Sqoop-hadoop200/> now builds and tests with
> Gradle*
> Since our Gradle build became much more stable and faster it made sense to
> reconfigure our Jenkins job to benefit from these improvements. The job is
> faster now (~30 minutes instead of ~40) and it executes all of the tests
> which can be run without external RDBMS or cloud systems (while the old Ant
> based job executed the unit test suite only).
>
> *Travis CI is enabled for Apache Sqoop*
> The new Travis CI job <https://travis-ci.org/apache/sqoop> now runs for
> every commit and every pull request on Apache Sqoop GitHub repository and
> it executes all of the tests except the Oracle third party test cases. One
> of the biggest benefit of Travis CI is that it can be really easily
> configured for the individual forks as well so contributors get a well
> configured CI job for their own feature branches for free. For more
> information please refer to COMPILING.txt
> <https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.
>
>
> Since we have a CI job now which integrates very well with GitHub pull
> requests I suggest deprecating the old Review Board and patch file based
> contribution process and use pull requests in the future. We had a mail
> chain about the same proposal last year and it seemed that the community
> was happy about the idea so I think we can evaluate it for some time and if
> everything goes well we can update our how to contribute wiki.
>
> Feel free to reply to this chain with your questions and suggestions on the
> above!
>
> Regards,
> Szabolcs
>

Re: Sqoop build infrastructure improvements

[Szabolcs Vasas <va...@apache.org>]

Hi Attila,

Thanks for raising this, I think it is a great idea to migrate as soon as
possible, I have started a separate mail chain which can be used as a proof
of community consensus.

On Sun, Dec 9, 2018 at 9:48 PM Attila Szabó <ma...@apache.org> wrote:

> Hello everyone,
>
> So if I'm not mistaken and according to the INFRA mail we've received
> during the weekend gitbox.apache.org is exactly what we wanted to have :
> Having both ASF commits and push privileges on Github side..
> We would just need to have an official consensus in the dev community (
> documented on the mail list), fire an INFRA Jira ticket, and after the
> merge update our site with the new "how to contribute" information.
>
> I think this would be a quite good timing for the Sqoop project as we're
> right now in the middle of these infrastructural reworks.
>
> Kind regards,
> Attila
>
> On Fri, Nov 30, 2018, 2:38 PM Szabolcs Vasas <vasas@apache.org wrote:
>
> > Thanks for the question, I think we should stick to the commit format we
> > had earlier so yes, we should go for squash and push.
> > The easiest solution could be to download the diff file instead of the
> > patch (e.g. https://github.com/apache/sqoop/pull/60.diff instead of
> > https://github.com/apache/sqoop/pull/60.patch) and that does the trick.
> >
> > The "This closes..." commit message just closes the PR but does not
> delete
> > the feature branch, asfgit most probably does not have delete permission
> > for these branches anyway.
> >
> >
> > On Fri, Nov 30, 2018 at 11:45 AM Attila Szabó <ma...@apache.org> wrote:
> >
> > > Hey Szabi,
> > >
> > > Thanks for the prompt response!
> > >
> > > I've thought the repos are connected back and forth and the close PR is
> > the
> > > official way. In case of we still stack to the patch file and git apply
> > > then commit and push approach.
> > >
> > > My only question in this case :
> > > Do we have any agreement on how we commit these PRs. I would vote for
> > > Squash and push, but of course I'm open for the discussion.
> > >
> > > BTW :
> > > Is "This closes" message deletes the branch automatically?
> > >
> > > On front of Github + Jira:
> > > I'm aware Github has the feature to connect with Jira so I'm pretty
> sure
> > it
> > > doable. Also I'm not sure if any ASF project has done it ever, but I'll
> > ask
> > > around in other communities.
> > >
> > > Cheers,
> > > Attila
> > >
> > >
> > >
> > > On Nov 30, 2018 11:03 AM, "Szabolcs Vasas" <va...@apache.org> wrote:
> > >
> > > Hi Attila,
> > >
> > > I think we won't be able to commit the pull requests on GitHub directly
> > > because our GitHub repo is just a mirror of the original Apache Sqoop
> > repo
> > > <https://git-wip-us.apache.org/repos/asf/sqoop.git>.
> > > However the commit process is still simplified, the ASF GitHub Bot JIRA
> > > comment contains the patch download link (e.g.
> > > https://github.com/apache/sqoop/pull/60.patch) and the commit message
> > > (e.g. This
> > > closes #60) you need to include to close the pull request. The rest of
> > the
> > > process remains the same, you need to apply the patch manually and push
> > the
> > > commit to git-wip-us repo.
> > >
> > > Regarding the GitHub UI improvement: I am not sure GitHub provides
> such a
> > > feature, do you know an example repository where this is implemented?
> > >
> > > Regards,
> > > Szabolcs
> > >
> > >
> > >
> > > On Fri, Nov 30, 2018 at 3:21 AM Attila Szabó <ma...@apache.org>
> wrote:
> > >
> > > > Hi Szabi,
> > > >
> > > > First of all:
> > > > Big Kudos for the more mature gradle build! I think this is a great
> > step
> > > > for the whole project!
> > > >
> > > > On the front of PRs:
> > > > I would only make it official if the user management / authorization
> > > > handling could be somehow automatically bound to the id.apache.org +
> > > > project privileges.
> > > > A good example:
> > > > Today I reviewed SQOOP-3396 but I would not had been able to merge it
> > > > because it seems on the Github project I do not have push / merge
> > > > privileges (regardless that I'm a Sqoop committer and also memeber of
> > the
> > > > ASF group on github with my user).
> > > > So if we can somehow bound these things together, and the majority of
> > the
> > > > ppl would like to use it instead of the Review Board then let it
> > happen!
> > > > I'm fine with both tools until there's no difference between the
> Github
> > > and
> > > > ASF repos from user management POV.
> > > >
> > > > On the top of this:
> > > > I'm not sure if it belongs to our table, or the ASF INFRA team, but
> it
> > > > would be nice if the PRs and the JIRA tickets would be connected
> > > > automatically on the Github UI as well, and thus the navigation to
> > > > issues.apache.org would be easier!
> > > >
> > > > On the front of the gradle build:
> > > > I've found a smaller issue with clean+unittest within the same
> command.
> > > > I've opened a ticket (SQOOP-3415) and a PR (just the follow the new
> > > > standard) with a solution proposal.
> > > >
> > > > My2cents,
> > > > Attila
> > > >
> > > > On Wed, Nov 28, 2018 at 4:54 PM Szabolcs Vasas
> > > <vasas@cloudera.com.invalid
> > > > >
> > > > wrote:
> > > >
> > > > > Dear Sqoop community,
> > > > >
> > > > > We have been working on quite a few exciting build infrastructure
> > > > > improvements recently, I am sending this email to summarize them.
> > > > >
> > > > > *Gradle can now execute all the Sqoop tests in a single JVM*
> > > > > This improvement makes the Gradle test tasks significantly faster
> > since
> > > > we
> > > > > do not have to start up a new JVM for every test class. It also
> made
> > > > > possible to introduce fine grained test categories which were
> > essential
> > > > to
> > > > > be able to parallelize the test execution in our CI systems. For
> more
> > > > > information please refer to COMPILING.txt
> > > > > <https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.
> > > > >
> > > > > *Apache Sqoop Jenkins job
> > > > > <https://builds.apache.org/job/Sqoop-hadoop200/> now builds and
> > tests
> > > > with
> > > > > Gradle*
> > > > > Since our Gradle build became much more stable and faster it made
> > sense
> > > > to
> > > > > reconfigure our Jenkins job to benefit from these improvements. The
> > job
> > > > is
> > > > > faster now (~30 minutes instead of ~40) and it executes all of the
> > > tests
> > > > > which can be run without external RDBMS or cloud systems (while the
> > old
> > > > Ant
> > > > > based job executed the unit test suite only).
> > > > >
> > > > > *Travis CI is enabled for Apache Sqoop*
> > > > > The new Travis CI job <https://travis-ci.org/apache/sqoop> now
> runs
> > > for
> > > > > every commit and every pull request on Apache Sqoop GitHub
> repository
> > > and
> > > > > it executes all of the tests except the Oracle third party test
> > cases.
> > > > One
> > > > > of the biggest benefit of Travis CI is that it can be really easily
> > > > > configured for the individual forks as well so contributors get a
> > well
> > > > > configured CI job for their own feature branches for free. For more
> > > > > information please refer to COMPILING.txt
> > > > > <https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.
> > > > >
> > > > >
> > > > > Since we have a CI job now which integrates very well with GitHub
> > pull
> > > > > requests I suggest deprecating the old Review Board and patch file
> > > based
> > > > > contribution process and use pull requests in the future. We had a
> > mail
> > > > > chain about the same proposal last year and it seemed that the
> > > community
> > > > > was happy about the idea so I think we can evaluate it for some
> time
> > > and
> > > > if
> > > > > everything goes well we can update our how to contribute wiki.
> > > > >
> > > > > Feel free to reply to this chain with your questions and
> suggestions
> > on
> > > > the
> > > > > above!
> > > > >
> > > > > Regards,
> > > > > Szabolcs
> > > > >
> > >
> > > <http://www.cloudera.com>
> > >
> >
>

Re: Sqoop build infrastructure improvements

[Attila Szabó <ma...@apache.org>]

Hello everyone,

So if I'm not mistaken and according to the INFRA mail we've received
during the weekend gitbox.apache.org is exactly what we wanted to have :
Having both ASF commits and push privileges on Github side..
We would just need to have an official consensus in the dev community (
documented on the mail list), fire an INFRA Jira ticket, and after the
merge update our site with the new "how to contribute" information.

I think this would be a quite good timing for the Sqoop project as we're
right now in the middle of these infrastructural reworks.

Kind regards,
Attila

On Fri, Nov 30, 2018, 2:38 PM Szabolcs Vasas <vasas@apache.org wrote:

> Thanks for the question, I think we should stick to the commit format we
> had earlier so yes, we should go for squash and push.
> The easiest solution could be to download the diff file instead of the
> patch (e.g. https://github.com/apache/sqoop/pull/60.diff instead of
> https://github.com/apache/sqoop/pull/60.patch) and that does the trick.
>
> The "This closes..." commit message just closes the PR but does not delete
> the feature branch, asfgit most probably does not have delete permission
> for these branches anyway.
>
>
> On Fri, Nov 30, 2018 at 11:45 AM Attila Szabó <ma...@apache.org> wrote:
>
> > Hey Szabi,
> >
> > Thanks for the prompt response!
> >
> > I've thought the repos are connected back and forth and the close PR is
> the
> > official way. In case of we still stack to the patch file and git apply
> > then commit and push approach.
> >
> > My only question in this case :
> > Do we have any agreement on how we commit these PRs. I would vote for
> > Squash and push, but of course I'm open for the discussion.
> >
> > BTW :
> > Is "This closes" message deletes the branch automatically?
> >
> > On front of Github + Jira:
> > I'm aware Github has the feature to connect with Jira so I'm pretty sure
> it
> > doable. Also I'm not sure if any ASF project has done it ever, but I'll
> ask
> > around in other communities.
> >
> > Cheers,
> > Attila
> >
> >
> >
> > On Nov 30, 2018 11:03 AM, "Szabolcs Vasas" <va...@apache.org> wrote:
> >
> > Hi Attila,
> >
> > I think we won't be able to commit the pull requests on GitHub directly
> > because our GitHub repo is just a mirror of the original Apache Sqoop
> repo
> > <https://git-wip-us.apache.org/repos/asf/sqoop.git>.
> > However the commit process is still simplified, the ASF GitHub Bot JIRA
> > comment contains the patch download link (e.g.
> > https://github.com/apache/sqoop/pull/60.patch) and the commit message
> > (e.g. This
> > closes #60) you need to include to close the pull request. The rest of
> the
> > process remains the same, you need to apply the patch manually and push
> the
> > commit to git-wip-us repo.
> >
> > Regarding the GitHub UI improvement: I am not sure GitHub provides such a
> > feature, do you know an example repository where this is implemented?
> >
> > Regards,
> > Szabolcs
> >
> >
> >
> > On Fri, Nov 30, 2018 at 3:21 AM Attila Szabó <ma...@apache.org> wrote:
> >
> > > Hi Szabi,
> > >
> > > First of all:
> > > Big Kudos for the more mature gradle build! I think this is a great
> step
> > > for the whole project!
> > >
> > > On the front of PRs:
> > > I would only make it official if the user management / authorization
> > > handling could be somehow automatically bound to the id.apache.org +
> > > project privileges.
> > > A good example:
> > > Today I reviewed SQOOP-3396 but I would not had been able to merge it
> > > because it seems on the Github project I do not have push / merge
> > > privileges (regardless that I'm a Sqoop committer and also memeber of
> the
> > > ASF group on github with my user).
> > > So if we can somehow bound these things together, and the majority of
> the
> > > ppl would like to use it instead of the Review Board then let it
> happen!
> > > I'm fine with both tools until there's no difference between the Github
> > and
> > > ASF repos from user management POV.
> > >
> > > On the top of this:
> > > I'm not sure if it belongs to our table, or the ASF INFRA team, but it
> > > would be nice if the PRs and the JIRA tickets would be connected
> > > automatically on the Github UI as well, and thus the navigation to
> > > issues.apache.org would be easier!
> > >
> > > On the front of the gradle build:
> > > I've found a smaller issue with clean+unittest within the same command.
> > > I've opened a ticket (SQOOP-3415) and a PR (just the follow the new
> > > standard) with a solution proposal.
> > >
> > > My2cents,
> > > Attila
> > >
> > > On Wed, Nov 28, 2018 at 4:54 PM Szabolcs Vasas
> > <vasas@cloudera.com.invalid
> > > >
> > > wrote:
> > >
> > > > Dear Sqoop community,
> > > >
> > > > We have been working on quite a few exciting build infrastructure
> > > > improvements recently, I am sending this email to summarize them.
> > > >
> > > > *Gradle can now execute all the Sqoop tests in a single JVM*
> > > > This improvement makes the Gradle test tasks significantly faster
> since
> > > we
> > > > do not have to start up a new JVM for every test class. It also made
> > > > possible to introduce fine grained test categories which were
> essential
> > > to
> > > > be able to parallelize the test execution in our CI systems. For more
> > > > information please refer to COMPILING.txt
> > > > <https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.
> > > >
> > > > *Apache Sqoop Jenkins job
> > > > <https://builds.apache.org/job/Sqoop-hadoop200/> now builds and
> tests
> > > with
> > > > Gradle*
> > > > Since our Gradle build became much more stable and faster it made
> sense
> > > to
> > > > reconfigure our Jenkins job to benefit from these improvements. The
> job
> > > is
> > > > faster now (~30 minutes instead of ~40) and it executes all of the
> > tests
> > > > which can be run without external RDBMS or cloud systems (while the
> old
> > > Ant
> > > > based job executed the unit test suite only).
> > > >
> > > > *Travis CI is enabled for Apache Sqoop*
> > > > The new Travis CI job <https://travis-ci.org/apache/sqoop> now runs
> > for
> > > > every commit and every pull request on Apache Sqoop GitHub repository
> > and
> > > > it executes all of the tests except the Oracle third party test
> cases.
> > > One
> > > > of the biggest benefit of Travis CI is that it can be really easily
> > > > configured for the individual forks as well so contributors get a
> well
> > > > configured CI job for their own feature branches for free. For more
> > > > information please refer to COMPILING.txt
> > > > <https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.
> > > >
> > > >
> > > > Since we have a CI job now which integrates very well with GitHub
> pull
> > > > requests I suggest deprecating the old Review Board and patch file
> > based
> > > > contribution process and use pull requests in the future. We had a
> mail
> > > > chain about the same proposal last year and it seemed that the
> > community
> > > > was happy about the idea so I think we can evaluate it for some time
> > and
> > > if
> > > > everything goes well we can update our how to contribute wiki.
> > > >
> > > > Feel free to reply to this chain with your questions and suggestions
> on
> > > the
> > > > above!
> > > >
> > > > Regards,
> > > > Szabolcs
> > > >
> >
> > <http://www.cloudera.com>
> >
>

Re: Sqoop build infrastructure improvements

[Szabolcs Vasas <va...@apache.org>]

Thanks for the question, I think we should stick to the commit format we
had earlier so yes, we should go for squash and push.
The easiest solution could be to download the diff file instead of the
patch (e.g. https://github.com/apache/sqoop/pull/60.diff instead of
https://github.com/apache/sqoop/pull/60.patch) and that does the trick.

The "This closes..." commit message just closes the PR but does not delete
the feature branch, asfgit most probably does not have delete permission
for these branches anyway.


On Fri, Nov 30, 2018 at 11:45 AM Attila Szabó <ma...@apache.org> wrote:

> Hey Szabi,
>
> Thanks for the prompt response!
>
> I've thought the repos are connected back and forth and the close PR is the
> official way. In case of we still stack to the patch file and git apply
> then commit and push approach.
>
> My only question in this case :
> Do we have any agreement on how we commit these PRs. I would vote for
> Squash and push, but of course I'm open for the discussion.
>
> BTW :
> Is "This closes" message deletes the branch automatically?
>
> On front of Github + Jira:
> I'm aware Github has the feature to connect with Jira so I'm pretty sure it
> doable. Also I'm not sure if any ASF project has done it ever, but I'll ask
> around in other communities.
>
> Cheers,
> Attila
>
>
>
> On Nov 30, 2018 11:03 AM, "Szabolcs Vasas" <va...@apache.org> wrote:
>
> Hi Attila,
>
> I think we won't be able to commit the pull requests on GitHub directly
> because our GitHub repo is just a mirror of the original Apache Sqoop repo
> <https://git-wip-us.apache.org/repos/asf/sqoop.git>.
> However the commit process is still simplified, the ASF GitHub Bot JIRA
> comment contains the patch download link (e.g.
> https://github.com/apache/sqoop/pull/60.patch) and the commit message
> (e.g. This
> closes #60) you need to include to close the pull request. The rest of the
> process remains the same, you need to apply the patch manually and push the
> commit to git-wip-us repo.
>
> Regarding the GitHub UI improvement: I am not sure GitHub provides such a
> feature, do you know an example repository where this is implemented?
>
> Regards,
> Szabolcs
>
>
>
> On Fri, Nov 30, 2018 at 3:21 AM Attila Szabó <ma...@apache.org> wrote:
>
> > Hi Szabi,
> >
> > First of all:
> > Big Kudos for the more mature gradle build! I think this is a great step
> > for the whole project!
> >
> > On the front of PRs:
> > I would only make it official if the user management / authorization
> > handling could be somehow automatically bound to the id.apache.org +
> > project privileges.
> > A good example:
> > Today I reviewed SQOOP-3396 but I would not had been able to merge it
> > because it seems on the Github project I do not have push / merge
> > privileges (regardless that I'm a Sqoop committer and also memeber of the
> > ASF group on github with my user).
> > So if we can somehow bound these things together, and the majority of the
> > ppl would like to use it instead of the Review Board then let it happen!
> > I'm fine with both tools until there's no difference between the Github
> and
> > ASF repos from user management POV.
> >
> > On the top of this:
> > I'm not sure if it belongs to our table, or the ASF INFRA team, but it
> > would be nice if the PRs and the JIRA tickets would be connected
> > automatically on the Github UI as well, and thus the navigation to
> > issues.apache.org would be easier!
> >
> > On the front of the gradle build:
> > I've found a smaller issue with clean+unittest within the same command.
> > I've opened a ticket (SQOOP-3415) and a PR (just the follow the new
> > standard) with a solution proposal.
> >
> > My2cents,
> > Attila
> >
> > On Wed, Nov 28, 2018 at 4:54 PM Szabolcs Vasas
> <vasas@cloudera.com.invalid
> > >
> > wrote:
> >
> > > Dear Sqoop community,
> > >
> > > We have been working on quite a few exciting build infrastructure
> > > improvements recently, I am sending this email to summarize them.
> > >
> > > *Gradle can now execute all the Sqoop tests in a single JVM*
> > > This improvement makes the Gradle test tasks significantly faster since
> > we
> > > do not have to start up a new JVM for every test class. It also made
> > > possible to introduce fine grained test categories which were essential
> > to
> > > be able to parallelize the test execution in our CI systems. For more
> > > information please refer to COMPILING.txt
> > > <https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.
> > >
> > > *Apache Sqoop Jenkins job
> > > <https://builds.apache.org/job/Sqoop-hadoop200/> now builds and tests
> > with
> > > Gradle*
> > > Since our Gradle build became much more stable and faster it made sense
> > to
> > > reconfigure our Jenkins job to benefit from these improvements. The job
> > is
> > > faster now (~30 minutes instead of ~40) and it executes all of the
> tests
> > > which can be run without external RDBMS or cloud systems (while the old
> > Ant
> > > based job executed the unit test suite only).
> > >
> > > *Travis CI is enabled for Apache Sqoop*
> > > The new Travis CI job <https://travis-ci.org/apache/sqoop> now runs
> for
> > > every commit and every pull request on Apache Sqoop GitHub repository
> and
> > > it executes all of the tests except the Oracle third party test cases.
> > One
> > > of the biggest benefit of Travis CI is that it can be really easily
> > > configured for the individual forks as well so contributors get a well
> > > configured CI job for their own feature branches for free. For more
> > > information please refer to COMPILING.txt
> > > <https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.
> > >
> > >
> > > Since we have a CI job now which integrates very well with GitHub pull
> > > requests I suggest deprecating the old Review Board and patch file
> based
> > > contribution process and use pull requests in the future. We had a mail
> > > chain about the same proposal last year and it seemed that the
> community
> > > was happy about the idea so I think we can evaluate it for some time
> and
> > if
> > > everything goes well we can update our how to contribute wiki.
> > >
> > > Feel free to reply to this chain with your questions and suggestions on
> > the
> > > above!
> > >
> > > Regards,
> > > Szabolcs
> > >
>
> <http://www.cloudera.com>
>

Re: Sqoop build infrastructure improvements

[Attila Szabó <ma...@apache.org>]

Hey Szabi,

Thanks for the prompt response!

I've thought the repos are connected back and forth and the close PR is the
official way. In case of we still stack to the patch file and git apply
then commit and push approach.

My only question in this case :
Do we have any agreement on how we commit these PRs. I would vote for
Squash and push, but of course I'm open for the discussion.

BTW :
Is "This closes" message deletes the branch automatically?

On front of Github + Jira:
I'm aware Github has the feature to connect with Jira so I'm pretty sure it
doable. Also I'm not sure if any ASF project has done it ever, but I'll ask
around in other communities.

Cheers,
Attila



On Nov 30, 2018 11:03 AM, "Szabolcs Vasas" <va...@apache.org> wrote:

Hi Attila,

I think we won't be able to commit the pull requests on GitHub directly
because our GitHub repo is just a mirror of the original Apache Sqoop repo
<https://git-wip-us.apache.org/repos/asf/sqoop.git>.
However the commit process is still simplified, the ASF GitHub Bot JIRA
comment contains the patch download link (e.g.
https://github.com/apache/sqoop/pull/60.patch) and the commit message
(e.g. This
closes #60) you need to include to close the pull request. The rest of the
process remains the same, you need to apply the patch manually and push the
commit to git-wip-us repo.

Regarding the GitHub UI improvement: I am not sure GitHub provides such a
feature, do you know an example repository where this is implemented?

Regards,
Szabolcs



On Fri, Nov 30, 2018 at 3:21 AM Attila Szabó <ma...@apache.org> wrote:

> Hi Szabi,
>
> First of all:
> Big Kudos for the more mature gradle build! I think this is a great step
> for the whole project!
>
> On the front of PRs:
> I would only make it official if the user management / authorization
> handling could be somehow automatically bound to the id.apache.org +
> project privileges.
> A good example:
> Today I reviewed SQOOP-3396 but I would not had been able to merge it
> because it seems on the Github project I do not have push / merge
> privileges (regardless that I'm a Sqoop committer and also memeber of the
> ASF group on github with my user).
> So if we can somehow bound these things together, and the majority of the
> ppl would like to use it instead of the Review Board then let it happen!
> I'm fine with both tools until there's no difference between the Github
and
> ASF repos from user management POV.
>
> On the top of this:
> I'm not sure if it belongs to our table, or the ASF INFRA team, but it
> would be nice if the PRs and the JIRA tickets would be connected
> automatically on the Github UI as well, and thus the navigation to
> issues.apache.org would be easier!
>
> On the front of the gradle build:
> I've found a smaller issue with clean+unittest within the same command.
> I've opened a ticket (SQOOP-3415) and a PR (just the follow the new
> standard) with a solution proposal.
>
> My2cents,
> Attila
>
> On Wed, Nov 28, 2018 at 4:54 PM Szabolcs Vasas <vasas@cloudera.com.invalid
> >
> wrote:
>
> > Dear Sqoop community,
> >
> > We have been working on quite a few exciting build infrastructure
> > improvements recently, I am sending this email to summarize them.
> >
> > *Gradle can now execute all the Sqoop tests in a single JVM*
> > This improvement makes the Gradle test tasks significantly faster since
> we
> > do not have to start up a new JVM for every test class. It also made
> > possible to introduce fine grained test categories which were essential
> to
> > be able to parallelize the test execution in our CI systems. For more
> > information please refer to COMPILING.txt
> > <https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.
> >
> > *Apache Sqoop Jenkins job
> > <https://builds.apache.org/job/Sqoop-hadoop200/> now builds and tests
> with
> > Gradle*
> > Since our Gradle build became much more stable and faster it made sense
> to
> > reconfigure our Jenkins job to benefit from these improvements. The job
> is
> > faster now (~30 minutes instead of ~40) and it executes all of the tests
> > which can be run without external RDBMS or cloud systems (while the old
> Ant
> > based job executed the unit test suite only).
> >
> > *Travis CI is enabled for Apache Sqoop*
> > The new Travis CI job <https://travis-ci.org/apache/sqoop> now runs for
> > every commit and every pull request on Apache Sqoop GitHub repository
and
> > it executes all of the tests except the Oracle third party test cases.
> One
> > of the biggest benefit of Travis CI is that it can be really easily
> > configured for the individual forks as well so contributors get a well
> > configured CI job for their own feature branches for free. For more
> > information please refer to COMPILING.txt
> > <https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.
> >
> >
> > Since we have a CI job now which integrates very well with GitHub pull
> > requests I suggest deprecating the old Review Board and patch file based
> > contribution process and use pull requests in the future. We had a mail
> > chain about the same proposal last year and it seemed that the community
> > was happy about the idea so I think we can evaluate it for some time and
> if
> > everything goes well we can update our how to contribute wiki.
> >
> > Feel free to reply to this chain with your questions and suggestions on
> the
> > above!
> >
> > Regards,
> > Szabolcs
> >

<http://www.cloudera.com>

Re: Sqoop build infrastructure improvements

[Szabolcs Vasas <va...@apache.org>]

Hi Attila,

I think we won't be able to commit the pull requests on GitHub directly
because our GitHub repo is just a mirror of the original Apache Sqoop repo
<https://git-wip-us.apache.org/repos/asf/sqoop.git>.
However the commit process is still simplified, the ASF GitHub Bot JIRA
comment contains the patch download link (e.g.
https://github.com/apache/sqoop/pull/60.patch) and the commit message
(e.g. This
closes #60) you need to include to close the pull request. The rest of the
process remains the same, you need to apply the patch manually and push the
commit to git-wip-us repo.

Regarding the GitHub UI improvement: I am not sure GitHub provides such a
feature, do you know an example repository where this is implemented?

Regards,
Szabolcs


On Fri, Nov 30, 2018 at 3:21 AM Attila Szabó <ma...@apache.org> wrote:

> Hi Szabi,
>
> First of all:
> Big Kudos for the more mature gradle build! I think this is a great step
> for the whole project!
>
> On the front of PRs:
> I would only make it official if the user management / authorization
> handling could be somehow automatically bound to the id.apache.org +
> project privileges.
> A good example:
> Today I reviewed SQOOP-3396 but I would not had been able to merge it
> because it seems on the Github project I do not have push / merge
> privileges (regardless that I'm a Sqoop committer and also memeber of the
> ASF group on github with my user).
> So if we can somehow bound these things together, and the majority of the
> ppl would like to use it instead of the Review Board then let it happen!
> I'm fine with both tools until there's no difference between the Github and
> ASF repos from user management POV.
>
> On the top of this:
> I'm not sure if it belongs to our table, or the ASF INFRA team, but it
> would be nice if the PRs and the JIRA tickets would be connected
> automatically on the Github UI as well, and thus the navigation to
> issues.apache.org would be easier!
>
> On the front of the gradle build:
> I've found a smaller issue with clean+unittest within the same command.
> I've opened a ticket (SQOOP-3415) and a PR (just the follow the new
> standard) with a solution proposal.
>
> My2cents,
> Attila
>
> On Wed, Nov 28, 2018 at 4:54 PM Szabolcs Vasas <vasas@cloudera.com.invalid
> >
> wrote:
>
> > Dear Sqoop community,
> >
> > We have been working on quite a few exciting build infrastructure
> > improvements recently, I am sending this email to summarize them.
> >
> > *Gradle can now execute all the Sqoop tests in a single JVM*
> > This improvement makes the Gradle test tasks significantly faster since
> we
> > do not have to start up a new JVM for every test class. It also made
> > possible to introduce fine grained test categories which were essential
> to
> > be able to parallelize the test execution in our CI systems. For more
> > information please refer to COMPILING.txt
> > <https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.
> >
> > *Apache Sqoop Jenkins job
> > <https://builds.apache.org/job/Sqoop-hadoop200/> now builds and tests
> with
> > Gradle*
> > Since our Gradle build became much more stable and faster it made sense
> to
> > reconfigure our Jenkins job to benefit from these improvements. The job
> is
> > faster now (~30 minutes instead of ~40) and it executes all of the tests
> > which can be run without external RDBMS or cloud systems (while the old
> Ant
> > based job executed the unit test suite only).
> >
> > *Travis CI is enabled for Apache Sqoop*
> > The new Travis CI job <https://travis-ci.org/apache/sqoop> now runs for
> > every commit and every pull request on Apache Sqoop GitHub repository and
> > it executes all of the tests except the Oracle third party test cases.
> One
> > of the biggest benefit of Travis CI is that it can be really easily
> > configured for the individual forks as well so contributors get a well
> > configured CI job for their own feature branches for free. For more
> > information please refer to COMPILING.txt
> > <https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.
> >
> >
> > Since we have a CI job now which integrates very well with GitHub pull
> > requests I suggest deprecating the old Review Board and patch file based
> > contribution process and use pull requests in the future. We had a mail
> > chain about the same proposal last year and it seemed that the community
> > was happy about the idea so I think we can evaluate it for some time and
> if
> > everything goes well we can update our how to contribute wiki.
> >
> > Feel free to reply to this chain with your questions and suggestions on
> the
> > above!
> >
> > Regards,
> > Szabolcs
> >

<http://www.cloudera.com>

Re: Sqoop build infrastructure improvements

[Attila Szabó <ma...@apache.org>]

Hi Szabi,

First of all:
Big Kudos for the more mature gradle build! I think this is a great step
for the whole project!

On the front of PRs:
I would only make it official if the user management / authorization
handling could be somehow automatically bound to the id.apache.org +
project privileges.
A good example:
Today I reviewed SQOOP-3396 but I would not had been able to merge it
because it seems on the Github project I do not have push / merge
privileges (regardless that I'm a Sqoop committer and also memeber of the
ASF group on github with my user).
So if we can somehow bound these things together, and the majority of the
ppl would like to use it instead of the Review Board then let it happen!
I'm fine with both tools until there's no difference between the Github and
ASF repos from user management POV.

On the top of this:
I'm not sure if it belongs to our table, or the ASF INFRA team, but it
would be nice if the PRs and the JIRA tickets would be connected
automatically on the Github UI as well, and thus the navigation to
issues.apache.org would be easier!

On the front of the gradle build:
I've found a smaller issue with clean+unittest within the same command.
I've opened a ticket (SQOOP-3415) and a PR (just the follow the new
standard) with a solution proposal.

My2cents,
Attila

On Wed, Nov 28, 2018 at 4:54 PM Szabolcs Vasas <va...@cloudera.com.invalid>
wrote:

> Dear Sqoop community,
>
> We have been working on quite a few exciting build infrastructure
> improvements recently, I am sending this email to summarize them.
>
> *Gradle can now execute all the Sqoop tests in a single JVM*
> This improvement makes the Gradle test tasks significantly faster since we
> do not have to start up a new JVM for every test class. It also made
> possible to introduce fine grained test categories which were essential to
> be able to parallelize the test execution in our CI systems. For more
> information please refer to COMPILING.txt
> <https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.
>
> *Apache Sqoop Jenkins job
> <https://builds.apache.org/job/Sqoop-hadoop200/> now builds and tests with
> Gradle*
> Since our Gradle build became much more stable and faster it made sense to
> reconfigure our Jenkins job to benefit from these improvements. The job is
> faster now (~30 minutes instead of ~40) and it executes all of the tests
> which can be run without external RDBMS or cloud systems (while the old Ant
> based job executed the unit test suite only).
>
> *Travis CI is enabled for Apache Sqoop*
> The new Travis CI job <https://travis-ci.org/apache/sqoop> now runs for
> every commit and every pull request on Apache Sqoop GitHub repository and
> it executes all of the tests except the Oracle third party test cases. One
> of the biggest benefit of Travis CI is that it can be really easily
> configured for the individual forks as well so contributors get a well
> configured CI job for their own feature branches for free. For more
> information please refer to COMPILING.txt
> <https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.
>
>
> Since we have a CI job now which integrates very well with GitHub pull
> requests I suggest deprecating the old Review Board and patch file based
> contribution process and use pull requests in the future. We had a mail
> chain about the same proposal last year and it seemed that the community
> was happy about the idea so I think we can evaluate it for some time and if
> everything goes well we can update our how to contribute wiki.
>
> Feel free to reply to this chain with your questions and suggestions on the
> above!
>
> Regards,
> Szabolcs
>