You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Szabolcs Vasas <va...@cloudera.com> on 2017/08/30 09:40:38 UTC
Add docker support to Sqoop test infrastructure to run third party
tests automatically
Hi All,
I have been working on making the third party test execution easier during
the last couple of months and I would like to share my results/plans and
ask for feedback.
Executing the Sqoop third party test cases is way too painful and we have
already discussed it in another chain that a possible solution to this
would be to use docker containers to run the necessary RDBMSs. This
solution could also enable us to run the third party test cases in our CI
system which would be a really important step to improve the quality of
Sqoop.
I have created an umbrella JIRA for this initiative (
https://issues.apache.org/jira/browse/SQOOP-3228) please feel free to take
a look and leave a comment with your suggestions and ideas.
My plan is to implement this initiative in 3 phases:
Phase 1 (subtasks 1-5):
The goal of this phase is to document how the contributors can setup the
docker environment and start the necessary docker containers on their
machines to be able to execute all the third party tests manually.
This phase includes the necessary fixes in the test infrastructure and
fixing the currently failing third party test cases too.
We are actually very close to complete this phase I have managed to run the
third party test cases with docker containers on my machine in 20 minutes,
the documentation is the only step left.
Phase 2 (subtasks 6-7):
The goal of this phase is to automate the dockerized third party test
execution. We want to have an ant task which starts up the necessary
containers, executes the third party tests and shuts down the containers.
Phase 3 (subtask 8):
Change the Sqoop CI build to execute the third party test cases as well.
Please let me know if you have any comments/concerns/suggestions regarding
this initiative.
Regards,
Szabolcs
Re: Add docker support to Sqoop test infrastructure to run third
party tests automatically
Posted by Szabolcs Vasas <va...@cloudera.com>.
Hi Bogi,
Thank you for the information!
Yes, I agree, the pre-commit hook executing all the tests is something we
should definitely work on.
Szabolcs
On Thu, Aug 31, 2017 at 1:08 PM, Boglarka Egyed <bo...@cloudera.com> wrote:
> Hi Szabolcs,
>
> This would be a huge improvement on quality side and it would definitely
> ease the every day life of the contributors - so many thanks for driving
> these efforts!
>
> Upon Anna's comment I would suggest to contact the Apache Infrastructure
> <https://www.apache.org/dev/infrastructure.html> and/or Apache Legal
> <https://www.apache.org/legal/> teams with the questionable items.
>
> And as a continuation of the three phases the pre-commit hook creation
> could be also started which has been brought up in another mail thread
> <http://mail-archives.apache.org/mod_mbox/sqoop-dev/201704.
> mbox/%3CC12DC713-E74F-4D39-A29B-CE8EA4A93515%40apache.org%3E>.
> What do you think?
>
> Thanks,
> Bogi
>
> On Wed, Aug 30, 2017 at 3:06 PM, Szabolcs Vasas <va...@cloudera.com>
> wrote:
>
> > Hi Anna,
> >
> > Thank you for your feedback! I have added these points to SQOOP-3231
> > <https://issues.apache.org/jira/browse/SQOOP-3231>.
> >
> > Szabolcs
> >
> > On Wed, Aug 30, 2017 at 1:54 PM, Anna Szonyi <sz...@cloudera.com>
> wrote:
> >
> > > Hi Szabolcs,
> > >
> > > The only thing I want to add/point out is that we will need to make
> sure
> > to
> > > reach out to apache legal regarding what we can add to the Sqoop repo
> and
> > > into the Sqoop CI. We also need to confirm that the jenkins infra is
> > ready
> > > to run docker.
> > >
> > > Other than this, this would be a very useful addition to our project,
> > > thanks for driving this!
> > >
> > > Thanks,
> > > Anna
> > >
> > > On Wed, Aug 30, 2017 at 11:40 AM, Szabolcs Vasas <va...@cloudera.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I have been working on making the third party test execution easier
> > > during
> > > > the last couple of months and I would like to share my results/plans
> > and
> > > > ask for feedback.
> > > > Executing the Sqoop third party test cases is way too painful and we
> > have
> > > > already discussed it in another chain that a possible solution to
> this
> > > > would be to use docker containers to run the necessary RDBMSs. This
> > > > solution could also enable us to run the third party test cases in
> our
> > CI
> > > > system which would be a really important step to improve the quality
> of
> > > > Sqoop.
> > > > I have created an umbrella JIRA for this initiative (
> > > > https://issues.apache.org/jira/browse/SQOOP-3228) please feel free
> to
> > > take
> > > > a look and leave a comment with your suggestions and ideas.
> > > > My plan is to implement this initiative in 3 phases:
> > > >
> > > > Phase 1 (subtasks 1-5):
> > > > The goal of this phase is to document how the contributors can setup
> > the
> > > > docker environment and start the necessary docker containers on their
> > > > machines to be able to execute all the third party tests manually.
> > > > This phase includes the necessary fixes in the test infrastructure
> and
> > > > fixing the currently failing third party test cases too.
> > > > We are actually very close to complete this phase I have managed to
> run
> > > the
> > > > third party test cases with docker containers on my machine in 20
> > > minutes,
> > > > the documentation is the only step left.
> > > >
> > > > Phase 2 (subtasks 6-7):
> > > > The goal of this phase is to automate the dockerized third party test
> > > > execution. We want to have an ant task which starts up the necessary
> > > > containers, executes the third party tests and shuts down the
> > containers.
> > > >
> > > > Phase 3 (subtask 8):
> > > > Change the Sqoop CI build to execute the third party test cases as
> > well.
> > > >
> > > >
> > > > Please let me know if you have any comments/concerns/suggestions
> > > regarding
> > > > this initiative.
> > > >
> > > > Regards,
> > > > Szabolcs
> > > >
> > >
> >
> >
> >
> > --
> > Szabolcs Vasas
> > Software Engineer
> > <http://www.cloudera.com>
> >
>
--
Szabolcs Vasas
Software Engineer
<http://www.cloudera.com>
Re: Add docker support to Sqoop test infrastructure to run third
party tests automatically
Posted by Boglarka Egyed <bo...@cloudera.com>.
Hi Szabolcs,
This would be a huge improvement on quality side and it would definitely
ease the every day life of the contributors - so many thanks for driving
these efforts!
Upon Anna's comment I would suggest to contact the Apache Infrastructure
<https://www.apache.org/dev/infrastructure.html> and/or Apache Legal
<https://www.apache.org/legal/> teams with the questionable items.
And as a continuation of the three phases the pre-commit hook creation
could be also started which has been brought up in another mail thread
<http://mail-archives.apache.org/mod_mbox/sqoop-dev/201704.mbox/%3CC12DC713-E74F-4D39-A29B-CE8EA4A93515%40apache.org%3E>.
What do you think?
Thanks,
Bogi
On Wed, Aug 30, 2017 at 3:06 PM, Szabolcs Vasas <va...@cloudera.com> wrote:
> Hi Anna,
>
> Thank you for your feedback! I have added these points to SQOOP-3231
> <https://issues.apache.org/jira/browse/SQOOP-3231>.
>
> Szabolcs
>
> On Wed, Aug 30, 2017 at 1:54 PM, Anna Szonyi <sz...@cloudera.com> wrote:
>
> > Hi Szabolcs,
> >
> > The only thing I want to add/point out is that we will need to make sure
> to
> > reach out to apache legal regarding what we can add to the Sqoop repo and
> > into the Sqoop CI. We also need to confirm that the jenkins infra is
> ready
> > to run docker.
> >
> > Other than this, this would be a very useful addition to our project,
> > thanks for driving this!
> >
> > Thanks,
> > Anna
> >
> > On Wed, Aug 30, 2017 at 11:40 AM, Szabolcs Vasas <va...@cloudera.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I have been working on making the third party test execution easier
> > during
> > > the last couple of months and I would like to share my results/plans
> and
> > > ask for feedback.
> > > Executing the Sqoop third party test cases is way too painful and we
> have
> > > already discussed it in another chain that a possible solution to this
> > > would be to use docker containers to run the necessary RDBMSs. This
> > > solution could also enable us to run the third party test cases in our
> CI
> > > system which would be a really important step to improve the quality of
> > > Sqoop.
> > > I have created an umbrella JIRA for this initiative (
> > > https://issues.apache.org/jira/browse/SQOOP-3228) please feel free to
> > take
> > > a look and leave a comment with your suggestions and ideas.
> > > My plan is to implement this initiative in 3 phases:
> > >
> > > Phase 1 (subtasks 1-5):
> > > The goal of this phase is to document how the contributors can setup
> the
> > > docker environment and start the necessary docker containers on their
> > > machines to be able to execute all the third party tests manually.
> > > This phase includes the necessary fixes in the test infrastructure and
> > > fixing the currently failing third party test cases too.
> > > We are actually very close to complete this phase I have managed to run
> > the
> > > third party test cases with docker containers on my machine in 20
> > minutes,
> > > the documentation is the only step left.
> > >
> > > Phase 2 (subtasks 6-7):
> > > The goal of this phase is to automate the dockerized third party test
> > > execution. We want to have an ant task which starts up the necessary
> > > containers, executes the third party tests and shuts down the
> containers.
> > >
> > > Phase 3 (subtask 8):
> > > Change the Sqoop CI build to execute the third party test cases as
> well.
> > >
> > >
> > > Please let me know if you have any comments/concerns/suggestions
> > regarding
> > > this initiative.
> > >
> > > Regards,
> > > Szabolcs
> > >
> >
>
>
>
> --
> Szabolcs Vasas
> Software Engineer
> <http://www.cloudera.com>
>
Re: Add docker support to Sqoop test infrastructure to run third
party tests automatically
Posted by Szabolcs Vasas <va...@cloudera.com>.
Hi Anna,
Thank you for your feedback! I have added these points to SQOOP-3231
<https://issues.apache.org/jira/browse/SQOOP-3231>.
Szabolcs
On Wed, Aug 30, 2017 at 1:54 PM, Anna Szonyi <sz...@cloudera.com> wrote:
> Hi Szabolcs,
>
> The only thing I want to add/point out is that we will need to make sure to
> reach out to apache legal regarding what we can add to the Sqoop repo and
> into the Sqoop CI. We also need to confirm that the jenkins infra is ready
> to run docker.
>
> Other than this, this would be a very useful addition to our project,
> thanks for driving this!
>
> Thanks,
> Anna
>
> On Wed, Aug 30, 2017 at 11:40 AM, Szabolcs Vasas <va...@cloudera.com>
> wrote:
>
> > Hi All,
> >
> > I have been working on making the third party test execution easier
> during
> > the last couple of months and I would like to share my results/plans and
> > ask for feedback.
> > Executing the Sqoop third party test cases is way too painful and we have
> > already discussed it in another chain that a possible solution to this
> > would be to use docker containers to run the necessary RDBMSs. This
> > solution could also enable us to run the third party test cases in our CI
> > system which would be a really important step to improve the quality of
> > Sqoop.
> > I have created an umbrella JIRA for this initiative (
> > https://issues.apache.org/jira/browse/SQOOP-3228) please feel free to
> take
> > a look and leave a comment with your suggestions and ideas.
> > My plan is to implement this initiative in 3 phases:
> >
> > Phase 1 (subtasks 1-5):
> > The goal of this phase is to document how the contributors can setup the
> > docker environment and start the necessary docker containers on their
> > machines to be able to execute all the third party tests manually.
> > This phase includes the necessary fixes in the test infrastructure and
> > fixing the currently failing third party test cases too.
> > We are actually very close to complete this phase I have managed to run
> the
> > third party test cases with docker containers on my machine in 20
> minutes,
> > the documentation is the only step left.
> >
> > Phase 2 (subtasks 6-7):
> > The goal of this phase is to automate the dockerized third party test
> > execution. We want to have an ant task which starts up the necessary
> > containers, executes the third party tests and shuts down the containers.
> >
> > Phase 3 (subtask 8):
> > Change the Sqoop CI build to execute the third party test cases as well.
> >
> >
> > Please let me know if you have any comments/concerns/suggestions
> regarding
> > this initiative.
> >
> > Regards,
> > Szabolcs
> >
>
--
Szabolcs Vasas
Software Engineer
<http://www.cloudera.com>
Re: Add docker support to Sqoop test infrastructure to run third
party tests automatically
Posted by Anna Szonyi <sz...@cloudera.com>.
Hi Szabolcs,
The only thing I want to add/point out is that we will need to make sure to
reach out to apache legal regarding what we can add to the Sqoop repo and
into the Sqoop CI. We also need to confirm that the jenkins infra is ready
to run docker.
Other than this, this would be a very useful addition to our project,
thanks for driving this!
Thanks,
Anna
On Wed, Aug 30, 2017 at 11:40 AM, Szabolcs Vasas <va...@cloudera.com> wrote:
> Hi All,
>
> I have been working on making the third party test execution easier during
> the last couple of months and I would like to share my results/plans and
> ask for feedback.
> Executing the Sqoop third party test cases is way too painful and we have
> already discussed it in another chain that a possible solution to this
> would be to use docker containers to run the necessary RDBMSs. This
> solution could also enable us to run the third party test cases in our CI
> system which would be a really important step to improve the quality of
> Sqoop.
> I have created an umbrella JIRA for this initiative (
> https://issues.apache.org/jira/browse/SQOOP-3228) please feel free to take
> a look and leave a comment with your suggestions and ideas.
> My plan is to implement this initiative in 3 phases:
>
> Phase 1 (subtasks 1-5):
> The goal of this phase is to document how the contributors can setup the
> docker environment and start the necessary docker containers on their
> machines to be able to execute all the third party tests manually.
> This phase includes the necessary fixes in the test infrastructure and
> fixing the currently failing third party test cases too.
> We are actually very close to complete this phase I have managed to run the
> third party test cases with docker containers on my machine in 20 minutes,
> the documentation is the only step left.
>
> Phase 2 (subtasks 6-7):
> The goal of this phase is to automate the dockerized third party test
> execution. We want to have an ant task which starts up the necessary
> containers, executes the third party tests and shuts down the containers.
>
> Phase 3 (subtask 8):
> Change the Sqoop CI build to execute the third party test cases as well.
>
>
> Please let me know if you have any comments/concerns/suggestions regarding
> this initiative.
>
> Regards,
> Szabolcs
>