You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Vladimir Sitnikov <si...@gmail.com> on 2015/05/14 17:57:48 UTC

Jenkins CI

Hi,

I think it would be nice to setup a Jenkins CI for the following:
1) Teting JIRA patches. I find it nice to have CI reports on the JIRA patches.
2) Testing github pull requests
(https://blogs.apache.org/infra/entry/github_pull_request_builds_now)
3) In the long run -- integration testing (ASF Jenkins supports
docker, so we can use it to test against postgresql, mysql, etc)

If I understand wiki properly, someone from PMC Chairs can grant admin
account to a commiter
http://wiki.apache.org/general/Jenkins?action=show&redirect=Hudson

Is it the case?
Are there any PMC members on the list that could grant me the account?

Vladimir

Re: Jenkins CI

Posted by Nick Dimiduk <nd...@gmail.com>.
On Fri, May 15, 2015 at 1:29 PM, Vladimir Sitnikov <
sitnikov.vladimir@gmail.com> wrote:

> >project chair blessing to a user's account
>
> Do you know who is chair for Calcite?
>

I'm not sure how it works in Incubator, but I think that would be Julian.

Do you have any idea who can bless me for creating Jenkins jobs?
>

User accounts for Jenkins are managed thusly:

https://wiki.apache.org/general/Jenkins#How_do_I_get_an_account

> pre-commit per project  and it's tied into a JIRA query that's currently
> owned by Jake (on CC).
>
> Does that mean we need Jake's help to add "Calcite" to the query?


As I understand it, yes.

Re: Jenkins CI

Posted by Nick Dimiduk <nd...@gmail.com>.
On Fri, May 15, 2015 at 1:29 PM, Vladimir Sitnikov <
sitnikov.vladimir@gmail.com> wrote:

> >project chair blessing to a user's account
>
> Do you know who is chair for Calcite?
>

I'm not sure how it works in Incubator, but I think that would be Julian.

Do you have any idea who can bless me for creating Jenkins jobs?
>

User accounts for Jenkins are managed thusly:

https://wiki.apache.org/general/Jenkins#How_do_I_get_an_account

> pre-commit per project  and it's tied into a JIRA query that's currently
> owned by Jake (on CC).
>
> Does that mean we need Jake's help to add "Calcite" to the query?


As I understand it, yes.

Re: Jenkins CI

Posted by Vladimir Sitnikov <si...@gmail.com>.
>project chair blessing to a user's account

Do you know who is chair for Calcite?

> see what other projects are doing

I was not able to look into build configuration of other projects.
I found no traces of other projects requesting a Jenkins access.

> INFRA is there to help us out, but don't abuse it ;)

Do you have any idea who can bless me for creating Jenkins jobs?

> pre-commit per project  and it's tied into a JIRA query that's currently owned by Jake (on CC).

Does that mean we need Jake's help to add "Calcite" to the query?

Vladimir

Re: Jenkins CI

Posted by Vladimir Sitnikov <si...@gmail.com>.
>project chair blessing to a user's account

Do you know who is chair for Calcite?

> see what other projects are doing

I was not able to look into build configuration of other projects.
I found no traces of other projects requesting a Jenkins access.

> INFRA is there to help us out, but don't abuse it ;)

Do you have any idea who can bless me for creating Jenkins jobs?

> pre-commit per project  and it's tied into a JIRA query that's currently owned by Jake (on CC).

Does that mean we need Jake's help to add "Calcite" to the query?

Vladimir

Re: Jenkins CI

Posted by Nick Dimiduk <nd...@apache.org>.
Forking the conversation onto dev@kylin (dev@calcite to bcc, some of these
details are useful there too)

builds.apache.org is for unit testing. It's up to the project to provide
unit tests that can run in the environment. It's a pretty standard (maybe a
little old?) Jenkins install. Projects like Phoenix and HBase run UT's
there, some of which spin in mini-HDFS and mini-MR clusters in the process
for support. Phoenix also has a IT suite that can run on either a mini-XX
cluster, or be invoked against a real cluster.

Has Kylin spent much time with surefire/failsafe yet? Or maybe you don't
use maven at all?

Access to builds.apache.org is provided via project chair blessing to a
user's account. After that, they can login with their LDAP (id.apache.org)
creds and administer to jenkins -- set up new jobs and such. Generally
projects have 2 kinds of jenkins jobs: pre-commit and post-commit. There's
a pre-commit per project, and it's tied into a JIRA query that's currently
owned by Jake (on CC). There's generally a post-commit job per active
branch, and it's driven by polling the src repo.

Poke around on builds.apache.org, see what other projects are doing, and
ask more questions. INFRA is there to help us out, but don't abuse it ;)

-n

On Thu, May 14, 2015 at 10:57 PM, hongbin ma <ma...@apache.org> wrote:

> hi Nick,
>
> Thank you for you message. I don't know if it's okay to discuss our(Apache
> Kylin) CI requirements at Calcite's mail list. Please do let me know if
> it's inappropriate.
>
> For us Kylin, it's best if we could be provided with hadoop environment
> which enables launching MR jobs, Hive queries and HBase scans. I'm not sure
> if https://builds.apache.org/ could provide us such a isolated/shared
> hadoop env.
>
> If that's not possible, we do have a maven profile that run all the
> integration tests with hbase minicluster. The concern is that minicluster
> generates a lot of logs, and it may take around 30min~60min to finish all
> the test cases. We once tried http://travis-ci.org/, but our job was
> terminated because of too much logs. I don't know if
> https://builds.apache.org/ has similar limitations.
>
> BTW, can you give me a link to the script you mentioned?
>
>
>
> On Fri, May 15, 2015 at 1:38 PM, Nick Dimiduk <nd...@gmail.com> wrote:
>
> > I just set up the basics for Phoenix. Now that hbase 1.1 is released, I
> can
> > look at these items over here too. Mahone, let me know what you need
> done.
> > On Phoenix we reused HBase's scripts, which were recycled from Hadoop and
> > others. I think Hadoop just rebooted theirs, so maybe have a look and see
> > if you like it.
> >
> > On Thursday, May 14, 2015, hongbin ma <ma...@apache.org> wrote:
> >
> > > +1
> > >
> > > this is Mahone from apache kylin, please keep updated on the progress
> so
> > > that we can borrow your experience:)
> > >
> > > On Fri, May 15, 2015 at 1:14 AM, Julian Hyde <julianhyde@gmail.com
> > > <javascript:;>> wrote:
> > >
> > > > +1
> > > >
> > > > There is already an issue:
> > > > https://issues.apache.org/jira/browse/CALCITE-623
> > > >
> > > > On Thu, May 14, 2015 at 8:57 AM, Vladimir Sitnikov
> > > > <sitnikov.vladimir@gmail.com <javascript:;>> wrote:
> > > > > Hi,
> > > > >
> > > > > I think it would be nice to setup a Jenkins CI for the following:
> > > > > 1) Teting JIRA patches. I find it nice to have CI reports on the
> JIRA
> > > > patches.
> > > > > 2) Testing github pull requests
> > > > > (
> https://blogs.apache.org/infra/entry/github_pull_request_builds_now
> > )
> > > > > 3) In the long run -- integration testing (ASF Jenkins supports
> > > > > docker, so we can use it to test against postgresql, mysql, etc)
> > > > >
> > > > > If I understand wiki properly, someone from PMC Chairs can grant
> > admin
> > > > > account to a commiter
> > > > > http://wiki.apache.org/general/Jenkins?action=show&redirect=Hudson
> > > > >
> > > > > Is it the case?
> > > > > Are there any PMC members on the list that could grant me the
> > account?
> > > > >
> > > > > Vladimir
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > *Bin Mahone | 马洪宾*
> > > Apache Kylin: http://kylin.io
> > > Github: https://github.com/binmahone
> > >
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>

Re: Jenkins CI

Posted by Nick Dimiduk <nd...@apache.org>.
Forking the conversation onto dev@kylin (dev@calcite to bcc, some of these
details are useful there too)

builds.apache.org is for unit testing. It's up to the project to provide
unit tests that can run in the environment. It's a pretty standard (maybe a
little old?) Jenkins install. Projects like Phoenix and HBase run UT's
there, some of which spin in mini-HDFS and mini-MR clusters in the process
for support. Phoenix also has a IT suite that can run on either a mini-XX
cluster, or be invoked against a real cluster.

Has Kylin spent much time with surefire/failsafe yet? Or maybe you don't
use maven at all?

Access to builds.apache.org is provided via project chair blessing to a
user's account. After that, they can login with their LDAP (id.apache.org)
creds and administer to jenkins -- set up new jobs and such. Generally
projects have 2 kinds of jenkins jobs: pre-commit and post-commit. There's
a pre-commit per project, and it's tied into a JIRA query that's currently
owned by Jake (on CC). There's generally a post-commit job per active
branch, and it's driven by polling the src repo.

Poke around on builds.apache.org, see what other projects are doing, and
ask more questions. INFRA is there to help us out, but don't abuse it ;)

-n

On Thu, May 14, 2015 at 10:57 PM, hongbin ma <ma...@apache.org> wrote:

> hi Nick,
>
> Thank you for you message. I don't know if it's okay to discuss our(Apache
> Kylin) CI requirements at Calcite's mail list. Please do let me know if
> it's inappropriate.
>
> For us Kylin, it's best if we could be provided with hadoop environment
> which enables launching MR jobs, Hive queries and HBase scans. I'm not sure
> if https://builds.apache.org/ could provide us such a isolated/shared
> hadoop env.
>
> If that's not possible, we do have a maven profile that run all the
> integration tests with hbase minicluster. The concern is that minicluster
> generates a lot of logs, and it may take around 30min~60min to finish all
> the test cases. We once tried http://travis-ci.org/, but our job was
> terminated because of too much logs. I don't know if
> https://builds.apache.org/ has similar limitations.
>
> BTW, can you give me a link to the script you mentioned?
>
>
>
> On Fri, May 15, 2015 at 1:38 PM, Nick Dimiduk <nd...@gmail.com> wrote:
>
> > I just set up the basics for Phoenix. Now that hbase 1.1 is released, I
> can
> > look at these items over here too. Mahone, let me know what you need
> done.
> > On Phoenix we reused HBase's scripts, which were recycled from Hadoop and
> > others. I think Hadoop just rebooted theirs, so maybe have a look and see
> > if you like it.
> >
> > On Thursday, May 14, 2015, hongbin ma <ma...@apache.org> wrote:
> >
> > > +1
> > >
> > > this is Mahone from apache kylin, please keep updated on the progress
> so
> > > that we can borrow your experience:)
> > >
> > > On Fri, May 15, 2015 at 1:14 AM, Julian Hyde <julianhyde@gmail.com
> > > <javascript:;>> wrote:
> > >
> > > > +1
> > > >
> > > > There is already an issue:
> > > > https://issues.apache.org/jira/browse/CALCITE-623
> > > >
> > > > On Thu, May 14, 2015 at 8:57 AM, Vladimir Sitnikov
> > > > <sitnikov.vladimir@gmail.com <javascript:;>> wrote:
> > > > > Hi,
> > > > >
> > > > > I think it would be nice to setup a Jenkins CI for the following:
> > > > > 1) Teting JIRA patches. I find it nice to have CI reports on the
> JIRA
> > > > patches.
> > > > > 2) Testing github pull requests
> > > > > (
> https://blogs.apache.org/infra/entry/github_pull_request_builds_now
> > )
> > > > > 3) In the long run -- integration testing (ASF Jenkins supports
> > > > > docker, so we can use it to test against postgresql, mysql, etc)
> > > > >
> > > > > If I understand wiki properly, someone from PMC Chairs can grant
> > admin
> > > > > account to a commiter
> > > > > http://wiki.apache.org/general/Jenkins?action=show&redirect=Hudson
> > > > >
> > > > > Is it the case?
> > > > > Are there any PMC members on the list that could grant me the
> > account?
> > > > >
> > > > > Vladimir
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > *Bin Mahone | 马洪宾*
> > > Apache Kylin: http://kylin.io
> > > Github: https://github.com/binmahone
> > >
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>

Re: Jenkins CI

Posted by hongbin ma <ma...@apache.org>.
hi Nick,

Thank you for you message. I don't know if it's okay to discuss our(Apache
Kylin) CI requirements at Calcite's mail list. Please do let me know if
it's inappropriate.

For us Kylin, it's best if we could be provided with hadoop environment
which enables launching MR jobs, Hive queries and HBase scans. I'm not sure
if https://builds.apache.org/ could provide us such a isolated/shared
hadoop env.

If that's not possible, we do have a maven profile that run all the
integration tests with hbase minicluster. The concern is that minicluster
generates a lot of logs, and it may take around 30min~60min to finish all
the test cases. We once tried http://travis-ci.org/, but our job was
terminated because of too much logs. I don't know if
https://builds.apache.org/ has similar limitations.

BTW, can you give me a link to the script you mentioned?



On Fri, May 15, 2015 at 1:38 PM, Nick Dimiduk <nd...@gmail.com> wrote:

> I just set up the basics for Phoenix. Now that hbase 1.1 is released, I can
> look at these items over here too. Mahone, let me know what you need done.
> On Phoenix we reused HBase's scripts, which were recycled from Hadoop and
> others. I think Hadoop just rebooted theirs, so maybe have a look and see
> if you like it.
>
> On Thursday, May 14, 2015, hongbin ma <ma...@apache.org> wrote:
>
> > +1
> >
> > this is Mahone from apache kylin, please keep updated on the progress so
> > that we can borrow your experience:)
> >
> > On Fri, May 15, 2015 at 1:14 AM, Julian Hyde <julianhyde@gmail.com
> > <javascript:;>> wrote:
> >
> > > +1
> > >
> > > There is already an issue:
> > > https://issues.apache.org/jira/browse/CALCITE-623
> > >
> > > On Thu, May 14, 2015 at 8:57 AM, Vladimir Sitnikov
> > > <sitnikov.vladimir@gmail.com <javascript:;>> wrote:
> > > > Hi,
> > > >
> > > > I think it would be nice to setup a Jenkins CI for the following:
> > > > 1) Teting JIRA patches. I find it nice to have CI reports on the JIRA
> > > patches.
> > > > 2) Testing github pull requests
> > > > (https://blogs.apache.org/infra/entry/github_pull_request_builds_now
> )
> > > > 3) In the long run -- integration testing (ASF Jenkins supports
> > > > docker, so we can use it to test against postgresql, mysql, etc)
> > > >
> > > > If I understand wiki properly, someone from PMC Chairs can grant
> admin
> > > > account to a commiter
> > > > http://wiki.apache.org/general/Jenkins?action=show&redirect=Hudson
> > > >
> > > > Is it the case?
> > > > Are there any PMC members on the list that could grant me the
> account?
> > > >
> > > > Vladimir
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > *Bin Mahone | 马洪宾*
> > Apache Kylin: http://kylin.io
> > Github: https://github.com/binmahone
> >
>



-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone

Re: Jenkins CI

Posted by Nick Dimiduk <nd...@gmail.com>.
I just set up the basics for Phoenix. Now that hbase 1.1 is released, I can
look at these items over here too. Mahone, let me know what you need done.
On Phoenix we reused HBase's scripts, which were recycled from Hadoop and
others. I think Hadoop just rebooted theirs, so maybe have a look and see
if you like it.

On Thursday, May 14, 2015, hongbin ma <ma...@apache.org> wrote:

> +1
>
> this is Mahone from apache kylin, please keep updated on the progress so
> that we can borrow your experience:)
>
> On Fri, May 15, 2015 at 1:14 AM, Julian Hyde <julianhyde@gmail.com
> <javascript:;>> wrote:
>
> > +1
> >
> > There is already an issue:
> > https://issues.apache.org/jira/browse/CALCITE-623
> >
> > On Thu, May 14, 2015 at 8:57 AM, Vladimir Sitnikov
> > <sitnikov.vladimir@gmail.com <javascript:;>> wrote:
> > > Hi,
> > >
> > > I think it would be nice to setup a Jenkins CI for the following:
> > > 1) Teting JIRA patches. I find it nice to have CI reports on the JIRA
> > patches.
> > > 2) Testing github pull requests
> > > (https://blogs.apache.org/infra/entry/github_pull_request_builds_now)
> > > 3) In the long run -- integration testing (ASF Jenkins supports
> > > docker, so we can use it to test against postgresql, mysql, etc)
> > >
> > > If I understand wiki properly, someone from PMC Chairs can grant admin
> > > account to a commiter
> > > http://wiki.apache.org/general/Jenkins?action=show&redirect=Hudson
> > >
> > > Is it the case?
> > > Are there any PMC members on the list that could grant me the account?
> > >
> > > Vladimir
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>

Re: Jenkins CI

Posted by hongbin ma <ma...@apache.org>.
+1

this is Mahone from apache kylin, please keep updated on the progress so
that we can borrow your experience:)

On Fri, May 15, 2015 at 1:14 AM, Julian Hyde <ju...@gmail.com> wrote:

> +1
>
> There is already an issue:
> https://issues.apache.org/jira/browse/CALCITE-623
>
> On Thu, May 14, 2015 at 8:57 AM, Vladimir Sitnikov
> <si...@gmail.com> wrote:
> > Hi,
> >
> > I think it would be nice to setup a Jenkins CI for the following:
> > 1) Teting JIRA patches. I find it nice to have CI reports on the JIRA
> patches.
> > 2) Testing github pull requests
> > (https://blogs.apache.org/infra/entry/github_pull_request_builds_now)
> > 3) In the long run -- integration testing (ASF Jenkins supports
> > docker, so we can use it to test against postgresql, mysql, etc)
> >
> > If I understand wiki properly, someone from PMC Chairs can grant admin
> > account to a commiter
> > http://wiki.apache.org/general/Jenkins?action=show&redirect=Hudson
> >
> > Is it the case?
> > Are there any PMC members on the list that could grant me the account?
> >
> > Vladimir
>



-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone

Re: Jenkins CI

Posted by Julian Hyde <ju...@gmail.com>.
+1

There is already an issue: https://issues.apache.org/jira/browse/CALCITE-623

On Thu, May 14, 2015 at 8:57 AM, Vladimir Sitnikov
<si...@gmail.com> wrote:
> Hi,
>
> I think it would be nice to setup a Jenkins CI for the following:
> 1) Teting JIRA patches. I find it nice to have CI reports on the JIRA patches.
> 2) Testing github pull requests
> (https://blogs.apache.org/infra/entry/github_pull_request_builds_now)
> 3) In the long run -- integration testing (ASF Jenkins supports
> docker, so we can use it to test against postgresql, mysql, etc)
>
> If I understand wiki properly, someone from PMC Chairs can grant admin
> account to a commiter
> http://wiki.apache.org/general/Jenkins?action=show&redirect=Hudson
>
> Is it the case?
> Are there any PMC members on the list that could grant me the account?
>
> Vladimir