You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by "Cheng, Hao" <ha...@intel.com> on 2016/01/26 06:37:32 UTC
HiBench as part of Bigtop
Dear BigTop Devs,
I am from Intel Big Data Technology team, and we are the owner of HiBench, an open source benchmark suite for Hadoop / Spark ecosystem; as widely used, more and more trivial requirements from HiBench users, due to the limited resources, particularly the ease of deployment, we are exploring the possibility of getting the code into BigTop.
HiBench code can be found at: https://github.com/intel-hadoop/HiBench
Looking forward to your reply.
Regards,
Hao
RE: HiBench as part of Bigtop
Posted by "Cheng, Hao" <ha...@intel.com>.
Thank you RJ, Cos. it's really worth to think about the datagen and deployment. I will talk with team and get back soon.
-----Original Message-----
From: RJ Nowling [mailto:rnowling@gmail.com]
Sent: Wednesday, January 27, 2016 12:09 PM
To: dev@bigtop.apache.org
Subject: Re: HiBench as part of Bigtop
If HiBench itself were contributed to Bigtop, maybe we could work on joining forces with our efforts on the Bigtop Data Generators? Currently, the data generators are used in the BigPetStore examples but also for some Kubernetes examples. We could share in the maintenance (and vice versa), and I would be love to talk about ways to reuse code.
I'm open to adopting any code you might have or helping incorporating our code and data generators into HiBench.
My strength tends to be building data generators (the math modeling and
implementation) but my weakness is providing them in an easy-to-use form to end users. HiBench has done a great job of the latter.
On Tue, Jan 26, 2016 at 10:00 PM, Konstantin Boudnik <co...@apache.org> wrote:
> it depends in part of how you position HiBench. Ie, does it make sense
> to make it part of the deployable stack and be able to install it on a
> node to benchmark the whole cluster? Or there is a different
> deployment scenario you have in mind?
>
> Cos
>
> On Wed, Jan 27, 2016 at 03:45AM, Cheng, Hao wrote:
> > Which way does the community prefer? Just for packaging or the put
> entire project into bigtop?
> >
> > -----Original Message-----
> > From: RJ Nowling [mailto:rnowling@gmail.com]
> > Sent: Wednesday, January 27, 2016 11:24 AM
> > To: dev@bigtop.apache.org
> > Subject: Re: HiBench as part of Bigtop
> >
> > A few of us have been working on BigPetStore, a family of example
> > applications (blueprints). We've also been working on data
> > generators to create relatively complex fake data for those demo apps.
> >
> > On Tue, Jan 26, 2016 at 9:10 PM, Cheng, Hao <ha...@intel.com> wrote:
> >
> > > Hi RJ,
> > >
> > > Currently, we are targeting packaging the HiBench in Bigtop; but
> > > is there any other subproject entirely be part of BigTop?
> > >
> > > Thanks,
> > > Hao
> > >
> > > -----Original Message-----
> > > From: RJ Nowling [mailto:rnowling@gmail.com]
> > > Sent: Wednesday, January 27, 2016 11:04 AM
> > > To: dev@bigtop.apache.org
> > > Subject: Re: HiBench as part of Bigtop
> > >
> > > Hi Hao,
> > >
> > > Are you interested in just packaging HiBench in Bigtop or
> > > contributing the entire project to Bigtop?
> > >
> > > Is there someone who would be willing to take responsibility for
> > > maintenance?
> > >
> > > We generally like to find a maintainer for each package and add
> > > them to our maintainers list. This way we know who to contact if
> > > a build fails and blocks a release. If there is no maintainer
> > > (and the build has problems), we use that as grounds for removing
> > > the package from
> Bigtop.
> > >
> > > Thanks!
> > > RJ
> > >
> > >
> > > On Tue, Jan 26, 2016 at 1:15 AM, Cheng, Hao <ha...@intel.com>
> wrote:
> > >
> > > > Thank you Cos for the reply, we will take a look at YCSB. :)
> > > >
> > > > Hao
> > > >
> > > > -----Original Message-----
> > > > From: Konstantin Boudnik [mailto:cos@apache.org]
> > > > Sent: Tuesday, January 26, 2016 2:33 PM
> > > > To: dev@bigtop.apache.org
> > > > Subject: Re: HiBench as part of Bigtop
> > > >
> > > > Hi Hao.
> > > >
> > > > Thanks for your interest in the project and for considering us
> > > > as the new home for this benchmarking tool! We already have YCSB
> > > > as the part of stack, but clearly the more the merrier ;)
> > > >
> > > > A quick look at the source code shows that HiBench is already
> > > > licensed under ASL2, so the rest of it should be just a SMOP ;)
> > > > As you know, we do package all our components for both deb and
> > > > rpm package managers, deployment code, as well as do have
> > > > requirements for package and integration testing. I think the
> > > > good starting point would be to take a look at YCSB
> > > >
> > > > bigtop-packages/src/*/ycsb/
> > > > bigtop-deploy/puppet/modules/ycsb/
> > > >
> > > > and/or other components in the stack. This link
> > > >
> > > > https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contri
> > > > bute
> > > >
> > > > should also be helpful to understand the process of contributing
> > > > into the project. And you already found dev@ list, so you'll set
> > > > ;)
> > > >
> > > > I wonder how others in the community see this?
> > > > Cos
> > > >
> > > > On Tue, Jan 26, 2016 at 05:37AM, Cheng, Hao wrote:
> > > > > Dear BigTop Devs,
> > > > >
> > > > > I am from Intel Big Data Technology team, and we are the owner
> > > > > of HiBench, an open source benchmark suite for Hadoop / Spark
> > > > > ecosystem; as widely used, more and more trivial requirements
> > > > > from HiBench users, due to the limited resources, particularly
> > > > > the ease of deployment, we are exploring the possibility of
> > > > > getting the code
> > > into BigTop.
> > > > >
> > > > > HiBench code can be found at:
> > > > > https://github.com/intel-hadoop/HiBench
> > > > >
> > > > > Looking forward to your reply.
> > > > >
> > > > > Regards,
> > > > > Hao
> > > > >
> > > >
> > >
>
Re: HiBench as part of Bigtop
Posted by RJ Nowling <rn...@gmail.com>.
If HiBench itself were contributed to Bigtop, maybe we could work on
joining forces with our efforts on the Bigtop Data Generators? Currently,
the data generators are used in the BigPetStore examples but also for some
Kubernetes examples. We could share in the maintenance (and vice versa),
and I would be love to talk about ways to reuse code.
I'm open to adopting any code you might have or helping incorporating our
code and data generators into HiBench.
My strength tends to be building data generators (the math modeling and
implementation) but my weakness is providing them in an easy-to-use form to
end users. HiBench has done a great job of the latter.
On Tue, Jan 26, 2016 at 10:00 PM, Konstantin Boudnik <co...@apache.org> wrote:
> it depends in part of how you position HiBench. Ie, does it make sense to
> make
> it part of the deployable stack and be able to install it on a node to
> benchmark the whole cluster? Or there is a different deployment scenario
> you
> have in mind?
>
> Cos
>
> On Wed, Jan 27, 2016 at 03:45AM, Cheng, Hao wrote:
> > Which way does the community prefer? Just for packaging or the put
> entire project into bigtop?
> >
> > -----Original Message-----
> > From: RJ Nowling [mailto:rnowling@gmail.com]
> > Sent: Wednesday, January 27, 2016 11:24 AM
> > To: dev@bigtop.apache.org
> > Subject: Re: HiBench as part of Bigtop
> >
> > A few of us have been working on BigPetStore, a family of example
> > applications (blueprints). We've also been working on data generators to
> > create relatively complex fake data for those demo apps.
> >
> > On Tue, Jan 26, 2016 at 9:10 PM, Cheng, Hao <ha...@intel.com> wrote:
> >
> > > Hi RJ,
> > >
> > > Currently, we are targeting packaging the HiBench in Bigtop; but is
> > > there any other subproject entirely be part of BigTop?
> > >
> > > Thanks,
> > > Hao
> > >
> > > -----Original Message-----
> > > From: RJ Nowling [mailto:rnowling@gmail.com]
> > > Sent: Wednesday, January 27, 2016 11:04 AM
> > > To: dev@bigtop.apache.org
> > > Subject: Re: HiBench as part of Bigtop
> > >
> > > Hi Hao,
> > >
> > > Are you interested in just packaging HiBench in Bigtop or contributing
> > > the entire project to Bigtop?
> > >
> > > Is there someone who would be willing to take responsibility for
> > > maintenance?
> > >
> > > We generally like to find a maintainer for each package and add them
> > > to our maintainers list. This way we know who to contact if a build
> > > fails and blocks a release. If there is no maintainer (and the build
> > > has problems), we use that as grounds for removing the package from
> Bigtop.
> > >
> > > Thanks!
> > > RJ
> > >
> > >
> > > On Tue, Jan 26, 2016 at 1:15 AM, Cheng, Hao <ha...@intel.com>
> wrote:
> > >
> > > > Thank you Cos for the reply, we will take a look at YCSB. :)
> > > >
> > > > Hao
> > > >
> > > > -----Original Message-----
> > > > From: Konstantin Boudnik [mailto:cos@apache.org]
> > > > Sent: Tuesday, January 26, 2016 2:33 PM
> > > > To: dev@bigtop.apache.org
> > > > Subject: Re: HiBench as part of Bigtop
> > > >
> > > > Hi Hao.
> > > >
> > > > Thanks for your interest in the project and for considering us as
> > > > the new home for this benchmarking tool! We already have YCSB as the
> > > > part of stack, but clearly the more the merrier ;)
> > > >
> > > > A quick look at the source code shows that HiBench is already
> > > > licensed under ASL2, so the rest of it should be just a SMOP ;) As
> > > > you know, we do package all our components for both deb and rpm
> > > > package managers, deployment code, as well as do have requirements
> > > > for package and integration testing. I think the good starting point
> > > > would be to take a look at YCSB
> > > >
> > > > bigtop-packages/src/*/ycsb/
> > > > bigtop-deploy/puppet/modules/ycsb/
> > > >
> > > > and/or other components in the stack. This link
> > > >
> > > > https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
> > > >
> > > > should also be helpful to understand the process of contributing
> > > > into the project. And you already found dev@ list, so you'll set ;)
> > > >
> > > > I wonder how others in the community see this?
> > > > Cos
> > > >
> > > > On Tue, Jan 26, 2016 at 05:37AM, Cheng, Hao wrote:
> > > > > Dear BigTop Devs,
> > > > >
> > > > > I am from Intel Big Data Technology team, and we are the owner of
> > > > > HiBench, an open source benchmark suite for Hadoop / Spark
> > > > > ecosystem; as widely used, more and more trivial requirements from
> > > > > HiBench users, due to the limited resources, particularly the ease
> > > > > of deployment, we are exploring the possibility of getting the
> > > > > code
> > > into BigTop.
> > > > >
> > > > > HiBench code can be found at:
> > > > > https://github.com/intel-hadoop/HiBench
> > > > >
> > > > > Looking forward to your reply.
> > > > >
> > > > > Regards,
> > > > > Hao
> > > > >
> > > >
> > >
>
Re: HiBench as part of Bigtop
Posted by Konstantin Boudnik <co...@apache.org>.
it depends in part of how you position HiBench. Ie, does it make sense to make
it part of the deployable stack and be able to install it on a node to
benchmark the whole cluster? Or there is a different deployment scenario you
have in mind?
Cos
On Wed, Jan 27, 2016 at 03:45AM, Cheng, Hao wrote:
> Which way does the community prefer? Just for packaging or the put entire project into bigtop?
>
> -----Original Message-----
> From: RJ Nowling [mailto:rnowling@gmail.com]
> Sent: Wednesday, January 27, 2016 11:24 AM
> To: dev@bigtop.apache.org
> Subject: Re: HiBench as part of Bigtop
>
> A few of us have been working on BigPetStore, a family of example
> applications (blueprints). We've also been working on data generators to
> create relatively complex fake data for those demo apps.
>
> On Tue, Jan 26, 2016 at 9:10 PM, Cheng, Hao <ha...@intel.com> wrote:
>
> > Hi RJ,
> >
> > Currently, we are targeting packaging the HiBench in Bigtop; but is
> > there any other subproject entirely be part of BigTop?
> >
> > Thanks,
> > Hao
> >
> > -----Original Message-----
> > From: RJ Nowling [mailto:rnowling@gmail.com]
> > Sent: Wednesday, January 27, 2016 11:04 AM
> > To: dev@bigtop.apache.org
> > Subject: Re: HiBench as part of Bigtop
> >
> > Hi Hao,
> >
> > Are you interested in just packaging HiBench in Bigtop or contributing
> > the entire project to Bigtop?
> >
> > Is there someone who would be willing to take responsibility for
> > maintenance?
> >
> > We generally like to find a maintainer for each package and add them
> > to our maintainers list. This way we know who to contact if a build
> > fails and blocks a release. If there is no maintainer (and the build
> > has problems), we use that as grounds for removing the package from Bigtop.
> >
> > Thanks!
> > RJ
> >
> >
> > On Tue, Jan 26, 2016 at 1:15 AM, Cheng, Hao <ha...@intel.com> wrote:
> >
> > > Thank you Cos for the reply, we will take a look at YCSB. :)
> > >
> > > Hao
> > >
> > > -----Original Message-----
> > > From: Konstantin Boudnik [mailto:cos@apache.org]
> > > Sent: Tuesday, January 26, 2016 2:33 PM
> > > To: dev@bigtop.apache.org
> > > Subject: Re: HiBench as part of Bigtop
> > >
> > > Hi Hao.
> > >
> > > Thanks for your interest in the project and for considering us as
> > > the new home for this benchmarking tool! We already have YCSB as the
> > > part of stack, but clearly the more the merrier ;)
> > >
> > > A quick look at the source code shows that HiBench is already
> > > licensed under ASL2, so the rest of it should be just a SMOP ;) As
> > > you know, we do package all our components for both deb and rpm
> > > package managers, deployment code, as well as do have requirements
> > > for package and integration testing. I think the good starting point
> > > would be to take a look at YCSB
> > >
> > > bigtop-packages/src/*/ycsb/
> > > bigtop-deploy/puppet/modules/ycsb/
> > >
> > > and/or other components in the stack. This link
> > >
> > > https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
> > >
> > > should also be helpful to understand the process of contributing
> > > into the project. And you already found dev@ list, so you'll set ;)
> > >
> > > I wonder how others in the community see this?
> > > Cos
> > >
> > > On Tue, Jan 26, 2016 at 05:37AM, Cheng, Hao wrote:
> > > > Dear BigTop Devs,
> > > >
> > > > I am from Intel Big Data Technology team, and we are the owner of
> > > > HiBench, an open source benchmark suite for Hadoop / Spark
> > > > ecosystem; as widely used, more and more trivial requirements from
> > > > HiBench users, due to the limited resources, particularly the ease
> > > > of deployment, we are exploring the possibility of getting the
> > > > code
> > into BigTop.
> > > >
> > > > HiBench code can be found at:
> > > > https://github.com/intel-hadoop/HiBench
> > > >
> > > > Looking forward to your reply.
> > > >
> > > > Regards,
> > > > Hao
> > > >
> > >
> >
Re: HiBench as part of Bigtop
Posted by Olaf Flebbe <of...@oflebbe.de>.
> Am 27.01.2016 um 05:58 schrieb Jay Vyas <ja...@gmail.com>:
>
> If hibench can run as a gradle task that would be far more useful to the typical audience IMO and less technical debt to carry
>
> .....I think the amount of boilerplate for a package only is justified if it's a distributed component.
>
Or if it is to be deployed in a production environment
Olaf
Re: HiBench as part of Bigtop
Posted by Konstantin Boudnik <co...@apache.org>.
On Tue, Jan 26, 2016 at 11:58PM, Jay Vyas wrote:
> If hibench can run as a gradle task that would be far more useful to the typical audience IMO and less technical debt to carry
>
> .....I think the amount of boilerplate for a package only is justified if it's a distributed component.
Well put!
> > On Jan 26, 2016, at 11:03 PM, RJ Nowling <rn...@gmail.com> wrote:
> >
> > My personal view would be to start with adding packages. Then send an
> > email to the dev@ list to open a separate discussion.
> >
> >> On Tue, Jan 26, 2016 at 9:45 PM, Cheng, Hao <ha...@intel.com> wrote:
> >>
> >> Which way does the community prefer? Just for packaging or the put entire
> >> project into bigtop?
> >>
> >> -----Original Message-----
> >> From: RJ Nowling [mailto:rnowling@gmail.com]
> >> Sent: Wednesday, January 27, 2016 11:24 AM
> >> To: dev@bigtop.apache.org
> >> Subject: Re: HiBench as part of Bigtop
> >>
> >> A few of us have been working on BigPetStore, a family of example
> >> applications (blueprints). We've also been working on data generators to
> >> create relatively complex fake data for those demo apps.
> >>
> >>> On Tue, Jan 26, 2016 at 9:10 PM, Cheng, Hao <ha...@intel.com> wrote:
> >>>
> >>> Hi RJ,
> >>>
> >>> Currently, we are targeting packaging the HiBench in Bigtop; but is
> >>> there any other subproject entirely be part of BigTop?
> >>>
> >>> Thanks,
> >>> Hao
> >>>
> >>> -----Original Message-----
> >>> From: RJ Nowling [mailto:rnowling@gmail.com]
> >>> Sent: Wednesday, January 27, 2016 11:04 AM
> >>> To: dev@bigtop.apache.org
> >>> Subject: Re: HiBench as part of Bigtop
> >>>
> >>> Hi Hao,
> >>>
> >>> Are you interested in just packaging HiBench in Bigtop or contributing
> >>> the entire project to Bigtop?
> >>>
> >>> Is there someone who would be willing to take responsibility for
> >>> maintenance?
> >>>
> >>> We generally like to find a maintainer for each package and add them
> >>> to our maintainers list. This way we know who to contact if a build
> >>> fails and blocks a release. If there is no maintainer (and the build
> >>> has problems), we use that as grounds for removing the package from
> >> Bigtop.
> >>>
> >>> Thanks!
> >>> RJ
> >>>
> >>>
> >>>> On Tue, Jan 26, 2016 at 1:15 AM, Cheng, Hao <ha...@intel.com> wrote:
> >>>>
> >>>> Thank you Cos for the reply, we will take a look at YCSB. :)
> >>>>
> >>>> Hao
> >>>>
> >>>> -----Original Message-----
> >>>> From: Konstantin Boudnik [mailto:cos@apache.org]
> >>>> Sent: Tuesday, January 26, 2016 2:33 PM
> >>>> To: dev@bigtop.apache.org
> >>>> Subject: Re: HiBench as part of Bigtop
> >>>>
> >>>> Hi Hao.
> >>>>
> >>>> Thanks for your interest in the project and for considering us as
> >>>> the new home for this benchmarking tool! We already have YCSB as the
> >>>> part of stack, but clearly the more the merrier ;)
> >>>>
> >>>> A quick look at the source code shows that HiBench is already
> >>>> licensed under ASL2, so the rest of it should be just a SMOP ;) As
> >>>> you know, we do package all our components for both deb and rpm
> >>>> package managers, deployment code, as well as do have requirements
> >>>> for package and integration testing. I think the good starting point
> >>>> would be to take a look at YCSB
> >>>>
> >>>> bigtop-packages/src/*/ycsb/
> >>>> bigtop-deploy/puppet/modules/ycsb/
> >>>>
> >>>> and/or other components in the stack. This link
> >>>>
> >>>> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
> >>>>
> >>>> should also be helpful to understand the process of contributing
> >>>> into the project. And you already found dev@ list, so you'll set ;)
> >>>>
> >>>> I wonder how others in the community see this?
> >>>> Cos
> >>>>
> >>>>> On Tue, Jan 26, 2016 at 05:37AM, Cheng, Hao wrote:
> >>>>> Dear BigTop Devs,
> >>>>>
> >>>>> I am from Intel Big Data Technology team, and we are the owner of
> >>>>> HiBench, an open source benchmark suite for Hadoop / Spark
> >>>>> ecosystem; as widely used, more and more trivial requirements from
> >>>>> HiBench users, due to the limited resources, particularly the ease
> >>>>> of deployment, we are exploring the possibility of getting the
> >>>>> code
> >>> into BigTop.
> >>>>>
> >>>>> HiBench code can be found at:
> >>>>> https://github.com/intel-hadoop/HiBench
> >>>>>
> >>>>> Looking forward to your reply.
> >>>>>
> >>>>> Regards,
> >>>>> Hao
> >>
Re: HiBench as part of Bigtop
Posted by Jay Vyas <ja...@gmail.com>.
If hibench can run as a gradle task that would be far more useful to the typical audience IMO and less technical debt to carry
.....I think the amount of boilerplate for a package only is justified if it's a distributed component.
> On Jan 26, 2016, at 11:03 PM, RJ Nowling <rn...@gmail.com> wrote:
>
> My personal view would be to start with adding packages. Then send an
> email to the dev@ list to open a separate discussion.
>
>> On Tue, Jan 26, 2016 at 9:45 PM, Cheng, Hao <ha...@intel.com> wrote:
>>
>> Which way does the community prefer? Just for packaging or the put entire
>> project into bigtop?
>>
>> -----Original Message-----
>> From: RJ Nowling [mailto:rnowling@gmail.com]
>> Sent: Wednesday, January 27, 2016 11:24 AM
>> To: dev@bigtop.apache.org
>> Subject: Re: HiBench as part of Bigtop
>>
>> A few of us have been working on BigPetStore, a family of example
>> applications (blueprints). We've also been working on data generators to
>> create relatively complex fake data for those demo apps.
>>
>>> On Tue, Jan 26, 2016 at 9:10 PM, Cheng, Hao <ha...@intel.com> wrote:
>>>
>>> Hi RJ,
>>>
>>> Currently, we are targeting packaging the HiBench in Bigtop; but is
>>> there any other subproject entirely be part of BigTop?
>>>
>>> Thanks,
>>> Hao
>>>
>>> -----Original Message-----
>>> From: RJ Nowling [mailto:rnowling@gmail.com]
>>> Sent: Wednesday, January 27, 2016 11:04 AM
>>> To: dev@bigtop.apache.org
>>> Subject: Re: HiBench as part of Bigtop
>>>
>>> Hi Hao,
>>>
>>> Are you interested in just packaging HiBench in Bigtop or contributing
>>> the entire project to Bigtop?
>>>
>>> Is there someone who would be willing to take responsibility for
>>> maintenance?
>>>
>>> We generally like to find a maintainer for each package and add them
>>> to our maintainers list. This way we know who to contact if a build
>>> fails and blocks a release. If there is no maintainer (and the build
>>> has problems), we use that as grounds for removing the package from
>> Bigtop.
>>>
>>> Thanks!
>>> RJ
>>>
>>>
>>>> On Tue, Jan 26, 2016 at 1:15 AM, Cheng, Hao <ha...@intel.com> wrote:
>>>>
>>>> Thank you Cos for the reply, we will take a look at YCSB. :)
>>>>
>>>> Hao
>>>>
>>>> -----Original Message-----
>>>> From: Konstantin Boudnik [mailto:cos@apache.org]
>>>> Sent: Tuesday, January 26, 2016 2:33 PM
>>>> To: dev@bigtop.apache.org
>>>> Subject: Re: HiBench as part of Bigtop
>>>>
>>>> Hi Hao.
>>>>
>>>> Thanks for your interest in the project and for considering us as
>>>> the new home for this benchmarking tool! We already have YCSB as the
>>>> part of stack, but clearly the more the merrier ;)
>>>>
>>>> A quick look at the source code shows that HiBench is already
>>>> licensed under ASL2, so the rest of it should be just a SMOP ;) As
>>>> you know, we do package all our components for both deb and rpm
>>>> package managers, deployment code, as well as do have requirements
>>>> for package and integration testing. I think the good starting point
>>>> would be to take a look at YCSB
>>>>
>>>> bigtop-packages/src/*/ycsb/
>>>> bigtop-deploy/puppet/modules/ycsb/
>>>>
>>>> and/or other components in the stack. This link
>>>>
>>>> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
>>>>
>>>> should also be helpful to understand the process of contributing
>>>> into the project. And you already found dev@ list, so you'll set ;)
>>>>
>>>> I wonder how others in the community see this?
>>>> Cos
>>>>
>>>>> On Tue, Jan 26, 2016 at 05:37AM, Cheng, Hao wrote:
>>>>> Dear BigTop Devs,
>>>>>
>>>>> I am from Intel Big Data Technology team, and we are the owner of
>>>>> HiBench, an open source benchmark suite for Hadoop / Spark
>>>>> ecosystem; as widely used, more and more trivial requirements from
>>>>> HiBench users, due to the limited resources, particularly the ease
>>>>> of deployment, we are exploring the possibility of getting the
>>>>> code
>>> into BigTop.
>>>>>
>>>>> HiBench code can be found at:
>>>>> https://github.com/intel-hadoop/HiBench
>>>>>
>>>>> Looking forward to your reply.
>>>>>
>>>>> Regards,
>>>>> Hao
>>
Re: HiBench as part of Bigtop
Posted by RJ Nowling <rn...@gmail.com>.
My personal view would be to start with adding packages. Then send an
email to the dev@ list to open a separate discussion.
On Tue, Jan 26, 2016 at 9:45 PM, Cheng, Hao <ha...@intel.com> wrote:
> Which way does the community prefer? Just for packaging or the put entire
> project into bigtop?
>
> -----Original Message-----
> From: RJ Nowling [mailto:rnowling@gmail.com]
> Sent: Wednesday, January 27, 2016 11:24 AM
> To: dev@bigtop.apache.org
> Subject: Re: HiBench as part of Bigtop
>
> A few of us have been working on BigPetStore, a family of example
> applications (blueprints). We've also been working on data generators to
> create relatively complex fake data for those demo apps.
>
> On Tue, Jan 26, 2016 at 9:10 PM, Cheng, Hao <ha...@intel.com> wrote:
>
> > Hi RJ,
> >
> > Currently, we are targeting packaging the HiBench in Bigtop; but is
> > there any other subproject entirely be part of BigTop?
> >
> > Thanks,
> > Hao
> >
> > -----Original Message-----
> > From: RJ Nowling [mailto:rnowling@gmail.com]
> > Sent: Wednesday, January 27, 2016 11:04 AM
> > To: dev@bigtop.apache.org
> > Subject: Re: HiBench as part of Bigtop
> >
> > Hi Hao,
> >
> > Are you interested in just packaging HiBench in Bigtop or contributing
> > the entire project to Bigtop?
> >
> > Is there someone who would be willing to take responsibility for
> > maintenance?
> >
> > We generally like to find a maintainer for each package and add them
> > to our maintainers list. This way we know who to contact if a build
> > fails and blocks a release. If there is no maintainer (and the build
> > has problems), we use that as grounds for removing the package from
> Bigtop.
> >
> > Thanks!
> > RJ
> >
> >
> > On Tue, Jan 26, 2016 at 1:15 AM, Cheng, Hao <ha...@intel.com> wrote:
> >
> > > Thank you Cos for the reply, we will take a look at YCSB. :)
> > >
> > > Hao
> > >
> > > -----Original Message-----
> > > From: Konstantin Boudnik [mailto:cos@apache.org]
> > > Sent: Tuesday, January 26, 2016 2:33 PM
> > > To: dev@bigtop.apache.org
> > > Subject: Re: HiBench as part of Bigtop
> > >
> > > Hi Hao.
> > >
> > > Thanks for your interest in the project and for considering us as
> > > the new home for this benchmarking tool! We already have YCSB as the
> > > part of stack, but clearly the more the merrier ;)
> > >
> > > A quick look at the source code shows that HiBench is already
> > > licensed under ASL2, so the rest of it should be just a SMOP ;) As
> > > you know, we do package all our components for both deb and rpm
> > > package managers, deployment code, as well as do have requirements
> > > for package and integration testing. I think the good starting point
> > > would be to take a look at YCSB
> > >
> > > bigtop-packages/src/*/ycsb/
> > > bigtop-deploy/puppet/modules/ycsb/
> > >
> > > and/or other components in the stack. This link
> > >
> > > https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
> > >
> > > should also be helpful to understand the process of contributing
> > > into the project. And you already found dev@ list, so you'll set ;)
> > >
> > > I wonder how others in the community see this?
> > > Cos
> > >
> > > On Tue, Jan 26, 2016 at 05:37AM, Cheng, Hao wrote:
> > > > Dear BigTop Devs,
> > > >
> > > > I am from Intel Big Data Technology team, and we are the owner of
> > > > HiBench, an open source benchmark suite for Hadoop / Spark
> > > > ecosystem; as widely used, more and more trivial requirements from
> > > > HiBench users, due to the limited resources, particularly the ease
> > > > of deployment, we are exploring the possibility of getting the
> > > > code
> > into BigTop.
> > > >
> > > > HiBench code can be found at:
> > > > https://github.com/intel-hadoop/HiBench
> > > >
> > > > Looking forward to your reply.
> > > >
> > > > Regards,
> > > > Hao
> > > >
> > >
> >
>
RE: HiBench as part of Bigtop
Posted by "Cheng, Hao" <ha...@intel.com>.
Which way does the community prefer? Just for packaging or the put entire project into bigtop?
-----Original Message-----
From: RJ Nowling [mailto:rnowling@gmail.com]
Sent: Wednesday, January 27, 2016 11:24 AM
To: dev@bigtop.apache.org
Subject: Re: HiBench as part of Bigtop
A few of us have been working on BigPetStore, a family of example applications (blueprints). We've also been working on data generators to create relatively complex fake data for those demo apps.
On Tue, Jan 26, 2016 at 9:10 PM, Cheng, Hao <ha...@intel.com> wrote:
> Hi RJ,
>
> Currently, we are targeting packaging the HiBench in Bigtop; but is
> there any other subproject entirely be part of BigTop?
>
> Thanks,
> Hao
>
> -----Original Message-----
> From: RJ Nowling [mailto:rnowling@gmail.com]
> Sent: Wednesday, January 27, 2016 11:04 AM
> To: dev@bigtop.apache.org
> Subject: Re: HiBench as part of Bigtop
>
> Hi Hao,
>
> Are you interested in just packaging HiBench in Bigtop or contributing
> the entire project to Bigtop?
>
> Is there someone who would be willing to take responsibility for
> maintenance?
>
> We generally like to find a maintainer for each package and add them
> to our maintainers list. This way we know who to contact if a build
> fails and blocks a release. If there is no maintainer (and the build
> has problems), we use that as grounds for removing the package from Bigtop.
>
> Thanks!
> RJ
>
>
> On Tue, Jan 26, 2016 at 1:15 AM, Cheng, Hao <ha...@intel.com> wrote:
>
> > Thank you Cos for the reply, we will take a look at YCSB. :)
> >
> > Hao
> >
> > -----Original Message-----
> > From: Konstantin Boudnik [mailto:cos@apache.org]
> > Sent: Tuesday, January 26, 2016 2:33 PM
> > To: dev@bigtop.apache.org
> > Subject: Re: HiBench as part of Bigtop
> >
> > Hi Hao.
> >
> > Thanks for your interest in the project and for considering us as
> > the new home for this benchmarking tool! We already have YCSB as the
> > part of stack, but clearly the more the merrier ;)
> >
> > A quick look at the source code shows that HiBench is already
> > licensed under ASL2, so the rest of it should be just a SMOP ;) As
> > you know, we do package all our components for both deb and rpm
> > package managers, deployment code, as well as do have requirements
> > for package and integration testing. I think the good starting point
> > would be to take a look at YCSB
> >
> > bigtop-packages/src/*/ycsb/
> > bigtop-deploy/puppet/modules/ycsb/
> >
> > and/or other components in the stack. This link
> >
> > https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
> >
> > should also be helpful to understand the process of contributing
> > into the project. And you already found dev@ list, so you'll set ;)
> >
> > I wonder how others in the community see this?
> > Cos
> >
> > On Tue, Jan 26, 2016 at 05:37AM, Cheng, Hao wrote:
> > > Dear BigTop Devs,
> > >
> > > I am from Intel Big Data Technology team, and we are the owner of
> > > HiBench, an open source benchmark suite for Hadoop / Spark
> > > ecosystem; as widely used, more and more trivial requirements from
> > > HiBench users, due to the limited resources, particularly the ease
> > > of deployment, we are exploring the possibility of getting the
> > > code
> into BigTop.
> > >
> > > HiBench code can be found at:
> > > https://github.com/intel-hadoop/HiBench
> > >
> > > Looking forward to your reply.
> > >
> > > Regards,
> > > Hao
> > >
> >
>
Re: HiBench as part of Bigtop
Posted by RJ Nowling <rn...@gmail.com>.
A few of us have been working on BigPetStore, a family of example
applications (blueprints). We've also been working on data generators to
create relatively complex fake data for those demo apps.
On Tue, Jan 26, 2016 at 9:10 PM, Cheng, Hao <ha...@intel.com> wrote:
> Hi RJ,
>
> Currently, we are targeting packaging the HiBench in Bigtop; but is there
> any other subproject entirely be part of BigTop?
>
> Thanks,
> Hao
>
> -----Original Message-----
> From: RJ Nowling [mailto:rnowling@gmail.com]
> Sent: Wednesday, January 27, 2016 11:04 AM
> To: dev@bigtop.apache.org
> Subject: Re: HiBench as part of Bigtop
>
> Hi Hao,
>
> Are you interested in just packaging HiBench in Bigtop or contributing the
> entire project to Bigtop?
>
> Is there someone who would be willing to take responsibility for
> maintenance?
>
> We generally like to find a maintainer for each package and add them to
> our maintainers list. This way we know who to contact if a build fails and
> blocks a release. If there is no maintainer (and the build has problems),
> we use that as grounds for removing the package from Bigtop.
>
> Thanks!
> RJ
>
>
> On Tue, Jan 26, 2016 at 1:15 AM, Cheng, Hao <ha...@intel.com> wrote:
>
> > Thank you Cos for the reply, we will take a look at YCSB. :)
> >
> > Hao
> >
> > -----Original Message-----
> > From: Konstantin Boudnik [mailto:cos@apache.org]
> > Sent: Tuesday, January 26, 2016 2:33 PM
> > To: dev@bigtop.apache.org
> > Subject: Re: HiBench as part of Bigtop
> >
> > Hi Hao.
> >
> > Thanks for your interest in the project and for considering us as the
> > new home for this benchmarking tool! We already have YCSB as the part
> > of stack, but clearly the more the merrier ;)
> >
> > A quick look at the source code shows that HiBench is already licensed
> > under ASL2, so the rest of it should be just a SMOP ;) As you know, we
> > do package all our components for both deb and rpm package managers,
> > deployment code, as well as do have requirements for package and
> > integration testing. I think the good starting point would be to take
> > a look at YCSB
> >
> > bigtop-packages/src/*/ycsb/
> > bigtop-deploy/puppet/modules/ycsb/
> >
> > and/or other components in the stack. This link
> >
> > https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
> >
> > should also be helpful to understand the process of contributing into
> > the project. And you already found dev@ list, so you'll set ;)
> >
> > I wonder how others in the community see this?
> > Cos
> >
> > On Tue, Jan 26, 2016 at 05:37AM, Cheng, Hao wrote:
> > > Dear BigTop Devs,
> > >
> > > I am from Intel Big Data Technology team, and we are the owner of
> > > HiBench, an open source benchmark suite for Hadoop / Spark
> > > ecosystem; as widely used, more and more trivial requirements from
> > > HiBench users, due to the limited resources, particularly the ease
> > > of deployment, we are exploring the possibility of getting the code
> into BigTop.
> > >
> > > HiBench code can be found at:
> > > https://github.com/intel-hadoop/HiBench
> > >
> > > Looking forward to your reply.
> > >
> > > Regards,
> > > Hao
> > >
> >
>
RE: HiBench as part of Bigtop
Posted by "Cheng, Hao" <ha...@intel.com>.
Hi RJ,
Currently, we are targeting packaging the HiBench in Bigtop; but is there any other subproject entirely be part of BigTop?
Thanks,
Hao
-----Original Message-----
From: RJ Nowling [mailto:rnowling@gmail.com]
Sent: Wednesday, January 27, 2016 11:04 AM
To: dev@bigtop.apache.org
Subject: Re: HiBench as part of Bigtop
Hi Hao,
Are you interested in just packaging HiBench in Bigtop or contributing the entire project to Bigtop?
Is there someone who would be willing to take responsibility for maintenance?
We generally like to find a maintainer for each package and add them to our maintainers list. This way we know who to contact if a build fails and blocks a release. If there is no maintainer (and the build has problems), we use that as grounds for removing the package from Bigtop.
Thanks!
RJ
On Tue, Jan 26, 2016 at 1:15 AM, Cheng, Hao <ha...@intel.com> wrote:
> Thank you Cos for the reply, we will take a look at YCSB. :)
>
> Hao
>
> -----Original Message-----
> From: Konstantin Boudnik [mailto:cos@apache.org]
> Sent: Tuesday, January 26, 2016 2:33 PM
> To: dev@bigtop.apache.org
> Subject: Re: HiBench as part of Bigtop
>
> Hi Hao.
>
> Thanks for your interest in the project and for considering us as the
> new home for this benchmarking tool! We already have YCSB as the part
> of stack, but clearly the more the merrier ;)
>
> A quick look at the source code shows that HiBench is already licensed
> under ASL2, so the rest of it should be just a SMOP ;) As you know, we
> do package all our components for both deb and rpm package managers,
> deployment code, as well as do have requirements for package and
> integration testing. I think the good starting point would be to take
> a look at YCSB
>
> bigtop-packages/src/*/ycsb/
> bigtop-deploy/puppet/modules/ycsb/
>
> and/or other components in the stack. This link
>
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
>
> should also be helpful to understand the process of contributing into
> the project. And you already found dev@ list, so you'll set ;)
>
> I wonder how others in the community see this?
> Cos
>
> On Tue, Jan 26, 2016 at 05:37AM, Cheng, Hao wrote:
> > Dear BigTop Devs,
> >
> > I am from Intel Big Data Technology team, and we are the owner of
> > HiBench, an open source benchmark suite for Hadoop / Spark
> > ecosystem; as widely used, more and more trivial requirements from
> > HiBench users, due to the limited resources, particularly the ease
> > of deployment, we are exploring the possibility of getting the code into BigTop.
> >
> > HiBench code can be found at:
> > https://github.com/intel-hadoop/HiBench
> >
> > Looking forward to your reply.
> >
> > Regards,
> > Hao
> >
>
Re: HiBench as part of Bigtop
Posted by RJ Nowling <rn...@gmail.com>.
Hi Hao,
Are you interested in just packaging HiBench in Bigtop or contributing the
entire project to Bigtop?
Is there someone who would be willing to take responsibility for
maintenance?
We generally like to find a maintainer for each package and add them to our
maintainers list. This way we know who to contact if a build fails and
blocks a release. If there is no maintainer (and the build has problems),
we use that as grounds for removing the package from Bigtop.
Thanks!
RJ
On Tue, Jan 26, 2016 at 1:15 AM, Cheng, Hao <ha...@intel.com> wrote:
> Thank you Cos for the reply, we will take a look at YCSB. :)
>
> Hao
>
> -----Original Message-----
> From: Konstantin Boudnik [mailto:cos@apache.org]
> Sent: Tuesday, January 26, 2016 2:33 PM
> To: dev@bigtop.apache.org
> Subject: Re: HiBench as part of Bigtop
>
> Hi Hao.
>
> Thanks for your interest in the project and for considering us as the new
> home for this benchmarking tool! We already have YCSB as the part of stack,
> but clearly the more the merrier ;)
>
> A quick look at the source code shows that HiBench is already licensed
> under ASL2, so the rest of it should be just a SMOP ;) As you know, we do
> package all our components for both deb and rpm package managers,
> deployment code, as well as do have requirements for package and
> integration testing. I think the good starting point would be to take a
> look at YCSB
>
> bigtop-packages/src/*/ycsb/
> bigtop-deploy/puppet/modules/ycsb/
>
> and/or other components in the stack. This link
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
>
> should also be helpful to understand the process of contributing into the
> project. And you already found dev@ list, so you'll set ;)
>
> I wonder how others in the community see this?
> Cos
>
> On Tue, Jan 26, 2016 at 05:37AM, Cheng, Hao wrote:
> > Dear BigTop Devs,
> >
> > I am from Intel Big Data Technology team, and we are the owner of
> > HiBench, an open source benchmark suite for Hadoop / Spark ecosystem;
> > as widely used, more and more trivial requirements from HiBench users,
> > due to the limited resources, particularly the ease of deployment, we
> > are exploring the possibility of getting the code into BigTop.
> >
> > HiBench code can be found at: https://github.com/intel-hadoop/HiBench
> >
> > Looking forward to your reply.
> >
> > Regards,
> > Hao
> >
>
RE: HiBench as part of Bigtop
Posted by "Cheng, Hao" <ha...@intel.com>.
Thank you Cos for the reply, we will take a look at YCSB. :)
Hao
-----Original Message-----
From: Konstantin Boudnik [mailto:cos@apache.org]
Sent: Tuesday, January 26, 2016 2:33 PM
To: dev@bigtop.apache.org
Subject: Re: HiBench as part of Bigtop
Hi Hao.
Thanks for your interest in the project and for considering us as the new home for this benchmarking tool! We already have YCSB as the part of stack, but clearly the more the merrier ;)
A quick look at the source code shows that HiBench is already licensed under ASL2, so the rest of it should be just a SMOP ;) As you know, we do package all our components for both deb and rpm package managers, deployment code, as well as do have requirements for package and integration testing. I think the good starting point would be to take a look at YCSB
bigtop-packages/src/*/ycsb/
bigtop-deploy/puppet/modules/ycsb/
and/or other components in the stack. This link
https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
should also be helpful to understand the process of contributing into the project. And you already found dev@ list, so you'll set ;)
I wonder how others in the community see this?
Cos
On Tue, Jan 26, 2016 at 05:37AM, Cheng, Hao wrote:
> Dear BigTop Devs,
>
> I am from Intel Big Data Technology team, and we are the owner of
> HiBench, an open source benchmark suite for Hadoop / Spark ecosystem;
> as widely used, more and more trivial requirements from HiBench users,
> due to the limited resources, particularly the ease of deployment, we
> are exploring the possibility of getting the code into BigTop.
>
> HiBench code can be found at: https://github.com/intel-hadoop/HiBench
>
> Looking forward to your reply.
>
> Regards,
> Hao
>
Re: HiBench as part of Bigtop
Posted by Konstantin Boudnik <co...@apache.org>.
Hi Hao.
Thanks for your interest in the project and for considering us as the new home
for this benchmarking tool! We already have YCSB as the part of stack, but
clearly the more the merrier ;)
A quick look at the source code shows that HiBench is already licensed under
ASL2, so the rest of it should be just a SMOP ;) As you know, we do package
all our components for both deb and rpm package managers, deployment code, as
well as do have requirements for package and integration testing. I think the
good starting point would be to take a look at YCSB
bigtop-packages/src/*/ycsb/
bigtop-deploy/puppet/modules/ycsb/
and/or other components in the stack. This link
https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
should also be helpful to understand the process of contributing into the
project. And you already found dev@ list, so you'll set ;)
I wonder how others in the community see this?
Cos
On Tue, Jan 26, 2016 at 05:37AM, Cheng, Hao wrote:
> Dear BigTop Devs,
>
> I am from Intel Big Data Technology team, and we are the owner of HiBench,
> an open source benchmark suite for Hadoop / Spark ecosystem; as widely used,
> more and more trivial requirements from HiBench users, due to the limited
> resources, particularly the ease of deployment, we are exploring the
> possibility of getting the code into BigTop.
>
> HiBench code can be found at: https://github.com/intel-hadoop/HiBench
>
> Looking forward to your reply.
>
> Regards,
> Hao
>
Re: HiBench as part of Bigtop
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
A very belated +1 from me as well!
Thanks,
Roman.
On Wed, Feb 1, 2017 at 7:40 AM, jay vyas <ja...@gmail.com> wrote:
> big +1 for integration , i don't work in the hadoop space at all anymore so
> i don't have any other feedback then that it would be a good addition to
> ASF bigtop!
>
> On Wed, Mar 2, 2016 at 8:18 PM, 김영우 (YoungWoo Kim) <yw...@apache.org> wrote:
>
>> Hao,
>>
>> I'm glad to hear that as a hibench user :-) Any progress on this?
>>
>> Thanks,
>> Youngwoo
>>
>> 2016년 1월 26일 화요일, Cheng, Hao<ha...@intel.com>님이 작성한 메시지:
>>
>> > Dear BigTop Devs,
>> >
>> > I am from Intel Big Data Technology team, and we are the owner of
>> HiBench,
>> > an open source benchmark suite for Hadoop / Spark ecosystem; as widely
>> > used, more and more trivial requirements from HiBench users, due to the
>> > limited resources, particularly the ease of deployment, we are exploring
>> > the possibility of getting the code into BigTop.
>> >
>> > HiBench code can be found at: https://github.com/intel-hadoop/HiBench
>> >
>> > Looking forward to your reply.
>> >
>> > Regards,
>> > Hao
>> >
>> >
>>
>
>
>
> --
> jay vyas
Re: HiBench as part of Bigtop
Posted by jay vyas <ja...@gmail.com>.
big +1 for integration , i don't work in the hadoop space at all anymore so
i don't have any other feedback then that it would be a good addition to
ASF bigtop!
On Wed, Mar 2, 2016 at 8:18 PM, 김영우 (YoungWoo Kim) <yw...@apache.org> wrote:
> Hao,
>
> I'm glad to hear that as a hibench user :-) Any progress on this?
>
> Thanks,
> Youngwoo
>
> 2016년 1월 26일 화요일, Cheng, Hao<ha...@intel.com>님이 작성한 메시지:
>
> > Dear BigTop Devs,
> >
> > I am from Intel Big Data Technology team, and we are the owner of
> HiBench,
> > an open source benchmark suite for Hadoop / Spark ecosystem; as widely
> > used, more and more trivial requirements from HiBench users, due to the
> > limited resources, particularly the ease of deployment, we are exploring
> > the possibility of getting the code into BigTop.
> >
> > HiBench code can be found at: https://github.com/intel-hadoop/HiBench
> >
> > Looking forward to your reply.
> >
> > Regards,
> > Hao
> >
> >
>
--
jay vyas
Re: HiBench as part of Bigtop
Posted by "김영우 (YoungWoo Kim)" <yw...@apache.org>.
Hao,
I'm glad to hear that as a hibench user :-) Any progress on this?
Thanks,
Youngwoo
2016년 1월 26일 화요일, Cheng, Hao<ha...@intel.com>님이 작성한 메시지:
> Dear BigTop Devs,
>
> I am from Intel Big Data Technology team, and we are the owner of HiBench,
> an open source benchmark suite for Hadoop / Spark ecosystem; as widely
> used, more and more trivial requirements from HiBench users, due to the
> limited resources, particularly the ease of deployment, we are exploring
> the possibility of getting the code into BigTop.
>
> HiBench code can be found at: https://github.com/intel-hadoop/HiBench
>
> Looking forward to your reply.
>
> Regards,
> Hao
>
>