You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Jason Wen <zh...@workday.com.INVALID> on 2021/02/24 23:11:28 UTC
How to build hadoop and all its dependent rpms
Hi team,
I build Hadoop rpms using `./gradlew hadoop-pkg-ind` successfully. When I view the rpm dependencies, I see hadoop rpms also depends on other bigtop rpms like `bigtop-utils`, `bigtop-jsvc`, etc.
I can deploy `bigtop-jsvc` using `./gradlew bigtop-jsvc-pkg-ind`, but I am wondering if there is a way that I can build all the dependent rpms for a specific component like hadoop.
Can anyone shed light on it?
Thanks,
Jason
Re: How to build hadoop and all its dependent rpms
Posted by Jason Wen <zh...@workday.com.INVALID>.
Hi Youngwoo,
Thanks for sharing the information.
Is init-hcfs.groovy the only case that depends on groovy in hadoop rpms? I am always confused on HCFS, but I never used it in a hadoop cluster. Also I don't see the same script in other distribution like CDH.
In the case that we don't use it, can we remove the dependency of bigtop-groovy on hadoop-hdfs rpm?
Thanks,
Jason
On 2/24/21, 6:48 PM, "Youngwoo Kim (김영우)" <yw...@apache.org> wrote:
Hi Jason,
See
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_BIGTOP-2D1097&d=DwIFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=FDi9-2BVmxFLsf0QUFmu8eL9gE_0bWxRCQLQy7fR4Bg&s=ZVDN9G6OOoysQGrPDzgA2dQQFU9PkvOTRdBExXYFUkM&e=
And also, init-hcfs is a groovy script.
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_bigtop_tree_master_bigtop-2Dpackages_src_common_hadoop&d=DwIFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=FDi9-2BVmxFLsf0QUFmu8eL9gE_0bWxRCQLQy7fR4Bg&s=1XYH9xKoUxy_o0wP13Im4cJBSJFJrhD9Yutj8_WJebY&e=
Thanks,
Youngwoo
2021년 2월 25일 (목) 오전 11:04, Jason Wen <zh...@workday.com.invalid>님이
작성:
> Hi Yuqi,
>
> Thanks for the information.
> I am curious why hadoop need depend on bigtop-groovy (I also find
> hadoop-hdfs rpm has a dependency to bigtop-groovy rpm). Is it needed for
> some specific feature in hdfs?
>
> Thanks,
> Jason
>
> On 2/24/21, 5:49 PM, "Yuqi Gu" <gu...@apache.org> wrote:
>
> Hi Jason,
>
> The dependencies of components have been defined in 'bigtop.bom' :
>
> * dependencies = [ 'bigtop-utils':['ambari', 'bigtop-groovy',
> 'bigtop-jsvc', 'bigtop-tomcat', 'elasticsearch',
> 'flink', 'logstash', 'qfs', 'zookeeper'],
> 'bigtop-groovy':['hadoop'],
> 'bigtop-jsvc':['hadoop'], 'bigtop-tomcat':['oozie', 'sqoop2'],
> zookeeper:['hadoop', 'hbase'], hadoop:['ignite-hadoop', 'hbase',
> 'hive',
> 'tez', 'sqoop', 'sqoop2', 'oozie', 'mahout', 'flume', 'giraph',
> 'solr', 'spark', 'phoenix', 'alluxio', 'kafka', 'ycsb', 'zeppelin'
> ], hbase:['phoenix','giraph','ycsb','hive'], hive:['oozie',
> 'zeppelin'], 'ignite-hadoop':['zeppelin'], spark:['livy',
> 'mahout',
> 'oozie', 'zeppelin'], flink:['mahout'] ]*
>
> Please check it before building related components. :)
> Thanks.
>
> BRs,
> Yuqi
>
>
> Jason Wen <zh...@workday.com.invalid> 于2021年2月25日周四 上午7:11写道：
>
> > Hi team,
> >
> > I build Hadoop rpms using `./gradlew hadoop-pkg-ind` successfully.
> When I
> > view the rpm dependencies, I see hadoop rpms also depends on other
> bigtop
> > rpms like `bigtop-utils`, `bigtop-jsvc`, etc.
> > I can deploy `bigtop-jsvc` using `./gradlew bigtop-jsvc-pkg-ind`,
> but I am
> > wondering if there is a way that I can build all the dependent rpms
> for a
> > specific component like hadoop.
> > Can anyone shed light on it?
> >
> > Thanks,
> > Jason
> >
>
>
Re: How to build hadoop and all its dependent rpms
Posted by "Youngwoo Kim (김영우)" <yw...@apache.org>.
Hi Jason,
See
https://issues.apache.org/jira/browse/BIGTOP-1097
And also, init-hcfs is a groovy script.
https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hadoop
Thanks,
Youngwoo
2021년 2월 25일 (목) 오전 11:04, Jason Wen <zh...@workday.com.invalid>님이
작성:
> Hi Yuqi,
>
> Thanks for the information.
> I am curious why hadoop need depend on bigtop-groovy (I also find
> hadoop-hdfs rpm has a dependency to bigtop-groovy rpm). Is it needed for
> some specific feature in hdfs?
>
> Thanks,
> Jason
>
> On 2/24/21, 5:49 PM, "Yuqi Gu" <gu...@apache.org> wrote:
>
> Hi Jason,
>
> The dependencies of components have been defined in 'bigtop.bom' :
>
> * dependencies = [ 'bigtop-utils':['ambari', 'bigtop-groovy',
> 'bigtop-jsvc', 'bigtop-tomcat', 'elasticsearch',
> 'flink', 'logstash', 'qfs', 'zookeeper'],
> 'bigtop-groovy':['hadoop'],
> 'bigtop-jsvc':['hadoop'], 'bigtop-tomcat':['oozie', 'sqoop2'],
> zookeeper:['hadoop', 'hbase'], hadoop:['ignite-hadoop', 'hbase',
> 'hive',
> 'tez', 'sqoop', 'sqoop2', 'oozie', 'mahout', 'flume', 'giraph',
> 'solr', 'spark', 'phoenix', 'alluxio', 'kafka', 'ycsb', 'zeppelin'
> ], hbase:['phoenix','giraph','ycsb','hive'], hive:['oozie',
> 'zeppelin'], 'ignite-hadoop':['zeppelin'], spark:['livy',
> 'mahout',
> 'oozie', 'zeppelin'], flink:['mahout'] ]*
>
> Please check it before building related components. :)
> Thanks.
>
> BRs,
> Yuqi
>
>
> Jason Wen <zh...@workday.com.invalid> 于2021年2月25日周四 上午7:11写道：
>
> > Hi team,
> >
> > I build Hadoop rpms using `./gradlew hadoop-pkg-ind` successfully.
> When I
> > view the rpm dependencies, I see hadoop rpms also depends on other
> bigtop
> > rpms like `bigtop-utils`, `bigtop-jsvc`, etc.
> > I can deploy `bigtop-jsvc` using `./gradlew bigtop-jsvc-pkg-ind`,
> but I am
> > wondering if there is a way that I can build all the dependent rpms
> for a
> > specific component like hadoop.
> > Can anyone shed light on it?
> >
> > Thanks,
> > Jason
> >
>
>
Re: How to build hadoop and all its dependent rpms
Posted by Jason Wen <zh...@workday.com.INVALID>.
Hi Yuqi,
Thanks for the information.
I am curious why hadoop need depend on bigtop-groovy (I also find hadoop-hdfs rpm has a dependency to bigtop-groovy rpm). Is it needed for some specific feature in hdfs?
Thanks,
Jason
On 2/24/21, 5:49 PM, "Yuqi Gu" <gu...@apache.org> wrote:
Hi Jason,
The dependencies of components have been defined in 'bigtop.bom' :
* dependencies = [ 'bigtop-utils':['ambari', 'bigtop-groovy',
'bigtop-jsvc', 'bigtop-tomcat', 'elasticsearch',
'flink', 'logstash', 'qfs', 'zookeeper'], 'bigtop-groovy':['hadoop'],
'bigtop-jsvc':['hadoop'], 'bigtop-tomcat':['oozie', 'sqoop2'],
zookeeper:['hadoop', 'hbase'], hadoop:['ignite-hadoop', 'hbase', 'hive',
'tez', 'sqoop', 'sqoop2', 'oozie', 'mahout', 'flume', 'giraph',
'solr', 'spark', 'phoenix', 'alluxio', 'kafka', 'ycsb', 'zeppelin'
], hbase:['phoenix','giraph','ycsb','hive'], hive:['oozie',
'zeppelin'], 'ignite-hadoop':['zeppelin'], spark:['livy', 'mahout',
'oozie', 'zeppelin'], flink:['mahout'] ]*
Please check it before building related components. :)
Thanks.
BRs,
Yuqi
Jason Wen <zh...@workday.com.invalid> 于2021年2月25日周四 上午7:11写道：
> Hi team,
>
> I build Hadoop rpms using `./gradlew hadoop-pkg-ind` successfully. When I
> view the rpm dependencies, I see hadoop rpms also depends on other bigtop
> rpms like `bigtop-utils`, `bigtop-jsvc`, etc.
> I can deploy `bigtop-jsvc` using `./gradlew bigtop-jsvc-pkg-ind`, but I am
> wondering if there is a way that I can build all the dependent rpms for a
> specific component like hadoop.
> Can anyone shed light on it?
>
> Thanks,
> Jason
>
Re: How to build hadoop and all its dependent rpms
Posted by Yuqi Gu <gu...@apache.org>.
Hi Jason,
The dependencies of components have been defined in 'bigtop.bom' :
* dependencies = [ 'bigtop-utils':['ambari', 'bigtop-groovy',
'bigtop-jsvc', 'bigtop-tomcat', 'elasticsearch',
'flink', 'logstash', 'qfs', 'zookeeper'], 'bigtop-groovy':['hadoop'],
'bigtop-jsvc':['hadoop'], 'bigtop-tomcat':['oozie', 'sqoop2'],
zookeeper:['hadoop', 'hbase'], hadoop:['ignite-hadoop', 'hbase', 'hive',
'tez', 'sqoop', 'sqoop2', 'oozie', 'mahout', 'flume', 'giraph',
'solr', 'spark', 'phoenix', 'alluxio', 'kafka', 'ycsb', 'zeppelin'
], hbase:['phoenix','giraph','ycsb','hive'], hive:['oozie',
'zeppelin'], 'ignite-hadoop':['zeppelin'], spark:['livy', 'mahout',
'oozie', 'zeppelin'], flink:['mahout'] ]*
Please check it before building related components. :)
Thanks.
BRs,
Yuqi
Jason Wen <zh...@workday.com.invalid> 于2021年2月25日周四 上午7:11写道：
> Hi team,
>
> I build Hadoop rpms using `./gradlew hadoop-pkg-ind` successfully. When I
> view the rpm dependencies, I see hadoop rpms also depends on other bigtop
> rpms like `bigtop-utils`, `bigtop-jsvc`, etc.
> I can deploy `bigtop-jsvc` using `./gradlew bigtop-jsvc-pkg-ind`, but I am
> wondering if there is a way that I can build all the dependent rpms for a
> specific component like hadoop.
> Can anyone shed light on it?
>
> Thanks,
> Jason
>
Re: How to build hadoop and all its dependent rpms
Posted by Jason Wen <zh...@workday.com.INVALID>.
Hi Masatake,
I tried `./gradlew hadoop-pkg-ind -Dbuildwithdeps=true -Pprefix=1.5.0`, but I only see hadoop rpms in build/ dir.
Thanks,
Jason
On 2/24/21, 5:42 PM, "Masatake Iwasaki" <iw...@oss.nttdata.co.jp> wrote:
Hi Jason,
Could you try -Dbuildwithdeps=true ?
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_bigtop_blob_master_bigtop.bom-23L104-2DL127&d=DwICaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=I33iIaqeUvsMsXaeWfBxG30_oeP8sIEeOa_GXvUg21s&s=EC1Ln7wb9yS8IClVe9XpBsHrLf4d6Mrn9y9hY4niklg&e=
If the dependency definition is incomplete, you can file a JIRA issue.
On 2021/02/25 8:11, Jason Wen wrote:
> Hi team,
>
> I build Hadoop rpms using `./gradlew hadoop-pkg-ind` successfully. When I view the rpm dependencies, I see hadoop rpms also depends on other bigtop rpms like `bigtop-utils`, `bigtop-jsvc`, etc.
> I can deploy `bigtop-jsvc` using `./gradlew bigtop-jsvc-pkg-ind`, but I am wondering if there is a way that I can build all the dependent rpms for a specific component like hadoop.
> Can anyone shed light on it?
>
> Thanks,
> Jason
>
Re: How to build hadoop and all its dependent rpms
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
Hi Jason,
Could you try -Dbuildwithdeps=true ?
https://github.com/apache/bigtop/blob/master/bigtop.bom#L104-L127
If the dependency definition is incomplete, you can file a JIRA issue.
On 2021/02/25 8:11, Jason Wen wrote:
> Hi team,
>
> I build Hadoop rpms using `./gradlew hadoop-pkg-ind` successfully. When I view the rpm dependencies, I see hadoop rpms also depends on other bigtop rpms like `bigtop-utils`, `bigtop-jsvc`, etc.
> I can deploy `bigtop-jsvc` using `./gradlew bigtop-jsvc-pkg-ind`, but I am wondering if there is a way that I can build all the dependent rpms for a specific component like hadoop.
> Can anyone shed light on it?
>
> Thanks,
> Jason
>