You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Dagang Wei <fu...@gmail.com> on 2018/10/16 20:57:48 UTC
Why the dependencies on bigtop-* packages are not reflected in bigtop.bom?
Hi folks,
I noticed that hadoop depends on bigtop-utils [1]:
Depends: ${shlibs:Depends}, ${misc:Depends}, adduser, bigtop-utils (>=
0.7), zookeeper (>= 3.4.0), psmisc, netcat-openbsd
But it is not reflected in bigtop.bom [2]:
dependencies = [
zookeeper:['hadoop', 'hbase'],
hadoop:['ignite-hadoop', 'hbase', 'crunch', 'hive', 'tez', 'sqoop',
'sqoop2',
'oozie', 'mahout', 'flume', 'giraph', 'solr', 'spark','spark1',
'phoenix', 'alluxio', 'kafka', 'ycsb', 'hama', 'zeppelin',
'tajo', 'apex'
],
hbase:['phoenix','giraph','ycsb','hive'],
hive:['oozie', 'zeppelin'],
'ignite-hadoop':['zeppelin'],
spark:['zeppelin']
]
Why is that? Should we add it?
[1]:
https://github.com/apache/bigtop/blob/master/bigtop-packages/src/deb/hadoop/control#L26
[2]: https://github.com/apache/bigtop/blob/master/bigtop.bom#L111
Re: Why the dependencies on bigtop-* packages are not reflected in
bigtop.bom?
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi Dagang,
If you look at the individual bigtop-packages/src/common/*/do-component-builds you may notice most of the Maven builds will call
"mvn install". As you may know this command installs the compiled maven artifacts in the local maven repository.
Zookeeper for instance populates itself into the maven cache with mvn install:install-file since it compile process is ant.
So if you are compiling bigtop in the right order without trashing mvn local repository, you end up picking up the artifacts you are compiled before. That's what I wanted to express as "build time dependency"
Our release builds are build that way ... btw
Best,
Olaf
> Am 17.10.2018 um 19:24 schrieb Dagang Wei <fu...@gmail.com>:
>
> Thanks Olaf! But how is build time dependencies reflected in Bigtop? e.g.,
> how does HBase depends on Hadoop at build time? If I clone a HBase git
> repo, I can build it without depending on Hadoop source, Hadoop JARs will
> be downloaded from Maven central.
>
> On Tue, Oct 16, 2018 at 10:29 PM Olaf Flebbe <of...@oflebbe.de> wrote:
>
>> Hi
>>
>> in bigtop.bom we note build time dependencies.
>>
>> package dependencies are for install time and runtime.
>>
>> best regards,
>> olaf
>>
>>
>>> Am 16.10.2018 um 22:57 schrieb Dagang Wei <fu...@gmail.com>:
>>>
>>> Hi folks,
>>>
>>> I noticed that hadoop depends on bigtop-utils [1]:
>>>
>>> Depends: ${shlibs:Depends}, ${misc:Depends}, adduser, bigtop-utils (>=
>>> 0.7), zookeeper (>= 3.4.0), psmisc, netcat-openbsd
>>>
>>> But it is not reflected in bigtop.bom [2]:
>>>
>>> dependencies = [
>>> zookeeper:['hadoop', 'hbase'],
>>> hadoop:['ignite-hadoop', 'hbase', 'crunch', 'hive', 'tez', 'sqoop',
>>> 'sqoop2',
>>> 'oozie', 'mahout', 'flume', 'giraph', 'solr', 'spark','spark1',
>>> 'phoenix', 'alluxio', 'kafka', 'ycsb', 'hama', 'zeppelin',
>>> 'tajo', 'apex'
>>> ],
>>> hbase:['phoenix','giraph','ycsb','hive'],
>>> hive:['oozie', 'zeppelin'],
>>> 'ignite-hadoop':['zeppelin'],
>>> spark:['zeppelin']
>>> ]
>>>
>>> Why is that? Should we add it?
>>>
>>> [1]:
>>>
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/deb/hadoop/control#L26
>>> [2]: https://github.com/apache/bigtop/blob/master/bigtop.bom#L111
>>
Re: Why the dependencies on bigtop-* packages are not reflected in bigtop.bom?
Posted by Konstantin Boudnik <co...@apache.org>.
Yes, you can but you'll end up with dependencies built elsewhere by
someone you might not trust in an environment you don't control.
That's the big problem right there. And we want Bigtop stack to be
consistent.
--
With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622
Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.
On Wed, Oct 17, 2018 at 8:24 PM, Dagang Wei <fu...@gmail.com> wrote:
> Thanks Olaf! But how is build time dependencies reflected in Bigtop? e.g.,
> how does HBase depends on Hadoop at build time? If I clone a HBase git
> repo, I can build it without depending on Hadoop source, Hadoop JARs will
> be downloaded from Maven central.
>
> On Tue, Oct 16, 2018 at 10:29 PM Olaf Flebbe <of...@oflebbe.de> wrote:
>
>> Hi
>>
>> in bigtop.bom we note build time dependencies.
>>
>> package dependencies are for install time and runtime.
>>
>> best regards,
>> olaf
>>
>>
>> > Am 16.10.2018 um 22:57 schrieb Dagang Wei <fu...@gmail.com>:
>> >
>> > Hi folks,
>> >
>> > I noticed that hadoop depends on bigtop-utils [1]:
>> >
>> > Depends: ${shlibs:Depends}, ${misc:Depends}, adduser, bigtop-utils (>=
>> > 0.7), zookeeper (>= 3.4.0), psmisc, netcat-openbsd
>> >
>> > But it is not reflected in bigtop.bom [2]:
>> >
>> > dependencies = [
>> > zookeeper:['hadoop', 'hbase'],
>> > hadoop:['ignite-hadoop', 'hbase', 'crunch', 'hive', 'tez', 'sqoop',
>> > 'sqoop2',
>> > 'oozie', 'mahout', 'flume', 'giraph', 'solr', 'spark','spark1',
>> > 'phoenix', 'alluxio', 'kafka', 'ycsb', 'hama', 'zeppelin',
>> > 'tajo', 'apex'
>> > ],
>> > hbase:['phoenix','giraph','ycsb','hive'],
>> > hive:['oozie', 'zeppelin'],
>> > 'ignite-hadoop':['zeppelin'],
>> > spark:['zeppelin']
>> > ]
>> >
>> > Why is that? Should we add it?
>> >
>> > [1]:
>> >
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/deb/hadoop/control#L26
>> > [2]: https://github.com/apache/bigtop/blob/master/bigtop.bom#L111
>>
Re: Why the dependencies on bigtop-* packages are not reflected in bigtop.bom?
Posted by Dagang Wei <fu...@gmail.com>.
Thanks Olaf! But how is build time dependencies reflected in Bigtop? e.g.,
how does HBase depends on Hadoop at build time? If I clone a HBase git
repo, I can build it without depending on Hadoop source, Hadoop JARs will
be downloaded from Maven central.
On Tue, Oct 16, 2018 at 10:29 PM Olaf Flebbe <of...@oflebbe.de> wrote:
> Hi
>
> in bigtop.bom we note build time dependencies.
>
> package dependencies are for install time and runtime.
>
> best regards,
> olaf
>
>
> > Am 16.10.2018 um 22:57 schrieb Dagang Wei <fu...@gmail.com>:
> >
> > Hi folks,
> >
> > I noticed that hadoop depends on bigtop-utils [1]:
> >
> > Depends: ${shlibs:Depends}, ${misc:Depends}, adduser, bigtop-utils (>=
> > 0.7), zookeeper (>= 3.4.0), psmisc, netcat-openbsd
> >
> > But it is not reflected in bigtop.bom [2]:
> >
> > dependencies = [
> > zookeeper:['hadoop', 'hbase'],
> > hadoop:['ignite-hadoop', 'hbase', 'crunch', 'hive', 'tez', 'sqoop',
> > 'sqoop2',
> > 'oozie', 'mahout', 'flume', 'giraph', 'solr', 'spark','spark1',
> > 'phoenix', 'alluxio', 'kafka', 'ycsb', 'hama', 'zeppelin',
> > 'tajo', 'apex'
> > ],
> > hbase:['phoenix','giraph','ycsb','hive'],
> > hive:['oozie', 'zeppelin'],
> > 'ignite-hadoop':['zeppelin'],
> > spark:['zeppelin']
> > ]
> >
> > Why is that? Should we add it?
> >
> > [1]:
> >
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/deb/hadoop/control#L26
> > [2]: https://github.com/apache/bigtop/blob/master/bigtop.bom#L111
>
Re: Why the dependencies on bigtop-* packages are not reflected in bigtop.bom?
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi
in bigtop.bom we note build time dependencies.
package dependencies are for install time and runtime.
best regards,
olaf
> Am 16.10.2018 um 22:57 schrieb Dagang Wei <fu...@gmail.com>:
>
> Hi folks,
>
> I noticed that hadoop depends on bigtop-utils [1]:
>
> Depends: ${shlibs:Depends}, ${misc:Depends}, adduser, bigtop-utils (>=
> 0.7), zookeeper (>= 3.4.0), psmisc, netcat-openbsd
>
> But it is not reflected in bigtop.bom [2]:
>
> dependencies = [
> zookeeper:['hadoop', 'hbase'],
> hadoop:['ignite-hadoop', 'hbase', 'crunch', 'hive', 'tez', 'sqoop',
> 'sqoop2',
> 'oozie', 'mahout', 'flume', 'giraph', 'solr', 'spark','spark1',
> 'phoenix', 'alluxio', 'kafka', 'ycsb', 'hama', 'zeppelin',
> 'tajo', 'apex'
> ],
> hbase:['phoenix','giraph','ycsb','hive'],
> hive:['oozie', 'zeppelin'],
> 'ignite-hadoop':['zeppelin'],
> spark:['zeppelin']
> ]
>
> Why is that? Should we add it?
>
> [1]:
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/deb/hadoop/control#L26
> [2]: https://github.com/apache/bigtop/blob/master/bigtop.bom#L111