You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "Kengo Seki (Jira)" <ji...@apache.org> on 2021/02/08 06:59:00 UTC
[jira] [Updated] (BIGTOP-3471) Define Bigtop 3.0 release BOM
[ https://issues.apache.org/jira/browse/BIGTOP-3471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kengo Seki updated BIGTOP-3471:
-------------------------------
Description:
It's about time to discuss the next release BOM :)
Based on the past discussions [1][2], we're going to focus "core" components in the next release and drop the ones (at least temporarily) which are
* not so widely used by our users
* not actively developed recently
* incompatible (or not sure if compatible and no enough time/resource to investigate it) with Hadoop 3.x
so that we can publish our releases more frequently with limited resources.
To drop such components, I marked the ones that matches any of the above conditions as "to be removed" in the below list.
(Tez, Phoenix and Livy seem to be inactive recently, but they significantly improve performance or usability of Hadoop/Hive, HBase and Spark respectively, so I'd like to keep them as an exception, from my biased perspective)
So *please update the BOM if your required components are going to be removed!*
h3. Components
{code:java}
Components in v1.5 in v3.0
alluxio 1.8.2 => 2.4.1
ambari 2.6.1 => 2.7.5
bigtop-ambari-mpack 2.6.1 => 2.7.5
bigtop_groovy 2.5.4 => 2.5.4
bigtop_jsvc 1.0.15 => 1.0.15
bigtop_tomcat 8.5.57 => 8.5.57
bigtop_utils 1.5.0 => 3.0.0
elasticsearch 5.6.14 => removed
flink 1.6.4 => 1.11.3 [3]
flume 1.9.0 => removed
giraph 1.2.0 => removed
gpdb 5.10.0 => removed
hadoop 2.10.1 => 3.2.1 (or 3.3.x (BIGTOP-3403)? see also the discussions on BIGTOP-3280 and [4])
hbase 1.5.0 => 2.2.6 (or 2.3.3 [5], if possible)
hive 2.3.6 => 3.1.2 [6]
ignite_hadoop 2.7.6 => removed
kafka 2.4.0 => 2.4.1
kibana 5.4.1 => removed
livy 0.7.0 => 0.7.0
logstash 5.4.1 => removed
mahout 0.13.0 => removed
oozie 4.3.0 => 5.2.0
phoenix 4.15.0-HBase-1.5 => 5.0.0-HBase-2.0 [7]
qfs 2.0.0 => removed
solr 6.6.6 => 8.7.0 (or maybe 8.8.0?)
spark 2.4.5 => 3.0.1
sqoop 1.4.6 => 1.4.7
sqoop2 1.99.4 => removed
tez 0.9.2 => 0.10.0
ycsb 0.12.0 => 0.17.0
zeppelin 0.8.2 => 0.9.0 [8]
zookeeper 3.4.13 => 3.4.14 (or 3.5+ [9] if possible)
{code}
h3. Distros
{code:java}
- CentOS 7, 8
- Debian 10, 11 (if officially released before this release)
- Fedora 31
- Ubuntu 16.04, 18.04
{code}
Some points of discussion:
* Should we still support CentOS 8 which is supposed to be EOL'd within 2021 [10]?
* Should we update Fedora to 33, the latest version as of Jan 2021?
* Should we also support Ubuntu 20.04 additionally?
h3. Archs
{code:java}
- x86_64
- aarch64
- ppc64le (if the CI server is recovered)
{code}
Also, I tentatively assigned 3.0 as its version for the following reasons, but let's discuss it too if someone has another option.
* We're going to adopt Hadoop 3.x (and hopefully Spark 3.x), so Bigtop 3.0 is aligned with them [11].
* Bigtop 2.x is (unofficially) reserved for the "Cloud Native Bigtop" (BIGTOP-3225) release line [12].
* In my opinion, this major version leap is good to impress users backward incompatibility described above.
[1]: [https://lists.apache.org/thread.html/fe54be35856c8f6a23fb7d35b979e3dc90ad58baf94d15a206cdaf63%40%3Cdev.bigtop.apache.org%3E]
[2]: [https://lists.apache.org/thread.html/r71869111448bf5f8bc82e7031ff25d5455ab957e8e536d000026fe16%40%3Cdev.bigtop.apache.org%3E]
[3]: The latest release which is officially supported by Zeppelin 0.9.0. [https://twitter.com/ApacheZeppelin/status/1288855217895698433]
[4]: [https://github.com/apache/bigtop/pull/704]
[5]: The latest release which is officially compatible with Hadoop 3.2.x as of now. [https://hbase.apache.org/book.html#hadoop]
[6]: The latest release which is officially compatible with Hadoop 3.x.y as of now. [https://hive.apache.org/downloads.html#26-august-2019-release-312-available]
[7]: The latest release which is officially compatible with HBase 2.x as of now. [https://phoenix.apache.org/download.html]
[8]: The latest release which officially supports Spark 3.0 as of now. [https://twitter.com/ApacheZeppelin/status/1288855217895698433]
[9]: HBase 2.3.3 is built with Zookeeper 3.5.7 by default and doesn't seem to be built with Zookeeper 3.4.x as it is.
[10]: [https://blog.centos.org/2020/12/future-is-centos-stream/]
[11]: https://issues.apache.org/jira/browse/BIGTOP-3287?focusedCommentId=17018665&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17018665
[12]: [https://lists.apache.org/thread.html/e0af97d83c33bf370e25fca61aee95d0ed63193893c5f0ff7d187ebb%40%3Cdev.bigtop.apache.org%3E]
was:
It's about time to discuss the next release BOM :)
Based on the past discussions [1][2], we're going to focus "core" components in the next release and drop the ones (at least temporarily) which are
* not so widely used by our users
* not actively developed recently
* incompatible (or not sure if compatible and no enough time/resource to investigate it) with Hadoop 3.x
so that we can publish our releases more frequently with limited resources.
To drop such components, I marked the ones that matches any of the above conditions as "to be removed" in the below list.
(Tez, Phoenix and Livy seem to be inactive recently, but they significantly improve performance or usability of Hadoop/Hive, HBase and Spark respectively, so I'd like to keep them as an exception, from my biased perspective)
So *please update the BOM if your required components are going to be removed!*
h3. Components
{code:java}
Components in v1.5 in v3.0
alluxio 1.8.2 => 2.4.1
ambari 2.6.1 => 2.7.5
bigtop-ambari-mpack 2.6.1 => 2.7.5
bigtop_groovy 2.5.4 => 2.5.4
bigtop_jsvc 1.0.15 => 1.0.15
bigtop_tomcat 8.5.57 => 8.5.57
bigtop_utils 1.5.0 => 3.0.0
elasticsearch 5.6.14 => removed
flink 1.6.4 => 1.11.3 [3]
flume 1.9.0 => removed
giraph 1.2.0 => removed
gpdb 5.10.0 => removed
hadoop 2.10.1 => 3.2.1 (or 3.3.x (BIGTOP-3403)? see also the discussions on BIGTOP-3280 and [4])
hbase 1.5.0 => 2.2.6 (or 2.3.3 [5], if possible)
hive 2.3.6 => 3.1.2 [6]
ignite_hadoop 2.7.6 => removed
kafka 2.4.0 => 2.7.0
kibana 5.4.1 => removed
livy 0.7.0 => 0.7.0
logstash 5.4.1 => removed
mahout 0.13.0 => removed
oozie 4.3.0 => 5.2.0
phoenix 4.15.0-HBase-1.5 => 5.0.0-HBase-2.0 [7]
qfs 2.0.0 => removed
solr 6.6.6 => removed
spark 2.4.5 => 3.0.1
sqoop 1.4.6 => 1.4.7
sqoop2 1.99.4 => removed
tez 0.9.2 => 0.10.0
ycsb 0.12.0 => removed
zeppelin 0.8.2 => 0.9.0 [8]
zookeeper 3.4.13 => 3.4.14 (or 3.5+ [9] if possible)
{code}
h3. Distros
{code:java}
- CentOS 7, 8
- Debian 10, 11 (if officially released before this release)
- Fedora 31
- Ubuntu 16.04, 18.04
{code}
Some points of discussion:
* Should we still support CentOS 8 which is supposed to be EOL'd within 2021 [10]?
* Should we update Fedora to 33, the latest version as of Jan 2021?
* Should we also support Ubuntu 20.04 additionally?
h3. Archs
{code:java}
- x86_64
- aarch64
- ppc64le (if the CI server is recovered)
{code}
Also, I tentatively assigned 3.0 as its version for the following reasons, but let's discuss it too if someone has another option.
* We're going to adopt Hadoop 3.x (and hopefully Spark 3.x), so Bigtop 3.0 is aligned with them [11].
* Bigtop 2.x is (unofficially) reserved for the "Cloud Native Bigtop" (BIGTOP-3225) release line [12].
* In my opinion, this major version leap is good to impress users backward incompatibility described above.
[1]: [https://lists.apache.org/thread.html/fe54be35856c8f6a23fb7d35b979e3dc90ad58baf94d15a206cdaf63%40%3Cdev.bigtop.apache.org%3E]
[2]: [https://lists.apache.org/thread.html/r71869111448bf5f8bc82e7031ff25d5455ab957e8e536d000026fe16%40%3Cdev.bigtop.apache.org%3E]
[3]: The latest release which is officially supported by Zeppelin 0.9.0. [https://twitter.com/ApacheZeppelin/status/1288855217895698433]
[4]: [https://github.com/apache/bigtop/pull/704]
[5]: The latest release which is officially compatible with Hadoop 3.2.x as of now. [https://hbase.apache.org/book.html#hadoop]
[6]: The latest release which is officially compatible with Hadoop 3.x.y as of now. [https://hive.apache.org/downloads.html#26-august-2019-release-312-available]
[7]: The latest release which is officially compatible with HBase 2.x as of now. [https://phoenix.apache.org/download.html]
[8]: The latest release which officially supports Spark 3.0 as of now. [https://twitter.com/ApacheZeppelin/status/1288855217895698433]
[9]: HBase 2.3.3 is built with Zookeeper 3.5.7 by default and doesn't seem to be built with Zookeeper 3.4.x as it is.
[10]: [https://blog.centos.org/2020/12/future-is-centos-stream/]
[11]: https://issues.apache.org/jira/browse/BIGTOP-3287?focusedCommentId=17018665&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17018665
[12]: [https://lists.apache.org/thread.html/e0af97d83c33bf370e25fca61aee95d0ed63193893c5f0ff7d187ebb%40%3Cdev.bigtop.apache.org%3E]
> Define Bigtop 3.0 release BOM
> -----------------------------
>
> Key: BIGTOP-3471
> URL: https://issues.apache.org/jira/browse/BIGTOP-3471
> Project: Bigtop
> Issue Type: Task
> Reporter: Kengo Seki
> Assignee: Yuqi Gu
> Priority: Major
>
> It's about time to discuss the next release BOM :)
> Based on the past discussions [1][2], we're going to focus "core" components in the next release and drop the ones (at least temporarily) which are
> * not so widely used by our users
> * not actively developed recently
> * incompatible (or not sure if compatible and no enough time/resource to investigate it) with Hadoop 3.x
> so that we can publish our releases more frequently with limited resources.
> To drop such components, I marked the ones that matches any of the above conditions as "to be removed" in the below list.
> (Tez, Phoenix and Livy seem to be inactive recently, but they significantly improve performance or usability of Hadoop/Hive, HBase and Spark respectively, so I'd like to keep them as an exception, from my biased perspective)
> So *please update the BOM if your required components are going to be removed!*
> h3. Components
> {code:java}
> Components in v1.5 in v3.0
> alluxio 1.8.2 => 2.4.1
> ambari 2.6.1 => 2.7.5
> bigtop-ambari-mpack 2.6.1 => 2.7.5
> bigtop_groovy 2.5.4 => 2.5.4
> bigtop_jsvc 1.0.15 => 1.0.15
> bigtop_tomcat 8.5.57 => 8.5.57
> bigtop_utils 1.5.0 => 3.0.0
> elasticsearch 5.6.14 => removed
> flink 1.6.4 => 1.11.3 [3]
> flume 1.9.0 => removed
> giraph 1.2.0 => removed
> gpdb 5.10.0 => removed
> hadoop 2.10.1 => 3.2.1 (or 3.3.x (BIGTOP-3403)? see also the discussions on BIGTOP-3280 and [4])
> hbase 1.5.0 => 2.2.6 (or 2.3.3 [5], if possible)
> hive 2.3.6 => 3.1.2 [6]
> ignite_hadoop 2.7.6 => removed
> kafka 2.4.0 => 2.4.1
> kibana 5.4.1 => removed
> livy 0.7.0 => 0.7.0
> logstash 5.4.1 => removed
> mahout 0.13.0 => removed
> oozie 4.3.0 => 5.2.0
> phoenix 4.15.0-HBase-1.5 => 5.0.0-HBase-2.0 [7]
> qfs 2.0.0 => removed
> solr 6.6.6 => 8.7.0 (or maybe 8.8.0?)
> spark 2.4.5 => 3.0.1
> sqoop 1.4.6 => 1.4.7
> sqoop2 1.99.4 => removed
> tez 0.9.2 => 0.10.0
> ycsb 0.12.0 => 0.17.0
> zeppelin 0.8.2 => 0.9.0 [8]
> zookeeper 3.4.13 => 3.4.14 (or 3.5+ [9] if possible)
> {code}
> h3. Distros
> {code:java}
> - CentOS 7, 8
> - Debian 10, 11 (if officially released before this release)
> - Fedora 31
> - Ubuntu 16.04, 18.04
> {code}
> Some points of discussion:
> * Should we still support CentOS 8 which is supposed to be EOL'd within 2021 [10]?
> * Should we update Fedora to 33, the latest version as of Jan 2021?
> * Should we also support Ubuntu 20.04 additionally?
> h3. Archs
> {code:java}
> - x86_64
> - aarch64
> - ppc64le (if the CI server is recovered)
> {code}
> Also, I tentatively assigned 3.0 as its version for the following reasons, but let's discuss it too if someone has another option.
> * We're going to adopt Hadoop 3.x (and hopefully Spark 3.x), so Bigtop 3.0 is aligned with them [11].
> * Bigtop 2.x is (unofficially) reserved for the "Cloud Native Bigtop" (BIGTOP-3225) release line [12].
> * In my opinion, this major version leap is good to impress users backward incompatibility described above.
> [1]: [https://lists.apache.org/thread.html/fe54be35856c8f6a23fb7d35b979e3dc90ad58baf94d15a206cdaf63%40%3Cdev.bigtop.apache.org%3E]
> [2]: [https://lists.apache.org/thread.html/r71869111448bf5f8bc82e7031ff25d5455ab957e8e536d000026fe16%40%3Cdev.bigtop.apache.org%3E]
> [3]: The latest release which is officially supported by Zeppelin 0.9.0. [https://twitter.com/ApacheZeppelin/status/1288855217895698433]
> [4]: [https://github.com/apache/bigtop/pull/704]
> [5]: The latest release which is officially compatible with Hadoop 3.2.x as of now. [https://hbase.apache.org/book.html#hadoop]
> [6]: The latest release which is officially compatible with Hadoop 3.x.y as of now. [https://hive.apache.org/downloads.html#26-august-2019-release-312-available]
> [7]: The latest release which is officially compatible with HBase 2.x as of now. [https://phoenix.apache.org/download.html]
> [8]: The latest release which officially supports Spark 3.0 as of now. [https://twitter.com/ApacheZeppelin/status/1288855217895698433]
> [9]: HBase 2.3.3 is built with Zookeeper 3.5.7 by default and doesn't seem to be built with Zookeeper 3.4.x as it is.
> [10]: [https://blog.centos.org/2020/12/future-is-centos-stream/]
> [11]: https://issues.apache.org/jira/browse/BIGTOP-3287?focusedCommentId=17018665&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17018665
> [12]: [https://lists.apache.org/thread.html/e0af97d83c33bf370e25fca61aee95d0ed63193893c5f0ff7d187ebb%40%3Cdev.bigtop.apache.org%3E]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)