You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "Battula, Brahma Reddy" <bb...@visa.com.INVALID> on 2022/06/19 17:07:48 UTC
[Discuss] Initial Draft for Roadmap
Hi All,
As Ambari back now, let’s start discussing roadmap. I will be closely tracking for Jira, GitHub(with Jenkins) and wiki to work normal.
Based on the conclusion of this thread will update wiki[1].
As 2.7.6 is released recently, we need to release 2.7.7
1. Stack Management. There are two approaches for this. Open to discuss on following..
* Entirely new stack
i. Define new stack name. (BDP: BigDataPlatform, BDS:BigDataStack, OSS:OpenSourceStack,TDP: TuskerDataPlatform…)
ii. Define stack versions (zk-3.5.9,Hadoop-3.2.3,hive-3.1.3,tez-0.10.1,spark-3.2..)
iii. Based above two we need to have select in bigtop code.
* Using the existing for current release.
i. Plugin the new stack into HDP self
ii. Still, we can have hdp select
I prefer the planA but planB takes less time. Users can easily adopt it upgrade their stack.
Note: Ideally bigtop mpack can be long term approach as stack id decoupled now, but currently it’s not ready for latest stack (upgrades can be risk). We’ll work in trunk for same. If this works shortly, we can choose.
1. Upgrade :
* Most of users will be deployed with HDP versions(HDP-2.6.5,HDP-3.1.0), we might need to support the upgrade to 2.7.7
2. Other work
* Any critical/blocker CVE’s Fixes
* CVEs Fixes : we’ll try to target some CVE fixes
* Review the open PR’s and Jira’s
Trunk Code : Parallel we can work for the trunk code, this we might give as 2.8.
* Removal of python2 support
* Removal of hard dependency on HDP
* Working on bigtop mpack
Can add more based on your inputs…
Any thoughts on this..? Please feel free to add anything which I missed.
Reference:
1.https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=30755705
Re: [Discuss] Initial Draft for Roadmap
Posted by "Battula, Brahma Reddy" <bb...@visa.com.INVALID>.
Hi All,
I raised for jira for 2.7.7 roadmap[1]. Please feel to add any other features which we need to consider. Will raise another jira for trunk roadmap also based on following mail.
Please let me know your thoughts on this.
1. https://issues.apache.org/jira/browse/AMBARI-25702
On 23/06/22, 11:57 AM, "Battula, Brahma Reddy" <bb...@visa.com.INVALID> wrote:
IMO, First way is not heavier when we consider all the efforts for mpack with upgrades. And first one is easiest to adopt now.
We'll start discussing the bigtop mailing list further.
On 21/06/22, 1:11 PM, "吴治国" <wz...@163.com> wrote:
Thanks for the details, Brahma and Kengo.
If I understand it correctly, there are two approaches under discussion:
1、Prepare to release 2.7.7, in this version, we won't use HDP components anymore, we'll replace it with the community version provided by Bigtop, and update RPM and DEB build scripts provided by Bigtop, change the installation path so hdp-select can be reused. (Due to component version changes, I don't know if the code under /stacks needs to be updated)
2、Develop Bigtop mpack for Ambari 2.7.5, we don't need to change Ambari's code (if the code under /stacks can be reused in the first way, maybe it also works in this way?)
Isn't the workload of the first way heavier?
About the meeting
I thought maybe email discussion would be better?
The discussion can be open for a few days and all interested people in the community can participate, not just a few of us.
And due to time zone and language issues, many people are hard to attending the meetings, email discussions may be more suitable for these people.
What do you think?
Best Regards,
Zhiguo Wu
> On Jun 20, 2022, at 23:15, Battula, Brahma Reddy <bb...@visa.com.INVALID> wrote:
>
>>> Assuming the understanding above is correct, my new concerns are:
>
> Yes, Hope we'll in same page now. Still we can discuss in weekly call.
>
>>> * Can we still call it as "HDP"? Doesn't it infringe Cloudera's trademark?
>>> If so, should we give a new stack name to it, as you mentioned as
>>> planA in [3]?
>>> (or it makes upgrading HDP cluster difficult? I don't understand it so much)
>
> Yes, it may be. Hence I introduced the planA and preferred it. one more idea(bad) came in my mind that can we treat "HDP: Hadoop Data Platform", not sure whether it should be ok or not(and we might not delivering as distro's).
>
>>> * It may be a bit tough work to fix Bigtop's build scripts so that its
>>> artifacts match Ambari's requirements.
>>> (package versioning, file layouts, included and not included files,
>>> file permissions, user/group creation, etc.)
>
> Only paths matter right everything else can be re-used from the hdp-select.
>
>
> On 20/06/22, 8:13 PM, "Kengo Seki" <sekikn@apache.org <ma...@apache.org>> wrote:
>
> Thanks for the elaboration Brahma, but I'm a bit confused.
> At this point, my understanding is as follows. Are these correct?
>
> * In the 2.7.x line, we'll still use a stack definition based on HDP's one [1].
>
> * But regarding RPM and DEB packages, we'll build them from source by leveraging
> Bigtop's build scripts and publish them on the ASF distribution site
> or somewhere.
> repoinfo.xml [2] will also be fixed to point that repository.
>
> Assuming the understanding above is correct, my new concerns are:
>
> * Can we still call it as "HDP"? Doesn't it infringe Cloudera's trademark?
> If so, should we give a new stack name to it, as you mentioned as
> planA in [3]?
> (or it makes upgrading HDP cluster difficult? I don't understand it so much)
>
> * It may be a bit tough work to fix Bigtop's build scripts so that its
> artifacts match Ambari's requirements.
> (package versioning, file layouts, included and not included files,
> file permissions, user/group creation, etc.)
> I'm not sure if it's faster than developing Mpack.
>
> [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=85IiJe3yZUlGO9h2WSWvUo6iDsOng5ZPTvxDfNHJ%2B54%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=85IiJe3yZUlGO9h2WSWvUo6iDsOng5ZPTvxDfNHJ%2B54%3D&reserved=0>
> [2]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Fblob%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP%2F2.6%2Frepos%2Frepoinfo.xml&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3HCXvWkXezwEJO9WHE9YNIsS5QLGBg5QvH%2B3UOvmKS8%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Fblob%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP%2F2.6%2Frepos%2Frepoinfo.xml&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3HCXvWkXezwEJO9WHE9YNIsS5QLGBg5QvH%2B3UOvmKS8%3D&reserved=0>
> [3]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fm77lhzo0njr2dhoock5twtm0c19j2py8&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zoRUStsrPL1G36Jk%2FsOVPPtPIgHrnTVf2lAed4QUoEc%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fm77lhzo0njr2dhoock5twtm0c19j2py8&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zoRUStsrPL1G36Jk%2FsOVPPtPIgHrnTVf2lAed4QUoEc%3D&reserved=0>
>
> Kengo Seki <sekikn@gmail.com <ma...@gmail.com>>
>
> On Mon, Jun 20, 2022 at 8:05 PM Battula, Brahma Reddy
> <bbattula@visa.com.invalid <ma...@visa.com.invalid>> wrote:
>>
>> We'll not use the HDP Packages (Only selects we'll use from hdp, which can be plugged in bigtop). Still we'll use opensource packages and bigtop to create the RPM's and selects.
>> Just we can use the source tar balls opensource components(e.g Hadoop-3.2.3 source tarballs from apache repo) and build rpm's and publish in following repo. Or from the apache github repo's.
>>
>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2Fbigtop%2Fbigtop-3.1.0%2Frepos%2F&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=eaE39YTI9Xr2eq4Zzy74Y0MdojmAg3lfR6mbBcFrzr8%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2Fbigtop%2Fbigtop-3.1.0%2Frepos%2F&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=eaE39YTI9Xr2eq4Zzy74Y0MdojmAg3lfR6mbBcFrzr8%3D&reserved=0>
>>
>> Planning to have sync up call on this week, once it's finalize, we can summarise here the approach. What do you think..?
>>
>>
>> On 20/06/22, 4:06 PM, "Kengo Seki" <sekikn@apache.org <ma...@apache.org>> wrote:
>>
>> Thanks for the reply, Brahma!
>>
>>> We'll use following repo, how other components in Apache are hosted.
>>
>> Using that site for distribution has no problem. My concern is, do we
>> have the source tree for building HDP packages and does it still work
>> without the artifacts behind the paywall?
>> According to ASF's release policy [1], we must publish the
>> corresponding source releases whenever we publish binary artifacts,
>> such as RPM and DEB packages for HDP.
>>
>> [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2Flegal%2Frelease-policy.html%23compiled-packages&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Q4F9VnSBxMX7zEXdyU7nPXcUYqACj1QLcJg1Dt1M9Z4%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2Flegal%2Frelease-policy.html%23compiled-packages&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Q4F9VnSBxMX7zEXdyU7nPXcUYqACj1QLcJg1Dt1M9Z4%3D&reserved=0>
>>
>> Kengo Seki <sekikn@apache.org <ma...@apache.org>>
>>
>> On Mon, Jun 20, 2022 at 6:19 PM Battula, Brahma Reddy
>> <bbattula@visa.com.invalid <ma...@visa.com.invalid>> wrote:
>>>
>>> Hi Kengo Seki,
>>>
>>> Thanks for pitching here.. We'll use following repo, how other components in Apache are hosted. Any other suggestions..?
>>>
>>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2F&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=eQkhwgujOanpkVvoFZ4aL6NygYDNqOK8vSZPoCZBSSA%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2F&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=eQkhwgujOanpkVvoFZ4aL6NygYDNqOK8vSZPoCZBSSA%3D&reserved=0>
>>>
>>>
>>>
>>> On 20/06/22, 2:32 PM, "Kengo Seki" <sekikn@apache.org <ma...@apache.org>> wrote:
>>>
>>> Oh, I've just noticed that I submitted an additional question into the
>>> wrong thread mistakenly.
>>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fj6kc2h1lphvjvsblcthcl5yzzmjk3tb5&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8IVF%2B%2FaS0jXRRmCvOT5H48%2B6DMy979TtrZHucKihfPo%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fj6kc2h1lphvjvsblcthcl5yzzmjk3tb5&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8IVF%2B%2FaS0jXRRmCvOT5H48%2B6DMy979TtrZHucKihfPo%3D&reserved=0> is
>>> originally intended as a reply to this thread. Sorry for the
>>> confusion.
>>>
>>> Kengo Seki <sekikn@apache.org <ma...@apache.org>>
>>>
>>> On Mon, Jun 20, 2022 at 11:31 AM Kengo Seki <sekikn@apache.org <ma...@apache.org>> wrote:
>>>>
>>>> Thank you so much for writing the draft and commenting on it, Brahma and Viraj!
>>>> Firstly, a minor comment:
>>>>
>>>>> Trunk Code : Parallel we can work for the trunk code, this we might give as 2.8.
>>>>> * Working on bigtop mpack
>>>>
>>>> This means adding a new built-in stack which uses the Bigtop
>>>> repository, and not developing Mpack on the Bigtop side, right?
>>>> If so, "bigtop mpack" sounds confusing a bit.
>>>>
>>>>> hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk branch. Thoughts?
>>>>
>>>> Sounds good to me. During the 2.7.x releases, users can use the HDP
>>>> stack by default. In addition, they can also add the Bigtop stack
>>>> through the Mpack, as Zhiguo mentioned in [1].
>>>> Since 2.8 or 3.0, we will drop HDP and users can use the built-in
>>>> Bigtop stack by default. It also allows us to drop the Mpack on the
>>>> Bigtop side.
>>>>
>>>> Regarding the built-in Bigtop stack developed on trunk, the current
>>>> BIGTOP stack in Ambari [2] may be a good start point to be upgraded.
>>>> I ensured that Ambari 2.7.6 was successfully built with the
>>>> `-Dstack.distribution=BIGTOP` option, though Bigtop 0.8 is too old to
>>>> use for now.
>>>>
>>>> [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fljnyymp2g9fg7cq4z7sfxd36bh8x7nlm&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zg1RgP6ofCnG%2FFxKnIUGGzyyB%2F7fyceGzi7ZgpV16zU%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fljnyymp2g9fg7cq4z7sfxd36bh8x7nlm&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zg1RgP6ofCnG%2FFxKnIUGGzyyB%2F7fyceGzi7ZgpV16zU%3D&reserved=0>
>>>> [2]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FBIGTOP%2F0.8&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Dm56vabeL2985wVVDbOQZjRPmyzXF9LvP%2FMioRPR09Q%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FBIGTOP%2F0.8&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Dm56vabeL2985wVVDbOQZjRPmyzXF9LvP%2FMioRPR09Q%3D&reserved=0>
>>>>
>>>> Kengo Seki <sekikn@apache.org <ma...@apache.org>>
>>>>
>>>> On Mon, Jun 20, 2022 at 9:26 AM Viraj Jasani <vjasani@apache.org <ma...@apache.org>> wrote:
>>>>>
>>>>> Thanks for the nice draft, Brahma!
>>>>>
>>>>> IMHO, as we narrow down to 2.7.7 release, it might be still worth sticking
>>>>> to HDP i.e. plugin the new stack for HDP for 2.7 release line. For 2.8.0
>>>>> (or 3.0.0), we can add entire new stack (BDP sounds good one) and start off
>>>>> with 2.8.0/3.0.0 stack version.
>>>>>
>>>>> hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk
>>>>> branch. Thoughts?
>>>>>
>>>>> Similarly, we can start-off with "python 3 only" support in trunk, whereas
>>>>> 2.7 release line can continue with python 2 to reduce complexities for
>>>>> maintenance releases on 2.7.
>>>>>
>>>>> All other points look good. Once again, thank you for drafting the roadmap
>>>>> and starting with this thread.
>>>>>
>>>>>
>>>>> On Sun, Jun 19, 2022 at 10:07 AM Battula, Brahma Reddy
>>>>> <bbattula@visa.com.invalid <ma...@visa.com.invalid>> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> As Ambari back now, let’s start discussing roadmap. I will be closely
>>>>>> tracking for Jira, GitHub(with Jenkins) and wiki to work normal.
>>>>>> Based on the conclusion of this thread will update wiki[1].
>>>>>>
>>>>>> As 2.7.6 is released recently, we need to release 2.7.7
>>>>>>
>>>>>>
>>>>>> 1. Stack Management. There are two approaches for this. Open to discuss
>>>>>> on following..
>>>>>> * Entirely new stack
>>>>>>
>>>>>> i.
>>>>>> Define new stack name. (BDP: BigDataPlatform, BDS:BigDataStack,
>>>>>> OSS:OpenSourceStack,TDP: TuskerDataPlatform…)
>>>>>>
>>>>>> ii.
>>>>>> Define stack versions
>>>>>> (zk-3.5.9,Hadoop-3.2.3,hive-3.1.3,tez-0.10.1,spark-3.2..)
>>>>>>
>>>>>> iii. Based
>>>>>> above two we need to have select in bigtop code.
>>>>>>
>>>>>> * Using the existing for current release.
>>>>>>
>>>>>> i.
>>>>>> Plugin the new stack into HDP self
>>>>>>
>>>>>> ii.
>>>>>> Still, we can have hdp select
>>>>>>
>>>>>>
>>>>>>
>>>>>> I prefer the planA but planB takes less time. Users can easily adopt it
>>>>>> upgrade their stack.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Note: Ideally bigtop mpack can be long term approach as stack id decoupled
>>>>>> now, but currently it’s not ready for latest stack (upgrades can be risk).
>>>>>> We’ll work in trunk for same. If this works shortly, we can choose.
>>>>>>
>>>>>>
>>>>>>
>>>>>> 1. Upgrade :
>>>>>> * Most of users will be deployed with HDP
>>>>>> versions(HDP-2.6.5,HDP-3.1.0), we might need to support the upgrade to 2.7.7
>>>>>> 2. Other work
>>>>>> * Any critical/blocker CVE’s Fixes
>>>>>> * CVEs Fixes : we’ll try to target some CVE fixes
>>>>>> * Review the open PR’s and Jira’s
>>>>>>
>>>>>>
>>>>>>
>>>>>> Trunk Code : Parallel we can work for the trunk code, this we might give
>>>>>> as 2.8.
>>>>>>
>>>>>> * Removal of python2 support
>>>>>> * Removal of hard dependency on HDP
>>>>>> * Working on bigtop mpack
>>>>>>
>>>>>> Can add more based on your inputs…
>>>>>>
>>>>>>
>>>>>> Any thoughts on this..? Please feel free to add anything which I missed.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Reference:
>>>>>>
>>>>>> 1.
>>>>>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D30755705&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nwpSqegK9X8K71R7CrroK%2FkNGz%2FfGBjCUVU2DNxxy1U%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D30755705&data=05%7C01%7Cbbattula%40visa.com%7Ca70f6e033f9948cdf2ba08da54e16b86%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637915624407026578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nwpSqegK9X8K71R7CrroK%2FkNGz%2FfGBjCUVU2DNxxy1U%3D&reserved=0>
>>>>>>
>>>>>>
>>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org <ma...@ambari.apache.org>
>>> For additional commands, e-mail: dev-help@ambari.apache.org <ma...@ambari.apache.org>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
>>> For additional commands, e-mail: dev-help@ambari.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
>> For additional commands, e-mail: dev-help@ambari.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
> For additional commands, e-mail: dev-help@ambari.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
> For additional commands, e-mail: dev-help@ambari.apache.org
?B�KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKCB�?�?[��X��ܚX�K??K[XZ[?�??]�][��X��ܚX�P?[X�\�K�\?X�?K�ܙ�B��܈?Y??]?[ۘ[??��[X[�?�??K[XZ[?�??]�Z?[???[X�\�K�\?X�?K�ܙ�B�B
Re: [Discuss] Initial Draft for Roadmap
Posted by "Battula, Brahma Reddy" <bb...@visa.com.INVALID>.
IMO, First way is not heavier when we consider all the efforts for mpack with upgrades. And first one is easiest to adopt now.
We'll start discussing the bigtop mailing list further.
On 21/06/22, 1:11 PM, "吴治国" <wz...@163.com> wrote:
Thanks for the details, Brahma and Kengo.
If I understand it correctly, there are two approaches under discussion:
1、Prepare to release 2.7.7, in this version, we won't use HDP components anymore, we'll replace it with the community version provided by Bigtop, and update RPM and DEB build scripts provided by Bigtop, change the installation path so hdp-select can be reused. (Due to component version changes, I don't know if the code under /stacks needs to be updated)
2、Develop Bigtop mpack for Ambari 2.7.5, we don't need to change Ambari's code (if the code under /stacks can be reused in the first way, maybe it also works in this way?)
Isn't the workload of the first way heavier?
About the meeting
I thought maybe email discussion would be better?
The discussion can be open for a few days and all interested people in the community can participate, not just a few of us.
And due to time zone and language issues, many people are hard to attending the meetings, email discussions may be more suitable for these people.
What do you think?
Best Regards,
Zhiguo Wu
> On Jun 20, 2022, at 23:15, Battula, Brahma Reddy <bb...@visa.com.INVALID> wrote:
>
>>> Assuming the understanding above is correct, my new concerns are:
>
> Yes, Hope we'll in same page now. Still we can discuss in weekly call.
>
>>> * Can we still call it as "HDP"? Doesn't it infringe Cloudera's trademark?
>>> If so, should we give a new stack name to it, as you mentioned as
>>> planA in [3]?
>>> (or it makes upgrading HDP cluster difficult? I don't understand it so much)
>
> Yes, it may be. Hence I introduced the planA and preferred it. one more idea(bad) came in my mind that can we treat "HDP: Hadoop Data Platform", not sure whether it should be ok or not(and we might not delivering as distro's).
>
>>> * It may be a bit tough work to fix Bigtop's build scripts so that its
>>> artifacts match Ambari's requirements.
>>> (package versioning, file layouts, included and not included files,
>>> file permissions, user/group creation, etc.)
>
> Only paths matter right everything else can be re-used from the hdp-select.
>
>
> On 20/06/22, 8:13 PM, "Kengo Seki" <sekikn@apache.org <ma...@apache.org>> wrote:
>
> Thanks for the elaboration Brahma, but I'm a bit confused.
> At this point, my understanding is as follows. Are these correct?
>
> * In the 2.7.x line, we'll still use a stack definition based on HDP's one [1].
>
> * But regarding RPM and DEB packages, we'll build them from source by leveraging
> Bigtop's build scripts and publish them on the ASF distribution site
> or somewhere.
> repoinfo.xml [2] will also be fixed to point that repository.
>
> Assuming the understanding above is correct, my new concerns are:
>
> * Can we still call it as "HDP"? Doesn't it infringe Cloudera's trademark?
> If so, should we give a new stack name to it, as you mentioned as
> planA in [3]?
> (or it makes upgrading HDP cluster difficult? I don't understand it so much)
>
> * It may be a bit tough work to fix Bigtop's build scripts so that its
> artifacts match Ambari's requirements.
> (package versioning, file layouts, included and not included files,
> file permissions, user/group creation, etc.)
> I'm not sure if it's faster than developing Mpack.
>
> [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dFX07wfIY8aukfEUcJe9GN0rds8t3%2Fpenmsiszglkn4%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dFX07wfIY8aukfEUcJe9GN0rds8t3%2Fpenmsiszglkn4%3D&reserved=0>
> [2]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Fblob%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP%2F2.6%2Frepos%2Frepoinfo.xml&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MDz57IloHfWYWcwmAI9h1pYDG45v1UbU6K2M%2BkpcDEA%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Fblob%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP%2F2.6%2Frepos%2Frepoinfo.xml&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MDz57IloHfWYWcwmAI9h1pYDG45v1UbU6K2M%2BkpcDEA%3D&reserved=0>
> [3]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fm77lhzo0njr2dhoock5twtm0c19j2py8&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=P1pPHwM1YggdPT9qD4aAvJdehjjZdZ8mLhvn0ZfR7Gw%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fm77lhzo0njr2dhoock5twtm0c19j2py8&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=P1pPHwM1YggdPT9qD4aAvJdehjjZdZ8mLhvn0ZfR7Gw%3D&reserved=0>
>
> Kengo Seki <sekikn@gmail.com <ma...@gmail.com>>
>
> On Mon, Jun 20, 2022 at 8:05 PM Battula, Brahma Reddy
> <bbattula@visa.com.invalid <ma...@visa.com.invalid>> wrote:
>>
>> We'll not use the HDP Packages (Only selects we'll use from hdp, which can be plugged in bigtop). Still we'll use opensource packages and bigtop to create the RPM's and selects.
>> Just we can use the source tar balls opensource components(e.g Hadoop-3.2.3 source tarballs from apache repo) and build rpm's and publish in following repo. Or from the apache github repo's.
>>
>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2Fbigtop%2Fbigtop-3.1.0%2Frepos%2F&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RyNyy7fOUlWBSFMJdd1iRMrmlKOVyA5NEGuMpSkCSEs%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2Fbigtop%2Fbigtop-3.1.0%2Frepos%2F&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RyNyy7fOUlWBSFMJdd1iRMrmlKOVyA5NEGuMpSkCSEs%3D&reserved=0>
>>
>> Planning to have sync up call on this week, once it's finalize, we can summarise here the approach. What do you think..?
>>
>>
>> On 20/06/22, 4:06 PM, "Kengo Seki" <sekikn@apache.org <ma...@apache.org>> wrote:
>>
>> Thanks for the reply, Brahma!
>>
>>> We'll use following repo, how other components in Apache are hosted.
>>
>> Using that site for distribution has no problem. My concern is, do we
>> have the source tree for building HDP packages and does it still work
>> without the artifacts behind the paywall?
>> According to ASF's release policy [1], we must publish the
>> corresponding source releases whenever we publish binary artifacts,
>> such as RPM and DEB packages for HDP.
>>
>> [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2Flegal%2Frelease-policy.html%23compiled-packages&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qPEbEEpIN8NCsrMnxW5xKbB%2B19bDJprq%2FiXjIF09vls%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2Flegal%2Frelease-policy.html%23compiled-packages&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qPEbEEpIN8NCsrMnxW5xKbB%2B19bDJprq%2FiXjIF09vls%3D&reserved=0>
>>
>> Kengo Seki <sekikn@apache.org <ma...@apache.org>>
>>
>> On Mon, Jun 20, 2022 at 6:19 PM Battula, Brahma Reddy
>> <bbattula@visa.com.invalid <ma...@visa.com.invalid>> wrote:
>>>
>>> Hi Kengo Seki,
>>>
>>> Thanks for pitching here.. We'll use following repo, how other components in Apache are hosted. Any other suggestions..?
>>>
>>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2F&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OUGhE45KsDvPum4%2Fd9cwlJPfH66OlKE%2BgD557%2BiiAC8%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2F&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OUGhE45KsDvPum4%2Fd9cwlJPfH66OlKE%2BgD557%2BiiAC8%3D&reserved=0>
>>>
>>>
>>>
>>> On 20/06/22, 2:32 PM, "Kengo Seki" <sekikn@apache.org <ma...@apache.org>> wrote:
>>>
>>> Oh, I've just noticed that I submitted an additional question into the
>>> wrong thread mistakenly.
>>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fj6kc2h1lphvjvsblcthcl5yzzmjk3tb5&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UTDhFu1Bloa3FzD0shWWHoDt%2FrQD22JziV1%2BNAWp8NQ%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fj6kc2h1lphvjvsblcthcl5yzzmjk3tb5&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UTDhFu1Bloa3FzD0shWWHoDt%2FrQD22JziV1%2BNAWp8NQ%3D&reserved=0> is
>>> originally intended as a reply to this thread. Sorry for the
>>> confusion.
>>>
>>> Kengo Seki <sekikn@apache.org <ma...@apache.org>>
>>>
>>> On Mon, Jun 20, 2022 at 11:31 AM Kengo Seki <sekikn@apache.org <ma...@apache.org>> wrote:
>>>>
>>>> Thank you so much for writing the draft and commenting on it, Brahma and Viraj!
>>>> Firstly, a minor comment:
>>>>
>>>>> Trunk Code : Parallel we can work for the trunk code, this we might give as 2.8.
>>>>> * Working on bigtop mpack
>>>>
>>>> This means adding a new built-in stack which uses the Bigtop
>>>> repository, and not developing Mpack on the Bigtop side, right?
>>>> If so, "bigtop mpack" sounds confusing a bit.
>>>>
>>>>> hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk branch. Thoughts?
>>>>
>>>> Sounds good to me. During the 2.7.x releases, users can use the HDP
>>>> stack by default. In addition, they can also add the Bigtop stack
>>>> through the Mpack, as Zhiguo mentioned in [1].
>>>> Since 2.8 or 3.0, we will drop HDP and users can use the built-in
>>>> Bigtop stack by default. It also allows us to drop the Mpack on the
>>>> Bigtop side.
>>>>
>>>> Regarding the built-in Bigtop stack developed on trunk, the current
>>>> BIGTOP stack in Ambari [2] may be a good start point to be upgraded.
>>>> I ensured that Ambari 2.7.6 was successfully built with the
>>>> `-Dstack.distribution=BIGTOP` option, though Bigtop 0.8 is too old to
>>>> use for now.
>>>>
>>>> [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fljnyymp2g9fg7cq4z7sfxd36bh8x7nlm&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iqQ6QCM1%2BMYJ61tU%2FFk%2BWC8GvftYYGhrnu6V%2FFJ3d7o%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fljnyymp2g9fg7cq4z7sfxd36bh8x7nlm&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iqQ6QCM1%2BMYJ61tU%2FFk%2BWC8GvftYYGhrnu6V%2FFJ3d7o%3D&reserved=0>
>>>> [2]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FBIGTOP%2F0.8&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FR5NTHCZt%2FQcqy9NyT63Q3NIYlEeJ6sIpbOqzfDOMRY%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FBIGTOP%2F0.8&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FR5NTHCZt%2FQcqy9NyT63Q3NIYlEeJ6sIpbOqzfDOMRY%3D&reserved=0>
>>>>
>>>> Kengo Seki <sekikn@apache.org <ma...@apache.org>>
>>>>
>>>> On Mon, Jun 20, 2022 at 9:26 AM Viraj Jasani <vjasani@apache.org <ma...@apache.org>> wrote:
>>>>>
>>>>> Thanks for the nice draft, Brahma!
>>>>>
>>>>> IMHO, as we narrow down to 2.7.7 release, it might be still worth sticking
>>>>> to HDP i.e. plugin the new stack for HDP for 2.7 release line. For 2.8.0
>>>>> (or 3.0.0), we can add entire new stack (BDP sounds good one) and start off
>>>>> with 2.8.0/3.0.0 stack version.
>>>>>
>>>>> hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk
>>>>> branch. Thoughts?
>>>>>
>>>>> Similarly, we can start-off with "python 3 only" support in trunk, whereas
>>>>> 2.7 release line can continue with python 2 to reduce complexities for
>>>>> maintenance releases on 2.7.
>>>>>
>>>>> All other points look good. Once again, thank you for drafting the roadmap
>>>>> and starting with this thread.
>>>>>
>>>>>
>>>>> On Sun, Jun 19, 2022 at 10:07 AM Battula, Brahma Reddy
>>>>> <bbattula@visa.com.invalid <ma...@visa.com.invalid>> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> As Ambari back now, let’s start discussing roadmap. I will be closely
>>>>>> tracking for Jira, GitHub(with Jenkins) and wiki to work normal.
>>>>>> Based on the conclusion of this thread will update wiki[1].
>>>>>>
>>>>>> As 2.7.6 is released recently, we need to release 2.7.7
>>>>>>
>>>>>>
>>>>>> 1. Stack Management. There are two approaches for this. Open to discuss
>>>>>> on following..
>>>>>> * Entirely new stack
>>>>>>
>>>>>> i.
>>>>>> Define new stack name. (BDP: BigDataPlatform, BDS:BigDataStack,
>>>>>> OSS:OpenSourceStack,TDP: TuskerDataPlatform…)
>>>>>>
>>>>>> ii.
>>>>>> Define stack versions
>>>>>> (zk-3.5.9,Hadoop-3.2.3,hive-3.1.3,tez-0.10.1,spark-3.2..)
>>>>>>
>>>>>> iii. Based
>>>>>> above two we need to have select in bigtop code.
>>>>>>
>>>>>> * Using the existing for current release.
>>>>>>
>>>>>> i.
>>>>>> Plugin the new stack into HDP self
>>>>>>
>>>>>> ii.
>>>>>> Still, we can have hdp select
>>>>>>
>>>>>>
>>>>>>
>>>>>> I prefer the planA but planB takes less time. Users can easily adopt it
>>>>>> upgrade their stack.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Note: Ideally bigtop mpack can be long term approach as stack id decoupled
>>>>>> now, but currently it’s not ready for latest stack (upgrades can be risk).
>>>>>> We’ll work in trunk for same. If this works shortly, we can choose.
>>>>>>
>>>>>>
>>>>>>
>>>>>> 1. Upgrade :
>>>>>> * Most of users will be deployed with HDP
>>>>>> versions(HDP-2.6.5,HDP-3.1.0), we might need to support the upgrade to 2.7.7
>>>>>> 2. Other work
>>>>>> * Any critical/blocker CVE’s Fixes
>>>>>> * CVEs Fixes : we’ll try to target some CVE fixes
>>>>>> * Review the open PR’s and Jira’s
>>>>>>
>>>>>>
>>>>>>
>>>>>> Trunk Code : Parallel we can work for the trunk code, this we might give
>>>>>> as 2.8.
>>>>>>
>>>>>> * Removal of python2 support
>>>>>> * Removal of hard dependency on HDP
>>>>>> * Working on bigtop mpack
>>>>>>
>>>>>> Can add more based on your inputs…
>>>>>>
>>>>>>
>>>>>> Any thoughts on this..? Please feel free to add anything which I missed.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Reference:
>>>>>>
>>>>>> 1.
>>>>>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D30755705&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PlGOgqOov%2FflDFyTRvQLGgK9NIj4SOOsSJH4jCQeeaA%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D30755705&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PlGOgqOov%2FflDFyTRvQLGgK9NIj4SOOsSJH4jCQeeaA%3D&reserved=0>
>>>>>>
>>>>>>
>>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org <ma...@ambari.apache.org>
>>> For additional commands, e-mail: dev-help@ambari.apache.org <ma...@ambari.apache.org>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
>>> For additional commands, e-mail: dev-help@ambari.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
>> For additional commands, e-mail: dev-help@ambari.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
> For additional commands, e-mail: dev-help@ambari.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
> For additional commands, e-mail: dev-help@ambari.apache.org
Re: [Discuss] Initial Draft for Roadmap
Posted by 吴治国 <wz...@163.com>.
Thanks for the details, Brahma and Kengo.
If I understand it correctly, there are two approaches under discussion:
1、Prepare to release 2.7.7, in this version, we won't use HDP components anymore, we'll replace it with the community version provided by Bigtop, and update RPM and DEB build scripts provided by Bigtop, change the installation path so hdp-select can be reused. (Due to component version changes, I don't know if the code under /stacks needs to be updated)
2、Develop Bigtop mpack for Ambari 2.7.5, we don't need to change Ambari's code (if the code under /stacks can be reused in the first way, maybe it also works in this way?)
Isn't the workload of the first way heavier?
About the meeting
I thought maybe email discussion would be better?
The discussion can be open for a few days and all interested people in the community can participate, not just a few of us.
And due to time zone and language issues, many people are hard to attending the meetings, email discussions may be more suitable for these people.
What do you think?
Best Regards,
Zhiguo Wu
> On Jun 20, 2022, at 23:15, Battula, Brahma Reddy <bb...@visa.com.INVALID> wrote:
>
>>> Assuming the understanding above is correct, my new concerns are:
>
> Yes, Hope we'll in same page now. Still we can discuss in weekly call.
>
>>> * Can we still call it as "HDP"? Doesn't it infringe Cloudera's trademark?
>>> If so, should we give a new stack name to it, as you mentioned as
>>> planA in [3]?
>>> (or it makes upgrading HDP cluster difficult? I don't understand it so much)
>
> Yes, it may be. Hence I introduced the planA and preferred it. one more idea(bad) came in my mind that can we treat "HDP: Hadoop Data Platform", not sure whether it should be ok or not(and we might not delivering as distro's).
>
>>> * It may be a bit tough work to fix Bigtop's build scripts so that its
>>> artifacts match Ambari's requirements.
>>> (package versioning, file layouts, included and not included files,
>>> file permissions, user/group creation, etc.)
>
> Only paths matter right everything else can be re-used from the hdp-select.
>
>
> On 20/06/22, 8:13 PM, "Kengo Seki" <sekikn@apache.org <ma...@apache.org>> wrote:
>
> Thanks for the elaboration Brahma, but I'm a bit confused.
> At this point, my understanding is as follows. Are these correct?
>
> * In the 2.7.x line, we'll still use a stack definition based on HDP's one [1].
>
> * But regarding RPM and DEB packages, we'll build them from source by leveraging
> Bigtop's build scripts and publish them on the ASF distribution site
> or somewhere.
> repoinfo.xml [2] will also be fixed to point that repository.
>
> Assuming the understanding above is correct, my new concerns are:
>
> * Can we still call it as "HDP"? Doesn't it infringe Cloudera's trademark?
> If so, should we give a new stack name to it, as you mentioned as
> planA in [3]?
> (or it makes upgrading HDP cluster difficult? I don't understand it so much)
>
> * It may be a bit tough work to fix Bigtop's build scripts so that its
> artifacts match Ambari's requirements.
> (package versioning, file layouts, included and not included files,
> file permissions, user/group creation, etc.)
> I'm not sure if it's faster than developing Mpack.
>
> [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6cm2kZCvgvI2%2Fc%2BJCxhBbVyBlHgxcUvl2brLX3PDz%2Bg%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6cm2kZCvgvI2%2Fc%2BJCxhBbVyBlHgxcUvl2brLX3PDz%2Bg%3D&reserved=0>
> [2]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Fblob%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP%2F2.6%2Frepos%2Frepoinfo.xml&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pkYcSo4Emv9%2BUQdP8L9uzgsgApV3IIua%2Bxbc3VNW7fQ%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Fblob%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP%2F2.6%2Frepos%2Frepoinfo.xml&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pkYcSo4Emv9%2BUQdP8L9uzgsgApV3IIua%2Bxbc3VNW7fQ%3D&reserved=0>
> [3]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fm77lhzo0njr2dhoock5twtm0c19j2py8&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=U1y4%2B%2FeckCFB2IMdgEei65DwI0rrEfCD7bFDHH3O2xQ%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fm77lhzo0njr2dhoock5twtm0c19j2py8&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=U1y4%2B%2FeckCFB2IMdgEei65DwI0rrEfCD7bFDHH3O2xQ%3D&reserved=0>
>
> Kengo Seki <sekikn@gmail.com <ma...@gmail.com>>
>
> On Mon, Jun 20, 2022 at 8:05 PM Battula, Brahma Reddy
> <bbattula@visa.com.invalid <ma...@visa.com.invalid>> wrote:
>>
>> We'll not use the HDP Packages (Only selects we'll use from hdp, which can be plugged in bigtop). Still we'll use opensource packages and bigtop to create the RPM's and selects.
>> Just we can use the source tar balls opensource components(e.g Hadoop-3.2.3 source tarballs from apache repo) and build rpm's and publish in following repo. Or from the apache github repo's.
>>
>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2Fbigtop%2Fbigtop-3.1.0%2Frepos%2F&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6kBAOFoW%2BsBHM8b4KX68e0UZ%2B%2BHg7Oxewx%2BJcCEdRXg%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2Fbigtop%2Fbigtop-3.1.0%2Frepos%2F&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6kBAOFoW%2BsBHM8b4KX68e0UZ%2B%2BHg7Oxewx%2BJcCEdRXg%3D&reserved=0>
>>
>> Planning to have sync up call on this week, once it's finalize, we can summarise here the approach. What do you think..?
>>
>>
>> On 20/06/22, 4:06 PM, "Kengo Seki" <sekikn@apache.org <ma...@apache.org>> wrote:
>>
>> Thanks for the reply, Brahma!
>>
>>> We'll use following repo, how other components in Apache are hosted.
>>
>> Using that site for distribution has no problem. My concern is, do we
>> have the source tree for building HDP packages and does it still work
>> without the artifacts behind the paywall?
>> According to ASF's release policy [1], we must publish the
>> corresponding source releases whenever we publish binary artifacts,
>> such as RPM and DEB packages for HDP.
>>
>> [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2Flegal%2Frelease-policy.html%23compiled-packages&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Cz%2BmZEjwjNnq9lICZmRtIDMLge9xZLQ0w3yK75AcHkc%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2Flegal%2Frelease-policy.html%23compiled-packages&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Cz%2BmZEjwjNnq9lICZmRtIDMLge9xZLQ0w3yK75AcHkc%3D&reserved=0>
>>
>> Kengo Seki <sekikn@apache.org <ma...@apache.org>>
>>
>> On Mon, Jun 20, 2022 at 6:19 PM Battula, Brahma Reddy
>> <bbattula@visa.com.invalid <ma...@visa.com.invalid>> wrote:
>>>
>>> Hi Kengo Seki,
>>>
>>> Thanks for pitching here.. We'll use following repo, how other components in Apache are hosted. Any other suggestions..?
>>>
>>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2F&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mQLNExpiw6fywDjtEbcUFWJlOuGJ6Ja45LWxjAWL6dY%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2F&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mQLNExpiw6fywDjtEbcUFWJlOuGJ6Ja45LWxjAWL6dY%3D&reserved=0>
>>>
>>>
>>>
>>> On 20/06/22, 2:32 PM, "Kengo Seki" <sekikn@apache.org <ma...@apache.org>> wrote:
>>>
>>> Oh, I've just noticed that I submitted an additional question into the
>>> wrong thread mistakenly.
>>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fj6kc2h1lphvjvsblcthcl5yzzmjk3tb5&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=R61kELeUwqNTvFHwQ5Gu4HuhnWuhrhPl07K40FLSrL0%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fj6kc2h1lphvjvsblcthcl5yzzmjk3tb5&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=R61kELeUwqNTvFHwQ5Gu4HuhnWuhrhPl07K40FLSrL0%3D&reserved=0> is
>>> originally intended as a reply to this thread. Sorry for the
>>> confusion.
>>>
>>> Kengo Seki <sekikn@apache.org <ma...@apache.org>>
>>>
>>> On Mon, Jun 20, 2022 at 11:31 AM Kengo Seki <sekikn@apache.org <ma...@apache.org>> wrote:
>>>>
>>>> Thank you so much for writing the draft and commenting on it, Brahma and Viraj!
>>>> Firstly, a minor comment:
>>>>
>>>>> Trunk Code : Parallel we can work for the trunk code, this we might give as 2.8.
>>>>> * Working on bigtop mpack
>>>>
>>>> This means adding a new built-in stack which uses the Bigtop
>>>> repository, and not developing Mpack on the Bigtop side, right?
>>>> If so, "bigtop mpack" sounds confusing a bit.
>>>>
>>>>> hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk branch. Thoughts?
>>>>
>>>> Sounds good to me. During the 2.7.x releases, users can use the HDP
>>>> stack by default. In addition, they can also add the Bigtop stack
>>>> through the Mpack, as Zhiguo mentioned in [1].
>>>> Since 2.8 or 3.0, we will drop HDP and users can use the built-in
>>>> Bigtop stack by default. It also allows us to drop the Mpack on the
>>>> Bigtop side.
>>>>
>>>> Regarding the built-in Bigtop stack developed on trunk, the current
>>>> BIGTOP stack in Ambari [2] may be a good start point to be upgraded.
>>>> I ensured that Ambari 2.7.6 was successfully built with the
>>>> `-Dstack.distribution=BIGTOP` option, though Bigtop 0.8 is too old to
>>>> use for now.
>>>>
>>>> [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fljnyymp2g9fg7cq4z7sfxd36bh8x7nlm&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FOPQS1G0t6phSKTW8LIVxgJqUEWFOMlehrQgh7kVjv8%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fljnyymp2g9fg7cq4z7sfxd36bh8x7nlm&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FOPQS1G0t6phSKTW8LIVxgJqUEWFOMlehrQgh7kVjv8%3D&reserved=0>
>>>> [2]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FBIGTOP%2F0.8&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MvxxrjL0F%2B7wLfw5Qr9bstYRBuiwma5mPMEPurCN3vw%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FBIGTOP%2F0.8&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MvxxrjL0F%2B7wLfw5Qr9bstYRBuiwma5mPMEPurCN3vw%3D&reserved=0>
>>>>
>>>> Kengo Seki <sekikn@apache.org <ma...@apache.org>>
>>>>
>>>> On Mon, Jun 20, 2022 at 9:26 AM Viraj Jasani <vjasani@apache.org <ma...@apache.org>> wrote:
>>>>>
>>>>> Thanks for the nice draft, Brahma!
>>>>>
>>>>> IMHO, as we narrow down to 2.7.7 release, it might be still worth sticking
>>>>> to HDP i.e. plugin the new stack for HDP for 2.7 release line. For 2.8.0
>>>>> (or 3.0.0), we can add entire new stack (BDP sounds good one) and start off
>>>>> with 2.8.0/3.0.0 stack version.
>>>>>
>>>>> hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk
>>>>> branch. Thoughts?
>>>>>
>>>>> Similarly, we can start-off with "python 3 only" support in trunk, whereas
>>>>> 2.7 release line can continue with python 2 to reduce complexities for
>>>>> maintenance releases on 2.7.
>>>>>
>>>>> All other points look good. Once again, thank you for drafting the roadmap
>>>>> and starting with this thread.
>>>>>
>>>>>
>>>>> On Sun, Jun 19, 2022 at 10:07 AM Battula, Brahma Reddy
>>>>> <bbattula@visa.com.invalid <ma...@visa.com.invalid>> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> As Ambari back now, let’s start discussing roadmap. I will be closely
>>>>>> tracking for Jira, GitHub(with Jenkins) and wiki to work normal.
>>>>>> Based on the conclusion of this thread will update wiki[1].
>>>>>>
>>>>>> As 2.7.6 is released recently, we need to release 2.7.7
>>>>>>
>>>>>>
>>>>>> 1. Stack Management. There are two approaches for this. Open to discuss
>>>>>> on following..
>>>>>> * Entirely new stack
>>>>>>
>>>>>> i.
>>>>>> Define new stack name. (BDP: BigDataPlatform, BDS:BigDataStack,
>>>>>> OSS:OpenSourceStack,TDP: TuskerDataPlatform…)
>>>>>>
>>>>>> ii.
>>>>>> Define stack versions
>>>>>> (zk-3.5.9,Hadoop-3.2.3,hive-3.1.3,tez-0.10.1,spark-3.2..)
>>>>>>
>>>>>> iii. Based
>>>>>> above two we need to have select in bigtop code.
>>>>>>
>>>>>> * Using the existing for current release.
>>>>>>
>>>>>> i.
>>>>>> Plugin the new stack into HDP self
>>>>>>
>>>>>> ii.
>>>>>> Still, we can have hdp select
>>>>>>
>>>>>>
>>>>>>
>>>>>> I prefer the planA but planB takes less time. Users can easily adopt it
>>>>>> upgrade their stack.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Note: Ideally bigtop mpack can be long term approach as stack id decoupled
>>>>>> now, but currently it’s not ready for latest stack (upgrades can be risk).
>>>>>> We’ll work in trunk for same. If this works shortly, we can choose.
>>>>>>
>>>>>>
>>>>>>
>>>>>> 1. Upgrade :
>>>>>> * Most of users will be deployed with HDP
>>>>>> versions(HDP-2.6.5,HDP-3.1.0), we might need to support the upgrade to 2.7.7
>>>>>> 2. Other work
>>>>>> * Any critical/blocker CVE’s Fixes
>>>>>> * CVEs Fixes : we’ll try to target some CVE fixes
>>>>>> * Review the open PR’s and Jira’s
>>>>>>
>>>>>>
>>>>>>
>>>>>> Trunk Code : Parallel we can work for the trunk code, this we might give
>>>>>> as 2.8.
>>>>>>
>>>>>> * Removal of python2 support
>>>>>> * Removal of hard dependency on HDP
>>>>>> * Working on bigtop mpack
>>>>>>
>>>>>> Can add more based on your inputs…
>>>>>>
>>>>>>
>>>>>> Any thoughts on this..? Please feel free to add anything which I missed.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Reference:
>>>>>>
>>>>>> 1.
>>>>>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D30755705&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Kv1AicBwjFxEcdYsVjMWgUpefukzFP46lYAs9odJyNM%3D&reserved=0 <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D30755705&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Kv1AicBwjFxEcdYsVjMWgUpefukzFP46lYAs9odJyNM%3D&reserved=0>
>>>>>>
>>>>>>
>>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org <ma...@ambari.apache.org>
>>> For additional commands, e-mail: dev-help@ambari.apache.org <ma...@ambari.apache.org>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
>>> For additional commands, e-mail: dev-help@ambari.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
>> For additional commands, e-mail: dev-help@ambari.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
> For additional commands, e-mail: dev-help@ambari.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
> For additional commands, e-mail: dev-help@ambari.apache.org
Re: [Discuss] Initial Draft for Roadmap
Posted by "Battula, Brahma Reddy" <bb...@visa.com.INVALID>.
>> Assuming the understanding above is correct, my new concerns are:
Yes, Hope we'll in same page now. Still we can discuss in weekly call.
>> * Can we still call it as "HDP"? Doesn't it infringe Cloudera's trademark?
>> If so, should we give a new stack name to it, as you mentioned as
>> planA in [3]?
>> (or it makes upgrading HDP cluster difficult? I don't understand it so much)
Yes, it may be. Hence I introduced the planA and preferred it. one more idea(bad) came in my mind that can we treat "HDP: Hadoop Data Platform", not sure whether it should be ok or not(and we might not delivering as distro's).
>>* It may be a bit tough work to fix Bigtop's build scripts so that its
>> artifacts match Ambari's requirements.
>> (package versioning, file layouts, included and not included files,
>> file permissions, user/group creation, etc.)
Only paths matter right everything else can be re-used from the hdp-select.
On 20/06/22, 8:13 PM, "Kengo Seki" <se...@apache.org> wrote:
Thanks for the elaboration Brahma, but I'm a bit confused.
At this point, my understanding is as follows. Are these correct?
* In the 2.7.x line, we'll still use a stack definition based on HDP's one [1].
* But regarding RPM and DEB packages, we'll build them from source by leveraging
Bigtop's build scripts and publish them on the ASF distribution site
or somewhere.
repoinfo.xml [2] will also be fixed to point that repository.
Assuming the understanding above is correct, my new concerns are:
* Can we still call it as "HDP"? Doesn't it infringe Cloudera's trademark?
If so, should we give a new stack name to it, as you mentioned as
planA in [3]?
(or it makes upgrading HDP cluster difficult? I don't understand it so much)
* It may be a bit tough work to fix Bigtop's build scripts so that its
artifacts match Ambari's requirements.
(package versioning, file layouts, included and not included files,
file permissions, user/group creation, etc.)
I'm not sure if it's faster than developing Mpack.
[1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6cm2kZCvgvI2%2Fc%2BJCxhBbVyBlHgxcUvl2brLX3PDz%2Bg%3D&reserved=0
[2]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Fblob%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP%2F2.6%2Frepos%2Frepoinfo.xml&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pkYcSo4Emv9%2BUQdP8L9uzgsgApV3IIua%2Bxbc3VNW7fQ%3D&reserved=0
[3]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fm77lhzo0njr2dhoock5twtm0c19j2py8&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=U1y4%2B%2FeckCFB2IMdgEei65DwI0rrEfCD7bFDHH3O2xQ%3D&reserved=0
Kengo Seki <se...@gmail.com>
On Mon, Jun 20, 2022 at 8:05 PM Battula, Brahma Reddy
<bb...@visa.com.invalid> wrote:
>
> We'll not use the HDP Packages (Only selects we'll use from hdp, which can be plugged in bigtop). Still we'll use opensource packages and bigtop to create the RPM's and selects.
> Just we can use the source tar balls opensource components(e.g Hadoop-3.2.3 source tarballs from apache repo) and build rpm's and publish in following repo. Or from the apache github repo's.
>
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2Fbigtop%2Fbigtop-3.1.0%2Frepos%2F&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6kBAOFoW%2BsBHM8b4KX68e0UZ%2B%2BHg7Oxewx%2BJcCEdRXg%3D&reserved=0
>
> Planning to have sync up call on this week, once it's finalize, we can summarise here the approach. What do you think..?
>
>
> On 20/06/22, 4:06 PM, "Kengo Seki" <se...@apache.org> wrote:
>
> Thanks for the reply, Brahma!
>
> > We'll use following repo, how other components in Apache are hosted.
>
> Using that site for distribution has no problem. My concern is, do we
> have the source tree for building HDP packages and does it still work
> without the artifacts behind the paywall?
> According to ASF's release policy [1], we must publish the
> corresponding source releases whenever we publish binary artifacts,
> such as RPM and DEB packages for HDP.
>
> [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2Flegal%2Frelease-policy.html%23compiled-packages&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Cz%2BmZEjwjNnq9lICZmRtIDMLge9xZLQ0w3yK75AcHkc%3D&reserved=0
>
> Kengo Seki <se...@apache.org>
>
> On Mon, Jun 20, 2022 at 6:19 PM Battula, Brahma Reddy
> <bb...@visa.com.invalid> wrote:
> >
> > Hi Kengo Seki,
> >
> > Thanks for pitching here.. We'll use following repo, how other components in Apache are hosted. Any other suggestions..?
> >
> > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2F&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mQLNExpiw6fywDjtEbcUFWJlOuGJ6Ja45LWxjAWL6dY%3D&reserved=0
> >
> >
> >
> > On 20/06/22, 2:32 PM, "Kengo Seki" <se...@apache.org> wrote:
> >
> > Oh, I've just noticed that I submitted an additional question into the
> > wrong thread mistakenly.
> > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fj6kc2h1lphvjvsblcthcl5yzzmjk3tb5&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=R61kELeUwqNTvFHwQ5Gu4HuhnWuhrhPl07K40FLSrL0%3D&reserved=0 is
> > originally intended as a reply to this thread. Sorry for the
> > confusion.
> >
> > Kengo Seki <se...@apache.org>
> >
> > On Mon, Jun 20, 2022 at 11:31 AM Kengo Seki <se...@apache.org> wrote:
> > >
> > > Thank you so much for writing the draft and commenting on it, Brahma and Viraj!
> > > Firstly, a minor comment:
> > >
> > > > Trunk Code : Parallel we can work for the trunk code, this we might give as 2.8.
> > > > * Working on bigtop mpack
> > >
> > > This means adding a new built-in stack which uses the Bigtop
> > > repository, and not developing Mpack on the Bigtop side, right?
> > > If so, "bigtop mpack" sounds confusing a bit.
> > >
> > > > hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk branch. Thoughts?
> > >
> > > Sounds good to me. During the 2.7.x releases, users can use the HDP
> > > stack by default. In addition, they can also add the Bigtop stack
> > > through the Mpack, as Zhiguo mentioned in [1].
> > > Since 2.8 or 3.0, we will drop HDP and users can use the built-in
> > > Bigtop stack by default. It also allows us to drop the Mpack on the
> > > Bigtop side.
> > >
> > > Regarding the built-in Bigtop stack developed on trunk, the current
> > > BIGTOP stack in Ambari [2] may be a good start point to be upgraded.
> > > I ensured that Ambari 2.7.6 was successfully built with the
> > > `-Dstack.distribution=BIGTOP` option, though Bigtop 0.8 is too old to
> > > use for now.
> > >
> > > [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fljnyymp2g9fg7cq4z7sfxd36bh8x7nlm&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FOPQS1G0t6phSKTW8LIVxgJqUEWFOMlehrQgh7kVjv8%3D&reserved=0
> > > [2]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FBIGTOP%2F0.8&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MvxxrjL0F%2B7wLfw5Qr9bstYRBuiwma5mPMEPurCN3vw%3D&reserved=0
> > >
> > > Kengo Seki <se...@apache.org>
> > >
> > > On Mon, Jun 20, 2022 at 9:26 AM Viraj Jasani <vj...@apache.org> wrote:
> > > >
> > > > Thanks for the nice draft, Brahma!
> > > >
> > > > IMHO, as we narrow down to 2.7.7 release, it might be still worth sticking
> > > > to HDP i.e. plugin the new stack for HDP for 2.7 release line. For 2.8.0
> > > > (or 3.0.0), we can add entire new stack (BDP sounds good one) and start off
> > > > with 2.8.0/3.0.0 stack version.
> > > >
> > > > hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk
> > > > branch. Thoughts?
> > > >
> > > > Similarly, we can start-off with "python 3 only" support in trunk, whereas
> > > > 2.7 release line can continue with python 2 to reduce complexities for
> > > > maintenance releases on 2.7.
> > > >
> > > > All other points look good. Once again, thank you for drafting the roadmap
> > > > and starting with this thread.
> > > >
> > > >
> > > > On Sun, Jun 19, 2022 at 10:07 AM Battula, Brahma Reddy
> > > > <bb...@visa.com.invalid> wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > As Ambari back now, let’s start discussing roadmap. I will be closely
> > > > > tracking for Jira, GitHub(with Jenkins) and wiki to work normal.
> > > > > Based on the conclusion of this thread will update wiki[1].
> > > > >
> > > > > As 2.7.6 is released recently, we need to release 2.7.7
> > > > >
> > > > >
> > > > > 1. Stack Management. There are two approaches for this. Open to discuss
> > > > > on following..
> > > > > * Entirely new stack
> > > > >
> > > > > i.
> > > > > Define new stack name. (BDP: BigDataPlatform, BDS:BigDataStack,
> > > > > OSS:OpenSourceStack,TDP: TuskerDataPlatform…)
> > > > >
> > > > > ii.
> > > > > Define stack versions
> > > > > (zk-3.5.9,Hadoop-3.2.3,hive-3.1.3,tez-0.10.1,spark-3.2..)
> > > > >
> > > > > iii. Based
> > > > > above two we need to have select in bigtop code.
> > > > >
> > > > > * Using the existing for current release.
> > > > >
> > > > > i.
> > > > > Plugin the new stack into HDP self
> > > > >
> > > > > ii.
> > > > > Still, we can have hdp select
> > > > >
> > > > >
> > > > >
> > > > > I prefer the planA but planB takes less time. Users can easily adopt it
> > > > > upgrade their stack.
> > > > >
> > > > >
> > > > >
> > > > > Note: Ideally bigtop mpack can be long term approach as stack id decoupled
> > > > > now, but currently it’s not ready for latest stack (upgrades can be risk).
> > > > > We’ll work in trunk for same. If this works shortly, we can choose.
> > > > >
> > > > >
> > > > >
> > > > > 1. Upgrade :
> > > > > * Most of users will be deployed with HDP
> > > > > versions(HDP-2.6.5,HDP-3.1.0), we might need to support the upgrade to 2.7.7
> > > > > 2. Other work
> > > > > * Any critical/blocker CVE’s Fixes
> > > > > * CVEs Fixes : we’ll try to target some CVE fixes
> > > > > * Review the open PR’s and Jira’s
> > > > >
> > > > >
> > > > >
> > > > > Trunk Code : Parallel we can work for the trunk code, this we might give
> > > > > as 2.8.
> > > > >
> > > > > * Removal of python2 support
> > > > > * Removal of hard dependency on HDP
> > > > > * Working on bigtop mpack
> > > > >
> > > > > Can add more based on your inputs…
> > > > >
> > > > >
> > > > > Any thoughts on this..? Please feel free to add anything which I missed.
> > > > >
> > > > >
> > > > >
> > > > > Reference:
> > > > >
> > > > > 1.
> > > > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D30755705&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Kv1AicBwjFxEcdYsVjMWgUpefukzFP46lYAs9odJyNM%3D&reserved=0
> > > > >
> > > > >
> > > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
> > For additional commands, e-mail: dev-help@ambari.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
> > For additional commands, e-mail: dev-help@ambari.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
> For additional commands, e-mail: dev-help@ambari.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
Re: [Discuss] Initial Draft for Roadmap
Posted by Kengo Seki <se...@apache.org>.
Thanks for the elaboration Brahma, but I'm a bit confused.
At this point, my understanding is as follows. Are these correct?
* In the 2.7.x line, we'll still use a stack definition based on HDP's one [1].
* But regarding RPM and DEB packages, we'll build them from source by leveraging
Bigtop's build scripts and publish them on the ASF distribution site
or somewhere.
repoinfo.xml [2] will also be fixed to point that repository.
Assuming the understanding above is correct, my new concerns are:
* Can we still call it as "HDP"? Doesn't it infringe Cloudera's trademark?
If so, should we give a new stack name to it, as you mentioned as
planA in [3]?
(or it makes upgrading HDP cluster difficult? I don't understand it so much)
* It may be a bit tough work to fix Bigtop's build scripts so that its
artifacts match Ambari's requirements.
(package versioning, file layouts, included and not included files,
file permissions, user/group creation, etc.)
I'm not sure if it's faster than developing Mpack.
[1]: https://github.com/apache/ambari/tree/release-2.7.6/ambari-server/src/main/resources/stacks/HDP
[2]: https://github.com/apache/ambari/blob/release-2.7.6/ambari-server/src/main/resources/stacks/HDP/2.6/repos/repoinfo.xml
[3]: https://lists.apache.org/thread/m77lhzo0njr2dhoock5twtm0c19j2py8
Kengo Seki <se...@gmail.com>
On Mon, Jun 20, 2022 at 8:05 PM Battula, Brahma Reddy
<bb...@visa.com.invalid> wrote:
>
> We'll not use the HDP Packages (Only selects we'll use from hdp, which can be plugged in bigtop). Still we'll use opensource packages and bigtop to create the RPM's and selects.
> Just we can use the source tar balls opensource components(e.g Hadoop-3.2.3 source tarballs from apache repo) and build rpm's and publish in following repo. Or from the apache github repo's.
>
> https://dlcdn.apache.org/bigtop/bigtop-3.1.0/repos/
>
> Planning to have sync up call on this week, once it's finalize, we can summarise here the approach. What do you think..?
>
>
> On 20/06/22, 4:06 PM, "Kengo Seki" <se...@apache.org> wrote:
>
> Thanks for the reply, Brahma!
>
> > We'll use following repo, how other components in Apache are hosted.
>
> Using that site for distribution has no problem. My concern is, do we
> have the source tree for building HDP packages and does it still work
> without the artifacts behind the paywall?
> According to ASF's release policy [1], we must publish the
> corresponding source releases whenever we publish binary artifacts,
> such as RPM and DEB packages for HDP.
>
> [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2Flegal%2Frelease-policy.html%23compiled-packages&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qwRVqj4GzMqB%2BfeVfzMemWJIEjDHz5okUrYXdH4vj00%3D&reserved=0
>
> Kengo Seki <se...@apache.org>
>
> On Mon, Jun 20, 2022 at 6:19 PM Battula, Brahma Reddy
> <bb...@visa.com.invalid> wrote:
> >
> > Hi Kengo Seki,
> >
> > Thanks for pitching here.. We'll use following repo, how other components in Apache are hosted. Any other suggestions..?
> >
> > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2F&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FWqjRiYl0WOptyA5P42xwP5S6KHiK1QrIi5avv1vtHw%3D&reserved=0
> >
> >
> >
> > On 20/06/22, 2:32 PM, "Kengo Seki" <se...@apache.org> wrote:
> >
> > Oh, I've just noticed that I submitted an additional question into the
> > wrong thread mistakenly.
> > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fj6kc2h1lphvjvsblcthcl5yzzmjk3tb5&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JxCXNEbqlknyBZES5PoFrk356zlDeO1uUtGGl677kqk%3D&reserved=0 is
> > originally intended as a reply to this thread. Sorry for the
> > confusion.
> >
> > Kengo Seki <se...@apache.org>
> >
> > On Mon, Jun 20, 2022 at 11:31 AM Kengo Seki <se...@apache.org> wrote:
> > >
> > > Thank you so much for writing the draft and commenting on it, Brahma and Viraj!
> > > Firstly, a minor comment:
> > >
> > > > Trunk Code : Parallel we can work for the trunk code, this we might give as 2.8.
> > > > * Working on bigtop mpack
> > >
> > > This means adding a new built-in stack which uses the Bigtop
> > > repository, and not developing Mpack on the Bigtop side, right?
> > > If so, "bigtop mpack" sounds confusing a bit.
> > >
> > > > hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk branch. Thoughts?
> > >
> > > Sounds good to me. During the 2.7.x releases, users can use the HDP
> > > stack by default. In addition, they can also add the Bigtop stack
> > > through the Mpack, as Zhiguo mentioned in [1].
> > > Since 2.8 or 3.0, we will drop HDP and users can use the built-in
> > > Bigtop stack by default. It also allows us to drop the Mpack on the
> > > Bigtop side.
> > >
> > > Regarding the built-in Bigtop stack developed on trunk, the current
> > > BIGTOP stack in Ambari [2] may be a good start point to be upgraded.
> > > I ensured that Ambari 2.7.6 was successfully built with the
> > > `-Dstack.distribution=BIGTOP` option, though Bigtop 0.8 is too old to
> > > use for now.
> > >
> > > [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fljnyymp2g9fg7cq4z7sfxd36bh8x7nlm&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5nWc%2BIgBeDof0a0wdth4X0JrGPNaUkse6ye66XpNFiM%3D&reserved=0
> > > [2]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FBIGTOP%2F0.8&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ExDSC%2BX%2BIhqdt8ok8w5jxFE2vDAlIiOQRmFkBXZ4XwY%3D&reserved=0
> > >
> > > Kengo Seki <se...@apache.org>
> > >
> > > On Mon, Jun 20, 2022 at 9:26 AM Viraj Jasani <vj...@apache.org> wrote:
> > > >
> > > > Thanks for the nice draft, Brahma!
> > > >
> > > > IMHO, as we narrow down to 2.7.7 release, it might be still worth sticking
> > > > to HDP i.e. plugin the new stack for HDP for 2.7 release line. For 2.8.0
> > > > (or 3.0.0), we can add entire new stack (BDP sounds good one) and start off
> > > > with 2.8.0/3.0.0 stack version.
> > > >
> > > > hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk
> > > > branch. Thoughts?
> > > >
> > > > Similarly, we can start-off with "python 3 only" support in trunk, whereas
> > > > 2.7 release line can continue with python 2 to reduce complexities for
> > > > maintenance releases on 2.7.
> > > >
> > > > All other points look good. Once again, thank you for drafting the roadmap
> > > > and starting with this thread.
> > > >
> > > >
> > > > On Sun, Jun 19, 2022 at 10:07 AM Battula, Brahma Reddy
> > > > <bb...@visa.com.invalid> wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > As Ambari back now, let’s start discussing roadmap. I will be closely
> > > > > tracking for Jira, GitHub(with Jenkins) and wiki to work normal.
> > > > > Based on the conclusion of this thread will update wiki[1].
> > > > >
> > > > > As 2.7.6 is released recently, we need to release 2.7.7
> > > > >
> > > > >
> > > > > 1. Stack Management. There are two approaches for this. Open to discuss
> > > > > on following..
> > > > > * Entirely new stack
> > > > >
> > > > > i.
> > > > > Define new stack name. (BDP: BigDataPlatform, BDS:BigDataStack,
> > > > > OSS:OpenSourceStack,TDP: TuskerDataPlatform…)
> > > > >
> > > > > ii.
> > > > > Define stack versions
> > > > > (zk-3.5.9,Hadoop-3.2.3,hive-3.1.3,tez-0.10.1,spark-3.2..)
> > > > >
> > > > > iii. Based
> > > > > above two we need to have select in bigtop code.
> > > > >
> > > > > * Using the existing for current release.
> > > > >
> > > > > i.
> > > > > Plugin the new stack into HDP self
> > > > >
> > > > > ii.
> > > > > Still, we can have hdp select
> > > > >
> > > > >
> > > > >
> > > > > I prefer the planA but planB takes less time. Users can easily adopt it
> > > > > upgrade their stack.
> > > > >
> > > > >
> > > > >
> > > > > Note: Ideally bigtop mpack can be long term approach as stack id decoupled
> > > > > now, but currently it’s not ready for latest stack (upgrades can be risk).
> > > > > We’ll work in trunk for same. If this works shortly, we can choose.
> > > > >
> > > > >
> > > > >
> > > > > 1. Upgrade :
> > > > > * Most of users will be deployed with HDP
> > > > > versions(HDP-2.6.5,HDP-3.1.0), we might need to support the upgrade to 2.7.7
> > > > > 2. Other work
> > > > > * Any critical/blocker CVE’s Fixes
> > > > > * CVEs Fixes : we’ll try to target some CVE fixes
> > > > > * Review the open PR’s and Jira’s
> > > > >
> > > > >
> > > > >
> > > > > Trunk Code : Parallel we can work for the trunk code, this we might give
> > > > > as 2.8.
> > > > >
> > > > > * Removal of python2 support
> > > > > * Removal of hard dependency on HDP
> > > > > * Working on bigtop mpack
> > > > >
> > > > > Can add more based on your inputs…
> > > > >
> > > > >
> > > > > Any thoughts on this..? Please feel free to add anything which I missed.
> > > > >
> > > > >
> > > > >
> > > > > Reference:
> > > > >
> > > > > 1.
> > > > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D30755705&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=s%2FI1zyAlaiuvWqx%2FjZo%2FNBu4KeHrPVmxtIfeYgLQAso%3D&reserved=0
> > > > >
> > > > >
> > > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
> > For additional commands, e-mail: dev-help@ambari.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
> > For additional commands, e-mail: dev-help@ambari.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
> For additional commands, e-mail: dev-help@ambari.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
Re: [Discuss] Initial Draft for Roadmap
Posted by "Battula, Brahma Reddy" <bb...@visa.com.INVALID>.
We'll not use the HDP Packages (Only selects we'll use from hdp, which can be plugged in bigtop). Still we'll use opensource packages and bigtop to create the RPM's and selects.
Just we can use the source tar balls opensource components(e.g Hadoop-3.2.3 source tarballs from apache repo) and build rpm's and publish in following repo. Or from the apache github repo's.
https://dlcdn.apache.org/bigtop/bigtop-3.1.0/repos/
Planning to have sync up call on this week, once it's finalize, we can summarise here the approach. What do you think..?
On 20/06/22, 4:06 PM, "Kengo Seki" <se...@apache.org> wrote:
Thanks for the reply, Brahma!
> We'll use following repo, how other components in Apache are hosted.
Using that site for distribution has no problem. My concern is, do we
have the source tree for building HDP packages and does it still work
without the artifacts behind the paywall?
According to ASF's release policy [1], we must publish the
corresponding source releases whenever we publish binary artifacts,
such as RPM and DEB packages for HDP.
[1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2Flegal%2Frelease-policy.html%23compiled-packages&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qwRVqj4GzMqB%2BfeVfzMemWJIEjDHz5okUrYXdH4vj00%3D&reserved=0
Kengo Seki <se...@apache.org>
On Mon, Jun 20, 2022 at 6:19 PM Battula, Brahma Reddy
<bb...@visa.com.invalid> wrote:
>
> Hi Kengo Seki,
>
> Thanks for pitching here.. We'll use following repo, how other components in Apache are hosted. Any other suggestions..?
>
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2F&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FWqjRiYl0WOptyA5P42xwP5S6KHiK1QrIi5avv1vtHw%3D&reserved=0
>
>
>
> On 20/06/22, 2:32 PM, "Kengo Seki" <se...@apache.org> wrote:
>
> Oh, I've just noticed that I submitted an additional question into the
> wrong thread mistakenly.
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fj6kc2h1lphvjvsblcthcl5yzzmjk3tb5&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JxCXNEbqlknyBZES5PoFrk356zlDeO1uUtGGl677kqk%3D&reserved=0 is
> originally intended as a reply to this thread. Sorry for the
> confusion.
>
> Kengo Seki <se...@apache.org>
>
> On Mon, Jun 20, 2022 at 11:31 AM Kengo Seki <se...@apache.org> wrote:
> >
> > Thank you so much for writing the draft and commenting on it, Brahma and Viraj!
> > Firstly, a minor comment:
> >
> > > Trunk Code : Parallel we can work for the trunk code, this we might give as 2.8.
> > > * Working on bigtop mpack
> >
> > This means adding a new built-in stack which uses the Bigtop
> > repository, and not developing Mpack on the Bigtop side, right?
> > If so, "bigtop mpack" sounds confusing a bit.
> >
> > > hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk branch. Thoughts?
> >
> > Sounds good to me. During the 2.7.x releases, users can use the HDP
> > stack by default. In addition, they can also add the Bigtop stack
> > through the Mpack, as Zhiguo mentioned in [1].
> > Since 2.8 or 3.0, we will drop HDP and users can use the built-in
> > Bigtop stack by default. It also allows us to drop the Mpack on the
> > Bigtop side.
> >
> > Regarding the built-in Bigtop stack developed on trunk, the current
> > BIGTOP stack in Ambari [2] may be a good start point to be upgraded.
> > I ensured that Ambari 2.7.6 was successfully built with the
> > `-Dstack.distribution=BIGTOP` option, though Bigtop 0.8 is too old to
> > use for now.
> >
> > [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fljnyymp2g9fg7cq4z7sfxd36bh8x7nlm&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5nWc%2BIgBeDof0a0wdth4X0JrGPNaUkse6ye66XpNFiM%3D&reserved=0
> > [2]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FBIGTOP%2F0.8&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ExDSC%2BX%2BIhqdt8ok8w5jxFE2vDAlIiOQRmFkBXZ4XwY%3D&reserved=0
> >
> > Kengo Seki <se...@apache.org>
> >
> > On Mon, Jun 20, 2022 at 9:26 AM Viraj Jasani <vj...@apache.org> wrote:
> > >
> > > Thanks for the nice draft, Brahma!
> > >
> > > IMHO, as we narrow down to 2.7.7 release, it might be still worth sticking
> > > to HDP i.e. plugin the new stack for HDP for 2.7 release line. For 2.8.0
> > > (or 3.0.0), we can add entire new stack (BDP sounds good one) and start off
> > > with 2.8.0/3.0.0 stack version.
> > >
> > > hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk
> > > branch. Thoughts?
> > >
> > > Similarly, we can start-off with "python 3 only" support in trunk, whereas
> > > 2.7 release line can continue with python 2 to reduce complexities for
> > > maintenance releases on 2.7.
> > >
> > > All other points look good. Once again, thank you for drafting the roadmap
> > > and starting with this thread.
> > >
> > >
> > > On Sun, Jun 19, 2022 at 10:07 AM Battula, Brahma Reddy
> > > <bb...@visa.com.invalid> wrote:
> > >
> > > > Hi All,
> > > >
> > > > As Ambari back now, let’s start discussing roadmap. I will be closely
> > > > tracking for Jira, GitHub(with Jenkins) and wiki to work normal.
> > > > Based on the conclusion of this thread will update wiki[1].
> > > >
> > > > As 2.7.6 is released recently, we need to release 2.7.7
> > > >
> > > >
> > > > 1. Stack Management. There are two approaches for this. Open to discuss
> > > > on following..
> > > > * Entirely new stack
> > > >
> > > > i.
> > > > Define new stack name. (BDP: BigDataPlatform, BDS:BigDataStack,
> > > > OSS:OpenSourceStack,TDP: TuskerDataPlatform…)
> > > >
> > > > ii.
> > > > Define stack versions
> > > > (zk-3.5.9,Hadoop-3.2.3,hive-3.1.3,tez-0.10.1,spark-3.2..)
> > > >
> > > > iii. Based
> > > > above two we need to have select in bigtop code.
> > > >
> > > > * Using the existing for current release.
> > > >
> > > > i.
> > > > Plugin the new stack into HDP self
> > > >
> > > > ii.
> > > > Still, we can have hdp select
> > > >
> > > >
> > > >
> > > > I prefer the planA but planB takes less time. Users can easily adopt it
> > > > upgrade their stack.
> > > >
> > > >
> > > >
> > > > Note: Ideally bigtop mpack can be long term approach as stack id decoupled
> > > > now, but currently it’s not ready for latest stack (upgrades can be risk).
> > > > We’ll work in trunk for same. If this works shortly, we can choose.
> > > >
> > > >
> > > >
> > > > 1. Upgrade :
> > > > * Most of users will be deployed with HDP
> > > > versions(HDP-2.6.5,HDP-3.1.0), we might need to support the upgrade to 2.7.7
> > > > 2. Other work
> > > > * Any critical/blocker CVE’s Fixes
> > > > * CVEs Fixes : we’ll try to target some CVE fixes
> > > > * Review the open PR’s and Jira’s
> > > >
> > > >
> > > >
> > > > Trunk Code : Parallel we can work for the trunk code, this we might give
> > > > as 2.8.
> > > >
> > > > * Removal of python2 support
> > > > * Removal of hard dependency on HDP
> > > > * Working on bigtop mpack
> > > >
> > > > Can add more based on your inputs…
> > > >
> > > >
> > > > Any thoughts on this..? Please feel free to add anything which I missed.
> > > >
> > > >
> > > >
> > > > Reference:
> > > >
> > > > 1.
> > > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D30755705&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=s%2FI1zyAlaiuvWqx%2FjZo%2FNBu4KeHrPVmxtIfeYgLQAso%3D&reserved=0
> > > >
> > > >
> > > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
> For additional commands, e-mail: dev-help@ambari.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
> For additional commands, e-mail: dev-help@ambari.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
Re: [Discuss] Initial Draft for Roadmap
Posted by Kengo Seki <se...@apache.org>.
Thanks for the reply, Brahma!
> We'll use following repo, how other components in Apache are hosted.
Using that site for distribution has no problem. My concern is, do we
have the source tree for building HDP packages and does it still work
without the artifacts behind the paywall?
According to ASF's release policy [1], we must publish the
corresponding source releases whenever we publish binary artifacts,
such as RPM and DEB packages for HDP.
[1]: https://www.apache.org/legal/release-policy.html#compiled-packages
Kengo Seki <se...@apache.org>
On Mon, Jun 20, 2022 at 6:19 PM Battula, Brahma Reddy
<bb...@visa.com.invalid> wrote:
>
> Hi Kengo Seki,
>
> Thanks for pitching here.. We'll use following repo, how other components in Apache are hosted. Any other suggestions..?
>
> https://dlcdn.apache.org/
>
>
>
> On 20/06/22, 2:32 PM, "Kengo Seki" <se...@apache.org> wrote:
>
> Oh, I've just noticed that I submitted an additional question into the
> wrong thread mistakenly.
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fj6kc2h1lphvjvsblcthcl5yzzmjk3tb5&data=05%7C01%7Cbbattula%40visa.com%7C8f8816799a4f4de1f58608da529b9c42%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913125540570268%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=kdYhCFVQVGowhmsHAagNOJFngJWkqmdL0bNQ03JBEGc%3D&reserved=0 is
> originally intended as a reply to this thread. Sorry for the
> confusion.
>
> Kengo Seki <se...@apache.org>
>
> On Mon, Jun 20, 2022 at 11:31 AM Kengo Seki <se...@apache.org> wrote:
> >
> > Thank you so much for writing the draft and commenting on it, Brahma and Viraj!
> > Firstly, a minor comment:
> >
> > > Trunk Code : Parallel we can work for the trunk code, this we might give as 2.8.
> > > * Working on bigtop mpack
> >
> > This means adding a new built-in stack which uses the Bigtop
> > repository, and not developing Mpack on the Bigtop side, right?
> > If so, "bigtop mpack" sounds confusing a bit.
> >
> > > hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk branch. Thoughts?
> >
> > Sounds good to me. During the 2.7.x releases, users can use the HDP
> > stack by default. In addition, they can also add the Bigtop stack
> > through the Mpack, as Zhiguo mentioned in [1].
> > Since 2.8 or 3.0, we will drop HDP and users can use the built-in
> > Bigtop stack by default. It also allows us to drop the Mpack on the
> > Bigtop side.
> >
> > Regarding the built-in Bigtop stack developed on trunk, the current
> > BIGTOP stack in Ambari [2] may be a good start point to be upgraded.
> > I ensured that Ambari 2.7.6 was successfully built with the
> > `-Dstack.distribution=BIGTOP` option, though Bigtop 0.8 is too old to
> > use for now.
> >
> > [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fljnyymp2g9fg7cq4z7sfxd36bh8x7nlm&data=05%7C01%7Cbbattula%40visa.com%7C8f8816799a4f4de1f58608da529b9c42%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913125540570268%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Zf6225sm1LNTsTEuXCkl3F1NiEsuN1afC9oFEsCpAoU%3D&reserved=0
> > [2]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FBIGTOP%2F0.8&data=05%7C01%7Cbbattula%40visa.com%7C8f8816799a4f4de1f58608da529b9c42%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913125540570268%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=kffTwEi%2FvLaQe7CjLzQQJVwc7K9nwZ3FZ4yJlsPr%2BFA%3D&reserved=0
> >
> > Kengo Seki <se...@apache.org>
> >
> > On Mon, Jun 20, 2022 at 9:26 AM Viraj Jasani <vj...@apache.org> wrote:
> > >
> > > Thanks for the nice draft, Brahma!
> > >
> > > IMHO, as we narrow down to 2.7.7 release, it might be still worth sticking
> > > to HDP i.e. plugin the new stack for HDP for 2.7 release line. For 2.8.0
> > > (or 3.0.0), we can add entire new stack (BDP sounds good one) and start off
> > > with 2.8.0/3.0.0 stack version.
> > >
> > > hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk
> > > branch. Thoughts?
> > >
> > > Similarly, we can start-off with "python 3 only" support in trunk, whereas
> > > 2.7 release line can continue with python 2 to reduce complexities for
> > > maintenance releases on 2.7.
> > >
> > > All other points look good. Once again, thank you for drafting the roadmap
> > > and starting with this thread.
> > >
> > >
> > > On Sun, Jun 19, 2022 at 10:07 AM Battula, Brahma Reddy
> > > <bb...@visa.com.invalid> wrote:
> > >
> > > > Hi All,
> > > >
> > > > As Ambari back now, let’s start discussing roadmap. I will be closely
> > > > tracking for Jira, GitHub(with Jenkins) and wiki to work normal.
> > > > Based on the conclusion of this thread will update wiki[1].
> > > >
> > > > As 2.7.6 is released recently, we need to release 2.7.7
> > > >
> > > >
> > > > 1. Stack Management. There are two approaches for this. Open to discuss
> > > > on following..
> > > > * Entirely new stack
> > > >
> > > > i.
> > > > Define new stack name. (BDP: BigDataPlatform, BDS:BigDataStack,
> > > > OSS:OpenSourceStack,TDP: TuskerDataPlatform…)
> > > >
> > > > ii.
> > > > Define stack versions
> > > > (zk-3.5.9,Hadoop-3.2.3,hive-3.1.3,tez-0.10.1,spark-3.2..)
> > > >
> > > > iii. Based
> > > > above two we need to have select in bigtop code.
> > > >
> > > > * Using the existing for current release.
> > > >
> > > > i.
> > > > Plugin the new stack into HDP self
> > > >
> > > > ii.
> > > > Still, we can have hdp select
> > > >
> > > >
> > > >
> > > > I prefer the planA but planB takes less time. Users can easily adopt it
> > > > upgrade their stack.
> > > >
> > > >
> > > >
> > > > Note: Ideally bigtop mpack can be long term approach as stack id decoupled
> > > > now, but currently it’s not ready for latest stack (upgrades can be risk).
> > > > We’ll work in trunk for same. If this works shortly, we can choose.
> > > >
> > > >
> > > >
> > > > 1. Upgrade :
> > > > * Most of users will be deployed with HDP
> > > > versions(HDP-2.6.5,HDP-3.1.0), we might need to support the upgrade to 2.7.7
> > > > 2. Other work
> > > > * Any critical/blocker CVE’s Fixes
> > > > * CVEs Fixes : we’ll try to target some CVE fixes
> > > > * Review the open PR’s and Jira’s
> > > >
> > > >
> > > >
> > > > Trunk Code : Parallel we can work for the trunk code, this we might give
> > > > as 2.8.
> > > >
> > > > * Removal of python2 support
> > > > * Removal of hard dependency on HDP
> > > > * Working on bigtop mpack
> > > >
> > > > Can add more based on your inputs…
> > > >
> > > >
> > > > Any thoughts on this..? Please feel free to add anything which I missed.
> > > >
> > > >
> > > >
> > > > Reference:
> > > >
> > > > 1.
> > > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D30755705&data=05%7C01%7Cbbattula%40visa.com%7C8f8816799a4f4de1f58608da529b9c42%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913125540570268%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8HISa3GK%2FDRU6%2BBkFg86WWXOVMOf3QOx36volDkTlWE%3D&reserved=0
> > > >
> > > >
> > > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
> For additional commands, e-mail: dev-help@ambari.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
> For additional commands, e-mail: dev-help@ambari.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
Re: [Discuss] Initial Draft for Roadmap
Posted by "Battula, Brahma Reddy" <bb...@visa.com.INVALID>.
Hi Kengo Seki,
Thanks for pitching here.. We'll use following repo, how other components in Apache are hosted. Any other suggestions..?
https://dlcdn.apache.org/
On 20/06/22, 2:32 PM, "Kengo Seki" <se...@apache.org> wrote:
Oh, I've just noticed that I submitted an additional question into the
wrong thread mistakenly.
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fj6kc2h1lphvjvsblcthcl5yzzmjk3tb5&data=05%7C01%7Cbbattula%40visa.com%7C8f8816799a4f4de1f58608da529b9c42%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913125540570268%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=kdYhCFVQVGowhmsHAagNOJFngJWkqmdL0bNQ03JBEGc%3D&reserved=0 is
originally intended as a reply to this thread. Sorry for the
confusion.
Kengo Seki <se...@apache.org>
On Mon, Jun 20, 2022 at 11:31 AM Kengo Seki <se...@apache.org> wrote:
>
> Thank you so much for writing the draft and commenting on it, Brahma and Viraj!
> Firstly, a minor comment:
>
> > Trunk Code : Parallel we can work for the trunk code, this we might give as 2.8.
> > * Working on bigtop mpack
>
> This means adding a new built-in stack which uses the Bigtop
> repository, and not developing Mpack on the Bigtop side, right?
> If so, "bigtop mpack" sounds confusing a bit.
>
> > hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk branch. Thoughts?
>
> Sounds good to me. During the 2.7.x releases, users can use the HDP
> stack by default. In addition, they can also add the Bigtop stack
> through the Mpack, as Zhiguo mentioned in [1].
> Since 2.8 or 3.0, we will drop HDP and users can use the built-in
> Bigtop stack by default. It also allows us to drop the Mpack on the
> Bigtop side.
>
> Regarding the built-in Bigtop stack developed on trunk, the current
> BIGTOP stack in Ambari [2] may be a good start point to be upgraded.
> I ensured that Ambari 2.7.6 was successfully built with the
> `-Dstack.distribution=BIGTOP` option, though Bigtop 0.8 is too old to
> use for now.
>
> [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fljnyymp2g9fg7cq4z7sfxd36bh8x7nlm&data=05%7C01%7Cbbattula%40visa.com%7C8f8816799a4f4de1f58608da529b9c42%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913125540570268%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Zf6225sm1LNTsTEuXCkl3F1NiEsuN1afC9oFEsCpAoU%3D&reserved=0
> [2]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FBIGTOP%2F0.8&data=05%7C01%7Cbbattula%40visa.com%7C8f8816799a4f4de1f58608da529b9c42%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913125540570268%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=kffTwEi%2FvLaQe7CjLzQQJVwc7K9nwZ3FZ4yJlsPr%2BFA%3D&reserved=0
>
> Kengo Seki <se...@apache.org>
>
> On Mon, Jun 20, 2022 at 9:26 AM Viraj Jasani <vj...@apache.org> wrote:
> >
> > Thanks for the nice draft, Brahma!
> >
> > IMHO, as we narrow down to 2.7.7 release, it might be still worth sticking
> > to HDP i.e. plugin the new stack for HDP for 2.7 release line. For 2.8.0
> > (or 3.0.0), we can add entire new stack (BDP sounds good one) and start off
> > with 2.8.0/3.0.0 stack version.
> >
> > hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk
> > branch. Thoughts?
> >
> > Similarly, we can start-off with "python 3 only" support in trunk, whereas
> > 2.7 release line can continue with python 2 to reduce complexities for
> > maintenance releases on 2.7.
> >
> > All other points look good. Once again, thank you for drafting the roadmap
> > and starting with this thread.
> >
> >
> > On Sun, Jun 19, 2022 at 10:07 AM Battula, Brahma Reddy
> > <bb...@visa.com.invalid> wrote:
> >
> > > Hi All,
> > >
> > > As Ambari back now, let’s start discussing roadmap. I will be closely
> > > tracking for Jira, GitHub(with Jenkins) and wiki to work normal.
> > > Based on the conclusion of this thread will update wiki[1].
> > >
> > > As 2.7.6 is released recently, we need to release 2.7.7
> > >
> > >
> > > 1. Stack Management. There are two approaches for this. Open to discuss
> > > on following..
> > > * Entirely new stack
> > >
> > > i.
> > > Define new stack name. (BDP: BigDataPlatform, BDS:BigDataStack,
> > > OSS:OpenSourceStack,TDP: TuskerDataPlatform…)
> > >
> > > ii.
> > > Define stack versions
> > > (zk-3.5.9,Hadoop-3.2.3,hive-3.1.3,tez-0.10.1,spark-3.2..)
> > >
> > > iii. Based
> > > above two we need to have select in bigtop code.
> > >
> > > * Using the existing for current release.
> > >
> > > i.
> > > Plugin the new stack into HDP self
> > >
> > > ii.
> > > Still, we can have hdp select
> > >
> > >
> > >
> > > I prefer the planA but planB takes less time. Users can easily adopt it
> > > upgrade their stack.
> > >
> > >
> > >
> > > Note: Ideally bigtop mpack can be long term approach as stack id decoupled
> > > now, but currently it’s not ready for latest stack (upgrades can be risk).
> > > We’ll work in trunk for same. If this works shortly, we can choose.
> > >
> > >
> > >
> > > 1. Upgrade :
> > > * Most of users will be deployed with HDP
> > > versions(HDP-2.6.5,HDP-3.1.0), we might need to support the upgrade to 2.7.7
> > > 2. Other work
> > > * Any critical/blocker CVE’s Fixes
> > > * CVEs Fixes : we’ll try to target some CVE fixes
> > > * Review the open PR’s and Jira’s
> > >
> > >
> > >
> > > Trunk Code : Parallel we can work for the trunk code, this we might give
> > > as 2.8.
> > >
> > > * Removal of python2 support
> > > * Removal of hard dependency on HDP
> > > * Working on bigtop mpack
> > >
> > > Can add more based on your inputs…
> > >
> > >
> > > Any thoughts on this..? Please feel free to add anything which I missed.
> > >
> > >
> > >
> > > Reference:
> > >
> > > 1.
> > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D30755705&data=05%7C01%7Cbbattula%40visa.com%7C8f8816799a4f4de1f58608da529b9c42%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913125540570268%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8HISa3GK%2FDRU6%2BBkFg86WWXOVMOf3QOx36volDkTlWE%3D&reserved=0
> > >
> > >
> > >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
Re: [Discuss] Initial Draft for Roadmap
Posted by Kengo Seki <se...@apache.org>.
Oh, I've just noticed that I submitted an additional question into the
wrong thread mistakenly.
https://lists.apache.org/thread/j6kc2h1lphvjvsblcthcl5yzzmjk3tb5 is
originally intended as a reply to this thread. Sorry for the
confusion.
Kengo Seki <se...@apache.org>
On Mon, Jun 20, 2022 at 11:31 AM Kengo Seki <se...@apache.org> wrote:
>
> Thank you so much for writing the draft and commenting on it, Brahma and Viraj!
> Firstly, a minor comment:
>
> > Trunk Code : Parallel we can work for the trunk code, this we might give as 2.8.
> > * Working on bigtop mpack
>
> This means adding a new built-in stack which uses the Bigtop
> repository, and not developing Mpack on the Bigtop side, right?
> If so, "bigtop mpack" sounds confusing a bit.
>
> > hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk branch. Thoughts?
>
> Sounds good to me. During the 2.7.x releases, users can use the HDP
> stack by default. In addition, they can also add the Bigtop stack
> through the Mpack, as Zhiguo mentioned in [1].
> Since 2.8 or 3.0, we will drop HDP and users can use the built-in
> Bigtop stack by default. It also allows us to drop the Mpack on the
> Bigtop side.
>
> Regarding the built-in Bigtop stack developed on trunk, the current
> BIGTOP stack in Ambari [2] may be a good start point to be upgraded.
> I ensured that Ambari 2.7.6 was successfully built with the
> `-Dstack.distribution=BIGTOP` option, though Bigtop 0.8 is too old to
> use for now.
>
> [1]: https://lists.apache.org/thread/ljnyymp2g9fg7cq4z7sfxd36bh8x7nlm
> [2]: https://github.com/apache/ambari/tree/release-2.7.6/ambari-server/src/main/resources/stacks/BIGTOP/0.8
>
> Kengo Seki <se...@apache.org>
>
> On Mon, Jun 20, 2022 at 9:26 AM Viraj Jasani <vj...@apache.org> wrote:
> >
> > Thanks for the nice draft, Brahma!
> >
> > IMHO, as we narrow down to 2.7.7 release, it might be still worth sticking
> > to HDP i.e. plugin the new stack for HDP for 2.7 release line. For 2.8.0
> > (or 3.0.0), we can add entire new stack (BDP sounds good one) and start off
> > with 2.8.0/3.0.0 stack version.
> >
> > hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk
> > branch. Thoughts?
> >
> > Similarly, we can start-off with "python 3 only" support in trunk, whereas
> > 2.7 release line can continue with python 2 to reduce complexities for
> > maintenance releases on 2.7.
> >
> > All other points look good. Once again, thank you for drafting the roadmap
> > and starting with this thread.
> >
> >
> > On Sun, Jun 19, 2022 at 10:07 AM Battula, Brahma Reddy
> > <bb...@visa.com.invalid> wrote:
> >
> > > Hi All,
> > >
> > > As Ambari back now, let’s start discussing roadmap. I will be closely
> > > tracking for Jira, GitHub(with Jenkins) and wiki to work normal.
> > > Based on the conclusion of this thread will update wiki[1].
> > >
> > > As 2.7.6 is released recently, we need to release 2.7.7
> > >
> > >
> > > 1. Stack Management. There are two approaches for this. Open to discuss
> > > on following..
> > > * Entirely new stack
> > >
> > > i.
> > > Define new stack name. (BDP: BigDataPlatform, BDS:BigDataStack,
> > > OSS:OpenSourceStack,TDP: TuskerDataPlatform…)
> > >
> > > ii.
> > > Define stack versions
> > > (zk-3.5.9,Hadoop-3.2.3,hive-3.1.3,tez-0.10.1,spark-3.2..)
> > >
> > > iii. Based
> > > above two we need to have select in bigtop code.
> > >
> > > * Using the existing for current release.
> > >
> > > i.
> > > Plugin the new stack into HDP self
> > >
> > > ii.
> > > Still, we can have hdp select
> > >
> > >
> > >
> > > I prefer the planA but planB takes less time. Users can easily adopt it
> > > upgrade their stack.
> > >
> > >
> > >
> > > Note: Ideally bigtop mpack can be long term approach as stack id decoupled
> > > now, but currently it’s not ready for latest stack (upgrades can be risk).
> > > We’ll work in trunk for same. If this works shortly, we can choose.
> > >
> > >
> > >
> > > 1. Upgrade :
> > > * Most of users will be deployed with HDP
> > > versions(HDP-2.6.5,HDP-3.1.0), we might need to support the upgrade to 2.7.7
> > > 2. Other work
> > > * Any critical/blocker CVE’s Fixes
> > > * CVEs Fixes : we’ll try to target some CVE fixes
> > > * Review the open PR’s and Jira’s
> > >
> > >
> > >
> > > Trunk Code : Parallel we can work for the trunk code, this we might give
> > > as 2.8.
> > >
> > > * Removal of python2 support
> > > * Removal of hard dependency on HDP
> > > * Working on bigtop mpack
> > >
> > > Can add more based on your inputs…
> > >
> > >
> > > Any thoughts on this..? Please feel free to add anything which I missed.
> > >
> > >
> > >
> > > Reference:
> > >
> > > 1.
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=30755705
> > >
> > >
> > >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
Re: [Discuss] Initial Draft for Roadmap
Posted by "Battula, Brahma Reddy" <bb...@visa.com.INVALID>.
>> This means adding a new built-in stack which uses the Bigtop
>> repository, and not developing Mpack on the Bigtop side, right?
>> If so, "bigtop mpack" sounds confusing a bit.
Looks you are confused with bigtop stack in ambari with bigtop mpack. What I mean is, As bigtop mpack which is currently in-progress in the bigtop community will take some more time develop and mature it, which can be used to trunk.
>>Sounds good to me. During the 2.7.x releases, users can use the HDP
>>stack by default. In addition, they can also add the Bigtop stack
>>through the Mpack, as Zhiguo mentioned in [1].
>>Since 2.8 or 3.0, we will drop HDP and users can use the built-in
>>Bigtop stack by default. It also allows us to drop the Mpack on the
>>Bigtop side.
Yes, this what mentioned. Bigtop mpack will be long term approach which we'll work together with bigtop community and will drop HDP.
>> Regarding the built-in Bigtop stack developed on trunk, the current
>> BIGTOP stack in Ambari [2] may be a good start point to be upgraded.
>> I ensured that Ambari 2.7.6 was successfully built with the
>> `-Dstack.distribution=BIGTOP` option, though Bigtop 0.8 is too old to
>> use for now.
Yes, this can be. But Existing cluster upgrades can be challenge right.(where those will be deployed with HDP-2.6.5,HDP-3.1.0 and HDP-3.1.5).?
On 20/06/22, 8:02 AM, "Kengo Seki" <se...@apache.org> wrote:
Thank you so much for writing the draft and commenting on it, Brahma and Viraj!
Firstly, a minor comment:
> Trunk Code : Parallel we can work for the trunk code, this we might give as 2.8.
> * Working on bigtop mpack
This means adding a new built-in stack which uses the Bigtop
repository, and not developing Mpack on the Bigtop side, right?
If so, "bigtop mpack" sounds confusing a bit.
> hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk branch. Thoughts?
Sounds good to me. During the 2.7.x releases, users can use the HDP
stack by default. In addition, they can also add the Bigtop stack
through the Mpack, as Zhiguo mentioned in [1].
Since 2.8 or 3.0, we will drop HDP and users can use the built-in
Bigtop stack by default. It also allows us to drop the Mpack on the
Bigtop side.
Regarding the built-in Bigtop stack developed on trunk, the current
BIGTOP stack in Ambari [2] may be a good start point to be upgraded.
I ensured that Ambari 2.7.6 was successfully built with the
`-Dstack.distribution=BIGTOP` option, though Bigtop 0.8 is too old to
use for now.
[1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fljnyymp2g9fg7cq4z7sfxd36bh8x7nlm&data=05%7C01%7Cbbattula%40visa.com%7Cd8a59968d77942985dab08da52651626%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637912891362493977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9HBlpQcv%2Bun2VrDGL%2F21MJTMIJz2%2Bs7ufkfFytIorPE%3D&reserved=0
[2]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FBIGTOP%2F0.8&data=05%7C01%7Cbbattula%40visa.com%7Cd8a59968d77942985dab08da52651626%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637912891362493977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nqRlKJ%2F%2FyTcqkmWk%2FDdYCmApMtIa0izybsnw26lPb%2BQ%3D&reserved=0
Kengo Seki <se...@apache.org>
On Mon, Jun 20, 2022 at 9:26 AM Viraj Jasani <vj...@apache.org> wrote:
>
> Thanks for the nice draft, Brahma!
>
> IMHO, as we narrow down to 2.7.7 release, it might be still worth sticking
> to HDP i.e. plugin the new stack for HDP for 2.7 release line. For 2.8.0
> (or 3.0.0), we can add entire new stack (BDP sounds good one) and start off
> with 2.8.0/3.0.0 stack version.
>
> hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk
> branch. Thoughts?
>
> Similarly, we can start-off with "python 3 only" support in trunk, whereas
> 2.7 release line can continue with python 2 to reduce complexities for
> maintenance releases on 2.7.
>
> All other points look good. Once again, thank you for drafting the roadmap
> and starting with this thread.
>
>
> On Sun, Jun 19, 2022 at 10:07 AM Battula, Brahma Reddy
> <bb...@visa.com.invalid> wrote:
>
> > Hi All,
> >
> > As Ambari back now, let’s start discussing roadmap. I will be closely
> > tracking for Jira, GitHub(with Jenkins) and wiki to work normal.
> > Based on the conclusion of this thread will update wiki[1].
> >
> > As 2.7.6 is released recently, we need to release 2.7.7
> >
> >
> > 1. Stack Management. There are two approaches for this. Open to discuss
> > on following..
> > * Entirely new stack
> >
> > i.
> > Define new stack name. (BDP: BigDataPlatform, BDS:BigDataStack,
> > OSS:OpenSourceStack,TDP: TuskerDataPlatform…)
> >
> > ii.
> > Define stack versions
> > (zk-3.5.9,Hadoop-3.2.3,hive-3.1.3,tez-0.10.1,spark-3.2..)
> >
> > iii. Based
> > above two we need to have select in bigtop code.
> >
> > * Using the existing for current release.
> >
> > i.
> > Plugin the new stack into HDP self
> >
> > ii.
> > Still, we can have hdp select
> >
> >
> >
> > I prefer the planA but planB takes less time. Users can easily adopt it
> > upgrade their stack.
> >
> >
> >
> > Note: Ideally bigtop mpack can be long term approach as stack id decoupled
> > now, but currently it’s not ready for latest stack (upgrades can be risk).
> > We’ll work in trunk for same. If this works shortly, we can choose.
> >
> >
> >
> > 1. Upgrade :
> > * Most of users will be deployed with HDP
> > versions(HDP-2.6.5,HDP-3.1.0), we might need to support the upgrade to 2.7.7
> > 2. Other work
> > * Any critical/blocker CVE’s Fixes
> > * CVEs Fixes : we’ll try to target some CVE fixes
> > * Review the open PR’s and Jira’s
> >
> >
> >
> > Trunk Code : Parallel we can work for the trunk code, this we might give
> > as 2.8.
> >
> > * Removal of python2 support
> > * Removal of hard dependency on HDP
> > * Working on bigtop mpack
> >
> > Can add more based on your inputs…
> >
> >
> > Any thoughts on this..? Please feel free to add anything which I missed.
> >
> >
> >
> > Reference:
> >
> > 1.
> > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D30755705&data=05%7C01%7Cbbattula%40visa.com%7Cd8a59968d77942985dab08da52651626%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637912891362493977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6leb0PfMOrmqscqUjV%2FUdc8nhJOEGogKUFrsg7XR7AU%3D&reserved=0
> >
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
Re: [Discuss] Initial Draft for Roadmap
Posted by Kengo Seki <se...@apache.org>.
Thank you so much for writing the draft and commenting on it, Brahma and Viraj!
Firstly, a minor comment:
> Trunk Code : Parallel we can work for the trunk code, this we might give as 2.8.
> * Working on bigtop mpack
This means adding a new built-in stack which uses the Bigtop
repository, and not developing Mpack on the Bigtop side, right?
If so, "bigtop mpack" sounds confusing a bit.
> hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk branch. Thoughts?
Sounds good to me. During the 2.7.x releases, users can use the HDP
stack by default. In addition, they can also add the Bigtop stack
through the Mpack, as Zhiguo mentioned in [1].
Since 2.8 or 3.0, we will drop HDP and users can use the built-in
Bigtop stack by default. It also allows us to drop the Mpack on the
Bigtop side.
Regarding the built-in Bigtop stack developed on trunk, the current
BIGTOP stack in Ambari [2] may be a good start point to be upgraded.
I ensured that Ambari 2.7.6 was successfully built with the
`-Dstack.distribution=BIGTOP` option, though Bigtop 0.8 is too old to
use for now.
[1]: https://lists.apache.org/thread/ljnyymp2g9fg7cq4z7sfxd36bh8x7nlm
[2]: https://github.com/apache/ambari/tree/release-2.7.6/ambari-server/src/main/resources/stacks/BIGTOP/0.8
Kengo Seki <se...@apache.org>
On Mon, Jun 20, 2022 at 9:26 AM Viraj Jasani <vj...@apache.org> wrote:
>
> Thanks for the nice draft, Brahma!
>
> IMHO, as we narrow down to 2.7.7 release, it might be still worth sticking
> to HDP i.e. plugin the new stack for HDP for 2.7 release line. For 2.8.0
> (or 3.0.0), we can add entire new stack (BDP sounds good one) and start off
> with 2.8.0/3.0.0 stack version.
>
> hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk
> branch. Thoughts?
>
> Similarly, we can start-off with "python 3 only" support in trunk, whereas
> 2.7 release line can continue with python 2 to reduce complexities for
> maintenance releases on 2.7.
>
> All other points look good. Once again, thank you for drafting the roadmap
> and starting with this thread.
>
>
> On Sun, Jun 19, 2022 at 10:07 AM Battula, Brahma Reddy
> <bb...@visa.com.invalid> wrote:
>
> > Hi All,
> >
> > As Ambari back now, let’s start discussing roadmap. I will be closely
> > tracking for Jira, GitHub(with Jenkins) and wiki to work normal.
> > Based on the conclusion of this thread will update wiki[1].
> >
> > As 2.7.6 is released recently, we need to release 2.7.7
> >
> >
> > 1. Stack Management. There are two approaches for this. Open to discuss
> > on following..
> > * Entirely new stack
> >
> > i.
> > Define new stack name. (BDP: BigDataPlatform, BDS:BigDataStack,
> > OSS:OpenSourceStack,TDP: TuskerDataPlatform…)
> >
> > ii.
> > Define stack versions
> > (zk-3.5.9,Hadoop-3.2.3,hive-3.1.3,tez-0.10.1,spark-3.2..)
> >
> > iii. Based
> > above two we need to have select in bigtop code.
> >
> > * Using the existing for current release.
> >
> > i.
> > Plugin the new stack into HDP self
> >
> > ii.
> > Still, we can have hdp select
> >
> >
> >
> > I prefer the planA but planB takes less time. Users can easily adopt it
> > upgrade their stack.
> >
> >
> >
> > Note: Ideally bigtop mpack can be long term approach as stack id decoupled
> > now, but currently it’s not ready for latest stack (upgrades can be risk).
> > We’ll work in trunk for same. If this works shortly, we can choose.
> >
> >
> >
> > 1. Upgrade :
> > * Most of users will be deployed with HDP
> > versions(HDP-2.6.5,HDP-3.1.0), we might need to support the upgrade to 2.7.7
> > 2. Other work
> > * Any critical/blocker CVE’s Fixes
> > * CVEs Fixes : we’ll try to target some CVE fixes
> > * Review the open PR’s and Jira’s
> >
> >
> >
> > Trunk Code : Parallel we can work for the trunk code, this we might give
> > as 2.8.
> >
> > * Removal of python2 support
> > * Removal of hard dependency on HDP
> > * Working on bigtop mpack
> >
> > Can add more based on your inputs…
> >
> >
> > Any thoughts on this..? Please feel free to add anything which I missed.
> >
> >
> >
> > Reference:
> >
> > 1.
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=30755705
> >
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
Re: [Discuss] Initial Draft for Roadmap
Posted by "Battula, Brahma Reddy" <bb...@visa.com.INVALID>.
Thanks viraj for pitching here!
>>hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk
>>branch. Thoughts?
Yes, we'll remove this in trunk..
>> Similarly, we can start-off with "python 3 only" support in trunk, whereas
>>2.7 release line can continue with python 2 to reduce complexities for
>> maintenance releases on 2.7.
Yes, sounds like a plan. We can still add some more trunk.
On 20/06/22, 5:56 AM, "Viraj Jasani" <vj...@apache.org> wrote:
Thanks for the nice draft, Brahma!
IMHO, as we narrow down to 2.7.7 release, it might be still worth sticking
to HDP i.e. plugin the new stack for HDP for 2.7 release line. For 2.8.0
(or 3.0.0), we can add entire new stack (BDP sounds good one) and start off
with 2.8.0/3.0.0 stack version.
hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk
branch. Thoughts?
Similarly, we can start-off with "python 3 only" support in trunk, whereas
2.7 release line can continue with python 2 to reduce complexities for
maintenance releases on 2.7.
All other points look good. Once again, thank you for drafting the roadmap
and starting with this thread.
On Sun, Jun 19, 2022 at 10:07 AM Battula, Brahma Reddy
<bb...@visa.com.invalid> wrote:
> Hi All,
>
> As Ambari back now, let’s start discussing roadmap. I will be closely
> tracking for Jira, GitHub(with Jenkins) and wiki to work normal.
> Based on the conclusion of this thread will update wiki[1].
>
> As 2.7.6 is released recently, we need to release 2.7.7
>
>
> 1. Stack Management. There are two approaches for this. Open to discuss
> on following..
> * Entirely new stack
>
> i.
> Define new stack name. (BDP: BigDataPlatform, BDS:BigDataStack,
> OSS:OpenSourceStack,TDP: TuskerDataPlatform…)
>
> ii.
> Define stack versions
> (zk-3.5.9,Hadoop-3.2.3,hive-3.1.3,tez-0.10.1,spark-3.2..)
>
> iii. Based
> above two we need to have select in bigtop code.
>
> * Using the existing for current release.
>
> i.
> Plugin the new stack into HDP self
>
> ii.
> Still, we can have hdp select
>
>
>
> I prefer the planA but planB takes less time. Users can easily adopt it
> upgrade their stack.
>
>
>
> Note: Ideally bigtop mpack can be long term approach as stack id decoupled
> now, but currently it’s not ready for latest stack (upgrades can be risk).
> We’ll work in trunk for same. If this works shortly, we can choose.
>
>
>
> 1. Upgrade :
> * Most of users will be deployed with HDP
> versions(HDP-2.6.5,HDP-3.1.0), we might need to support the upgrade to 2.7.7
> 2. Other work
> * Any critical/blocker CVE’s Fixes
> * CVEs Fixes : we’ll try to target some CVE fixes
> * Review the open PR’s and Jira’s
>
>
>
> Trunk Code : Parallel we can work for the trunk code, this we might give
> as 2.8.
>
> * Removal of python2 support
> * Removal of hard dependency on HDP
> * Working on bigtop mpack
>
> Can add more based on your inputs…
>
>
> Any thoughts on this..? Please feel free to add anything which I missed.
>
>
>
> Reference:
>
> 1.
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D30755705&data=05%7C01%7Cbbattula%40visa.com%7C85aa186aef084ddb868b08da52537aa0%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637912815739777112%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=POv14glYghx1tjtBVdC6zQGovsU9OKRzXRrgygCRS0g%3D&reserved=0
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
Re: [Discuss] Initial Draft for Roadmap
Posted by Viraj Jasani <vj...@apache.org>.
Thanks for the nice draft, Brahma!
IMHO, as we narrow down to 2.7.7 release, it might be still worth sticking
to HDP i.e. plugin the new stack for HDP for 2.7 release line. For 2.8.0
(or 3.0.0), we can add entire new stack (BDP sounds good one) and start off
with 2.8.0/3.0.0 stack version.
hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk
branch. Thoughts?
Similarly, we can start-off with "python 3 only" support in trunk, whereas
2.7 release line can continue with python 2 to reduce complexities for
maintenance releases on 2.7.
All other points look good. Once again, thank you for drafting the roadmap
and starting with this thread.
On Sun, Jun 19, 2022 at 10:07 AM Battula, Brahma Reddy
<bb...@visa.com.invalid> wrote:
> Hi All,
>
> As Ambari back now, let’s start discussing roadmap. I will be closely
> tracking for Jira, GitHub(with Jenkins) and wiki to work normal.
> Based on the conclusion of this thread will update wiki[1].
>
> As 2.7.6 is released recently, we need to release 2.7.7
>
>
> 1. Stack Management. There are two approaches for this. Open to discuss
> on following..
> * Entirely new stack
>
> i.
> Define new stack name. (BDP: BigDataPlatform, BDS:BigDataStack,
> OSS:OpenSourceStack,TDP: TuskerDataPlatform…)
>
> ii.
> Define stack versions
> (zk-3.5.9,Hadoop-3.2.3,hive-3.1.3,tez-0.10.1,spark-3.2..)
>
> iii. Based
> above two we need to have select in bigtop code.
>
> * Using the existing for current release.
>
> i.
> Plugin the new stack into HDP self
>
> ii.
> Still, we can have hdp select
>
>
>
> I prefer the planA but planB takes less time. Users can easily adopt it
> upgrade their stack.
>
>
>
> Note: Ideally bigtop mpack can be long term approach as stack id decoupled
> now, but currently it’s not ready for latest stack (upgrades can be risk).
> We’ll work in trunk for same. If this works shortly, we can choose.
>
>
>
> 1. Upgrade :
> * Most of users will be deployed with HDP
> versions(HDP-2.6.5,HDP-3.1.0), we might need to support the upgrade to 2.7.7
> 2. Other work
> * Any critical/blocker CVE’s Fixes
> * CVEs Fixes : we’ll try to target some CVE fixes
> * Review the open PR’s and Jira’s
>
>
>
> Trunk Code : Parallel we can work for the trunk code, this we might give
> as 2.8.
>
> * Removal of python2 support
> * Removal of hard dependency on HDP
> * Working on bigtop mpack
>
> Can add more based on your inputs…
>
>
> Any thoughts on this..? Please feel free to add anything which I missed.
>
>
>
> Reference:
>
> 1.
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=30755705
>
>
>