You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@incubator.apache.org by larry mccay <la...@gmail.com> on 2022/10/14 18:38:25 UTC
Proposal to Revive Apache Livy Community
Abstract
Livy is a web service that exposes a REST interface for managing long
running Apache Spark contexts in your cluster. With Livy, new applications
can be built on top of Apache Spark that require fine grained interaction
with many Spark contexts [1].
While this project has been well regarded and used in many contexts as the
defacto standard API to Spark environments, it has been incubating for over
5 years without graduation to a TLP and it has become difficult to
impossible for fixes and improvements to be contributed as the current
community seems to have moved on.
There has been discussion regarding retirement of this podling where there
seems to be some increasing interest in joining and reviving the community
[2].
The intent of this proposal is to avoid retiring a well regarded, actively
used and rather mature project by reviving the PPMC and community with new
folks that have a vested interest in the project and health of the
community.
Proposal
We propose to revive the PPMC with a set of contributors and maintainers as
mentors, PPMC members and committers.
The retirement DISCUSS thread [2] has shown a growing interest in providing
new committers and bringing improvements and fixes from organization’s
internally maintained forks back to a revived community.
General Approach to Revival:
-
Add new Mentors
-
Larry McCay, lmccay@apache.org , Cloudera
-
Sunil Govindan, sunilg@apache.org, Cloudera
-
Imran Rashid - irashid@apache.org, Cloudera
-
Add new Committers/PPMC
-
Larry McCay, lmccay@apache.org, Cloudera
-
Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
-
Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
-
Wing Yew Poon, wypoon@cloudera.com, Cloudera
-
Xilang Yan, xilang.yan@gmail.com, Shopee
-
Jianzhen Wu, myjianzhen@gmail.com, Shopee
-
Nagella Jagadeewara Rao, jnagella@visa.com, Visa
-
Pralab Kumar, pralkuma@visa.com, Visa
-
Prasad Shrikant, shrikant.ism@gmail.com, Visa
-
Brahma Reddy Battula, brahma@apache.org, Visa
-
Invite existing PPMC members to opt-in or otherwise go emeritus
-
Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in via
Retirement DISCUSS thread [2])
-
Invite existing Committers/PPMC members to opt-in or otherwise go
emeritus
-
Establish Roadmap via follow up DISCUSS thread
-
Known Improvements from Forks which will need proposals and
discussion:
-
Adding HA for Livy
-
Updating security capabilities (eg. kerberos for jdbc, fixing bugs
in encryption)
-
Expanding the support for kubernetes
-
Responding to CVEs in dependencies (eg. log4j, thrift)
-
Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
-
Support multi Spark versions
-
Implemented a metrics system for Livy
-
Support customize batch/interactive session lifecycle event
handler, default log event with log4j, very helpful for
trouble shooting
-
Optimize log to track which session id the log message came from,
also very helpful for trouble shooting
-
Support customize Spark config optimization rules, can be used to
optimize config for users’ job
-
A set of command line tool which can be used to replace Spark’s
spark-submit, pyspark, spark-sql but actually submit
application in Livy
-
We are planning to implement a JDBC state store, and allow multi
Livy Thrift sessions to share one backend Spark application
in the next few
months.
-
These items and others that are brought to community may need
consolidation or multiple configurable options and will need to
be part of
the discussion
-
One-pager Livy Improvement Proposals (LIP) may make sense to drive
these discussions and convergence
-
Feature Branch Strategy for large changes
-
Large features are hard to review we will need to define a
strategy
-
Determine the Improvements to be delivered across first 3 Releases
with Target Release Dates
-
Ensure CVE and Dependency management hygiene is in place
The above approach will usher the community back to an active status with a
Roadmap of 3 or more release plans and security hygiene in place.
Development Practices
The Livy project follows a review before commit philosophy. Every commit
automatically runs through the unit tests and generates coverage reports
presented as a pull request comment. Our experience with this process leads
us to believe that it helps ease new contributors into the project. They
get feedback quickly on common mistakes, lowering the burden on reviewers.
Those same reviewers get to lead by example, showing the new contributors
that we value feedback within our community even when changes are done by
more experienced folks. Taken from the original Apache Livy Proposal [1],
this should continue to be true. As mentioned, Livy is a mature project and
as such RTC is the most appropriate for continued quality and awareness.
1.
Original Apache Livy Proposal
https://cwiki.apache.org/confluence/display/incubator/LivyProposal
2.
Retirement DISCUSS thread
https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
Re: Proposal to Revive Apache Livy Community
Posted by Justin Mclean <ju...@classsoftware.com>.
Hi,
Madhawa is an IPMC member.
Kind Regards,
Justin
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [EXTERNAL] Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
Hi Renat -
More active contributors/committers would be great!
Let's get the names before I craft the NOTICE to add new members email
later today.
If that isn't realistic then we can always add more after the fact as well.
That's what is great about re/building the community in the incubator!
thanks!
--larry
On Mon, Oct 24, 2022 at 10:30 PM Renat Bekbolatov
<re...@microsoft.com.invalid> wrote:
> I welcome this development - honestly, possibly long overdue.
> Would it be possible to propose additional committers for a wider
> representation?
>
>
> Thank you,
> Renat
>
> -----Original Message-----
> From: larry mccay <lm...@apache.org>
> Sent: Monday, October 24, 2022 4:41 PM
> To: dev@livy.apache.org
> Subject: [EXTERNAL] Re: Proposal to Revive Apache Livy Community
>
> Again, I have no objections to this and will be happy to include you.
> Thanks for calling out!
>
> On Mon, Oct 24, 2022 at 5:38 PM Marco Gaido <mg...@apache.org> wrote:
>
> > Hi all,
> >
> > I am also busy with different projects at the moment, as such I am
> > more than happy if ai can help somehow, e.g. reviewing code in parts
> > that I know, but I cannot allocate much bandwidth to it.
> >
> > Thanks,
> > Marco
> >
> > Il lun 24 ott 2022, 22:07 larry mccay <lm...@apache.org> ha scritto:
> >
> > > Personally, I would welcome it.
> > > Any notes of guidance and support as we ramp up the team and make
> > > the
> > first
> > > few releases would be invaluable.
> > >
> > >
> > > On Mon, Oct 24, 2022 at 3:49 PM Alex Bozarth <aj...@us.ibm.com>
> > wrote:
> > >
> > > > I would love to stay on as a PPMC, but if my lack of availability
> > > concerns
> > > > the new PPMC then I'm willing to be emeritus.
> > > >
> > > >
> > > > Alex Bozarth
> > > > Jupyter Architect, IBM CODAIT
> > > > GitHub: ajbozarth
> > > >
> > > > On 10/24/22, 2:46 PM, "larry mccay" <lm...@apache.org> wrote:
> > > >
> > > > Oh, very good, Alex!
> > > > Be sure to let us know if you want to remain on the project in
> > > > any capacity.
> > > >
> > > >
> > > > On Mon, Oct 24, 2022 at 3:18 PM Alex Bozarth
> > > > <aj...@us.ibm.com>
> > > > wrote:
> > > >
> > > > > As the only Livy PPMC still responding to the Mentors on the
> > > private
> > > > list,
> > > > > I have updated the Livy Podling report for November with the
> > status
> > > > of the
> > > > > project and a request to the IPMC to review this proposal
> > > > since there are
> > > > > not enough active Livy PPMC members to reach a quorum to
> > > > pass the proposal.
> > > > >
> > > > > As a current Livy PPMC I strongly support this proposal for
> > > > revitalization
> > > > > as I do not have enough bandwidth to dedicate to Livy.
> > > > >
> > > > >
> > > > > Alex Bozarth
> > > > > Jupyter Architect, IBM CODAIT
> > > > > GitHub: ajbozarth
> > > > >
> > > > > On 10/24/22, 1:41 PM, "larry mccay" <lm...@apache.org> wrote:
> > > > >
> > > > > Gentle reminder that we need to determine next steps here.
> > > > > We have an updated proposal on this thread.
> > > > > Do we need a VOTE or can we move forward directly to
> > adjusting
> > > > the
> > > > > members,
> > > > > etc?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > --larry
> > > > >
> > > > > On Thu, Oct 20, 2022 at 3:26 PM larry mccay <
> > lmccay@apache.org
> > > >
> > > > wrote:
> > > > >
> > > > > > @Justin Mclean <ju...@me.com> - any insights on
> > next
> > > > steps
> > > > > here?
> > > > > >
> > > > > >
> > > > > > On Tue, Oct 18, 2022 at 5:44 PM larry mccay <
> > > lmccay@apache.org
> > > > >
> > > > > wrote:
> > > > > >
> > > > > >> Very good, here is the latest revision with updated
> > Mentors.
> > > > > >> Sunil and I have been added to the IPMC as well.
> > > > > >> Welcome Madhawa and thanks for stepping up as a Mentor
> for
> > > > Livy!
> > > > > >>
> > > > > >> Abstract
> > > > > >>
> > > > > >> Livy is a web service that exposes a REST interface for
> > > > managing
> > > > > long
> > > > > >> running Apache Spark contexts in your cluster. With
> Livy,
> > > new
> > > > > applications
> > > > > >> can be built on top of Apache Spark that require fine
> > > grained
> > > > > interaction
> > > > > >> with many Spark contexts [1].
> > > > > >>
> > > > > >> While this project has been well regarded and used in
> many
> > > > contexts
> > > > > as
> > > > > >> the defacto standard API to Spark environments, it has
> > been
> > > > > incubating for
> > > > > >> over 5 years without graduation to a TLP and it has
> become
> > > > > difficult to
> > > > > >> impossible for fixes and improvements to be contributed
> as
> > > the
> > > > > current
> > > > > >> community seems to have moved on.
> > > > > >>
> > > > > >> There has been discussion regarding retirement of this
> > > > podling where
> > > > > >> there seems to be some increasing interest in joining
> and
> > > > reviving
> > > > > the
> > > > > >> community [2].
> > > > > >>
> > > > > >> The intent of this proposal is to avoid retiring a well
> > > > regarded,
> > > > > >> actively used and rather mature project by reviving the
> > PPMC
> > > > and
> > > > > community
> > > > > >> with new folks that have a vested interest in the
> project
> > > and
> > > > > health of the
> > > > > >> community.
> > > > > >> Proposal
> > > > > >>
> > > > > >> We propose to revive the PPMC with a set of contributors
> > and
> > > > > maintainers
> > > > > >> as mentors, PPMC members and committers.
> > > > > >>
> > > > > >> The retirement DISCUSS thread [2] has shown a growing
> > > > interest in
> > > > > >> providing new committers and bringing improvements and
> > fixes
> > > > from
> > > > > >> organization’s internally maintained forks back to a
> > revived
> > > > > community.
> > > > > >>
> > > > > >> General Approach to Revival:
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> Add new Mentors
> > > > > >> -
> > > > > >>
> > > > > >> Larry McCay, lmccay@apache.org , Cloudera
> > > > > >> -
> > > > > >>
> > > > > >> Sunil Govindan, sunilg@apache.org, Cloudera
> > > > > >> -
> > > > > >>
> > > > > >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
> > > > > >> -
> > > > > >>
> > > > > >> Madhawa Gunasekara, madhawa30@gmail.com,
> > Independent
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> Add new Committers/PPMC
> > > > > >> -
> > > > > >>
> > > > > >> Larry McCay, lmccay@apache.org, Cloudera
> > > > > >> -
> > > > > >>
> > > > > >> Vinod Kumar Vavilapalli, vinodkv@cloudera.com,
> > > Cloudera
> > > > > >> -
> > > > > >>
> > > > > >> Imran Rashid - irashid@apache.org, Cloudera
> > > > > >> -
> > > > > >>
> > > > > >> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
> > > > > >> -
> > > > > >>
> > > > > >> Wing Yew Poon, wypoon@cloudera.com, Cloudera
> > > > > >> -
> > > > > >>
> > > > > >> Xilang Yan, xilang.yan@gmail.com, Shopee
> > > > > >> -
> > > > > >>
> > > > > >> Jianzhen Wu, myjianzhen@gmail.com, Shopee
> > > > > >> -
> > > > > >>
> > > > > >> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
> > > > > >> -
> > > > > >>
> > > > > >> Pralab Kumar, pralkuma@visa.com, Visa
> > > > > >> -
> > > > > >>
> > > > > >> Prasad Shrikant, shrikant.ism@gmail.com, Visa
> > > > > >> -
> > > > > >>
> > > > > >> Brahma Reddy Battula, brahma@apache.org, Visa
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> Invite existing PPMC members to opt-in or otherwise
> go
> > > > emeritus
> > > > > >> -
> > > > > >>
> > > > > >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
> > > > (opted-in
> > > > > via
> > > > > >> Retirement DISCUSS thread [2])
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> Invite existing Committers to opt-out or otherwise
> > > continue
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> Establish Roadmap via follow up DISCUSS thread
> > > > > >> -
> > > > > >>
> > > > > >> Known Improvements from Forks which will need
> > > proposals
> > > > and
> > > > > >> discussion:
> > > > > >> -
> > > > > >>
> > > > > >> Adding HA for Livy
> > > > > >> -
> > > > > >>
> > > > > >> Updating security capabilities (eg. kerberos
> for
> > > > jdbc,
> > > > > fixing
> > > > > >> bugs in encryption)
> > > > > >> -
> > > > > >>
> > > > > >> Expanding the support for kubernetes
> > > > > >> -
> > > > > >>
> > > > > >> Responding to CVEs in dependencies (eg. log4j,
> > > > thrift)
> > > > > >> -
> > > > > >>
> > > > > >> Livy rest cluster - IS THIS SAME AS HA for Livy
> > > > ABOVE?
> > > > > >> -
> > > > > >>
> > > > > >> Support multi Spark versions
> > > > > >> -
> > > > > >>
> > > > > >> Implemented a metrics system for Livy
> > > > > >> -
> > > > > >>
> > > > > >> Support customize batch/interactive session
> > > > lifecycle event
> > > > > >> handler, default log event with log4j, very
> > helpful
> > > > for
> > > > > troubleshooting
> > > > > >> -
> > > > > >>
> > > > > >> Optimize log to track which session id the log
> > > > message came
> > > > > >> from, also very helpful for troubleshooting
> > > > > >> -
> > > > > >>
> > > > > >> Support customize Spark config optimization
> > rules,
> > > > can be
> > > > > used
> > > > > >> to optimize config for users’ job
> > > > > >> -
> > > > > >>
> > > > > >> A set of command line tool which can be used to
> > > > replace
> > > > > Spark’s
> > > > > >> spark-submit, pyspark, spark-sql but actually
> > > submit
> > > > > application in Livy
> > > > > >> -
> > > > > >>
> > > > > >> We are planning to implement a JDBC state
> store,
> > > and
> > > > allow
> > > > > >> multi Livy Thrift sessions to share one backend
> > > Spark
> > > > > application in the
> > > > > >> next few months.
> > > > > >> -
> > > > > >>
> > > > > >> These items and others that are brought to
> community
> > > > may need
> > > > > >> consolidation or multiple configurable options and
> > > will
> > > > need
> > > > > to be part of
> > > > > >> the discussion
> > > > > >> -
> > > > > >>
> > > > > >> One-pager Livy Improvement Proposals (LIP) may
> > make
> > > > sense
> > > > > to
> > > > > >> drive these discussions and convergence
> > > > > >> -
> > > > > >>
> > > > > >> Feature Branch Strategy for large changes
> > > > > >> -
> > > > > >>
> > > > > >> Large features are hard to review we will
> need
> > > to
> > > > > define a
> > > > > >> strategy
> > > > > >> -
> > > > > >>
> > > > > >> Determine the Improvements to be delivered across
> > > first
> > > > 3
> > > > > Releases
> > > > > >> with Target Release Dates
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> Ensure CVE and Dependency management hygiene is in
> > place
> > > > > >>
> > > > > >>
> > > > > >> The above approach will usher the community back to an
> > > active
> > > > > status with
> > > > > >> a Roadmap of 3 or more release plans and security
> hygiene
> > in
> > > > place.
> > > > > >> Development Practices
> > > > > >>
> > > > > >> The Livy project follows a review before commit
> > philosophy.
> > > > Every
> > > > > commit
> > > > > >> automatically runs through the unit tests and generates
> > > > coverage
> > > > > reports
> > > > > >> presented as a pull request comment. Our experience with
> > > this
> > > > > process leads
> > > > > >> us to believe that it helps ease new contributors into
> the
> > > > project.
> > > > > They
> > > > > >> get feedback quickly on common mistakes, lowering the
> > burden
> > > > on
> > > > > reviewers.
> > > > > >> Those same reviewers get to lead by example, showing the
> > new
> > > > > contributors
> > > > > >> that we value feedback within our community even when
> > > changes
> > > > are
> > > > > done by
> > > > > >> more experienced folks. Taken from the original Apache
> > Livy
> > > > > Proposal [1],
> > > > > >> this should continue to be true. As mentioned, Livy is a
> > > > mature
> > > > > project and
> > > > > >> as such RTC is the most appropriate for continued
> quality
> > > and
> > > > > awareness.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> 1.
> > > > > >>
> > > > > >> Original Apache Livy Proposal
> > > > > >>
> > > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik
> > i.apache.org%2Fconfluence%2Fdisplay%2Fincubator%2FLivyProposal&dat
> > a=05%7C01%7Crebekbol%40microsoft.com%7C5ead1754350a408bceba08dab6192ee
> > 9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638022516535784379%7CUn
> > known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haW
> > wiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rbccZA%2BUAUHjpEolIiY8ig%2B
> > 5nRw5abmw9oiwJGkz8TM%3D&reserved=0
> > > > > >> 2.
> > > > > >>
> > > > > >> Retirement DISCUSS thread
> > > > > >>
> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fgcstsrhbp91c5mm55htqn1l3djv8m7o0&data=05%7C01%7Crebekbol%40microsoft.com%7C5ead1754350a408bceba08dab6192ee9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638022516535940162%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dgaMvjevmL84UXIvyHZ3OgCDX%2FXs7ZdKZvNXR1UR5A8%3D&reserved=0
> > > > >
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <
> > > > > madhawa30@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hi Larry,
> > > > > >>>
> > > > > >>> I'm an IPMC Member. madhawa30 at gmail dot com is my
> > > > preferred
> > > > > email
> > > > > >>> address.
> > > > > >>> apache id: madhawa
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>> Madhawa
> > > > > >>>
> > > > > >>>
> > > > > >>> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <
> > > > lmccay@apache.org>
> > > > > wrote:
> > > > > >>>
> > > > > >>> > Hi Madhawa -
> > > > > >>> >
> > > > > >>> > That's awesome!
> > > > > >>> > Are you already a member of IPMC?
> > > > > >>> > If not, are you an ASF member?
> > > > > >>> > If you are an ASF member you can request that you be
> > > added
> > > > as an
> > > > > IPMC
> > > > > >>> > member.
> > > > > >>> >
> > > > > >>> > Can you provide your company affiliation for the
> > proposal
> > > > and
> > > > > preferred
> > > > > >>> > email?
> > > > > >>> >
> > > > > >>> > thanks!
> > > > > >>> >
> > > > > >>> > --larry
> > > > > >>> >
> > > > > >>> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <
> > > > > >>> madhawa30@gmail.com>
> > > > > >>> > wrote:
> > > > > >>> >
> > > > > >>> > > Hi Larry,
> > > > > >>> > >
> > > > > >>> > > I'm interested in working with Livy and would like
> to
> > > > join as a
> > > > > >>> Mentor.
> > > > > >>> > >
> > > > > >>> > > Thanks,
> > > > > >>> > > Madhawa
> > > > > >>> > >
> > > > > >>> > >
> > > > > >>> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <
> > > > lmccay@apache.org
> > > > > >
> > > > > >>> wrote:
> > > > > >>> > >
> > > > > >>> > > > Sorry, I missed commenting on this:
> > > > > >>> > > >
> > > > > >>> > > > "There is also no concept as an emeritus PPMC
> > member
> > > > at the
> > > > > ASF."
> > > > > >>> > > >
> > > > > >>> > > > I assume that we can remove PPMC members that do
> > not
> > > > opt-in
> > > > > >>> explicitly
> > > > > >>> > at
> > > > > >>> > > > this point.
> > > > > >>> > > > They will have every opportunity to rejoin.
> > > > > >>> > > >
> > > > > >>> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <
> > > > > lmccay@apache.org>
> > > > > >>> > wrote:
> > > > > >>> > > >
> > > > > >>> > > > > I will ask in a separate thread, @Justin
> Mclean <
> > > > > >>> justinmclean@me.com
> > > > > >>> > >
> > > > > >>> > > -
> > > > > >>> > > > > thanks.
> > > > > >>> > > > > Adding JB adds another company and we are
> > certainly
> > > > open to
> > > > > >>> anyone
> > > > > >>> > else
> > > > > >>> > > > > that would like to join as a mentor.
> > > > > >>> > > > > At the end of the day, the mentors are for
> > > > instilling the
> > > > > Apache
> > > > > >>> Way
> > > > > >>> > > > > knowledge and steering toward graduation.
> > > > > >>> > > > > I feel that this diversity, while nice to have,
> > is
> > > > less
> > > > > important
> > > > > >>> > than
> > > > > >>> > > > > that of the PPMC and committers for the long
> term
> > > > health
> > > > > of the
> > > > > >>> > > > community.
> > > > > >>> > > > >
> > > > > >>> > > > > We need to push this podling to graduation as
> > > > quickly as
> > > > > possible
> > > > > >>> > since
> > > > > >>> > > > it
> > > > > >>> > > > > is rather mature and needs to get to the next
> > > level.
> > > > > >>> > > > >
> > > > > >>> > > > > Again, any potential Mentors that would like to
> > > join
> > > > are
> > > > > more
> > > > > >>> than
> > > > > >>> > > > welcome.
> > > > > >>> > > > >
> > > > > >>> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean
> <
> > > > > >>> > > justin@classsoftware.com
> > > > > >>> > > > >
> > > > > >>> > > > > wrote:
> > > > > >>> > > > >
> > > > > >>> > > > >> Hi,
> > > > > >>> > > > >>
> > > > > >>> > > > >> I’m sorry, but Imran Rashid can’t be a mentor
> > for
> > > > the
> > > > > project as
> > > > > >>> > they
> > > > > >>> > > > are
> > > > > >>> > > > >> not an IPMC member. Currently, both Sunil and
> > > Larry
> > > > (as
> > > > > they
> > > > > >>> are ASF
> > > > > >>> > > > >> members) need to ask to join the IPMC and
> NOTICE
> > > > sent to
> > > > > the ASF
> > > > > >>> > > board.
> > > > > >>> > > > I
> > > > > >>> > > > >> would also prefer that mentors come from
> > different
> > > > > companies.
> > > > > >>> > > > >>
> > > > > >>> > > > >> There is also no concept as an emeritus PPMC
> > > member
> > > > at
> > > > > the ASF.
> > > > > >>> > > > >>
> > > > > >>> > > > >> Kind Regards,
> > > > > >>> > > > >> Justin
> > > > > >>> > > > >>
> > > > > >>> > > > >>
> > > > > >>> > > > >>
> > > > > >>> >
> > > > >
> > > --------------------------------------------------------------------
> > > -
> > > > > >>> > > > >> To unsubscribe, e-mail:
> > > > > >>> general-unsubscribe@incubator.apache.org
> > > > > >>> > > > >> For additional commands, e-mail:
> > > > > >>> general-help@incubator.apache.org
> > > > > >>> > > > >>
> > > > > >>> > > > >>
> > > > > >>> > > >
> > > > > >>> > >
> > > > > >>> >
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > > >
> > > --------------------------------------------------------------------
> > > -
> > > > > To unsubscribe, e-mail:
> general-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail:
> > general-help@incubator.apache.org
> > > > >
> > > >
> > > >
> > >
> >
>
Re: [EXTERNAL] Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
Hi Renat -
More active contributors/committers would be great!
Let's get the names before I craft the NOTICE to add new members email
later today.
If that isn't realistic then we can always add more after the fact as well.
That's what is great about re/building the community in the incubator!
thanks!
--larry
On Mon, Oct 24, 2022 at 10:30 PM Renat Bekbolatov
<re...@microsoft.com.invalid> wrote:
> I welcome this development - honestly, possibly long overdue.
> Would it be possible to propose additional committers for a wider
> representation?
>
>
> Thank you,
> Renat
>
> -----Original Message-----
> From: larry mccay <lm...@apache.org>
> Sent: Monday, October 24, 2022 4:41 PM
> To: dev@livy.apache.org
> Subject: [EXTERNAL] Re: Proposal to Revive Apache Livy Community
>
> Again, I have no objections to this and will be happy to include you.
> Thanks for calling out!
>
> On Mon, Oct 24, 2022 at 5:38 PM Marco Gaido <mg...@apache.org> wrote:
>
> > Hi all,
> >
> > I am also busy with different projects at the moment, as such I am
> > more than happy if ai can help somehow, e.g. reviewing code in parts
> > that I know, but I cannot allocate much bandwidth to it.
> >
> > Thanks,
> > Marco
> >
> > Il lun 24 ott 2022, 22:07 larry mccay <lm...@apache.org> ha scritto:
> >
> > > Personally, I would welcome it.
> > > Any notes of guidance and support as we ramp up the team and make
> > > the
> > first
> > > few releases would be invaluable.
> > >
> > >
> > > On Mon, Oct 24, 2022 at 3:49 PM Alex Bozarth <aj...@us.ibm.com>
> > wrote:
> > >
> > > > I would love to stay on as a PPMC, but if my lack of availability
> > > concerns
> > > > the new PPMC then I'm willing to be emeritus.
> > > >
> > > >
> > > > Alex Bozarth
> > > > Jupyter Architect, IBM CODAIT
> > > > GitHub: ajbozarth
> > > >
> > > > On 10/24/22, 2:46 PM, "larry mccay" <lm...@apache.org> wrote:
> > > >
> > > > Oh, very good, Alex!
> > > > Be sure to let us know if you want to remain on the project in
> > > > any capacity.
> > > >
> > > >
> > > > On Mon, Oct 24, 2022 at 3:18 PM Alex Bozarth
> > > > <aj...@us.ibm.com>
> > > > wrote:
> > > >
> > > > > As the only Livy PPMC still responding to the Mentors on the
> > > private
> > > > list,
> > > > > I have updated the Livy Podling report for November with the
> > status
> > > > of the
> > > > > project and a request to the IPMC to review this proposal
> > > > since there are
> > > > > not enough active Livy PPMC members to reach a quorum to
> > > > pass the proposal.
> > > > >
> > > > > As a current Livy PPMC I strongly support this proposal for
> > > > revitalization
> > > > > as I do not have enough bandwidth to dedicate to Livy.
> > > > >
> > > > >
> > > > > Alex Bozarth
> > > > > Jupyter Architect, IBM CODAIT
> > > > > GitHub: ajbozarth
> > > > >
> > > > > On 10/24/22, 1:41 PM, "larry mccay" <lm...@apache.org> wrote:
> > > > >
> > > > > Gentle reminder that we need to determine next steps here.
> > > > > We have an updated proposal on this thread.
> > > > > Do we need a VOTE or can we move forward directly to
> > adjusting
> > > > the
> > > > > members,
> > > > > etc?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > --larry
> > > > >
> > > > > On Thu, Oct 20, 2022 at 3:26 PM larry mccay <
> > lmccay@apache.org
> > > >
> > > > wrote:
> > > > >
> > > > > > @Justin Mclean <ju...@me.com> - any insights on
> > next
> > > > steps
> > > > > here?
> > > > > >
> > > > > >
> > > > > > On Tue, Oct 18, 2022 at 5:44 PM larry mccay <
> > > lmccay@apache.org
> > > > >
> > > > > wrote:
> > > > > >
> > > > > >> Very good, here is the latest revision with updated
> > Mentors.
> > > > > >> Sunil and I have been added to the IPMC as well.
> > > > > >> Welcome Madhawa and thanks for stepping up as a Mentor
> for
> > > > Livy!
> > > > > >>
> > > > > >> Abstract
> > > > > >>
> > > > > >> Livy is a web service that exposes a REST interface for
> > > > managing
> > > > > long
> > > > > >> running Apache Spark contexts in your cluster. With
> Livy,
> > > new
> > > > > applications
> > > > > >> can be built on top of Apache Spark that require fine
> > > grained
> > > > > interaction
> > > > > >> with many Spark contexts [1].
> > > > > >>
> > > > > >> While this project has been well regarded and used in
> many
> > > > contexts
> > > > > as
> > > > > >> the defacto standard API to Spark environments, it has
> > been
> > > > > incubating for
> > > > > >> over 5 years without graduation to a TLP and it has
> become
> > > > > difficult to
> > > > > >> impossible for fixes and improvements to be contributed
> as
> > > the
> > > > > current
> > > > > >> community seems to have moved on.
> > > > > >>
> > > > > >> There has been discussion regarding retirement of this
> > > > podling where
> > > > > >> there seems to be some increasing interest in joining
> and
> > > > reviving
> > > > > the
> > > > > >> community [2].
> > > > > >>
> > > > > >> The intent of this proposal is to avoid retiring a well
> > > > regarded,
> > > > > >> actively used and rather mature project by reviving the
> > PPMC
> > > > and
> > > > > community
> > > > > >> with new folks that have a vested interest in the
> project
> > > and
> > > > > health of the
> > > > > >> community.
> > > > > >> Proposal
> > > > > >>
> > > > > >> We propose to revive the PPMC with a set of contributors
> > and
> > > > > maintainers
> > > > > >> as mentors, PPMC members and committers.
> > > > > >>
> > > > > >> The retirement DISCUSS thread [2] has shown a growing
> > > > interest in
> > > > > >> providing new committers and bringing improvements and
> > fixes
> > > > from
> > > > > >> organization’s internally maintained forks back to a
> > revived
> > > > > community.
> > > > > >>
> > > > > >> General Approach to Revival:
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> Add new Mentors
> > > > > >> -
> > > > > >>
> > > > > >> Larry McCay, lmccay@apache.org , Cloudera
> > > > > >> -
> > > > > >>
> > > > > >> Sunil Govindan, sunilg@apache.org, Cloudera
> > > > > >> -
> > > > > >>
> > > > > >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
> > > > > >> -
> > > > > >>
> > > > > >> Madhawa Gunasekara, madhawa30@gmail.com,
> > Independent
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> Add new Committers/PPMC
> > > > > >> -
> > > > > >>
> > > > > >> Larry McCay, lmccay@apache.org, Cloudera
> > > > > >> -
> > > > > >>
> > > > > >> Vinod Kumar Vavilapalli, vinodkv@cloudera.com,
> > > Cloudera
> > > > > >> -
> > > > > >>
> > > > > >> Imran Rashid - irashid@apache.org, Cloudera
> > > > > >> -
> > > > > >>
> > > > > >> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
> > > > > >> -
> > > > > >>
> > > > > >> Wing Yew Poon, wypoon@cloudera.com, Cloudera
> > > > > >> -
> > > > > >>
> > > > > >> Xilang Yan, xilang.yan@gmail.com, Shopee
> > > > > >> -
> > > > > >>
> > > > > >> Jianzhen Wu, myjianzhen@gmail.com, Shopee
> > > > > >> -
> > > > > >>
> > > > > >> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
> > > > > >> -
> > > > > >>
> > > > > >> Pralab Kumar, pralkuma@visa.com, Visa
> > > > > >> -
> > > > > >>
> > > > > >> Prasad Shrikant, shrikant.ism@gmail.com, Visa
> > > > > >> -
> > > > > >>
> > > > > >> Brahma Reddy Battula, brahma@apache.org, Visa
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> Invite existing PPMC members to opt-in or otherwise
> go
> > > > emeritus
> > > > > >> -
> > > > > >>
> > > > > >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
> > > > (opted-in
> > > > > via
> > > > > >> Retirement DISCUSS thread [2])
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> Invite existing Committers to opt-out or otherwise
> > > continue
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> Establish Roadmap via follow up DISCUSS thread
> > > > > >> -
> > > > > >>
> > > > > >> Known Improvements from Forks which will need
> > > proposals
> > > > and
> > > > > >> discussion:
> > > > > >> -
> > > > > >>
> > > > > >> Adding HA for Livy
> > > > > >> -
> > > > > >>
> > > > > >> Updating security capabilities (eg. kerberos
> for
> > > > jdbc,
> > > > > fixing
> > > > > >> bugs in encryption)
> > > > > >> -
> > > > > >>
> > > > > >> Expanding the support for kubernetes
> > > > > >> -
> > > > > >>
> > > > > >> Responding to CVEs in dependencies (eg. log4j,
> > > > thrift)
> > > > > >> -
> > > > > >>
> > > > > >> Livy rest cluster - IS THIS SAME AS HA for Livy
> > > > ABOVE?
> > > > > >> -
> > > > > >>
> > > > > >> Support multi Spark versions
> > > > > >> -
> > > > > >>
> > > > > >> Implemented a metrics system for Livy
> > > > > >> -
> > > > > >>
> > > > > >> Support customize batch/interactive session
> > > > lifecycle event
> > > > > >> handler, default log event with log4j, very
> > helpful
> > > > for
> > > > > troubleshooting
> > > > > >> -
> > > > > >>
> > > > > >> Optimize log to track which session id the log
> > > > message came
> > > > > >> from, also very helpful for troubleshooting
> > > > > >> -
> > > > > >>
> > > > > >> Support customize Spark config optimization
> > rules,
> > > > can be
> > > > > used
> > > > > >> to optimize config for users’ job
> > > > > >> -
> > > > > >>
> > > > > >> A set of command line tool which can be used to
> > > > replace
> > > > > Spark’s
> > > > > >> spark-submit, pyspark, spark-sql but actually
> > > submit
> > > > > application in Livy
> > > > > >> -
> > > > > >>
> > > > > >> We are planning to implement a JDBC state
> store,
> > > and
> > > > allow
> > > > > >> multi Livy Thrift sessions to share one backend
> > > Spark
> > > > > application in the
> > > > > >> next few months.
> > > > > >> -
> > > > > >>
> > > > > >> These items and others that are brought to
> community
> > > > may need
> > > > > >> consolidation or multiple configurable options and
> > > will
> > > > need
> > > > > to be part of
> > > > > >> the discussion
> > > > > >> -
> > > > > >>
> > > > > >> One-pager Livy Improvement Proposals (LIP) may
> > make
> > > > sense
> > > > > to
> > > > > >> drive these discussions and convergence
> > > > > >> -
> > > > > >>
> > > > > >> Feature Branch Strategy for large changes
> > > > > >> -
> > > > > >>
> > > > > >> Large features are hard to review we will
> need
> > > to
> > > > > define a
> > > > > >> strategy
> > > > > >> -
> > > > > >>
> > > > > >> Determine the Improvements to be delivered across
> > > first
> > > > 3
> > > > > Releases
> > > > > >> with Target Release Dates
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> Ensure CVE and Dependency management hygiene is in
> > place
> > > > > >>
> > > > > >>
> > > > > >> The above approach will usher the community back to an
> > > active
> > > > > status with
> > > > > >> a Roadmap of 3 or more release plans and security
> hygiene
> > in
> > > > place.
> > > > > >> Development Practices
> > > > > >>
> > > > > >> The Livy project follows a review before commit
> > philosophy.
> > > > Every
> > > > > commit
> > > > > >> automatically runs through the unit tests and generates
> > > > coverage
> > > > > reports
> > > > > >> presented as a pull request comment. Our experience with
> > > this
> > > > > process leads
> > > > > >> us to believe that it helps ease new contributors into
> the
> > > > project.
> > > > > They
> > > > > >> get feedback quickly on common mistakes, lowering the
> > burden
> > > > on
> > > > > reviewers.
> > > > > >> Those same reviewers get to lead by example, showing the
> > new
> > > > > contributors
> > > > > >> that we value feedback within our community even when
> > > changes
> > > > are
> > > > > done by
> > > > > >> more experienced folks. Taken from the original Apache
> > Livy
> > > > > Proposal [1],
> > > > > >> this should continue to be true. As mentioned, Livy is a
> > > > mature
> > > > > project and
> > > > > >> as such RTC is the most appropriate for continued
> quality
> > > and
> > > > > awareness.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> 1.
> > > > > >>
> > > > > >> Original Apache Livy Proposal
> > > > > >>
> > > > >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik
> > i.apache.org%2Fconfluence%2Fdisplay%2Fincubator%2FLivyProposal&dat
> > a=05%7C01%7Crebekbol%40microsoft.com%7C5ead1754350a408bceba08dab6192ee
> > 9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638022516535784379%7CUn
> > known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haW
> > wiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rbccZA%2BUAUHjpEolIiY8ig%2B
> > 5nRw5abmw9oiwJGkz8TM%3D&reserved=0
> > > > > >> 2.
> > > > > >>
> > > > > >> Retirement DISCUSS thread
> > > > > >>
> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fgcstsrhbp91c5mm55htqn1l3djv8m7o0&data=05%7C01%7Crebekbol%40microsoft.com%7C5ead1754350a408bceba08dab6192ee9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638022516535940162%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dgaMvjevmL84UXIvyHZ3OgCDX%2FXs7ZdKZvNXR1UR5A8%3D&reserved=0
> > > > >
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <
> > > > > madhawa30@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hi Larry,
> > > > > >>>
> > > > > >>> I'm an IPMC Member. madhawa30 at gmail dot com is my
> > > > preferred
> > > > > email
> > > > > >>> address.
> > > > > >>> apache id: madhawa
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>> Madhawa
> > > > > >>>
> > > > > >>>
> > > > > >>> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <
> > > > lmccay@apache.org>
> > > > > wrote:
> > > > > >>>
> > > > > >>> > Hi Madhawa -
> > > > > >>> >
> > > > > >>> > That's awesome!
> > > > > >>> > Are you already a member of IPMC?
> > > > > >>> > If not, are you an ASF member?
> > > > > >>> > If you are an ASF member you can request that you be
> > > added
> > > > as an
> > > > > IPMC
> > > > > >>> > member.
> > > > > >>> >
> > > > > >>> > Can you provide your company affiliation for the
> > proposal
> > > > and
> > > > > preferred
> > > > > >>> > email?
> > > > > >>> >
> > > > > >>> > thanks!
> > > > > >>> >
> > > > > >>> > --larry
> > > > > >>> >
> > > > > >>> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <
> > > > > >>> madhawa30@gmail.com>
> > > > > >>> > wrote:
> > > > > >>> >
> > > > > >>> > > Hi Larry,
> > > > > >>> > >
> > > > > >>> > > I'm interested in working with Livy and would like
> to
> > > > join as a
> > > > > >>> Mentor.
> > > > > >>> > >
> > > > > >>> > > Thanks,
> > > > > >>> > > Madhawa
> > > > > >>> > >
> > > > > >>> > >
> > > > > >>> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <
> > > > lmccay@apache.org
> > > > > >
> > > > > >>> wrote:
> > > > > >>> > >
> > > > > >>> > > > Sorry, I missed commenting on this:
> > > > > >>> > > >
> > > > > >>> > > > "There is also no concept as an emeritus PPMC
> > member
> > > > at the
> > > > > ASF."
> > > > > >>> > > >
> > > > > >>> > > > I assume that we can remove PPMC members that do
> > not
> > > > opt-in
> > > > > >>> explicitly
> > > > > >>> > at
> > > > > >>> > > > this point.
> > > > > >>> > > > They will have every opportunity to rejoin.
> > > > > >>> > > >
> > > > > >>> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <
> > > > > lmccay@apache.org>
> > > > > >>> > wrote:
> > > > > >>> > > >
> > > > > >>> > > > > I will ask in a separate thread, @Justin
> Mclean <
> > > > > >>> justinmclean@me.com
> > > > > >>> > >
> > > > > >>> > > -
> > > > > >>> > > > > thanks.
> > > > > >>> > > > > Adding JB adds another company and we are
> > certainly
> > > > open to
> > > > > >>> anyone
> > > > > >>> > else
> > > > > >>> > > > > that would like to join as a mentor.
> > > > > >>> > > > > At the end of the day, the mentors are for
> > > > instilling the
> > > > > Apache
> > > > > >>> Way
> > > > > >>> > > > > knowledge and steering toward graduation.
> > > > > >>> > > > > I feel that this diversity, while nice to have,
> > is
> > > > less
> > > > > important
> > > > > >>> > than
> > > > > >>> > > > > that of the PPMC and committers for the long
> term
> > > > health
> > > > > of the
> > > > > >>> > > > community.
> > > > > >>> > > > >
> > > > > >>> > > > > We need to push this podling to graduation as
> > > > quickly as
> > > > > possible
> > > > > >>> > since
> > > > > >>> > > > it
> > > > > >>> > > > > is rather mature and needs to get to the next
> > > level.
> > > > > >>> > > > >
> > > > > >>> > > > > Again, any potential Mentors that would like to
> > > join
> > > > are
> > > > > more
> > > > > >>> than
> > > > > >>> > > > welcome.
> > > > > >>> > > > >
> > > > > >>> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean
> <
> > > > > >>> > > justin@classsoftware.com
> > > > > >>> > > > >
> > > > > >>> > > > > wrote:
> > > > > >>> > > > >
> > > > > >>> > > > >> Hi,
> > > > > >>> > > > >>
> > > > > >>> > > > >> I’m sorry, but Imran Rashid can’t be a mentor
> > for
> > > > the
> > > > > project as
> > > > > >>> > they
> > > > > >>> > > > are
> > > > > >>> > > > >> not an IPMC member. Currently, both Sunil and
> > > Larry
> > > > (as
> > > > > they
> > > > > >>> are ASF
> > > > > >>> > > > >> members) need to ask to join the IPMC and
> NOTICE
> > > > sent to
> > > > > the ASF
> > > > > >>> > > board.
> > > > > >>> > > > I
> > > > > >>> > > > >> would also prefer that mentors come from
> > different
> > > > > companies.
> > > > > >>> > > > >>
> > > > > >>> > > > >> There is also no concept as an emeritus PPMC
> > > member
> > > > at
> > > > > the ASF.
> > > > > >>> > > > >>
> > > > > >>> > > > >> Kind Regards,
> > > > > >>> > > > >> Justin
> > > > > >>> > > > >>
> > > > > >>> > > > >>
> > > > > >>> > > > >>
> > > > > >>> >
> > > > >
> > > --------------------------------------------------------------------
> > > -
> > > > > >>> > > > >> To unsubscribe, e-mail:
> > > > > >>> general-unsubscribe@incubator.apache.org
> > > > > >>> > > > >> For additional commands, e-mail:
> > > > > >>> general-help@incubator.apache.org
> > > > > >>> > > > >>
> > > > > >>> > > > >>
> > > > > >>> > > >
> > > > > >>> > >
> > > > > >>> >
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > > >
> > > --------------------------------------------------------------------
> > > -
> > > > > To unsubscribe, e-mail:
> general-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail:
> > general-help@incubator.apache.org
> > > > >
> > > >
> > > >
> > >
> >
>
RE: [EXTERNAL] Re: Proposal to Revive Apache Livy Community
Posted by Renat Bekbolatov <re...@microsoft.com.INVALID>.
I welcome this development - honestly, possibly long overdue.
Would it be possible to propose additional committers for a wider representation?
Thank you,
Renat
-----Original Message-----
From: larry mccay <lm...@apache.org>
Sent: Monday, October 24, 2022 4:41 PM
To: dev@livy.apache.org
Subject: [EXTERNAL] Re: Proposal to Revive Apache Livy Community
Again, I have no objections to this and will be happy to include you.
Thanks for calling out!
On Mon, Oct 24, 2022 at 5:38 PM Marco Gaido <mg...@apache.org> wrote:
> Hi all,
>
> I am also busy with different projects at the moment, as such I am
> more than happy if ai can help somehow, e.g. reviewing code in parts
> that I know, but I cannot allocate much bandwidth to it.
>
> Thanks,
> Marco
>
> Il lun 24 ott 2022, 22:07 larry mccay <lm...@apache.org> ha scritto:
>
> > Personally, I would welcome it.
> > Any notes of guidance and support as we ramp up the team and make
> > the
> first
> > few releases would be invaluable.
> >
> >
> > On Mon, Oct 24, 2022 at 3:49 PM Alex Bozarth <aj...@us.ibm.com>
> wrote:
> >
> > > I would love to stay on as a PPMC, but if my lack of availability
> > concerns
> > > the new PPMC then I'm willing to be emeritus.
> > >
> > >
> > > Alex Bozarth
> > > Jupyter Architect, IBM CODAIT
> > > GitHub: ajbozarth
> > >
> > > On 10/24/22, 2:46 PM, "larry mccay" <lm...@apache.org> wrote:
> > >
> > > Oh, very good, Alex!
> > > Be sure to let us know if you want to remain on the project in
> > > any capacity.
> > >
> > >
> > > On Mon, Oct 24, 2022 at 3:18 PM Alex Bozarth
> > > <aj...@us.ibm.com>
> > > wrote:
> > >
> > > > As the only Livy PPMC still responding to the Mentors on the
> > private
> > > list,
> > > > I have updated the Livy Podling report for November with the
> status
> > > of the
> > > > project and a request to the IPMC to review this proposal
> > > since there are
> > > > not enough active Livy PPMC members to reach a quorum to
> > > pass the proposal.
> > > >
> > > > As a current Livy PPMC I strongly support this proposal for
> > > revitalization
> > > > as I do not have enough bandwidth to dedicate to Livy.
> > > >
> > > >
> > > > Alex Bozarth
> > > > Jupyter Architect, IBM CODAIT
> > > > GitHub: ajbozarth
> > > >
> > > > On 10/24/22, 1:41 PM, "larry mccay" <lm...@apache.org> wrote:
> > > >
> > > > Gentle reminder that we need to determine next steps here.
> > > > We have an updated proposal on this thread.
> > > > Do we need a VOTE or can we move forward directly to
> adjusting
> > > the
> > > > members,
> > > > etc?
> > > >
> > > > Thanks!
> > > >
> > > > --larry
> > > >
> > > > On Thu, Oct 20, 2022 at 3:26 PM larry mccay <
> lmccay@apache.org
> > >
> > > wrote:
> > > >
> > > > > @Justin Mclean <ju...@me.com> - any insights on
> next
> > > steps
> > > > here?
> > > > >
> > > > >
> > > > > On Tue, Oct 18, 2022 at 5:44 PM larry mccay <
> > lmccay@apache.org
> > > >
> > > > wrote:
> > > > >
> > > > >> Very good, here is the latest revision with updated
> Mentors.
> > > > >> Sunil and I have been added to the IPMC as well.
> > > > >> Welcome Madhawa and thanks for stepping up as a Mentor for
> > > Livy!
> > > > >>
> > > > >> Abstract
> > > > >>
> > > > >> Livy is a web service that exposes a REST interface for
> > > managing
> > > > long
> > > > >> running Apache Spark contexts in your cluster. With Livy,
> > new
> > > > applications
> > > > >> can be built on top of Apache Spark that require fine
> > grained
> > > > interaction
> > > > >> with many Spark contexts [1].
> > > > >>
> > > > >> While this project has been well regarded and used in many
> > > contexts
> > > > as
> > > > >> the defacto standard API to Spark environments, it has
> been
> > > > incubating for
> > > > >> over 5 years without graduation to a TLP and it has become
> > > > difficult to
> > > > >> impossible for fixes and improvements to be contributed as
> > the
> > > > current
> > > > >> community seems to have moved on.
> > > > >>
> > > > >> There has been discussion regarding retirement of this
> > > podling where
> > > > >> there seems to be some increasing interest in joining and
> > > reviving
> > > > the
> > > > >> community [2].
> > > > >>
> > > > >> The intent of this proposal is to avoid retiring a well
> > > regarded,
> > > > >> actively used and rather mature project by reviving the
> PPMC
> > > and
> > > > community
> > > > >> with new folks that have a vested interest in the project
> > and
> > > > health of the
> > > > >> community.
> > > > >> Proposal
> > > > >>
> > > > >> We propose to revive the PPMC with a set of contributors
> and
> > > > maintainers
> > > > >> as mentors, PPMC members and committers.
> > > > >>
> > > > >> The retirement DISCUSS thread [2] has shown a growing
> > > interest in
> > > > >> providing new committers and bringing improvements and
> fixes
> > > from
> > > > >> organization’s internally maintained forks back to a
> revived
> > > > community.
> > > > >>
> > > > >> General Approach to Revival:
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Add new Mentors
> > > > >> -
> > > > >>
> > > > >> Larry McCay, lmccay@apache.org , Cloudera
> > > > >> -
> > > > >>
> > > > >> Sunil Govindan, sunilg@apache.org, Cloudera
> > > > >> -
> > > > >>
> > > > >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
> > > > >> -
> > > > >>
> > > > >> Madhawa Gunasekara, madhawa30@gmail.com,
> Independent
> > > > >>
> > > > >>
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Add new Committers/PPMC
> > > > >> -
> > > > >>
> > > > >> Larry McCay, lmccay@apache.org, Cloudera
> > > > >> -
> > > > >>
> > > > >> Vinod Kumar Vavilapalli, vinodkv@cloudera.com,
> > Cloudera
> > > > >> -
> > > > >>
> > > > >> Imran Rashid - irashid@apache.org, Cloudera
> > > > >> -
> > > > >>
> > > > >> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
> > > > >> -
> > > > >>
> > > > >> Wing Yew Poon, wypoon@cloudera.com, Cloudera
> > > > >> -
> > > > >>
> > > > >> Xilang Yan, xilang.yan@gmail.com, Shopee
> > > > >> -
> > > > >>
> > > > >> Jianzhen Wu, myjianzhen@gmail.com, Shopee
> > > > >> -
> > > > >>
> > > > >> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
> > > > >> -
> > > > >>
> > > > >> Pralab Kumar, pralkuma@visa.com, Visa
> > > > >> -
> > > > >>
> > > > >> Prasad Shrikant, shrikant.ism@gmail.com, Visa
> > > > >> -
> > > > >>
> > > > >> Brahma Reddy Battula, brahma@apache.org, Visa
> > > > >>
> > > > >>
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Invite existing PPMC members to opt-in or otherwise go
> > > emeritus
> > > > >> -
> > > > >>
> > > > >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
> > > (opted-in
> > > > via
> > > > >> Retirement DISCUSS thread [2])
> > > > >>
> > > > >>
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Invite existing Committers to opt-out or otherwise
> > continue
> > > > >>
> > > > >>
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Establish Roadmap via follow up DISCUSS thread
> > > > >> -
> > > > >>
> > > > >> Known Improvements from Forks which will need
> > proposals
> > > and
> > > > >> discussion:
> > > > >> -
> > > > >>
> > > > >> Adding HA for Livy
> > > > >> -
> > > > >>
> > > > >> Updating security capabilities (eg. kerberos for
> > > jdbc,
> > > > fixing
> > > > >> bugs in encryption)
> > > > >> -
> > > > >>
> > > > >> Expanding the support for kubernetes
> > > > >> -
> > > > >>
> > > > >> Responding to CVEs in dependencies (eg. log4j,
> > > thrift)
> > > > >> -
> > > > >>
> > > > >> Livy rest cluster - IS THIS SAME AS HA for Livy
> > > ABOVE?
> > > > >> -
> > > > >>
> > > > >> Support multi Spark versions
> > > > >> -
> > > > >>
> > > > >> Implemented a metrics system for Livy
> > > > >> -
> > > > >>
> > > > >> Support customize batch/interactive session
> > > lifecycle event
> > > > >> handler, default log event with log4j, very
> helpful
> > > for
> > > > troubleshooting
> > > > >> -
> > > > >>
> > > > >> Optimize log to track which session id the log
> > > message came
> > > > >> from, also very helpful for troubleshooting
> > > > >> -
> > > > >>
> > > > >> Support customize Spark config optimization
> rules,
> > > can be
> > > > used
> > > > >> to optimize config for users’ job
> > > > >> -
> > > > >>
> > > > >> A set of command line tool which can be used to
> > > replace
> > > > Spark’s
> > > > >> spark-submit, pyspark, spark-sql but actually
> > submit
> > > > application in Livy
> > > > >> -
> > > > >>
> > > > >> We are planning to implement a JDBC state store,
> > and
> > > allow
> > > > >> multi Livy Thrift sessions to share one backend
> > Spark
> > > > application in the
> > > > >> next few months.
> > > > >> -
> > > > >>
> > > > >> These items and others that are brought to community
> > > may need
> > > > >> consolidation or multiple configurable options and
> > will
> > > need
> > > > to be part of
> > > > >> the discussion
> > > > >> -
> > > > >>
> > > > >> One-pager Livy Improvement Proposals (LIP) may
> make
> > > sense
> > > > to
> > > > >> drive these discussions and convergence
> > > > >> -
> > > > >>
> > > > >> Feature Branch Strategy for large changes
> > > > >> -
> > > > >>
> > > > >> Large features are hard to review we will need
> > to
> > > > define a
> > > > >> strategy
> > > > >> -
> > > > >>
> > > > >> Determine the Improvements to be delivered across
> > first
> > > 3
> > > > Releases
> > > > >> with Target Release Dates
> > > > >>
> > > > >>
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Ensure CVE and Dependency management hygiene is in
> place
> > > > >>
> > > > >>
> > > > >> The above approach will usher the community back to an
> > active
> > > > status with
> > > > >> a Roadmap of 3 or more release plans and security hygiene
> in
> > > place.
> > > > >> Development Practices
> > > > >>
> > > > >> The Livy project follows a review before commit
> philosophy.
> > > Every
> > > > commit
> > > > >> automatically runs through the unit tests and generates
> > > coverage
> > > > reports
> > > > >> presented as a pull request comment. Our experience with
> > this
> > > > process leads
> > > > >> us to believe that it helps ease new contributors into the
> > > project.
> > > > They
> > > > >> get feedback quickly on common mistakes, lowering the
> burden
> > > on
> > > > reviewers.
> > > > >> Those same reviewers get to lead by example, showing the
> new
> > > > contributors
> > > > >> that we value feedback within our community even when
> > changes
> > > are
> > > > done by
> > > > >> more experienced folks. Taken from the original Apache
> Livy
> > > > Proposal [1],
> > > > >> this should continue to be true. As mentioned, Livy is a
> > > mature
> > > > project and
> > > > >> as such RTC is the most appropriate for continued quality
> > and
> > > > awareness.
> > > > >>
> > > > >>
> > > > >>
> > > > >> 1.
> > > > >>
> > > > >> Original Apache Livy Proposal
> > > > >>
> > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik
> i.apache.org%2Fconfluence%2Fdisplay%2Fincubator%2FLivyProposal&dat
> a=05%7C01%7Crebekbol%40microsoft.com%7C5ead1754350a408bceba08dab6192ee
> 9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638022516535784379%7CUn
> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haW
> wiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rbccZA%2BUAUHjpEolIiY8ig%2B
> 5nRw5abmw9oiwJGkz8TM%3D&reserved=0
> > > > >> 2.
> > > > >>
> > > > >> Retirement DISCUSS thread
> > > > >>
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fgcstsrhbp91c5mm55htqn1l3djv8m7o0&data=05%7C01%7Crebekbol%40microsoft.com%7C5ead1754350a408bceba08dab6192ee9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638022516535940162%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dgaMvjevmL84UXIvyHZ3OgCDX%2FXs7ZdKZvNXR1UR5A8%3D&reserved=0
> > > >
> > > > >>
> > > > >>
> > > > >> On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <
> > > > madhawa30@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Hi Larry,
> > > > >>>
> > > > >>> I'm an IPMC Member. madhawa30 at gmail dot com is my
> > > preferred
> > > > email
> > > > >>> address.
> > > > >>> apache id: madhawa
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Madhawa
> > > > >>>
> > > > >>>
> > > > >>> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <
> > > lmccay@apache.org>
> > > > wrote:
> > > > >>>
> > > > >>> > Hi Madhawa -
> > > > >>> >
> > > > >>> > That's awesome!
> > > > >>> > Are you already a member of IPMC?
> > > > >>> > If not, are you an ASF member?
> > > > >>> > If you are an ASF member you can request that you be
> > added
> > > as an
> > > > IPMC
> > > > >>> > member.
> > > > >>> >
> > > > >>> > Can you provide your company affiliation for the
> proposal
> > > and
> > > > preferred
> > > > >>> > email?
> > > > >>> >
> > > > >>> > thanks!
> > > > >>> >
> > > > >>> > --larry
> > > > >>> >
> > > > >>> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <
> > > > >>> madhawa30@gmail.com>
> > > > >>> > wrote:
> > > > >>> >
> > > > >>> > > Hi Larry,
> > > > >>> > >
> > > > >>> > > I'm interested in working with Livy and would like to
> > > join as a
> > > > >>> Mentor.
> > > > >>> > >
> > > > >>> > > Thanks,
> > > > >>> > > Madhawa
> > > > >>> > >
> > > > >>> > >
> > > > >>> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <
> > > lmccay@apache.org
> > > > >
> > > > >>> wrote:
> > > > >>> > >
> > > > >>> > > > Sorry, I missed commenting on this:
> > > > >>> > > >
> > > > >>> > > > "There is also no concept as an emeritus PPMC
> member
> > > at the
> > > > ASF."
> > > > >>> > > >
> > > > >>> > > > I assume that we can remove PPMC members that do
> not
> > > opt-in
> > > > >>> explicitly
> > > > >>> > at
> > > > >>> > > > this point.
> > > > >>> > > > They will have every opportunity to rejoin.
> > > > >>> > > >
> > > > >>> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <
> > > > lmccay@apache.org>
> > > > >>> > wrote:
> > > > >>> > > >
> > > > >>> > > > > I will ask in a separate thread, @Justin Mclean <
> > > > >>> justinmclean@me.com
> > > > >>> > >
> > > > >>> > > -
> > > > >>> > > > > thanks.
> > > > >>> > > > > Adding JB adds another company and we are
> certainly
> > > open to
> > > > >>> anyone
> > > > >>> > else
> > > > >>> > > > > that would like to join as a mentor.
> > > > >>> > > > > At the end of the day, the mentors are for
> > > instilling the
> > > > Apache
> > > > >>> Way
> > > > >>> > > > > knowledge and steering toward graduation.
> > > > >>> > > > > I feel that this diversity, while nice to have,
> is
> > > less
> > > > important
> > > > >>> > than
> > > > >>> > > > > that of the PPMC and committers for the long term
> > > health
> > > > of the
> > > > >>> > > > community.
> > > > >>> > > > >
> > > > >>> > > > > We need to push this podling to graduation as
> > > quickly as
> > > > possible
> > > > >>> > since
> > > > >>> > > > it
> > > > >>> > > > > is rather mature and needs to get to the next
> > level.
> > > > >>> > > > >
> > > > >>> > > > > Again, any potential Mentors that would like to
> > join
> > > are
> > > > more
> > > > >>> than
> > > > >>> > > > welcome.
> > > > >>> > > > >
> > > > >>> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
> > > > >>> > > justin@classsoftware.com
> > > > >>> > > > >
> > > > >>> > > > > wrote:
> > > > >>> > > > >
> > > > >>> > > > >> Hi,
> > > > >>> > > > >>
> > > > >>> > > > >> I’m sorry, but Imran Rashid can’t be a mentor
> for
> > > the
> > > > project as
> > > > >>> > they
> > > > >>> > > > are
> > > > >>> > > > >> not an IPMC member. Currently, both Sunil and
> > Larry
> > > (as
> > > > they
> > > > >>> are ASF
> > > > >>> > > > >> members) need to ask to join the IPMC and NOTICE
> > > sent to
> > > > the ASF
> > > > >>> > > board.
> > > > >>> > > > I
> > > > >>> > > > >> would also prefer that mentors come from
> different
> > > > companies.
> > > > >>> > > > >>
> > > > >>> > > > >> There is also no concept as an emeritus PPMC
> > member
> > > at
> > > > the ASF.
> > > > >>> > > > >>
> > > > >>> > > > >> Kind Regards,
> > > > >>> > > > >> Justin
> > > > >>> > > > >>
> > > > >>> > > > >>
> > > > >>> > > > >>
> > > > >>> >
> > > >
> > --------------------------------------------------------------------
> > -
> > > > >>> > > > >> To unsubscribe, e-mail:
> > > > >>> general-unsubscribe@incubator.apache.org
> > > > >>> > > > >> For additional commands, e-mail:
> > > > >>> general-help@incubator.apache.org
> > > > >>> > > > >>
> > > > >>> > > > >>
> > > > >>> > > >
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > > >>
> > > >
> > > >
> > > >
> > --------------------------------------------------------------------
> > -
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail:
> general-help@incubator.apache.org
> > > >
> > >
> > >
> >
>
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
Again, I have no objections to this and will be happy to include you.
Thanks for calling out!
On Mon, Oct 24, 2022 at 5:38 PM Marco Gaido <mg...@apache.org> wrote:
> Hi all,
>
> I am also busy with different projects at the moment, as such I am more
> than happy if ai can help somehow, e.g. reviewing code in parts that I
> know, but I cannot allocate much bandwidth to it.
>
> Thanks,
> Marco
>
> Il lun 24 ott 2022, 22:07 larry mccay <lm...@apache.org> ha scritto:
>
> > Personally, I would welcome it.
> > Any notes of guidance and support as we ramp up the team and make the
> first
> > few releases would be invaluable.
> >
> >
> > On Mon, Oct 24, 2022 at 3:49 PM Alex Bozarth <aj...@us.ibm.com>
> wrote:
> >
> > > I would love to stay on as a PPMC, but if my lack of availability
> > concerns
> > > the new PPMC then I'm willing to be emeritus.
> > >
> > >
> > > Alex Bozarth
> > > Jupyter Architect, IBM CODAIT
> > > GitHub: ajbozarth
> > >
> > > On 10/24/22, 2:46 PM, "larry mccay" <lm...@apache.org> wrote:
> > >
> > > Oh, very good, Alex!
> > > Be sure to let us know if you want to remain on the project in any
> > > capacity.
> > >
> > >
> > > On Mon, Oct 24, 2022 at 3:18 PM Alex Bozarth <aj...@us.ibm.com>
> > > wrote:
> > >
> > > > As the only Livy PPMC still responding to the Mentors on the
> > private
> > > list,
> > > > I have updated the Livy Podling report for November with the
> status
> > > of the
> > > > project and a request to the IPMC to review this proposal since
> > > there are
> > > > not enough active Livy PPMC members to reach a quorum to pass the
> > > proposal.
> > > >
> > > > As a current Livy PPMC I strongly support this proposal for
> > > revitalization
> > > > as I do not have enough bandwidth to dedicate to Livy.
> > > >
> > > >
> > > > Alex Bozarth
> > > > Jupyter Architect, IBM CODAIT
> > > > GitHub: ajbozarth
> > > >
> > > > On 10/24/22, 1:41 PM, "larry mccay" <lm...@apache.org> wrote:
> > > >
> > > > Gentle reminder that we need to determine next steps here.
> > > > We have an updated proposal on this thread.
> > > > Do we need a VOTE or can we move forward directly to
> adjusting
> > > the
> > > > members,
> > > > etc?
> > > >
> > > > Thanks!
> > > >
> > > > --larry
> > > >
> > > > On Thu, Oct 20, 2022 at 3:26 PM larry mccay <
> lmccay@apache.org
> > >
> > > wrote:
> > > >
> > > > > @Justin Mclean <ju...@me.com> - any insights on
> next
> > > steps
> > > > here?
> > > > >
> > > > >
> > > > > On Tue, Oct 18, 2022 at 5:44 PM larry mccay <
> > lmccay@apache.org
> > > >
> > > > wrote:
> > > > >
> > > > >> Very good, here is the latest revision with updated
> Mentors.
> > > > >> Sunil and I have been added to the IPMC as well.
> > > > >> Welcome Madhawa and thanks for stepping up as a Mentor for
> > > Livy!
> > > > >>
> > > > >> Abstract
> > > > >>
> > > > >> Livy is a web service that exposes a REST interface for
> > > managing
> > > > long
> > > > >> running Apache Spark contexts in your cluster. With Livy,
> > new
> > > > applications
> > > > >> can be built on top of Apache Spark that require fine
> > grained
> > > > interaction
> > > > >> with many Spark contexts [1].
> > > > >>
> > > > >> While this project has been well regarded and used in many
> > > contexts
> > > > as
> > > > >> the defacto standard API to Spark environments, it has
> been
> > > > incubating for
> > > > >> over 5 years without graduation to a TLP and it has become
> > > > difficult to
> > > > >> impossible for fixes and improvements to be contributed as
> > the
> > > > current
> > > > >> community seems to have moved on.
> > > > >>
> > > > >> There has been discussion regarding retirement of this
> > > podling where
> > > > >> there seems to be some increasing interest in joining and
> > > reviving
> > > > the
> > > > >> community [2].
> > > > >>
> > > > >> The intent of this proposal is to avoid retiring a well
> > > regarded,
> > > > >> actively used and rather mature project by reviving the
> PPMC
> > > and
> > > > community
> > > > >> with new folks that have a vested interest in the project
> > and
> > > > health of the
> > > > >> community.
> > > > >> Proposal
> > > > >>
> > > > >> We propose to revive the PPMC with a set of contributors
> and
> > > > maintainers
> > > > >> as mentors, PPMC members and committers.
> > > > >>
> > > > >> The retirement DISCUSS thread [2] has shown a growing
> > > interest in
> > > > >> providing new committers and bringing improvements and
> fixes
> > > from
> > > > >> organization’s internally maintained forks back to a
> revived
> > > > community.
> > > > >>
> > > > >> General Approach to Revival:
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Add new Mentors
> > > > >> -
> > > > >>
> > > > >> Larry McCay, lmccay@apache.org , Cloudera
> > > > >> -
> > > > >>
> > > > >> Sunil Govindan, sunilg@apache.org, Cloudera
> > > > >> -
> > > > >>
> > > > >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
> > > > >> -
> > > > >>
> > > > >> Madhawa Gunasekara, madhawa30@gmail.com,
> Independent
> > > > >>
> > > > >>
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Add new Committers/PPMC
> > > > >> -
> > > > >>
> > > > >> Larry McCay, lmccay@apache.org, Cloudera
> > > > >> -
> > > > >>
> > > > >> Vinod Kumar Vavilapalli, vinodkv@cloudera.com,
> > Cloudera
> > > > >> -
> > > > >>
> > > > >> Imran Rashid - irashid@apache.org, Cloudera
> > > > >> -
> > > > >>
> > > > >> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
> > > > >> -
> > > > >>
> > > > >> Wing Yew Poon, wypoon@cloudera.com, Cloudera
> > > > >> -
> > > > >>
> > > > >> Xilang Yan, xilang.yan@gmail.com, Shopee
> > > > >> -
> > > > >>
> > > > >> Jianzhen Wu, myjianzhen@gmail.com, Shopee
> > > > >> -
> > > > >>
> > > > >> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
> > > > >> -
> > > > >>
> > > > >> Pralab Kumar, pralkuma@visa.com, Visa
> > > > >> -
> > > > >>
> > > > >> Prasad Shrikant, shrikant.ism@gmail.com, Visa
> > > > >> -
> > > > >>
> > > > >> Brahma Reddy Battula, brahma@apache.org, Visa
> > > > >>
> > > > >>
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Invite existing PPMC members to opt-in or otherwise go
> > > emeritus
> > > > >> -
> > > > >>
> > > > >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
> > > (opted-in
> > > > via
> > > > >> Retirement DISCUSS thread [2])
> > > > >>
> > > > >>
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Invite existing Committers to opt-out or otherwise
> > continue
> > > > >>
> > > > >>
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Establish Roadmap via follow up DISCUSS thread
> > > > >> -
> > > > >>
> > > > >> Known Improvements from Forks which will need
> > proposals
> > > and
> > > > >> discussion:
> > > > >> -
> > > > >>
> > > > >> Adding HA for Livy
> > > > >> -
> > > > >>
> > > > >> Updating security capabilities (eg. kerberos for
> > > jdbc,
> > > > fixing
> > > > >> bugs in encryption)
> > > > >> -
> > > > >>
> > > > >> Expanding the support for kubernetes
> > > > >> -
> > > > >>
> > > > >> Responding to CVEs in dependencies (eg. log4j,
> > > thrift)
> > > > >> -
> > > > >>
> > > > >> Livy rest cluster - IS THIS SAME AS HA for Livy
> > > ABOVE?
> > > > >> -
> > > > >>
> > > > >> Support multi Spark versions
> > > > >> -
> > > > >>
> > > > >> Implemented a metrics system for Livy
> > > > >> -
> > > > >>
> > > > >> Support customize batch/interactive session
> > > lifecycle event
> > > > >> handler, default log event with log4j, very
> helpful
> > > for
> > > > troubleshooting
> > > > >> -
> > > > >>
> > > > >> Optimize log to track which session id the log
> > > message came
> > > > >> from, also very helpful for troubleshooting
> > > > >> -
> > > > >>
> > > > >> Support customize Spark config optimization
> rules,
> > > can be
> > > > used
> > > > >> to optimize config for users’ job
> > > > >> -
> > > > >>
> > > > >> A set of command line tool which can be used to
> > > replace
> > > > Spark’s
> > > > >> spark-submit, pyspark, spark-sql but actually
> > submit
> > > > application in Livy
> > > > >> -
> > > > >>
> > > > >> We are planning to implement a JDBC state store,
> > and
> > > allow
> > > > >> multi Livy Thrift sessions to share one backend
> > Spark
> > > > application in the
> > > > >> next few months.
> > > > >> -
> > > > >>
> > > > >> These items and others that are brought to community
> > > may need
> > > > >> consolidation or multiple configurable options and
> > will
> > > need
> > > > to be part of
> > > > >> the discussion
> > > > >> -
> > > > >>
> > > > >> One-pager Livy Improvement Proposals (LIP) may
> make
> > > sense
> > > > to
> > > > >> drive these discussions and convergence
> > > > >> -
> > > > >>
> > > > >> Feature Branch Strategy for large changes
> > > > >> -
> > > > >>
> > > > >> Large features are hard to review we will need
> > to
> > > > define a
> > > > >> strategy
> > > > >> -
> > > > >>
> > > > >> Determine the Improvements to be delivered across
> > first
> > > 3
> > > > Releases
> > > > >> with Target Release Dates
> > > > >>
> > > > >>
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Ensure CVE and Dependency management hygiene is in
> place
> > > > >>
> > > > >>
> > > > >> The above approach will usher the community back to an
> > active
> > > > status with
> > > > >> a Roadmap of 3 or more release plans and security hygiene
> in
> > > place.
> > > > >> Development Practices
> > > > >>
> > > > >> The Livy project follows a review before commit
> philosophy.
> > > Every
> > > > commit
> > > > >> automatically runs through the unit tests and generates
> > > coverage
> > > > reports
> > > > >> presented as a pull request comment. Our experience with
> > this
> > > > process leads
> > > > >> us to believe that it helps ease new contributors into the
> > > project.
> > > > They
> > > > >> get feedback quickly on common mistakes, lowering the
> burden
> > > on
> > > > reviewers.
> > > > >> Those same reviewers get to lead by example, showing the
> new
> > > > contributors
> > > > >> that we value feedback within our community even when
> > changes
> > > are
> > > > done by
> > > > >> more experienced folks. Taken from the original Apache
> Livy
> > > > Proposal [1],
> > > > >> this should continue to be true. As mentioned, Livy is a
> > > mature
> > > > project and
> > > > >> as such RTC is the most appropriate for continued quality
> > and
> > > > awareness.
> > > > >>
> > > > >>
> > > > >>
> > > > >> 1.
> > > > >>
> > > > >> Original Apache Livy Proposal
> > > > >>
> > > >
> https://cwiki.apache.org/confluence/display/incubator/LivyProposal
> > > > >> 2.
> > > > >>
> > > > >> Retirement DISCUSS thread
> > > > >>
> > > https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
> > > >
> > > > >>
> > > > >>
> > > > >> On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <
> > > > madhawa30@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Hi Larry,
> > > > >>>
> > > > >>> I'm an IPMC Member. madhawa30 at gmail dot com is my
> > > preferred
> > > > email
> > > > >>> address.
> > > > >>> apache id: madhawa
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Madhawa
> > > > >>>
> > > > >>>
> > > > >>> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <
> > > lmccay@apache.org>
> > > > wrote:
> > > > >>>
> > > > >>> > Hi Madhawa -
> > > > >>> >
> > > > >>> > That's awesome!
> > > > >>> > Are you already a member of IPMC?
> > > > >>> > If not, are you an ASF member?
> > > > >>> > If you are an ASF member you can request that you be
> > added
> > > as an
> > > > IPMC
> > > > >>> > member.
> > > > >>> >
> > > > >>> > Can you provide your company affiliation for the
> proposal
> > > and
> > > > preferred
> > > > >>> > email?
> > > > >>> >
> > > > >>> > thanks!
> > > > >>> >
> > > > >>> > --larry
> > > > >>> >
> > > > >>> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <
> > > > >>> madhawa30@gmail.com>
> > > > >>> > wrote:
> > > > >>> >
> > > > >>> > > Hi Larry,
> > > > >>> > >
> > > > >>> > > I'm interested in working with Livy and would like to
> > > join as a
> > > > >>> Mentor.
> > > > >>> > >
> > > > >>> > > Thanks,
> > > > >>> > > Madhawa
> > > > >>> > >
> > > > >>> > >
> > > > >>> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <
> > > lmccay@apache.org
> > > > >
> > > > >>> wrote:
> > > > >>> > >
> > > > >>> > > > Sorry, I missed commenting on this:
> > > > >>> > > >
> > > > >>> > > > "There is also no concept as an emeritus PPMC
> member
> > > at the
> > > > ASF."
> > > > >>> > > >
> > > > >>> > > > I assume that we can remove PPMC members that do
> not
> > > opt-in
> > > > >>> explicitly
> > > > >>> > at
> > > > >>> > > > this point.
> > > > >>> > > > They will have every opportunity to rejoin.
> > > > >>> > > >
> > > > >>> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <
> > > > lmccay@apache.org>
> > > > >>> > wrote:
> > > > >>> > > >
> > > > >>> > > > > I will ask in a separate thread, @Justin Mclean <
> > > > >>> justinmclean@me.com
> > > > >>> > >
> > > > >>> > > -
> > > > >>> > > > > thanks.
> > > > >>> > > > > Adding JB adds another company and we are
> certainly
> > > open to
> > > > >>> anyone
> > > > >>> > else
> > > > >>> > > > > that would like to join as a mentor.
> > > > >>> > > > > At the end of the day, the mentors are for
> > > instilling the
> > > > Apache
> > > > >>> Way
> > > > >>> > > > > knowledge and steering toward graduation.
> > > > >>> > > > > I feel that this diversity, while nice to have,
> is
> > > less
> > > > important
> > > > >>> > than
> > > > >>> > > > > that of the PPMC and committers for the long term
> > > health
> > > > of the
> > > > >>> > > > community.
> > > > >>> > > > >
> > > > >>> > > > > We need to push this podling to graduation as
> > > quickly as
> > > > possible
> > > > >>> > since
> > > > >>> > > > it
> > > > >>> > > > > is rather mature and needs to get to the next
> > level.
> > > > >>> > > > >
> > > > >>> > > > > Again, any potential Mentors that would like to
> > join
> > > are
> > > > more
> > > > >>> than
> > > > >>> > > > welcome.
> > > > >>> > > > >
> > > > >>> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
> > > > >>> > > justin@classsoftware.com
> > > > >>> > > > >
> > > > >>> > > > > wrote:
> > > > >>> > > > >
> > > > >>> > > > >> Hi,
> > > > >>> > > > >>
> > > > >>> > > > >> I’m sorry, but Imran Rashid can’t be a mentor
> for
> > > the
> > > > project as
> > > > >>> > they
> > > > >>> > > > are
> > > > >>> > > > >> not an IPMC member. Currently, both Sunil and
> > Larry
> > > (as
> > > > they
> > > > >>> are ASF
> > > > >>> > > > >> members) need to ask to join the IPMC and NOTICE
> > > sent to
> > > > the ASF
> > > > >>> > > board.
> > > > >>> > > > I
> > > > >>> > > > >> would also prefer that mentors come from
> different
> > > > companies.
> > > > >>> > > > >>
> > > > >>> > > > >> There is also no concept as an emeritus PPMC
> > member
> > > at
> > > > the ASF.
> > > > >>> > > > >>
> > > > >>> > > > >> Kind Regards,
> > > > >>> > > > >> Justin
> > > > >>> > > > >>
> > > > >>> > > > >>
> > > > >>> > > > >>
> > > > >>> >
> > > >
> > ---------------------------------------------------------------------
> > > > >>> > > > >> To unsubscribe, e-mail:
> > > > >>> general-unsubscribe@incubator.apache.org
> > > > >>> > > > >> For additional commands, e-mail:
> > > > >>> general-help@incubator.apache.org
> > > > >>> > > > >>
> > > > >>> > > > >>
> > > > >>> > > >
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > > >>
> > > >
> > > >
> > > >
> > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail:
> general-help@incubator.apache.org
> > > >
> > >
> > >
> >
>
Re: Proposal to Revive Apache Livy Community
Posted by Marco Gaido <mg...@apache.org>.
Hi all,
I am also busy with different projects at the moment, as such I am more
than happy if ai can help somehow, e.g. reviewing code in parts that I
know, but I cannot allocate much bandwidth to it.
Thanks,
Marco
Il lun 24 ott 2022, 22:07 larry mccay <lm...@apache.org> ha scritto:
> Personally, I would welcome it.
> Any notes of guidance and support as we ramp up the team and make the first
> few releases would be invaluable.
>
>
> On Mon, Oct 24, 2022 at 3:49 PM Alex Bozarth <aj...@us.ibm.com> wrote:
>
> > I would love to stay on as a PPMC, but if my lack of availability
> concerns
> > the new PPMC then I'm willing to be emeritus.
> >
> >
> > Alex Bozarth
> > Jupyter Architect, IBM CODAIT
> > GitHub: ajbozarth
> >
> > On 10/24/22, 2:46 PM, "larry mccay" <lm...@apache.org> wrote:
> >
> > Oh, very good, Alex!
> > Be sure to let us know if you want to remain on the project in any
> > capacity.
> >
> >
> > On Mon, Oct 24, 2022 at 3:18 PM Alex Bozarth <aj...@us.ibm.com>
> > wrote:
> >
> > > As the only Livy PPMC still responding to the Mentors on the
> private
> > list,
> > > I have updated the Livy Podling report for November with the status
> > of the
> > > project and a request to the IPMC to review this proposal since
> > there are
> > > not enough active Livy PPMC members to reach a quorum to pass the
> > proposal.
> > >
> > > As a current Livy PPMC I strongly support this proposal for
> > revitalization
> > > as I do not have enough bandwidth to dedicate to Livy.
> > >
> > >
> > > Alex Bozarth
> > > Jupyter Architect, IBM CODAIT
> > > GitHub: ajbozarth
> > >
> > > On 10/24/22, 1:41 PM, "larry mccay" <lm...@apache.org> wrote:
> > >
> > > Gentle reminder that we need to determine next steps here.
> > > We have an updated proposal on this thread.
> > > Do we need a VOTE or can we move forward directly to adjusting
> > the
> > > members,
> > > etc?
> > >
> > > Thanks!
> > >
> > > --larry
> > >
> > > On Thu, Oct 20, 2022 at 3:26 PM larry mccay <lmccay@apache.org
> >
> > wrote:
> > >
> > > > @Justin Mclean <ju...@me.com> - any insights on next
> > steps
> > > here?
> > > >
> > > >
> > > > On Tue, Oct 18, 2022 at 5:44 PM larry mccay <
> lmccay@apache.org
> > >
> > > wrote:
> > > >
> > > >> Very good, here is the latest revision with updated Mentors.
> > > >> Sunil and I have been added to the IPMC as well.
> > > >> Welcome Madhawa and thanks for stepping up as a Mentor for
> > Livy!
> > > >>
> > > >> Abstract
> > > >>
> > > >> Livy is a web service that exposes a REST interface for
> > managing
> > > long
> > > >> running Apache Spark contexts in your cluster. With Livy,
> new
> > > applications
> > > >> can be built on top of Apache Spark that require fine
> grained
> > > interaction
> > > >> with many Spark contexts [1].
> > > >>
> > > >> While this project has been well regarded and used in many
> > contexts
> > > as
> > > >> the defacto standard API to Spark environments, it has been
> > > incubating for
> > > >> over 5 years without graduation to a TLP and it has become
> > > difficult to
> > > >> impossible for fixes and improvements to be contributed as
> the
> > > current
> > > >> community seems to have moved on.
> > > >>
> > > >> There has been discussion regarding retirement of this
> > podling where
> > > >> there seems to be some increasing interest in joining and
> > reviving
> > > the
> > > >> community [2].
> > > >>
> > > >> The intent of this proposal is to avoid retiring a well
> > regarded,
> > > >> actively used and rather mature project by reviving the PPMC
> > and
> > > community
> > > >> with new folks that have a vested interest in the project
> and
> > > health of the
> > > >> community.
> > > >> Proposal
> > > >>
> > > >> We propose to revive the PPMC with a set of contributors and
> > > maintainers
> > > >> as mentors, PPMC members and committers.
> > > >>
> > > >> The retirement DISCUSS thread [2] has shown a growing
> > interest in
> > > >> providing new committers and bringing improvements and fixes
> > from
> > > >> organization’s internally maintained forks back to a revived
> > > community.
> > > >>
> > > >> General Approach to Revival:
> > > >>
> > > >> -
> > > >>
> > > >> Add new Mentors
> > > >> -
> > > >>
> > > >> Larry McCay, lmccay@apache.org , Cloudera
> > > >> -
> > > >>
> > > >> Sunil Govindan, sunilg@apache.org, Cloudera
> > > >> -
> > > >>
> > > >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
> > > >> -
> > > >>
> > > >> Madhawa Gunasekara, madhawa30@gmail.com, Independent
> > > >>
> > > >>
> > > >>
> > > >> -
> > > >>
> > > >> Add new Committers/PPMC
> > > >> -
> > > >>
> > > >> Larry McCay, lmccay@apache.org, Cloudera
> > > >> -
> > > >>
> > > >> Vinod Kumar Vavilapalli, vinodkv@cloudera.com,
> Cloudera
> > > >> -
> > > >>
> > > >> Imran Rashid - irashid@apache.org, Cloudera
> > > >> -
> > > >>
> > > >> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
> > > >> -
> > > >>
> > > >> Wing Yew Poon, wypoon@cloudera.com, Cloudera
> > > >> -
> > > >>
> > > >> Xilang Yan, xilang.yan@gmail.com, Shopee
> > > >> -
> > > >>
> > > >> Jianzhen Wu, myjianzhen@gmail.com, Shopee
> > > >> -
> > > >>
> > > >> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
> > > >> -
> > > >>
> > > >> Pralab Kumar, pralkuma@visa.com, Visa
> > > >> -
> > > >>
> > > >> Prasad Shrikant, shrikant.ism@gmail.com, Visa
> > > >> -
> > > >>
> > > >> Brahma Reddy Battula, brahma@apache.org, Visa
> > > >>
> > > >>
> > > >>
> > > >> -
> > > >>
> > > >> Invite existing PPMC members to opt-in or otherwise go
> > emeritus
> > > >> -
> > > >>
> > > >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
> > (opted-in
> > > via
> > > >> Retirement DISCUSS thread [2])
> > > >>
> > > >>
> > > >>
> > > >> -
> > > >>
> > > >> Invite existing Committers to opt-out or otherwise
> continue
> > > >>
> > > >>
> > > >>
> > > >> -
> > > >>
> > > >> Establish Roadmap via follow up DISCUSS thread
> > > >> -
> > > >>
> > > >> Known Improvements from Forks which will need
> proposals
> > and
> > > >> discussion:
> > > >> -
> > > >>
> > > >> Adding HA for Livy
> > > >> -
> > > >>
> > > >> Updating security capabilities (eg. kerberos for
> > jdbc,
> > > fixing
> > > >> bugs in encryption)
> > > >> -
> > > >>
> > > >> Expanding the support for kubernetes
> > > >> -
> > > >>
> > > >> Responding to CVEs in dependencies (eg. log4j,
> > thrift)
> > > >> -
> > > >>
> > > >> Livy rest cluster - IS THIS SAME AS HA for Livy
> > ABOVE?
> > > >> -
> > > >>
> > > >> Support multi Spark versions
> > > >> -
> > > >>
> > > >> Implemented a metrics system for Livy
> > > >> -
> > > >>
> > > >> Support customize batch/interactive session
> > lifecycle event
> > > >> handler, default log event with log4j, very helpful
> > for
> > > troubleshooting
> > > >> -
> > > >>
> > > >> Optimize log to track which session id the log
> > message came
> > > >> from, also very helpful for troubleshooting
> > > >> -
> > > >>
> > > >> Support customize Spark config optimization rules,
> > can be
> > > used
> > > >> to optimize config for users’ job
> > > >> -
> > > >>
> > > >> A set of command line tool which can be used to
> > replace
> > > Spark’s
> > > >> spark-submit, pyspark, spark-sql but actually
> submit
> > > application in Livy
> > > >> -
> > > >>
> > > >> We are planning to implement a JDBC state store,
> and
> > allow
> > > >> multi Livy Thrift sessions to share one backend
> Spark
> > > application in the
> > > >> next few months.
> > > >> -
> > > >>
> > > >> These items and others that are brought to community
> > may need
> > > >> consolidation or multiple configurable options and
> will
> > need
> > > to be part of
> > > >> the discussion
> > > >> -
> > > >>
> > > >> One-pager Livy Improvement Proposals (LIP) may make
> > sense
> > > to
> > > >> drive these discussions and convergence
> > > >> -
> > > >>
> > > >> Feature Branch Strategy for large changes
> > > >> -
> > > >>
> > > >> Large features are hard to review we will need
> to
> > > define a
> > > >> strategy
> > > >> -
> > > >>
> > > >> Determine the Improvements to be delivered across
> first
> > 3
> > > Releases
> > > >> with Target Release Dates
> > > >>
> > > >>
> > > >>
> > > >> -
> > > >>
> > > >> Ensure CVE and Dependency management hygiene is in place
> > > >>
> > > >>
> > > >> The above approach will usher the community back to an
> active
> > > status with
> > > >> a Roadmap of 3 or more release plans and security hygiene in
> > place.
> > > >> Development Practices
> > > >>
> > > >> The Livy project follows a review before commit philosophy.
> > Every
> > > commit
> > > >> automatically runs through the unit tests and generates
> > coverage
> > > reports
> > > >> presented as a pull request comment. Our experience with
> this
> > > process leads
> > > >> us to believe that it helps ease new contributors into the
> > project.
> > > They
> > > >> get feedback quickly on common mistakes, lowering the burden
> > on
> > > reviewers.
> > > >> Those same reviewers get to lead by example, showing the new
> > > contributors
> > > >> that we value feedback within our community even when
> changes
> > are
> > > done by
> > > >> more experienced folks. Taken from the original Apache Livy
> > > Proposal [1],
> > > >> this should continue to be true. As mentioned, Livy is a
> > mature
> > > project and
> > > >> as such RTC is the most appropriate for continued quality
> and
> > > awareness.
> > > >>
> > > >>
> > > >>
> > > >> 1.
> > > >>
> > > >> Original Apache Livy Proposal
> > > >>
> > > https://cwiki.apache.org/confluence/display/incubator/LivyProposal
> > > >> 2.
> > > >>
> > > >> Retirement DISCUSS thread
> > > >>
> > https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
> > >
> > > >>
> > > >>
> > > >> On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <
> > > madhawa30@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Hi Larry,
> > > >>>
> > > >>> I'm an IPMC Member. madhawa30 at gmail dot com is my
> > preferred
> > > email
> > > >>> address.
> > > >>> apache id: madhawa
> > > >>>
> > > >>> Thanks,
> > > >>> Madhawa
> > > >>>
> > > >>>
> > > >>> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <
> > lmccay@apache.org>
> > > wrote:
> > > >>>
> > > >>> > Hi Madhawa -
> > > >>> >
> > > >>> > That's awesome!
> > > >>> > Are you already a member of IPMC?
> > > >>> > If not, are you an ASF member?
> > > >>> > If you are an ASF member you can request that you be
> added
> > as an
> > > IPMC
> > > >>> > member.
> > > >>> >
> > > >>> > Can you provide your company affiliation for the proposal
> > and
> > > preferred
> > > >>> > email?
> > > >>> >
> > > >>> > thanks!
> > > >>> >
> > > >>> > --larry
> > > >>> >
> > > >>> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <
> > > >>> madhawa30@gmail.com>
> > > >>> > wrote:
> > > >>> >
> > > >>> > > Hi Larry,
> > > >>> > >
> > > >>> > > I'm interested in working with Livy and would like to
> > join as a
> > > >>> Mentor.
> > > >>> > >
> > > >>> > > Thanks,
> > > >>> > > Madhawa
> > > >>> > >
> > > >>> > >
> > > >>> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <
> > lmccay@apache.org
> > > >
> > > >>> wrote:
> > > >>> > >
> > > >>> > > > Sorry, I missed commenting on this:
> > > >>> > > >
> > > >>> > > > "There is also no concept as an emeritus PPMC member
> > at the
> > > ASF."
> > > >>> > > >
> > > >>> > > > I assume that we can remove PPMC members that do not
> > opt-in
> > > >>> explicitly
> > > >>> > at
> > > >>> > > > this point.
> > > >>> > > > They will have every opportunity to rejoin.
> > > >>> > > >
> > > >>> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <
> > > lmccay@apache.org>
> > > >>> > wrote:
> > > >>> > > >
> > > >>> > > > > I will ask in a separate thread, @Justin Mclean <
> > > >>> justinmclean@me.com
> > > >>> > >
> > > >>> > > -
> > > >>> > > > > thanks.
> > > >>> > > > > Adding JB adds another company and we are certainly
> > open to
> > > >>> anyone
> > > >>> > else
> > > >>> > > > > that would like to join as a mentor.
> > > >>> > > > > At the end of the day, the mentors are for
> > instilling the
> > > Apache
> > > >>> Way
> > > >>> > > > > knowledge and steering toward graduation.
> > > >>> > > > > I feel that this diversity, while nice to have, is
> > less
> > > important
> > > >>> > than
> > > >>> > > > > that of the PPMC and committers for the long term
> > health
> > > of the
> > > >>> > > > community.
> > > >>> > > > >
> > > >>> > > > > We need to push this podling to graduation as
> > quickly as
> > > possible
> > > >>> > since
> > > >>> > > > it
> > > >>> > > > > is rather mature and needs to get to the next
> level.
> > > >>> > > > >
> > > >>> > > > > Again, any potential Mentors that would like to
> join
> > are
> > > more
> > > >>> than
> > > >>> > > > welcome.
> > > >>> > > > >
> > > >>> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
> > > >>> > > justin@classsoftware.com
> > > >>> > > > >
> > > >>> > > > > wrote:
> > > >>> > > > >
> > > >>> > > > >> Hi,
> > > >>> > > > >>
> > > >>> > > > >> I’m sorry, but Imran Rashid can’t be a mentor for
> > the
> > > project as
> > > >>> > they
> > > >>> > > > are
> > > >>> > > > >> not an IPMC member. Currently, both Sunil and
> Larry
> > (as
> > > they
> > > >>> are ASF
> > > >>> > > > >> members) need to ask to join the IPMC and NOTICE
> > sent to
> > > the ASF
> > > >>> > > board.
> > > >>> > > > I
> > > >>> > > > >> would also prefer that mentors come from different
> > > companies.
> > > >>> > > > >>
> > > >>> > > > >> There is also no concept as an emeritus PPMC
> member
> > at
> > > the ASF.
> > > >>> > > > >>
> > > >>> > > > >> Kind Regards,
> > > >>> > > > >> Justin
> > > >>> > > > >>
> > > >>> > > > >>
> > > >>> > > > >>
> > > >>> >
> > >
> ---------------------------------------------------------------------
> > > >>> > > > >> To unsubscribe, e-mail:
> > > >>> general-unsubscribe@incubator.apache.org
> > > >>> > > > >> For additional commands, e-mail:
> > > >>> general-help@incubator.apache.org
> > > >>> > > > >>
> > > >>> > > > >>
> > > >>> > > >
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>
> > >
> > >
> > >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> >
> >
>
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
Personally, I would welcome it.
Any notes of guidance and support as we ramp up the team and make the first
few releases would be invaluable.
On Mon, Oct 24, 2022 at 3:49 PM Alex Bozarth <aj...@us.ibm.com> wrote:
> I would love to stay on as a PPMC, but if my lack of availability concerns
> the new PPMC then I'm willing to be emeritus.
>
>
> Alex Bozarth
> Jupyter Architect, IBM CODAIT
> GitHub: ajbozarth
>
> On 10/24/22, 2:46 PM, "larry mccay" <lm...@apache.org> wrote:
>
> Oh, very good, Alex!
> Be sure to let us know if you want to remain on the project in any
> capacity.
>
>
> On Mon, Oct 24, 2022 at 3:18 PM Alex Bozarth <aj...@us.ibm.com>
> wrote:
>
> > As the only Livy PPMC still responding to the Mentors on the private
> list,
> > I have updated the Livy Podling report for November with the status
> of the
> > project and a request to the IPMC to review this proposal since
> there are
> > not enough active Livy PPMC members to reach a quorum to pass the
> proposal.
> >
> > As a current Livy PPMC I strongly support this proposal for
> revitalization
> > as I do not have enough bandwidth to dedicate to Livy.
> >
> >
> > Alex Bozarth
> > Jupyter Architect, IBM CODAIT
> > GitHub: ajbozarth
> >
> > On 10/24/22, 1:41 PM, "larry mccay" <lm...@apache.org> wrote:
> >
> > Gentle reminder that we need to determine next steps here.
> > We have an updated proposal on this thread.
> > Do we need a VOTE or can we move forward directly to adjusting
> the
> > members,
> > etc?
> >
> > Thanks!
> >
> > --larry
> >
> > On Thu, Oct 20, 2022 at 3:26 PM larry mccay <lm...@apache.org>
> wrote:
> >
> > > @Justin Mclean <ju...@me.com> - any insights on next
> steps
> > here?
> > >
> > >
> > > On Tue, Oct 18, 2022 at 5:44 PM larry mccay <lmccay@apache.org
> >
> > wrote:
> > >
> > >> Very good, here is the latest revision with updated Mentors.
> > >> Sunil and I have been added to the IPMC as well.
> > >> Welcome Madhawa and thanks for stepping up as a Mentor for
> Livy!
> > >>
> > >> Abstract
> > >>
> > >> Livy is a web service that exposes a REST interface for
> managing
> > long
> > >> running Apache Spark contexts in your cluster. With Livy, new
> > applications
> > >> can be built on top of Apache Spark that require fine grained
> > interaction
> > >> with many Spark contexts [1].
> > >>
> > >> While this project has been well regarded and used in many
> contexts
> > as
> > >> the defacto standard API to Spark environments, it has been
> > incubating for
> > >> over 5 years without graduation to a TLP and it has become
> > difficult to
> > >> impossible for fixes and improvements to be contributed as the
> > current
> > >> community seems to have moved on.
> > >>
> > >> There has been discussion regarding retirement of this
> podling where
> > >> there seems to be some increasing interest in joining and
> reviving
> > the
> > >> community [2].
> > >>
> > >> The intent of this proposal is to avoid retiring a well
> regarded,
> > >> actively used and rather mature project by reviving the PPMC
> and
> > community
> > >> with new folks that have a vested interest in the project and
> > health of the
> > >> community.
> > >> Proposal
> > >>
> > >> We propose to revive the PPMC with a set of contributors and
> > maintainers
> > >> as mentors, PPMC members and committers.
> > >>
> > >> The retirement DISCUSS thread [2] has shown a growing
> interest in
> > >> providing new committers and bringing improvements and fixes
> from
> > >> organization’s internally maintained forks back to a revived
> > community.
> > >>
> > >> General Approach to Revival:
> > >>
> > >> -
> > >>
> > >> Add new Mentors
> > >> -
> > >>
> > >> Larry McCay, lmccay@apache.org , Cloudera
> > >> -
> > >>
> > >> Sunil Govindan, sunilg@apache.org, Cloudera
> > >> -
> > >>
> > >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
> > >> -
> > >>
> > >> Madhawa Gunasekara, madhawa30@gmail.com, Independent
> > >>
> > >>
> > >>
> > >> -
> > >>
> > >> Add new Committers/PPMC
> > >> -
> > >>
> > >> Larry McCay, lmccay@apache.org, Cloudera
> > >> -
> > >>
> > >> Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
> > >> -
> > >>
> > >> Imran Rashid - irashid@apache.org, Cloudera
> > >> -
> > >>
> > >> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
> > >> -
> > >>
> > >> Wing Yew Poon, wypoon@cloudera.com, Cloudera
> > >> -
> > >>
> > >> Xilang Yan, xilang.yan@gmail.com, Shopee
> > >> -
> > >>
> > >> Jianzhen Wu, myjianzhen@gmail.com, Shopee
> > >> -
> > >>
> > >> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
> > >> -
> > >>
> > >> Pralab Kumar, pralkuma@visa.com, Visa
> > >> -
> > >>
> > >> Prasad Shrikant, shrikant.ism@gmail.com, Visa
> > >> -
> > >>
> > >> Brahma Reddy Battula, brahma@apache.org, Visa
> > >>
> > >>
> > >>
> > >> -
> > >>
> > >> Invite existing PPMC members to opt-in or otherwise go
> emeritus
> > >> -
> > >>
> > >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
> (opted-in
> > via
> > >> Retirement DISCUSS thread [2])
> > >>
> > >>
> > >>
> > >> -
> > >>
> > >> Invite existing Committers to opt-out or otherwise continue
> > >>
> > >>
> > >>
> > >> -
> > >>
> > >> Establish Roadmap via follow up DISCUSS thread
> > >> -
> > >>
> > >> Known Improvements from Forks which will need proposals
> and
> > >> discussion:
> > >> -
> > >>
> > >> Adding HA for Livy
> > >> -
> > >>
> > >> Updating security capabilities (eg. kerberos for
> jdbc,
> > fixing
> > >> bugs in encryption)
> > >> -
> > >>
> > >> Expanding the support for kubernetes
> > >> -
> > >>
> > >> Responding to CVEs in dependencies (eg. log4j,
> thrift)
> > >> -
> > >>
> > >> Livy rest cluster - IS THIS SAME AS HA for Livy
> ABOVE?
> > >> -
> > >>
> > >> Support multi Spark versions
> > >> -
> > >>
> > >> Implemented a metrics system for Livy
> > >> -
> > >>
> > >> Support customize batch/interactive session
> lifecycle event
> > >> handler, default log event with log4j, very helpful
> for
> > troubleshooting
> > >> -
> > >>
> > >> Optimize log to track which session id the log
> message came
> > >> from, also very helpful for troubleshooting
> > >> -
> > >>
> > >> Support customize Spark config optimization rules,
> can be
> > used
> > >> to optimize config for users’ job
> > >> -
> > >>
> > >> A set of command line tool which can be used to
> replace
> > Spark’s
> > >> spark-submit, pyspark, spark-sql but actually submit
> > application in Livy
> > >> -
> > >>
> > >> We are planning to implement a JDBC state store, and
> allow
> > >> multi Livy Thrift sessions to share one backend Spark
> > application in the
> > >> next few months.
> > >> -
> > >>
> > >> These items and others that are brought to community
> may need
> > >> consolidation or multiple configurable options and will
> need
> > to be part of
> > >> the discussion
> > >> -
> > >>
> > >> One-pager Livy Improvement Proposals (LIP) may make
> sense
> > to
> > >> drive these discussions and convergence
> > >> -
> > >>
> > >> Feature Branch Strategy for large changes
> > >> -
> > >>
> > >> Large features are hard to review we will need to
> > define a
> > >> strategy
> > >> -
> > >>
> > >> Determine the Improvements to be delivered across first
> 3
> > Releases
> > >> with Target Release Dates
> > >>
> > >>
> > >>
> > >> -
> > >>
> > >> Ensure CVE and Dependency management hygiene is in place
> > >>
> > >>
> > >> The above approach will usher the community back to an active
> > status with
> > >> a Roadmap of 3 or more release plans and security hygiene in
> place.
> > >> Development Practices
> > >>
> > >> The Livy project follows a review before commit philosophy.
> Every
> > commit
> > >> automatically runs through the unit tests and generates
> coverage
> > reports
> > >> presented as a pull request comment. Our experience with this
> > process leads
> > >> us to believe that it helps ease new contributors into the
> project.
> > They
> > >> get feedback quickly on common mistakes, lowering the burden
> on
> > reviewers.
> > >> Those same reviewers get to lead by example, showing the new
> > contributors
> > >> that we value feedback within our community even when changes
> are
> > done by
> > >> more experienced folks. Taken from the original Apache Livy
> > Proposal [1],
> > >> this should continue to be true. As mentioned, Livy is a
> mature
> > project and
> > >> as such RTC is the most appropriate for continued quality and
> > awareness.
> > >>
> > >>
> > >>
> > >> 1.
> > >>
> > >> Original Apache Livy Proposal
> > >>
> > https://cwiki.apache.org/confluence/display/incubator/LivyProposal
> > >> 2.
> > >>
> > >> Retirement DISCUSS thread
> > >>
> https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
> >
> > >>
> > >>
> > >> On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <
> > madhawa30@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi Larry,
> > >>>
> > >>> I'm an IPMC Member. madhawa30 at gmail dot com is my
> preferred
> > email
> > >>> address.
> > >>> apache id: madhawa
> > >>>
> > >>> Thanks,
> > >>> Madhawa
> > >>>
> > >>>
> > >>> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <
> lmccay@apache.org>
> > wrote:
> > >>>
> > >>> > Hi Madhawa -
> > >>> >
> > >>> > That's awesome!
> > >>> > Are you already a member of IPMC?
> > >>> > If not, are you an ASF member?
> > >>> > If you are an ASF member you can request that you be added
> as an
> > IPMC
> > >>> > member.
> > >>> >
> > >>> > Can you provide your company affiliation for the proposal
> and
> > preferred
> > >>> > email?
> > >>> >
> > >>> > thanks!
> > >>> >
> > >>> > --larry
> > >>> >
> > >>> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <
> > >>> madhawa30@gmail.com>
> > >>> > wrote:
> > >>> >
> > >>> > > Hi Larry,
> > >>> > >
> > >>> > > I'm interested in working with Livy and would like to
> join as a
> > >>> Mentor.
> > >>> > >
> > >>> > > Thanks,
> > >>> > > Madhawa
> > >>> > >
> > >>> > >
> > >>> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <
> lmccay@apache.org
> > >
> > >>> wrote:
> > >>> > >
> > >>> > > > Sorry, I missed commenting on this:
> > >>> > > >
> > >>> > > > "There is also no concept as an emeritus PPMC member
> at the
> > ASF."
> > >>> > > >
> > >>> > > > I assume that we can remove PPMC members that do not
> opt-in
> > >>> explicitly
> > >>> > at
> > >>> > > > this point.
> > >>> > > > They will have every opportunity to rejoin.
> > >>> > > >
> > >>> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <
> > lmccay@apache.org>
> > >>> > wrote:
> > >>> > > >
> > >>> > > > > I will ask in a separate thread, @Justin Mclean <
> > >>> justinmclean@me.com
> > >>> > >
> > >>> > > -
> > >>> > > > > thanks.
> > >>> > > > > Adding JB adds another company and we are certainly
> open to
> > >>> anyone
> > >>> > else
> > >>> > > > > that would like to join as a mentor.
> > >>> > > > > At the end of the day, the mentors are for
> instilling the
> > Apache
> > >>> Way
> > >>> > > > > knowledge and steering toward graduation.
> > >>> > > > > I feel that this diversity, while nice to have, is
> less
> > important
> > >>> > than
> > >>> > > > > that of the PPMC and committers for the long term
> health
> > of the
> > >>> > > > community.
> > >>> > > > >
> > >>> > > > > We need to push this podling to graduation as
> quickly as
> > possible
> > >>> > since
> > >>> > > > it
> > >>> > > > > is rather mature and needs to get to the next level.
> > >>> > > > >
> > >>> > > > > Again, any potential Mentors that would like to join
> are
> > more
> > >>> than
> > >>> > > > welcome.
> > >>> > > > >
> > >>> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
> > >>> > > justin@classsoftware.com
> > >>> > > > >
> > >>> > > > > wrote:
> > >>> > > > >
> > >>> > > > >> Hi,
> > >>> > > > >>
> > >>> > > > >> I’m sorry, but Imran Rashid can’t be a mentor for
> the
> > project as
> > >>> > they
> > >>> > > > are
> > >>> > > > >> not an IPMC member. Currently, both Sunil and Larry
> (as
> > they
> > >>> are ASF
> > >>> > > > >> members) need to ask to join the IPMC and NOTICE
> sent to
> > the ASF
> > >>> > > board.
> > >>> > > > I
> > >>> > > > >> would also prefer that mentors come from different
> > companies.
> > >>> > > > >>
> > >>> > > > >> There is also no concept as an emeritus PPMC member
> at
> > the ASF.
> > >>> > > > >>
> > >>> > > > >> Kind Regards,
> > >>> > > > >> Justin
> > >>> > > > >>
> > >>> > > > >>
> > >>> > > > >>
> > >>> >
> > ---------------------------------------------------------------------
> > >>> > > > >> To unsubscribe, e-mail:
> > >>> general-unsubscribe@incubator.apache.org
> > >>> > > > >> For additional commands, e-mail:
> > >>> general-help@incubator.apache.org
> > >>> > > > >>
> > >>> > > > >>
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
>
>
RE: Proposal to Revive Apache Livy Community
Posted by Alex Bozarth <aj...@us.ibm.com>.
I would love to stay on as a PPMC, but if my lack of availability concerns the new PPMC then I'm willing to be emeritus.
Alex Bozarth
Jupyter Architect, IBM CODAIT
GitHub: ajbozarth
On 10/24/22, 2:46 PM, "larry mccay" <lm...@apache.org> wrote:
Oh, very good, Alex!
Be sure to let us know if you want to remain on the project in any capacity.
On Mon, Oct 24, 2022 at 3:18 PM Alex Bozarth <aj...@us.ibm.com> wrote:
> As the only Livy PPMC still responding to the Mentors on the private list,
> I have updated the Livy Podling report for November with the status of the
> project and a request to the IPMC to review this proposal since there are
> not enough active Livy PPMC members to reach a quorum to pass the proposal.
>
> As a current Livy PPMC I strongly support this proposal for revitalization
> as I do not have enough bandwidth to dedicate to Livy.
>
>
> Alex Bozarth
> Jupyter Architect, IBM CODAIT
> GitHub: ajbozarth
>
> On 10/24/22, 1:41 PM, "larry mccay" <lm...@apache.org> wrote:
>
> Gentle reminder that we need to determine next steps here.
> We have an updated proposal on this thread.
> Do we need a VOTE or can we move forward directly to adjusting the
> members,
> etc?
>
> Thanks!
>
> --larry
>
> On Thu, Oct 20, 2022 at 3:26 PM larry mccay <lm...@apache.org> wrote:
>
> > @Justin Mclean <ju...@me.com> - any insights on next steps
> here?
> >
> >
> > On Tue, Oct 18, 2022 at 5:44 PM larry mccay <lm...@apache.org>
> wrote:
> >
> >> Very good, here is the latest revision with updated Mentors.
> >> Sunil and I have been added to the IPMC as well.
> >> Welcome Madhawa and thanks for stepping up as a Mentor for Livy!
> >>
> >> Abstract
> >>
> >> Livy is a web service that exposes a REST interface for managing
> long
> >> running Apache Spark contexts in your cluster. With Livy, new
> applications
> >> can be built on top of Apache Spark that require fine grained
> interaction
> >> with many Spark contexts [1].
> >>
> >> While this project has been well regarded and used in many contexts
> as
> >> the defacto standard API to Spark environments, it has been
> incubating for
> >> over 5 years without graduation to a TLP and it has become
> difficult to
> >> impossible for fixes and improvements to be contributed as the
> current
> >> community seems to have moved on.
> >>
> >> There has been discussion regarding retirement of this podling where
> >> there seems to be some increasing interest in joining and reviving
> the
> >> community [2].
> >>
> >> The intent of this proposal is to avoid retiring a well regarded,
> >> actively used and rather mature project by reviving the PPMC and
> community
> >> with new folks that have a vested interest in the project and
> health of the
> >> community.
> >> Proposal
> >>
> >> We propose to revive the PPMC with a set of contributors and
> maintainers
> >> as mentors, PPMC members and committers.
> >>
> >> The retirement DISCUSS thread [2] has shown a growing interest in
> >> providing new committers and bringing improvements and fixes from
> >> organization’s internally maintained forks back to a revived
> community.
> >>
> >> General Approach to Revival:
> >>
> >> -
> >>
> >> Add new Mentors
> >> -
> >>
> >> Larry McCay, lmccay@apache.org , Cloudera
> >> -
> >>
> >> Sunil Govindan, sunilg@apache.org, Cloudera
> >> -
> >>
> >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
> >> -
> >>
> >> Madhawa Gunasekara, madhawa30@gmail.com, Independent
> >>
> >>
> >>
> >> -
> >>
> >> Add new Committers/PPMC
> >> -
> >>
> >> Larry McCay, lmccay@apache.org, Cloudera
> >> -
> >>
> >> Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
> >> -
> >>
> >> Imran Rashid - irashid@apache.org, Cloudera
> >> -
> >>
> >> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
> >> -
> >>
> >> Wing Yew Poon, wypoon@cloudera.com, Cloudera
> >> -
> >>
> >> Xilang Yan, xilang.yan@gmail.com, Shopee
> >> -
> >>
> >> Jianzhen Wu, myjianzhen@gmail.com, Shopee
> >> -
> >>
> >> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
> >> -
> >>
> >> Pralab Kumar, pralkuma@visa.com, Visa
> >> -
> >>
> >> Prasad Shrikant, shrikant.ism@gmail.com, Visa
> >> -
> >>
> >> Brahma Reddy Battula, brahma@apache.org, Visa
> >>
> >>
> >>
> >> -
> >>
> >> Invite existing PPMC members to opt-in or otherwise go emeritus
> >> -
> >>
> >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in
> via
> >> Retirement DISCUSS thread [2])
> >>
> >>
> >>
> >> -
> >>
> >> Invite existing Committers to opt-out or otherwise continue
> >>
> >>
> >>
> >> -
> >>
> >> Establish Roadmap via follow up DISCUSS thread
> >> -
> >>
> >> Known Improvements from Forks which will need proposals and
> >> discussion:
> >> -
> >>
> >> Adding HA for Livy
> >> -
> >>
> >> Updating security capabilities (eg. kerberos for jdbc,
> fixing
> >> bugs in encryption)
> >> -
> >>
> >> Expanding the support for kubernetes
> >> -
> >>
> >> Responding to CVEs in dependencies (eg. log4j, thrift)
> >> -
> >>
> >> Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
> >> -
> >>
> >> Support multi Spark versions
> >> -
> >>
> >> Implemented a metrics system for Livy
> >> -
> >>
> >> Support customize batch/interactive session lifecycle event
> >> handler, default log event with log4j, very helpful for
> troubleshooting
> >> -
> >>
> >> Optimize log to track which session id the log message came
> >> from, also very helpful for troubleshooting
> >> -
> >>
> >> Support customize Spark config optimization rules, can be
> used
> >> to optimize config for users’ job
> >> -
> >>
> >> A set of command line tool which can be used to replace
> Spark’s
> >> spark-submit, pyspark, spark-sql but actually submit
> application in Livy
> >> -
> >>
> >> We are planning to implement a JDBC state store, and allow
> >> multi Livy Thrift sessions to share one backend Spark
> application in the
> >> next few months.
> >> -
> >>
> >> These items and others that are brought to community may need
> >> consolidation or multiple configurable options and will need
> to be part of
> >> the discussion
> >> -
> >>
> >> One-pager Livy Improvement Proposals (LIP) may make sense
> to
> >> drive these discussions and convergence
> >> -
> >>
> >> Feature Branch Strategy for large changes
> >> -
> >>
> >> Large features are hard to review we will need to
> define a
> >> strategy
> >> -
> >>
> >> Determine the Improvements to be delivered across first 3
> Releases
> >> with Target Release Dates
> >>
> >>
> >>
> >> -
> >>
> >> Ensure CVE and Dependency management hygiene is in place
> >>
> >>
> >> The above approach will usher the community back to an active
> status with
> >> a Roadmap of 3 or more release plans and security hygiene in place.
> >> Development Practices
> >>
> >> The Livy project follows a review before commit philosophy. Every
> commit
> >> automatically runs through the unit tests and generates coverage
> reports
> >> presented as a pull request comment. Our experience with this
> process leads
> >> us to believe that it helps ease new contributors into the project.
> They
> >> get feedback quickly on common mistakes, lowering the burden on
> reviewers.
> >> Those same reviewers get to lead by example, showing the new
> contributors
> >> that we value feedback within our community even when changes are
> done by
> >> more experienced folks. Taken from the original Apache Livy
> Proposal [1],
> >> this should continue to be true. As mentioned, Livy is a mature
> project and
> >> as such RTC is the most appropriate for continued quality and
> awareness.
> >>
> >>
> >>
> >> 1.
> >>
> >> Original Apache Livy Proposal
> >>
> https://cwiki.apache.org/confluence/display/incubator/LivyProposal
> >> 2.
> >>
> >> Retirement DISCUSS thread
> >> https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
>
> >>
> >>
> >> On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <
> madhawa30@gmail.com>
> >> wrote:
> >>
> >>> Hi Larry,
> >>>
> >>> I'm an IPMC Member. madhawa30 at gmail dot com is my preferred
> email
> >>> address.
> >>> apache id: madhawa
> >>>
> >>> Thanks,
> >>> Madhawa
> >>>
> >>>
> >>> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <lm...@apache.org>
> wrote:
> >>>
> >>> > Hi Madhawa -
> >>> >
> >>> > That's awesome!
> >>> > Are you already a member of IPMC?
> >>> > If not, are you an ASF member?
> >>> > If you are an ASF member you can request that you be added as an
> IPMC
> >>> > member.
> >>> >
> >>> > Can you provide your company affiliation for the proposal and
> preferred
> >>> > email?
> >>> >
> >>> > thanks!
> >>> >
> >>> > --larry
> >>> >
> >>> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <
> >>> madhawa30@gmail.com>
> >>> > wrote:
> >>> >
> >>> > > Hi Larry,
> >>> > >
> >>> > > I'm interested in working with Livy and would like to join as a
> >>> Mentor.
> >>> > >
> >>> > > Thanks,
> >>> > > Madhawa
> >>> > >
> >>> > >
> >>> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <lmccay@apache.org
> >
> >>> wrote:
> >>> > >
> >>> > > > Sorry, I missed commenting on this:
> >>> > > >
> >>> > > > "There is also no concept as an emeritus PPMC member at the
> ASF."
> >>> > > >
> >>> > > > I assume that we can remove PPMC members that do not opt-in
> >>> explicitly
> >>> > at
> >>> > > > this point.
> >>> > > > They will have every opportunity to rejoin.
> >>> > > >
> >>> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <
> lmccay@apache.org>
> >>> > wrote:
> >>> > > >
> >>> > > > > I will ask in a separate thread, @Justin Mclean <
> >>> justinmclean@me.com
> >>> > >
> >>> > > -
> >>> > > > > thanks.
> >>> > > > > Adding JB adds another company and we are certainly open to
> >>> anyone
> >>> > else
> >>> > > > > that would like to join as a mentor.
> >>> > > > > At the end of the day, the mentors are for instilling the
> Apache
> >>> Way
> >>> > > > > knowledge and steering toward graduation.
> >>> > > > > I feel that this diversity, while nice to have, is less
> important
> >>> > than
> >>> > > > > that of the PPMC and committers for the long term health
> of the
> >>> > > > community.
> >>> > > > >
> >>> > > > > We need to push this podling to graduation as quickly as
> possible
> >>> > since
> >>> > > > it
> >>> > > > > is rather mature and needs to get to the next level.
> >>> > > > >
> >>> > > > > Again, any potential Mentors that would like to join are
> more
> >>> than
> >>> > > > welcome.
> >>> > > > >
> >>> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
> >>> > > justin@classsoftware.com
> >>> > > > >
> >>> > > > > wrote:
> >>> > > > >
> >>> > > > >> Hi,
> >>> > > > >>
> >>> > > > >> I’m sorry, but Imran Rashid can’t be a mentor for the
> project as
> >>> > they
> >>> > > > are
> >>> > > > >> not an IPMC member. Currently, both Sunil and Larry (as
> they
> >>> are ASF
> >>> > > > >> members) need to ask to join the IPMC and NOTICE sent to
> the ASF
> >>> > > board.
> >>> > > > I
> >>> > > > >> would also prefer that mentors come from different
> companies.
> >>> > > > >>
> >>> > > > >> There is also no concept as an emeritus PPMC member at
> the ASF.
> >>> > > > >>
> >>> > > > >> Kind Regards,
> >>> > > > >> Justin
> >>> > > > >>
> >>> > > > >>
> >>> > > > >>
> >>> >
> ---------------------------------------------------------------------
> >>> > > > >> To unsubscribe, e-mail:
> >>> general-unsubscribe@incubator.apache.org
> >>> > > > >> For additional commands, e-mail:
> >>> general-help@incubator.apache.org
> >>> > > > >>
> >>> > > > >>
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
Oh, very good, Alex!
Be sure to let us know if you want to remain on the project in any capacity.
On Mon, Oct 24, 2022 at 3:18 PM Alex Bozarth <aj...@us.ibm.com> wrote:
> As the only Livy PPMC still responding to the Mentors on the private list,
> I have updated the Livy Podling report for November with the status of the
> project and a request to the IPMC to review this proposal since there are
> not enough active Livy PPMC members to reach a quorum to pass the proposal.
>
> As a current Livy PPMC I strongly support this proposal for revitalization
> as I do not have enough bandwidth to dedicate to Livy.
>
>
> Alex Bozarth
> Jupyter Architect, IBM CODAIT
> GitHub: ajbozarth
>
> On 10/24/22, 1:41 PM, "larry mccay" <lm...@apache.org> wrote:
>
> Gentle reminder that we need to determine next steps here.
> We have an updated proposal on this thread.
> Do we need a VOTE or can we move forward directly to adjusting the
> members,
> etc?
>
> Thanks!
>
> --larry
>
> On Thu, Oct 20, 2022 at 3:26 PM larry mccay <lm...@apache.org> wrote:
>
> > @Justin Mclean <ju...@me.com> - any insights on next steps
> here?
> >
> >
> > On Tue, Oct 18, 2022 at 5:44 PM larry mccay <lm...@apache.org>
> wrote:
> >
> >> Very good, here is the latest revision with updated Mentors.
> >> Sunil and I have been added to the IPMC as well.
> >> Welcome Madhawa and thanks for stepping up as a Mentor for Livy!
> >>
> >> Abstract
> >>
> >> Livy is a web service that exposes a REST interface for managing
> long
> >> running Apache Spark contexts in your cluster. With Livy, new
> applications
> >> can be built on top of Apache Spark that require fine grained
> interaction
> >> with many Spark contexts [1].
> >>
> >> While this project has been well regarded and used in many contexts
> as
> >> the defacto standard API to Spark environments, it has been
> incubating for
> >> over 5 years without graduation to a TLP and it has become
> difficult to
> >> impossible for fixes and improvements to be contributed as the
> current
> >> community seems to have moved on.
> >>
> >> There has been discussion regarding retirement of this podling where
> >> there seems to be some increasing interest in joining and reviving
> the
> >> community [2].
> >>
> >> The intent of this proposal is to avoid retiring a well regarded,
> >> actively used and rather mature project by reviving the PPMC and
> community
> >> with new folks that have a vested interest in the project and
> health of the
> >> community.
> >> Proposal
> >>
> >> We propose to revive the PPMC with a set of contributors and
> maintainers
> >> as mentors, PPMC members and committers.
> >>
> >> The retirement DISCUSS thread [2] has shown a growing interest in
> >> providing new committers and bringing improvements and fixes from
> >> organization’s internally maintained forks back to a revived
> community.
> >>
> >> General Approach to Revival:
> >>
> >> -
> >>
> >> Add new Mentors
> >> -
> >>
> >> Larry McCay, lmccay@apache.org , Cloudera
> >> -
> >>
> >> Sunil Govindan, sunilg@apache.org, Cloudera
> >> -
> >>
> >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
> >> -
> >>
> >> Madhawa Gunasekara, madhawa30@gmail.com, Independent
> >>
> >>
> >>
> >> -
> >>
> >> Add new Committers/PPMC
> >> -
> >>
> >> Larry McCay, lmccay@apache.org, Cloudera
> >> -
> >>
> >> Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
> >> -
> >>
> >> Imran Rashid - irashid@apache.org, Cloudera
> >> -
> >>
> >> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
> >> -
> >>
> >> Wing Yew Poon, wypoon@cloudera.com, Cloudera
> >> -
> >>
> >> Xilang Yan, xilang.yan@gmail.com, Shopee
> >> -
> >>
> >> Jianzhen Wu, myjianzhen@gmail.com, Shopee
> >> -
> >>
> >> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
> >> -
> >>
> >> Pralab Kumar, pralkuma@visa.com, Visa
> >> -
> >>
> >> Prasad Shrikant, shrikant.ism@gmail.com, Visa
> >> -
> >>
> >> Brahma Reddy Battula, brahma@apache.org, Visa
> >>
> >>
> >>
> >> -
> >>
> >> Invite existing PPMC members to opt-in or otherwise go emeritus
> >> -
> >>
> >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in
> via
> >> Retirement DISCUSS thread [2])
> >>
> >>
> >>
> >> -
> >>
> >> Invite existing Committers to opt-out or otherwise continue
> >>
> >>
> >>
> >> -
> >>
> >> Establish Roadmap via follow up DISCUSS thread
> >> -
> >>
> >> Known Improvements from Forks which will need proposals and
> >> discussion:
> >> -
> >>
> >> Adding HA for Livy
> >> -
> >>
> >> Updating security capabilities (eg. kerberos for jdbc,
> fixing
> >> bugs in encryption)
> >> -
> >>
> >> Expanding the support for kubernetes
> >> -
> >>
> >> Responding to CVEs in dependencies (eg. log4j, thrift)
> >> -
> >>
> >> Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
> >> -
> >>
> >> Support multi Spark versions
> >> -
> >>
> >> Implemented a metrics system for Livy
> >> -
> >>
> >> Support customize batch/interactive session lifecycle event
> >> handler, default log event with log4j, very helpful for
> troubleshooting
> >> -
> >>
> >> Optimize log to track which session id the log message came
> >> from, also very helpful for troubleshooting
> >> -
> >>
> >> Support customize Spark config optimization rules, can be
> used
> >> to optimize config for users’ job
> >> -
> >>
> >> A set of command line tool which can be used to replace
> Spark’s
> >> spark-submit, pyspark, spark-sql but actually submit
> application in Livy
> >> -
> >>
> >> We are planning to implement a JDBC state store, and allow
> >> multi Livy Thrift sessions to share one backend Spark
> application in the
> >> next few months.
> >> -
> >>
> >> These items and others that are brought to community may need
> >> consolidation or multiple configurable options and will need
> to be part of
> >> the discussion
> >> -
> >>
> >> One-pager Livy Improvement Proposals (LIP) may make sense
> to
> >> drive these discussions and convergence
> >> -
> >>
> >> Feature Branch Strategy for large changes
> >> -
> >>
> >> Large features are hard to review we will need to
> define a
> >> strategy
> >> -
> >>
> >> Determine the Improvements to be delivered across first 3
> Releases
> >> with Target Release Dates
> >>
> >>
> >>
> >> -
> >>
> >> Ensure CVE and Dependency management hygiene is in place
> >>
> >>
> >> The above approach will usher the community back to an active
> status with
> >> a Roadmap of 3 or more release plans and security hygiene in place.
> >> Development Practices
> >>
> >> The Livy project follows a review before commit philosophy. Every
> commit
> >> automatically runs through the unit tests and generates coverage
> reports
> >> presented as a pull request comment. Our experience with this
> process leads
> >> us to believe that it helps ease new contributors into the project.
> They
> >> get feedback quickly on common mistakes, lowering the burden on
> reviewers.
> >> Those same reviewers get to lead by example, showing the new
> contributors
> >> that we value feedback within our community even when changes are
> done by
> >> more experienced folks. Taken from the original Apache Livy
> Proposal [1],
> >> this should continue to be true. As mentioned, Livy is a mature
> project and
> >> as such RTC is the most appropriate for continued quality and
> awareness.
> >>
> >>
> >>
> >> 1.
> >>
> >> Original Apache Livy Proposal
> >>
> https://cwiki.apache.org/confluence/display/incubator/LivyProposal
> >> 2.
> >>
> >> Retirement DISCUSS thread
> >> https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
>
> >>
> >>
> >> On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <
> madhawa30@gmail.com>
> >> wrote:
> >>
> >>> Hi Larry,
> >>>
> >>> I'm an IPMC Member. madhawa30 at gmail dot com is my preferred
> email
> >>> address.
> >>> apache id: madhawa
> >>>
> >>> Thanks,
> >>> Madhawa
> >>>
> >>>
> >>> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <lm...@apache.org>
> wrote:
> >>>
> >>> > Hi Madhawa -
> >>> >
> >>> > That's awesome!
> >>> > Are you already a member of IPMC?
> >>> > If not, are you an ASF member?
> >>> > If you are an ASF member you can request that you be added as an
> IPMC
> >>> > member.
> >>> >
> >>> > Can you provide your company affiliation for the proposal and
> preferred
> >>> > email?
> >>> >
> >>> > thanks!
> >>> >
> >>> > --larry
> >>> >
> >>> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <
> >>> madhawa30@gmail.com>
> >>> > wrote:
> >>> >
> >>> > > Hi Larry,
> >>> > >
> >>> > > I'm interested in working with Livy and would like to join as a
> >>> Mentor.
> >>> > >
> >>> > > Thanks,
> >>> > > Madhawa
> >>> > >
> >>> > >
> >>> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <lmccay@apache.org
> >
> >>> wrote:
> >>> > >
> >>> > > > Sorry, I missed commenting on this:
> >>> > > >
> >>> > > > "There is also no concept as an emeritus PPMC member at the
> ASF."
> >>> > > >
> >>> > > > I assume that we can remove PPMC members that do not opt-in
> >>> explicitly
> >>> > at
> >>> > > > this point.
> >>> > > > They will have every opportunity to rejoin.
> >>> > > >
> >>> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <
> lmccay@apache.org>
> >>> > wrote:
> >>> > > >
> >>> > > > > I will ask in a separate thread, @Justin Mclean <
> >>> justinmclean@me.com
> >>> > >
> >>> > > -
> >>> > > > > thanks.
> >>> > > > > Adding JB adds another company and we are certainly open to
> >>> anyone
> >>> > else
> >>> > > > > that would like to join as a mentor.
> >>> > > > > At the end of the day, the mentors are for instilling the
> Apache
> >>> Way
> >>> > > > > knowledge and steering toward graduation.
> >>> > > > > I feel that this diversity, while nice to have, is less
> important
> >>> > than
> >>> > > > > that of the PPMC and committers for the long term health
> of the
> >>> > > > community.
> >>> > > > >
> >>> > > > > We need to push this podling to graduation as quickly as
> possible
> >>> > since
> >>> > > > it
> >>> > > > > is rather mature and needs to get to the next level.
> >>> > > > >
> >>> > > > > Again, any potential Mentors that would like to join are
> more
> >>> than
> >>> > > > welcome.
> >>> > > > >
> >>> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
> >>> > > justin@classsoftware.com
> >>> > > > >
> >>> > > > > wrote:
> >>> > > > >
> >>> > > > >> Hi,
> >>> > > > >>
> >>> > > > >> I’m sorry, but Imran Rashid can’t be a mentor for the
> project as
> >>> > they
> >>> > > > are
> >>> > > > >> not an IPMC member. Currently, both Sunil and Larry (as
> they
> >>> are ASF
> >>> > > > >> members) need to ask to join the IPMC and NOTICE sent to
> the ASF
> >>> > > board.
> >>> > > > I
> >>> > > > >> would also prefer that mentors come from different
> companies.
> >>> > > > >>
> >>> > > > >> There is also no concept as an emeritus PPMC member at
> the ASF.
> >>> > > > >>
> >>> > > > >> Kind Regards,
> >>> > > > >> Justin
> >>> > > > >>
> >>> > > > >>
> >>> > > > >>
> >>> >
> ---------------------------------------------------------------------
> >>> > > > >> To unsubscribe, e-mail:
> >>> general-unsubscribe@incubator.apache.org
> >>> > > > >> For additional commands, e-mail:
> >>> general-help@incubator.apache.org
> >>> > > > >>
> >>> > > > >>
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
Oh, very good, Alex!
Be sure to let us know if you want to remain on the project in any capacity.
On Mon, Oct 24, 2022 at 3:18 PM Alex Bozarth <aj...@us.ibm.com> wrote:
> As the only Livy PPMC still responding to the Mentors on the private list,
> I have updated the Livy Podling report for November with the status of the
> project and a request to the IPMC to review this proposal since there are
> not enough active Livy PPMC members to reach a quorum to pass the proposal.
>
> As a current Livy PPMC I strongly support this proposal for revitalization
> as I do not have enough bandwidth to dedicate to Livy.
>
>
> Alex Bozarth
> Jupyter Architect, IBM CODAIT
> GitHub: ajbozarth
>
> On 10/24/22, 1:41 PM, "larry mccay" <lm...@apache.org> wrote:
>
> Gentle reminder that we need to determine next steps here.
> We have an updated proposal on this thread.
> Do we need a VOTE or can we move forward directly to adjusting the
> members,
> etc?
>
> Thanks!
>
> --larry
>
> On Thu, Oct 20, 2022 at 3:26 PM larry mccay <lm...@apache.org> wrote:
>
> > @Justin Mclean <ju...@me.com> - any insights on next steps
> here?
> >
> >
> > On Tue, Oct 18, 2022 at 5:44 PM larry mccay <lm...@apache.org>
> wrote:
> >
> >> Very good, here is the latest revision with updated Mentors.
> >> Sunil and I have been added to the IPMC as well.
> >> Welcome Madhawa and thanks for stepping up as a Mentor for Livy!
> >>
> >> Abstract
> >>
> >> Livy is a web service that exposes a REST interface for managing
> long
> >> running Apache Spark contexts in your cluster. With Livy, new
> applications
> >> can be built on top of Apache Spark that require fine grained
> interaction
> >> with many Spark contexts [1].
> >>
> >> While this project has been well regarded and used in many contexts
> as
> >> the defacto standard API to Spark environments, it has been
> incubating for
> >> over 5 years without graduation to a TLP and it has become
> difficult to
> >> impossible for fixes and improvements to be contributed as the
> current
> >> community seems to have moved on.
> >>
> >> There has been discussion regarding retirement of this podling where
> >> there seems to be some increasing interest in joining and reviving
> the
> >> community [2].
> >>
> >> The intent of this proposal is to avoid retiring a well regarded,
> >> actively used and rather mature project by reviving the PPMC and
> community
> >> with new folks that have a vested interest in the project and
> health of the
> >> community.
> >> Proposal
> >>
> >> We propose to revive the PPMC with a set of contributors and
> maintainers
> >> as mentors, PPMC members and committers.
> >>
> >> The retirement DISCUSS thread [2] has shown a growing interest in
> >> providing new committers and bringing improvements and fixes from
> >> organization’s internally maintained forks back to a revived
> community.
> >>
> >> General Approach to Revival:
> >>
> >> -
> >>
> >> Add new Mentors
> >> -
> >>
> >> Larry McCay, lmccay@apache.org , Cloudera
> >> -
> >>
> >> Sunil Govindan, sunilg@apache.org, Cloudera
> >> -
> >>
> >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
> >> -
> >>
> >> Madhawa Gunasekara, madhawa30@gmail.com, Independent
> >>
> >>
> >>
> >> -
> >>
> >> Add new Committers/PPMC
> >> -
> >>
> >> Larry McCay, lmccay@apache.org, Cloudera
> >> -
> >>
> >> Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
> >> -
> >>
> >> Imran Rashid - irashid@apache.org, Cloudera
> >> -
> >>
> >> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
> >> -
> >>
> >> Wing Yew Poon, wypoon@cloudera.com, Cloudera
> >> -
> >>
> >> Xilang Yan, xilang.yan@gmail.com, Shopee
> >> -
> >>
> >> Jianzhen Wu, myjianzhen@gmail.com, Shopee
> >> -
> >>
> >> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
> >> -
> >>
> >> Pralab Kumar, pralkuma@visa.com, Visa
> >> -
> >>
> >> Prasad Shrikant, shrikant.ism@gmail.com, Visa
> >> -
> >>
> >> Brahma Reddy Battula, brahma@apache.org, Visa
> >>
> >>
> >>
> >> -
> >>
> >> Invite existing PPMC members to opt-in or otherwise go emeritus
> >> -
> >>
> >> Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in
> via
> >> Retirement DISCUSS thread [2])
> >>
> >>
> >>
> >> -
> >>
> >> Invite existing Committers to opt-out or otherwise continue
> >>
> >>
> >>
> >> -
> >>
> >> Establish Roadmap via follow up DISCUSS thread
> >> -
> >>
> >> Known Improvements from Forks which will need proposals and
> >> discussion:
> >> -
> >>
> >> Adding HA for Livy
> >> -
> >>
> >> Updating security capabilities (eg. kerberos for jdbc,
> fixing
> >> bugs in encryption)
> >> -
> >>
> >> Expanding the support for kubernetes
> >> -
> >>
> >> Responding to CVEs in dependencies (eg. log4j, thrift)
> >> -
> >>
> >> Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
> >> -
> >>
> >> Support multi Spark versions
> >> -
> >>
> >> Implemented a metrics system for Livy
> >> -
> >>
> >> Support customize batch/interactive session lifecycle event
> >> handler, default log event with log4j, very helpful for
> troubleshooting
> >> -
> >>
> >> Optimize log to track which session id the log message came
> >> from, also very helpful for troubleshooting
> >> -
> >>
> >> Support customize Spark config optimization rules, can be
> used
> >> to optimize config for users’ job
> >> -
> >>
> >> A set of command line tool which can be used to replace
> Spark’s
> >> spark-submit, pyspark, spark-sql but actually submit
> application in Livy
> >> -
> >>
> >> We are planning to implement a JDBC state store, and allow
> >> multi Livy Thrift sessions to share one backend Spark
> application in the
> >> next few months.
> >> -
> >>
> >> These items and others that are brought to community may need
> >> consolidation or multiple configurable options and will need
> to be part of
> >> the discussion
> >> -
> >>
> >> One-pager Livy Improvement Proposals (LIP) may make sense
> to
> >> drive these discussions and convergence
> >> -
> >>
> >> Feature Branch Strategy for large changes
> >> -
> >>
> >> Large features are hard to review we will need to
> define a
> >> strategy
> >> -
> >>
> >> Determine the Improvements to be delivered across first 3
> Releases
> >> with Target Release Dates
> >>
> >>
> >>
> >> -
> >>
> >> Ensure CVE and Dependency management hygiene is in place
> >>
> >>
> >> The above approach will usher the community back to an active
> status with
> >> a Roadmap of 3 or more release plans and security hygiene in place.
> >> Development Practices
> >>
> >> The Livy project follows a review before commit philosophy. Every
> commit
> >> automatically runs through the unit tests and generates coverage
> reports
> >> presented as a pull request comment. Our experience with this
> process leads
> >> us to believe that it helps ease new contributors into the project.
> They
> >> get feedback quickly on common mistakes, lowering the burden on
> reviewers.
> >> Those same reviewers get to lead by example, showing the new
> contributors
> >> that we value feedback within our community even when changes are
> done by
> >> more experienced folks. Taken from the original Apache Livy
> Proposal [1],
> >> this should continue to be true. As mentioned, Livy is a mature
> project and
> >> as such RTC is the most appropriate for continued quality and
> awareness.
> >>
> >>
> >>
> >> 1.
> >>
> >> Original Apache Livy Proposal
> >>
> https://cwiki.apache.org/confluence/display/incubator/LivyProposal
> >> 2.
> >>
> >> Retirement DISCUSS thread
> >> https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
>
> >>
> >>
> >> On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <
> madhawa30@gmail.com>
> >> wrote:
> >>
> >>> Hi Larry,
> >>>
> >>> I'm an IPMC Member. madhawa30 at gmail dot com is my preferred
> email
> >>> address.
> >>> apache id: madhawa
> >>>
> >>> Thanks,
> >>> Madhawa
> >>>
> >>>
> >>> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <lm...@apache.org>
> wrote:
> >>>
> >>> > Hi Madhawa -
> >>> >
> >>> > That's awesome!
> >>> > Are you already a member of IPMC?
> >>> > If not, are you an ASF member?
> >>> > If you are an ASF member you can request that you be added as an
> IPMC
> >>> > member.
> >>> >
> >>> > Can you provide your company affiliation for the proposal and
> preferred
> >>> > email?
> >>> >
> >>> > thanks!
> >>> >
> >>> > --larry
> >>> >
> >>> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <
> >>> madhawa30@gmail.com>
> >>> > wrote:
> >>> >
> >>> > > Hi Larry,
> >>> > >
> >>> > > I'm interested in working with Livy and would like to join as a
> >>> Mentor.
> >>> > >
> >>> > > Thanks,
> >>> > > Madhawa
> >>> > >
> >>> > >
> >>> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <lmccay@apache.org
> >
> >>> wrote:
> >>> > >
> >>> > > > Sorry, I missed commenting on this:
> >>> > > >
> >>> > > > "There is also no concept as an emeritus PPMC member at the
> ASF."
> >>> > > >
> >>> > > > I assume that we can remove PPMC members that do not opt-in
> >>> explicitly
> >>> > at
> >>> > > > this point.
> >>> > > > They will have every opportunity to rejoin.
> >>> > > >
> >>> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <
> lmccay@apache.org>
> >>> > wrote:
> >>> > > >
> >>> > > > > I will ask in a separate thread, @Justin Mclean <
> >>> justinmclean@me.com
> >>> > >
> >>> > > -
> >>> > > > > thanks.
> >>> > > > > Adding JB adds another company and we are certainly open to
> >>> anyone
> >>> > else
> >>> > > > > that would like to join as a mentor.
> >>> > > > > At the end of the day, the mentors are for instilling the
> Apache
> >>> Way
> >>> > > > > knowledge and steering toward graduation.
> >>> > > > > I feel that this diversity, while nice to have, is less
> important
> >>> > than
> >>> > > > > that of the PPMC and committers for the long term health
> of the
> >>> > > > community.
> >>> > > > >
> >>> > > > > We need to push this podling to graduation as quickly as
> possible
> >>> > since
> >>> > > > it
> >>> > > > > is rather mature and needs to get to the next level.
> >>> > > > >
> >>> > > > > Again, any potential Mentors that would like to join are
> more
> >>> than
> >>> > > > welcome.
> >>> > > > >
> >>> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
> >>> > > justin@classsoftware.com
> >>> > > > >
> >>> > > > > wrote:
> >>> > > > >
> >>> > > > >> Hi,
> >>> > > > >>
> >>> > > > >> I’m sorry, but Imran Rashid can’t be a mentor for the
> project as
> >>> > they
> >>> > > > are
> >>> > > > >> not an IPMC member. Currently, both Sunil and Larry (as
> they
> >>> are ASF
> >>> > > > >> members) need to ask to join the IPMC and NOTICE sent to
> the ASF
> >>> > > board.
> >>> > > > I
> >>> > > > >> would also prefer that mentors come from different
> companies.
> >>> > > > >>
> >>> > > > >> There is also no concept as an emeritus PPMC member at
> the ASF.
> >>> > > > >>
> >>> > > > >> Kind Regards,
> >>> > > > >> Justin
> >>> > > > >>
> >>> > > > >>
> >>> > > > >>
> >>> >
> ---------------------------------------------------------------------
> >>> > > > >> To unsubscribe, e-mail:
> >>> general-unsubscribe@incubator.apache.org
> >>> > > > >> For additional commands, e-mail:
> >>> general-help@incubator.apache.org
> >>> > > > >>
> >>> > > > >>
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
RE: Proposal to Revive Apache Livy Community
Posted by Alex Bozarth <aj...@us.ibm.com>.
As the only Livy PPMC still responding to the Mentors on the private list, I have updated the Livy Podling report for November with the status of the project and a request to the IPMC to review this proposal since there are not enough active Livy PPMC members to reach a quorum to pass the proposal.
As a current Livy PPMC I strongly support this proposal for revitalization as I do not have enough bandwidth to dedicate to Livy.
Alex Bozarth
Jupyter Architect, IBM CODAIT
GitHub: ajbozarth
On 10/24/22, 1:41 PM, "larry mccay" <lm...@apache.org> wrote:
Gentle reminder that we need to determine next steps here.
We have an updated proposal on this thread.
Do we need a VOTE or can we move forward directly to adjusting the members,
etc?
Thanks!
--larry
On Thu, Oct 20, 2022 at 3:26 PM larry mccay <lm...@apache.org> wrote:
> @Justin Mclean <ju...@me.com> - any insights on next steps here?
>
>
> On Tue, Oct 18, 2022 at 5:44 PM larry mccay <lm...@apache.org> wrote:
>
>> Very good, here is the latest revision with updated Mentors.
>> Sunil and I have been added to the IPMC as well.
>> Welcome Madhawa and thanks for stepping up as a Mentor for Livy!
>>
>> Abstract
>>
>> Livy is a web service that exposes a REST interface for managing long
>> running Apache Spark contexts in your cluster. With Livy, new applications
>> can be built on top of Apache Spark that require fine grained interaction
>> with many Spark contexts [1].
>>
>> While this project has been well regarded and used in many contexts as
>> the defacto standard API to Spark environments, it has been incubating for
>> over 5 years without graduation to a TLP and it has become difficult to
>> impossible for fixes and improvements to be contributed as the current
>> community seems to have moved on.
>>
>> There has been discussion regarding retirement of this podling where
>> there seems to be some increasing interest in joining and reviving the
>> community [2].
>>
>> The intent of this proposal is to avoid retiring a well regarded,
>> actively used and rather mature project by reviving the PPMC and community
>> with new folks that have a vested interest in the project and health of the
>> community.
>> Proposal
>>
>> We propose to revive the PPMC with a set of contributors and maintainers
>> as mentors, PPMC members and committers.
>>
>> The retirement DISCUSS thread [2] has shown a growing interest in
>> providing new committers and bringing improvements and fixes from
>> organization’s internally maintained forks back to a revived community.
>>
>> General Approach to Revival:
>>
>> -
>>
>> Add new Mentors
>> -
>>
>> Larry McCay, lmccay@apache.org , Cloudera
>> -
>>
>> Sunil Govindan, sunilg@apache.org, Cloudera
>> -
>>
>> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
>> -
>>
>> Madhawa Gunasekara, madhawa30@gmail.com, Independent
>>
>>
>>
>> -
>>
>> Add new Committers/PPMC
>> -
>>
>> Larry McCay, lmccay@apache.org, Cloudera
>> -
>>
>> Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
>> -
>>
>> Imran Rashid - irashid@apache.org, Cloudera
>> -
>>
>> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
>> -
>>
>> Wing Yew Poon, wypoon@cloudera.com, Cloudera
>> -
>>
>> Xilang Yan, xilang.yan@gmail.com, Shopee
>> -
>>
>> Jianzhen Wu, myjianzhen@gmail.com, Shopee
>> -
>>
>> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
>> -
>>
>> Pralab Kumar, pralkuma@visa.com, Visa
>> -
>>
>> Prasad Shrikant, shrikant.ism@gmail.com, Visa
>> -
>>
>> Brahma Reddy Battula, brahma@apache.org, Visa
>>
>>
>>
>> -
>>
>> Invite existing PPMC members to opt-in or otherwise go emeritus
>> -
>>
>> Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in via
>> Retirement DISCUSS thread [2])
>>
>>
>>
>> -
>>
>> Invite existing Committers to opt-out or otherwise continue
>>
>>
>>
>> -
>>
>> Establish Roadmap via follow up DISCUSS thread
>> -
>>
>> Known Improvements from Forks which will need proposals and
>> discussion:
>> -
>>
>> Adding HA for Livy
>> -
>>
>> Updating security capabilities (eg. kerberos for jdbc, fixing
>> bugs in encryption)
>> -
>>
>> Expanding the support for kubernetes
>> -
>>
>> Responding to CVEs in dependencies (eg. log4j, thrift)
>> -
>>
>> Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
>> -
>>
>> Support multi Spark versions
>> -
>>
>> Implemented a metrics system for Livy
>> -
>>
>> Support customize batch/interactive session lifecycle event
>> handler, default log event with log4j, very helpful for troubleshooting
>> -
>>
>> Optimize log to track which session id the log message came
>> from, also very helpful for troubleshooting
>> -
>>
>> Support customize Spark config optimization rules, can be used
>> to optimize config for users’ job
>> -
>>
>> A set of command line tool which can be used to replace Spark’s
>> spark-submit, pyspark, spark-sql but actually submit application in Livy
>> -
>>
>> We are planning to implement a JDBC state store, and allow
>> multi Livy Thrift sessions to share one backend Spark application in the
>> next few months.
>> -
>>
>> These items and others that are brought to community may need
>> consolidation or multiple configurable options and will need to be part of
>> the discussion
>> -
>>
>> One-pager Livy Improvement Proposals (LIP) may make sense to
>> drive these discussions and convergence
>> -
>>
>> Feature Branch Strategy for large changes
>> -
>>
>> Large features are hard to review we will need to define a
>> strategy
>> -
>>
>> Determine the Improvements to be delivered across first 3 Releases
>> with Target Release Dates
>>
>>
>>
>> -
>>
>> Ensure CVE and Dependency management hygiene is in place
>>
>>
>> The above approach will usher the community back to an active status with
>> a Roadmap of 3 or more release plans and security hygiene in place.
>> Development Practices
>>
>> The Livy project follows a review before commit philosophy. Every commit
>> automatically runs through the unit tests and generates coverage reports
>> presented as a pull request comment. Our experience with this process leads
>> us to believe that it helps ease new contributors into the project. They
>> get feedback quickly on common mistakes, lowering the burden on reviewers.
>> Those same reviewers get to lead by example, showing the new contributors
>> that we value feedback within our community even when changes are done by
>> more experienced folks. Taken from the original Apache Livy Proposal [1],
>> this should continue to be true. As mentioned, Livy is a mature project and
>> as such RTC is the most appropriate for continued quality and awareness.
>>
>>
>>
>> 1.
>>
>> Original Apache Livy Proposal
>> https://cwiki.apache.org/confluence/display/incubator/LivyProposal
>> 2.
>>
>> Retirement DISCUSS thread
>> https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
>>
>>
>> On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <ma...@gmail.com>
>> wrote:
>>
>>> Hi Larry,
>>>
>>> I'm an IPMC Member. madhawa30 at gmail dot com is my preferred email
>>> address.
>>> apache id: madhawa
>>>
>>> Thanks,
>>> Madhawa
>>>
>>>
>>> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <lm...@apache.org> wrote:
>>>
>>> > Hi Madhawa -
>>> >
>>> > That's awesome!
>>> > Are you already a member of IPMC?
>>> > If not, are you an ASF member?
>>> > If you are an ASF member you can request that you be added as an IPMC
>>> > member.
>>> >
>>> > Can you provide your company affiliation for the proposal and preferred
>>> > email?
>>> >
>>> > thanks!
>>> >
>>> > --larry
>>> >
>>> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <
>>> madhawa30@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi Larry,
>>> > >
>>> > > I'm interested in working with Livy and would like to join as a
>>> Mentor.
>>> > >
>>> > > Thanks,
>>> > > Madhawa
>>> > >
>>> > >
>>> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <lm...@apache.org>
>>> wrote:
>>> > >
>>> > > > Sorry, I missed commenting on this:
>>> > > >
>>> > > > "There is also no concept as an emeritus PPMC member at the ASF."
>>> > > >
>>> > > > I assume that we can remove PPMC members that do not opt-in
>>> explicitly
>>> > at
>>> > > > this point.
>>> > > > They will have every opportunity to rejoin.
>>> > > >
>>> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <lm...@apache.org>
>>> > wrote:
>>> > > >
>>> > > > > I will ask in a separate thread, @Justin Mclean <
>>> justinmclean@me.com
>>> > >
>>> > > -
>>> > > > > thanks.
>>> > > > > Adding JB adds another company and we are certainly open to
>>> anyone
>>> > else
>>> > > > > that would like to join as a mentor.
>>> > > > > At the end of the day, the mentors are for instilling the Apache
>>> Way
>>> > > > > knowledge and steering toward graduation.
>>> > > > > I feel that this diversity, while nice to have, is less important
>>> > than
>>> > > > > that of the PPMC and committers for the long term health of the
>>> > > > community.
>>> > > > >
>>> > > > > We need to push this podling to graduation as quickly as possible
>>> > since
>>> > > > it
>>> > > > > is rather mature and needs to get to the next level.
>>> > > > >
>>> > > > > Again, any potential Mentors that would like to join are more
>>> than
>>> > > > welcome.
>>> > > > >
>>> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
>>> > > justin@classsoftware.com
>>> > > > >
>>> > > > > wrote:
>>> > > > >
>>> > > > >> Hi,
>>> > > > >>
>>> > > > >> I’m sorry, but Imran Rashid can’t be a mentor for the project as
>>> > they
>>> > > > are
>>> > > > >> not an IPMC member. Currently, both Sunil and Larry (as they
>>> are ASF
>>> > > > >> members) need to ask to join the IPMC and NOTICE sent to the ASF
>>> > > board.
>>> > > > I
>>> > > > >> would also prefer that mentors come from different companies.
>>> > > > >>
>>> > > > >> There is also no concept as an emeritus PPMC member at the ASF.
>>> > > > >>
>>> > > > >> Kind Regards,
>>> > > > >> Justin
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > ---------------------------------------------------------------------
>>> > > > >> To unsubscribe, e-mail:
>>> general-unsubscribe@incubator.apache.org
>>> > > > >> For additional commands, e-mail:
>>> general-help@incubator.apache.org
>>> > > > >>
>>> > > > >>
>>> > > >
>>> > >
>>> >
>>>
>>
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
Great, I'll move that forward.
That sounds like a NOTICE thread?
On Mon, Oct 24, 2022 at 4:16 PM Justin Mclean <ju...@classsoftware.com>
wrote:
> Hi,
>
> Usually, the PPMC would need to vote on adding new PPMC members, but it
> seems Livy no longer has 3 active PPMC members. The chair of the project
> can add PMC members. But PPMC members only have standing in the Incubator,
> so I would guess (as it isn’t documented) that any IPMC could add those new
> PPMC members.
>
> A VOTE is probably not required but is helpful to confirm consensus. I
> would write an email to the general list containing who will be added to
> the Livy PPMC. iI there are no objections by any IPMC members after 72
> hours add them.
>
> Kind Regards,
> Justin
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: Proposal to Revive Apache Livy Community
Posted by Justin Mclean <ju...@classsoftware.com>.
Hi,
Usually, the PPMC would need to vote on adding new PPMC members, but it seems Livy no longer has 3 active PPMC members. The chair of the project can add PMC members. But PPMC members only have standing in the Incubator, so I would guess (as it isn’t documented) that any IPMC could add those new PPMC members.
A VOTE is probably not required but is helpful to confirm consensus. I would write an email to the general list containing who will be added to the Livy PPMC. iI there are no objections by any IPMC members after 72 hours add them.
Kind Regards,
Justin
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
RE: Proposal to Revive Apache Livy Community
Posted by Alex Bozarth <aj...@us.ibm.com>.
As the only Livy PPMC still responding to the Mentors on the private list, I have updated the Livy Podling report for November with the status of the project and a request to the IPMC to review this proposal since there are not enough active Livy PPMC members to reach a quorum to pass the proposal.
As a current Livy PPMC I strongly support this proposal for revitalization as I do not have enough bandwidth to dedicate to Livy.
Alex Bozarth
Jupyter Architect, IBM CODAIT
GitHub: ajbozarth
On 10/24/22, 1:41 PM, "larry mccay" <lm...@apache.org> wrote:
Gentle reminder that we need to determine next steps here.
We have an updated proposal on this thread.
Do we need a VOTE or can we move forward directly to adjusting the members,
etc?
Thanks!
--larry
On Thu, Oct 20, 2022 at 3:26 PM larry mccay <lm...@apache.org> wrote:
> @Justin Mclean <ju...@me.com> - any insights on next steps here?
>
>
> On Tue, Oct 18, 2022 at 5:44 PM larry mccay <lm...@apache.org> wrote:
>
>> Very good, here is the latest revision with updated Mentors.
>> Sunil and I have been added to the IPMC as well.
>> Welcome Madhawa and thanks for stepping up as a Mentor for Livy!
>>
>> Abstract
>>
>> Livy is a web service that exposes a REST interface for managing long
>> running Apache Spark contexts in your cluster. With Livy, new applications
>> can be built on top of Apache Spark that require fine grained interaction
>> with many Spark contexts [1].
>>
>> While this project has been well regarded and used in many contexts as
>> the defacto standard API to Spark environments, it has been incubating for
>> over 5 years without graduation to a TLP and it has become difficult to
>> impossible for fixes and improvements to be contributed as the current
>> community seems to have moved on.
>>
>> There has been discussion regarding retirement of this podling where
>> there seems to be some increasing interest in joining and reviving the
>> community [2].
>>
>> The intent of this proposal is to avoid retiring a well regarded,
>> actively used and rather mature project by reviving the PPMC and community
>> with new folks that have a vested interest in the project and health of the
>> community.
>> Proposal
>>
>> We propose to revive the PPMC with a set of contributors and maintainers
>> as mentors, PPMC members and committers.
>>
>> The retirement DISCUSS thread [2] has shown a growing interest in
>> providing new committers and bringing improvements and fixes from
>> organization’s internally maintained forks back to a revived community.
>>
>> General Approach to Revival:
>>
>> -
>>
>> Add new Mentors
>> -
>>
>> Larry McCay, lmccay@apache.org , Cloudera
>> -
>>
>> Sunil Govindan, sunilg@apache.org, Cloudera
>> -
>>
>> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
>> -
>>
>> Madhawa Gunasekara, madhawa30@gmail.com, Independent
>>
>>
>>
>> -
>>
>> Add new Committers/PPMC
>> -
>>
>> Larry McCay, lmccay@apache.org, Cloudera
>> -
>>
>> Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
>> -
>>
>> Imran Rashid - irashid@apache.org, Cloudera
>> -
>>
>> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
>> -
>>
>> Wing Yew Poon, wypoon@cloudera.com, Cloudera
>> -
>>
>> Xilang Yan, xilang.yan@gmail.com, Shopee
>> -
>>
>> Jianzhen Wu, myjianzhen@gmail.com, Shopee
>> -
>>
>> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
>> -
>>
>> Pralab Kumar, pralkuma@visa.com, Visa
>> -
>>
>> Prasad Shrikant, shrikant.ism@gmail.com, Visa
>> -
>>
>> Brahma Reddy Battula, brahma@apache.org, Visa
>>
>>
>>
>> -
>>
>> Invite existing PPMC members to opt-in or otherwise go emeritus
>> -
>>
>> Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in via
>> Retirement DISCUSS thread [2])
>>
>>
>>
>> -
>>
>> Invite existing Committers to opt-out or otherwise continue
>>
>>
>>
>> -
>>
>> Establish Roadmap via follow up DISCUSS thread
>> -
>>
>> Known Improvements from Forks which will need proposals and
>> discussion:
>> -
>>
>> Adding HA for Livy
>> -
>>
>> Updating security capabilities (eg. kerberos for jdbc, fixing
>> bugs in encryption)
>> -
>>
>> Expanding the support for kubernetes
>> -
>>
>> Responding to CVEs in dependencies (eg. log4j, thrift)
>> -
>>
>> Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
>> -
>>
>> Support multi Spark versions
>> -
>>
>> Implemented a metrics system for Livy
>> -
>>
>> Support customize batch/interactive session lifecycle event
>> handler, default log event with log4j, very helpful for troubleshooting
>> -
>>
>> Optimize log to track which session id the log message came
>> from, also very helpful for troubleshooting
>> -
>>
>> Support customize Spark config optimization rules, can be used
>> to optimize config for users’ job
>> -
>>
>> A set of command line tool which can be used to replace Spark’s
>> spark-submit, pyspark, spark-sql but actually submit application in Livy
>> -
>>
>> We are planning to implement a JDBC state store, and allow
>> multi Livy Thrift sessions to share one backend Spark application in the
>> next few months.
>> -
>>
>> These items and others that are brought to community may need
>> consolidation or multiple configurable options and will need to be part of
>> the discussion
>> -
>>
>> One-pager Livy Improvement Proposals (LIP) may make sense to
>> drive these discussions and convergence
>> -
>>
>> Feature Branch Strategy for large changes
>> -
>>
>> Large features are hard to review we will need to define a
>> strategy
>> -
>>
>> Determine the Improvements to be delivered across first 3 Releases
>> with Target Release Dates
>>
>>
>>
>> -
>>
>> Ensure CVE and Dependency management hygiene is in place
>>
>>
>> The above approach will usher the community back to an active status with
>> a Roadmap of 3 or more release plans and security hygiene in place.
>> Development Practices
>>
>> The Livy project follows a review before commit philosophy. Every commit
>> automatically runs through the unit tests and generates coverage reports
>> presented as a pull request comment. Our experience with this process leads
>> us to believe that it helps ease new contributors into the project. They
>> get feedback quickly on common mistakes, lowering the burden on reviewers.
>> Those same reviewers get to lead by example, showing the new contributors
>> that we value feedback within our community even when changes are done by
>> more experienced folks. Taken from the original Apache Livy Proposal [1],
>> this should continue to be true. As mentioned, Livy is a mature project and
>> as such RTC is the most appropriate for continued quality and awareness.
>>
>>
>>
>> 1.
>>
>> Original Apache Livy Proposal
>> https://cwiki.apache.org/confluence/display/incubator/LivyProposal
>> 2.
>>
>> Retirement DISCUSS thread
>> https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
>>
>>
>> On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <ma...@gmail.com>
>> wrote:
>>
>>> Hi Larry,
>>>
>>> I'm an IPMC Member. madhawa30 at gmail dot com is my preferred email
>>> address.
>>> apache id: madhawa
>>>
>>> Thanks,
>>> Madhawa
>>>
>>>
>>> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <lm...@apache.org> wrote:
>>>
>>> > Hi Madhawa -
>>> >
>>> > That's awesome!
>>> > Are you already a member of IPMC?
>>> > If not, are you an ASF member?
>>> > If you are an ASF member you can request that you be added as an IPMC
>>> > member.
>>> >
>>> > Can you provide your company affiliation for the proposal and preferred
>>> > email?
>>> >
>>> > thanks!
>>> >
>>> > --larry
>>> >
>>> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <
>>> madhawa30@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi Larry,
>>> > >
>>> > > I'm interested in working with Livy and would like to join as a
>>> Mentor.
>>> > >
>>> > > Thanks,
>>> > > Madhawa
>>> > >
>>> > >
>>> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <lm...@apache.org>
>>> wrote:
>>> > >
>>> > > > Sorry, I missed commenting on this:
>>> > > >
>>> > > > "There is also no concept as an emeritus PPMC member at the ASF."
>>> > > >
>>> > > > I assume that we can remove PPMC members that do not opt-in
>>> explicitly
>>> > at
>>> > > > this point.
>>> > > > They will have every opportunity to rejoin.
>>> > > >
>>> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <lm...@apache.org>
>>> > wrote:
>>> > > >
>>> > > > > I will ask in a separate thread, @Justin Mclean <
>>> justinmclean@me.com
>>> > >
>>> > > -
>>> > > > > thanks.
>>> > > > > Adding JB adds another company and we are certainly open to
>>> anyone
>>> > else
>>> > > > > that would like to join as a mentor.
>>> > > > > At the end of the day, the mentors are for instilling the Apache
>>> Way
>>> > > > > knowledge and steering toward graduation.
>>> > > > > I feel that this diversity, while nice to have, is less important
>>> > than
>>> > > > > that of the PPMC and committers for the long term health of the
>>> > > > community.
>>> > > > >
>>> > > > > We need to push this podling to graduation as quickly as possible
>>> > since
>>> > > > it
>>> > > > > is rather mature and needs to get to the next level.
>>> > > > >
>>> > > > > Again, any potential Mentors that would like to join are more
>>> than
>>> > > > welcome.
>>> > > > >
>>> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
>>> > > justin@classsoftware.com
>>> > > > >
>>> > > > > wrote:
>>> > > > >
>>> > > > >> Hi,
>>> > > > >>
>>> > > > >> I’m sorry, but Imran Rashid can’t be a mentor for the project as
>>> > they
>>> > > > are
>>> > > > >> not an IPMC member. Currently, both Sunil and Larry (as they
>>> are ASF
>>> > > > >> members) need to ask to join the IPMC and NOTICE sent to the ASF
>>> > > board.
>>> > > > I
>>> > > > >> would also prefer that mentors come from different companies.
>>> > > > >>
>>> > > > >> There is also no concept as an emeritus PPMC member at the ASF.
>>> > > > >>
>>> > > > >> Kind Regards,
>>> > > > >> Justin
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > ---------------------------------------------------------------------
>>> > > > >> To unsubscribe, e-mail:
>>> general-unsubscribe@incubator.apache.org
>>> > > > >> For additional commands, e-mail:
>>> general-help@incubator.apache.org
>>> > > > >>
>>> > > > >>
>>> > > >
>>> > >
>>> >
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
Gentle reminder that we need to determine next steps here.
We have an updated proposal on this thread.
Do we need a VOTE or can we move forward directly to adjusting the members,
etc?
Thanks!
--larry
On Thu, Oct 20, 2022 at 3:26 PM larry mccay <lm...@apache.org> wrote:
> @Justin Mclean <ju...@me.com> - any insights on next steps here?
>
>
> On Tue, Oct 18, 2022 at 5:44 PM larry mccay <lm...@apache.org> wrote:
>
>> Very good, here is the latest revision with updated Mentors.
>> Sunil and I have been added to the IPMC as well.
>> Welcome Madhawa and thanks for stepping up as a Mentor for Livy!
>>
>> Abstract
>>
>> Livy is a web service that exposes a REST interface for managing long
>> running Apache Spark contexts in your cluster. With Livy, new applications
>> can be built on top of Apache Spark that require fine grained interaction
>> with many Spark contexts [1].
>>
>> While this project has been well regarded and used in many contexts as
>> the defacto standard API to Spark environments, it has been incubating for
>> over 5 years without graduation to a TLP and it has become difficult to
>> impossible for fixes and improvements to be contributed as the current
>> community seems to have moved on.
>>
>> There has been discussion regarding retirement of this podling where
>> there seems to be some increasing interest in joining and reviving the
>> community [2].
>>
>> The intent of this proposal is to avoid retiring a well regarded,
>> actively used and rather mature project by reviving the PPMC and community
>> with new folks that have a vested interest in the project and health of the
>> community.
>> Proposal
>>
>> We propose to revive the PPMC with a set of contributors and maintainers
>> as mentors, PPMC members and committers.
>>
>> The retirement DISCUSS thread [2] has shown a growing interest in
>> providing new committers and bringing improvements and fixes from
>> organization’s internally maintained forks back to a revived community.
>>
>> General Approach to Revival:
>>
>> -
>>
>> Add new Mentors
>> -
>>
>> Larry McCay, lmccay@apache.org , Cloudera
>> -
>>
>> Sunil Govindan, sunilg@apache.org, Cloudera
>> -
>>
>> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
>> -
>>
>> Madhawa Gunasekara, madhawa30@gmail.com, Independent
>>
>>
>>
>> -
>>
>> Add new Committers/PPMC
>> -
>>
>> Larry McCay, lmccay@apache.org, Cloudera
>> -
>>
>> Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
>> -
>>
>> Imran Rashid - irashid@apache.org, Cloudera
>> -
>>
>> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
>> -
>>
>> Wing Yew Poon, wypoon@cloudera.com, Cloudera
>> -
>>
>> Xilang Yan, xilang.yan@gmail.com, Shopee
>> -
>>
>> Jianzhen Wu, myjianzhen@gmail.com, Shopee
>> -
>>
>> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
>> -
>>
>> Pralab Kumar, pralkuma@visa.com, Visa
>> -
>>
>> Prasad Shrikant, shrikant.ism@gmail.com, Visa
>> -
>>
>> Brahma Reddy Battula, brahma@apache.org, Visa
>>
>>
>>
>> -
>>
>> Invite existing PPMC members to opt-in or otherwise go emeritus
>> -
>>
>> Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in via
>> Retirement DISCUSS thread [2])
>>
>>
>>
>> -
>>
>> Invite existing Committers to opt-out or otherwise continue
>>
>>
>>
>> -
>>
>> Establish Roadmap via follow up DISCUSS thread
>> -
>>
>> Known Improvements from Forks which will need proposals and
>> discussion:
>> -
>>
>> Adding HA for Livy
>> -
>>
>> Updating security capabilities (eg. kerberos for jdbc, fixing
>> bugs in encryption)
>> -
>>
>> Expanding the support for kubernetes
>> -
>>
>> Responding to CVEs in dependencies (eg. log4j, thrift)
>> -
>>
>> Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
>> -
>>
>> Support multi Spark versions
>> -
>>
>> Implemented a metrics system for Livy
>> -
>>
>> Support customize batch/interactive session lifecycle event
>> handler, default log event with log4j, very helpful for troubleshooting
>> -
>>
>> Optimize log to track which session id the log message came
>> from, also very helpful for troubleshooting
>> -
>>
>> Support customize Spark config optimization rules, can be used
>> to optimize config for users’ job
>> -
>>
>> A set of command line tool which can be used to replace Spark’s
>> spark-submit, pyspark, spark-sql but actually submit application in Livy
>> -
>>
>> We are planning to implement a JDBC state store, and allow
>> multi Livy Thrift sessions to share one backend Spark application in the
>> next few months.
>> -
>>
>> These items and others that are brought to community may need
>> consolidation or multiple configurable options and will need to be part of
>> the discussion
>> -
>>
>> One-pager Livy Improvement Proposals (LIP) may make sense to
>> drive these discussions and convergence
>> -
>>
>> Feature Branch Strategy for large changes
>> -
>>
>> Large features are hard to review we will need to define a
>> strategy
>> -
>>
>> Determine the Improvements to be delivered across first 3 Releases
>> with Target Release Dates
>>
>>
>>
>> -
>>
>> Ensure CVE and Dependency management hygiene is in place
>>
>>
>> The above approach will usher the community back to an active status with
>> a Roadmap of 3 or more release plans and security hygiene in place.
>> Development Practices
>>
>> The Livy project follows a review before commit philosophy. Every commit
>> automatically runs through the unit tests and generates coverage reports
>> presented as a pull request comment. Our experience with this process leads
>> us to believe that it helps ease new contributors into the project. They
>> get feedback quickly on common mistakes, lowering the burden on reviewers.
>> Those same reviewers get to lead by example, showing the new contributors
>> that we value feedback within our community even when changes are done by
>> more experienced folks. Taken from the original Apache Livy Proposal [1],
>> this should continue to be true. As mentioned, Livy is a mature project and
>> as such RTC is the most appropriate for continued quality and awareness.
>>
>>
>>
>> 1.
>>
>> Original Apache Livy Proposal
>> https://cwiki.apache.org/confluence/display/incubator/LivyProposal
>> 2.
>>
>> Retirement DISCUSS thread
>> https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
>>
>>
>> On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <ma...@gmail.com>
>> wrote:
>>
>>> Hi Larry,
>>>
>>> I'm an IPMC Member. madhawa30 at gmail dot com is my preferred email
>>> address.
>>> apache id: madhawa
>>>
>>> Thanks,
>>> Madhawa
>>>
>>>
>>> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <lm...@apache.org> wrote:
>>>
>>> > Hi Madhawa -
>>> >
>>> > That's awesome!
>>> > Are you already a member of IPMC?
>>> > If not, are you an ASF member?
>>> > If you are an ASF member you can request that you be added as an IPMC
>>> > member.
>>> >
>>> > Can you provide your company affiliation for the proposal and preferred
>>> > email?
>>> >
>>> > thanks!
>>> >
>>> > --larry
>>> >
>>> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <
>>> madhawa30@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi Larry,
>>> > >
>>> > > I'm interested in working with Livy and would like to join as a
>>> Mentor.
>>> > >
>>> > > Thanks,
>>> > > Madhawa
>>> > >
>>> > >
>>> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <lm...@apache.org>
>>> wrote:
>>> > >
>>> > > > Sorry, I missed commenting on this:
>>> > > >
>>> > > > "There is also no concept as an emeritus PPMC member at the ASF."
>>> > > >
>>> > > > I assume that we can remove PPMC members that do not opt-in
>>> explicitly
>>> > at
>>> > > > this point.
>>> > > > They will have every opportunity to rejoin.
>>> > > >
>>> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <lm...@apache.org>
>>> > wrote:
>>> > > >
>>> > > > > I will ask in a separate thread, @Justin Mclean <
>>> justinmclean@me.com
>>> > >
>>> > > -
>>> > > > > thanks.
>>> > > > > Adding JB adds another company and we are certainly open to
>>> anyone
>>> > else
>>> > > > > that would like to join as a mentor.
>>> > > > > At the end of the day, the mentors are for instilling the Apache
>>> Way
>>> > > > > knowledge and steering toward graduation.
>>> > > > > I feel that this diversity, while nice to have, is less important
>>> > than
>>> > > > > that of the PPMC and committers for the long term health of the
>>> > > > community.
>>> > > > >
>>> > > > > We need to push this podling to graduation as quickly as possible
>>> > since
>>> > > > it
>>> > > > > is rather mature and needs to get to the next level.
>>> > > > >
>>> > > > > Again, any potential Mentors that would like to join are more
>>> than
>>> > > > welcome.
>>> > > > >
>>> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
>>> > > justin@classsoftware.com
>>> > > > >
>>> > > > > wrote:
>>> > > > >
>>> > > > >> Hi,
>>> > > > >>
>>> > > > >> I’m sorry, but Imran Rashid can’t be a mentor for the project as
>>> > they
>>> > > > are
>>> > > > >> not an IPMC member. Currently, both Sunil and Larry (as they
>>> are ASF
>>> > > > >> members) need to ask to join the IPMC and NOTICE sent to the ASF
>>> > > board.
>>> > > > I
>>> > > > >> would also prefer that mentors come from different companies.
>>> > > > >>
>>> > > > >> There is also no concept as an emeritus PPMC member at the ASF.
>>> > > > >>
>>> > > > >> Kind Regards,
>>> > > > >> Justin
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > ---------------------------------------------------------------------
>>> > > > >> To unsubscribe, e-mail:
>>> general-unsubscribe@incubator.apache.org
>>> > > > >> For additional commands, e-mail:
>>> general-help@incubator.apache.org
>>> > > > >>
>>> > > > >>
>>> > > >
>>> > >
>>> >
>>>
>>
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
Gentle reminder that we need to determine next steps here.
We have an updated proposal on this thread.
Do we need a VOTE or can we move forward directly to adjusting the members,
etc?
Thanks!
--larry
On Thu, Oct 20, 2022 at 3:26 PM larry mccay <lm...@apache.org> wrote:
> @Justin Mclean <ju...@me.com> - any insights on next steps here?
>
>
> On Tue, Oct 18, 2022 at 5:44 PM larry mccay <lm...@apache.org> wrote:
>
>> Very good, here is the latest revision with updated Mentors.
>> Sunil and I have been added to the IPMC as well.
>> Welcome Madhawa and thanks for stepping up as a Mentor for Livy!
>>
>> Abstract
>>
>> Livy is a web service that exposes a REST interface for managing long
>> running Apache Spark contexts in your cluster. With Livy, new applications
>> can be built on top of Apache Spark that require fine grained interaction
>> with many Spark contexts [1].
>>
>> While this project has been well regarded and used in many contexts as
>> the defacto standard API to Spark environments, it has been incubating for
>> over 5 years without graduation to a TLP and it has become difficult to
>> impossible for fixes and improvements to be contributed as the current
>> community seems to have moved on.
>>
>> There has been discussion regarding retirement of this podling where
>> there seems to be some increasing interest in joining and reviving the
>> community [2].
>>
>> The intent of this proposal is to avoid retiring a well regarded,
>> actively used and rather mature project by reviving the PPMC and community
>> with new folks that have a vested interest in the project and health of the
>> community.
>> Proposal
>>
>> We propose to revive the PPMC with a set of contributors and maintainers
>> as mentors, PPMC members and committers.
>>
>> The retirement DISCUSS thread [2] has shown a growing interest in
>> providing new committers and bringing improvements and fixes from
>> organization’s internally maintained forks back to a revived community.
>>
>> General Approach to Revival:
>>
>> -
>>
>> Add new Mentors
>> -
>>
>> Larry McCay, lmccay@apache.org , Cloudera
>> -
>>
>> Sunil Govindan, sunilg@apache.org, Cloudera
>> -
>>
>> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
>> -
>>
>> Madhawa Gunasekara, madhawa30@gmail.com, Independent
>>
>>
>>
>> -
>>
>> Add new Committers/PPMC
>> -
>>
>> Larry McCay, lmccay@apache.org, Cloudera
>> -
>>
>> Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
>> -
>>
>> Imran Rashid - irashid@apache.org, Cloudera
>> -
>>
>> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
>> -
>>
>> Wing Yew Poon, wypoon@cloudera.com, Cloudera
>> -
>>
>> Xilang Yan, xilang.yan@gmail.com, Shopee
>> -
>>
>> Jianzhen Wu, myjianzhen@gmail.com, Shopee
>> -
>>
>> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
>> -
>>
>> Pralab Kumar, pralkuma@visa.com, Visa
>> -
>>
>> Prasad Shrikant, shrikant.ism@gmail.com, Visa
>> -
>>
>> Brahma Reddy Battula, brahma@apache.org, Visa
>>
>>
>>
>> -
>>
>> Invite existing PPMC members to opt-in or otherwise go emeritus
>> -
>>
>> Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in via
>> Retirement DISCUSS thread [2])
>>
>>
>>
>> -
>>
>> Invite existing Committers to opt-out or otherwise continue
>>
>>
>>
>> -
>>
>> Establish Roadmap via follow up DISCUSS thread
>> -
>>
>> Known Improvements from Forks which will need proposals and
>> discussion:
>> -
>>
>> Adding HA for Livy
>> -
>>
>> Updating security capabilities (eg. kerberos for jdbc, fixing
>> bugs in encryption)
>> -
>>
>> Expanding the support for kubernetes
>> -
>>
>> Responding to CVEs in dependencies (eg. log4j, thrift)
>> -
>>
>> Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
>> -
>>
>> Support multi Spark versions
>> -
>>
>> Implemented a metrics system for Livy
>> -
>>
>> Support customize batch/interactive session lifecycle event
>> handler, default log event with log4j, very helpful for troubleshooting
>> -
>>
>> Optimize log to track which session id the log message came
>> from, also very helpful for troubleshooting
>> -
>>
>> Support customize Spark config optimization rules, can be used
>> to optimize config for users’ job
>> -
>>
>> A set of command line tool which can be used to replace Spark’s
>> spark-submit, pyspark, spark-sql but actually submit application in Livy
>> -
>>
>> We are planning to implement a JDBC state store, and allow
>> multi Livy Thrift sessions to share one backend Spark application in the
>> next few months.
>> -
>>
>> These items and others that are brought to community may need
>> consolidation or multiple configurable options and will need to be part of
>> the discussion
>> -
>>
>> One-pager Livy Improvement Proposals (LIP) may make sense to
>> drive these discussions and convergence
>> -
>>
>> Feature Branch Strategy for large changes
>> -
>>
>> Large features are hard to review we will need to define a
>> strategy
>> -
>>
>> Determine the Improvements to be delivered across first 3 Releases
>> with Target Release Dates
>>
>>
>>
>> -
>>
>> Ensure CVE and Dependency management hygiene is in place
>>
>>
>> The above approach will usher the community back to an active status with
>> a Roadmap of 3 or more release plans and security hygiene in place.
>> Development Practices
>>
>> The Livy project follows a review before commit philosophy. Every commit
>> automatically runs through the unit tests and generates coverage reports
>> presented as a pull request comment. Our experience with this process leads
>> us to believe that it helps ease new contributors into the project. They
>> get feedback quickly on common mistakes, lowering the burden on reviewers.
>> Those same reviewers get to lead by example, showing the new contributors
>> that we value feedback within our community even when changes are done by
>> more experienced folks. Taken from the original Apache Livy Proposal [1],
>> this should continue to be true. As mentioned, Livy is a mature project and
>> as such RTC is the most appropriate for continued quality and awareness.
>>
>>
>>
>> 1.
>>
>> Original Apache Livy Proposal
>> https://cwiki.apache.org/confluence/display/incubator/LivyProposal
>> 2.
>>
>> Retirement DISCUSS thread
>> https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
>>
>>
>> On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <ma...@gmail.com>
>> wrote:
>>
>>> Hi Larry,
>>>
>>> I'm an IPMC Member. madhawa30 at gmail dot com is my preferred email
>>> address.
>>> apache id: madhawa
>>>
>>> Thanks,
>>> Madhawa
>>>
>>>
>>> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <lm...@apache.org> wrote:
>>>
>>> > Hi Madhawa -
>>> >
>>> > That's awesome!
>>> > Are you already a member of IPMC?
>>> > If not, are you an ASF member?
>>> > If you are an ASF member you can request that you be added as an IPMC
>>> > member.
>>> >
>>> > Can you provide your company affiliation for the proposal and preferred
>>> > email?
>>> >
>>> > thanks!
>>> >
>>> > --larry
>>> >
>>> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <
>>> madhawa30@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi Larry,
>>> > >
>>> > > I'm interested in working with Livy and would like to join as a
>>> Mentor.
>>> > >
>>> > > Thanks,
>>> > > Madhawa
>>> > >
>>> > >
>>> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <lm...@apache.org>
>>> wrote:
>>> > >
>>> > > > Sorry, I missed commenting on this:
>>> > > >
>>> > > > "There is also no concept as an emeritus PPMC member at the ASF."
>>> > > >
>>> > > > I assume that we can remove PPMC members that do not opt-in
>>> explicitly
>>> > at
>>> > > > this point.
>>> > > > They will have every opportunity to rejoin.
>>> > > >
>>> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <lm...@apache.org>
>>> > wrote:
>>> > > >
>>> > > > > I will ask in a separate thread, @Justin Mclean <
>>> justinmclean@me.com
>>> > >
>>> > > -
>>> > > > > thanks.
>>> > > > > Adding JB adds another company and we are certainly open to
>>> anyone
>>> > else
>>> > > > > that would like to join as a mentor.
>>> > > > > At the end of the day, the mentors are for instilling the Apache
>>> Way
>>> > > > > knowledge and steering toward graduation.
>>> > > > > I feel that this diversity, while nice to have, is less important
>>> > than
>>> > > > > that of the PPMC and committers for the long term health of the
>>> > > > community.
>>> > > > >
>>> > > > > We need to push this podling to graduation as quickly as possible
>>> > since
>>> > > > it
>>> > > > > is rather mature and needs to get to the next level.
>>> > > > >
>>> > > > > Again, any potential Mentors that would like to join are more
>>> than
>>> > > > welcome.
>>> > > > >
>>> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
>>> > > justin@classsoftware.com
>>> > > > >
>>> > > > > wrote:
>>> > > > >
>>> > > > >> Hi,
>>> > > > >>
>>> > > > >> I’m sorry, but Imran Rashid can’t be a mentor for the project as
>>> > they
>>> > > > are
>>> > > > >> not an IPMC member. Currently, both Sunil and Larry (as they
>>> are ASF
>>> > > > >> members) need to ask to join the IPMC and NOTICE sent to the ASF
>>> > > board.
>>> > > > I
>>> > > > >> would also prefer that mentors come from different companies.
>>> > > > >>
>>> > > > >> There is also no concept as an emeritus PPMC member at the ASF.
>>> > > > >>
>>> > > > >> Kind Regards,
>>> > > > >> Justin
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > ---------------------------------------------------------------------
>>> > > > >> To unsubscribe, e-mail:
>>> general-unsubscribe@incubator.apache.org
>>> > > > >> For additional commands, e-mail:
>>> general-help@incubator.apache.org
>>> > > > >>
>>> > > > >>
>>> > > >
>>> > >
>>> >
>>>
>>
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
@Justin Mclean <ju...@me.com> - any insights on next steps here?
On Tue, Oct 18, 2022 at 5:44 PM larry mccay <lm...@apache.org> wrote:
> Very good, here is the latest revision with updated Mentors.
> Sunil and I have been added to the IPMC as well.
> Welcome Madhawa and thanks for stepping up as a Mentor for Livy!
>
> Abstract
>
> Livy is a web service that exposes a REST interface for managing long
> running Apache Spark contexts in your cluster. With Livy, new applications
> can be built on top of Apache Spark that require fine grained interaction
> with many Spark contexts [1].
>
> While this project has been well regarded and used in many contexts as the
> defacto standard API to Spark environments, it has been incubating for over
> 5 years without graduation to a TLP and it has become difficult to
> impossible for fixes and improvements to be contributed as the current
> community seems to have moved on.
>
> There has been discussion regarding retirement of this podling where there
> seems to be some increasing interest in joining and reviving the community
> [2].
>
> The intent of this proposal is to avoid retiring a well regarded, actively
> used and rather mature project by reviving the PPMC and community with new
> folks that have a vested interest in the project and health of the
> community.
> Proposal
>
> We propose to revive the PPMC with a set of contributors and maintainers
> as mentors, PPMC members and committers.
>
> The retirement DISCUSS thread [2] has shown a growing interest in
> providing new committers and bringing improvements and fixes from
> organization’s internally maintained forks back to a revived community.
>
> General Approach to Revival:
>
> -
>
> Add new Mentors
> -
>
> Larry McCay, lmccay@apache.org , Cloudera
> -
>
> Sunil Govindan, sunilg@apache.org, Cloudera
> -
>
> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
> -
>
> Madhawa Gunasekara, madhawa30@gmail.com, Independent
>
>
>
> -
>
> Add new Committers/PPMC
> -
>
> Larry McCay, lmccay@apache.org, Cloudera
> -
>
> Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
> -
>
> Imran Rashid - irashid@apache.org, Cloudera
> -
>
> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
> -
>
> Wing Yew Poon, wypoon@cloudera.com, Cloudera
> -
>
> Xilang Yan, xilang.yan@gmail.com, Shopee
> -
>
> Jianzhen Wu, myjianzhen@gmail.com, Shopee
> -
>
> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
> -
>
> Pralab Kumar, pralkuma@visa.com, Visa
> -
>
> Prasad Shrikant, shrikant.ism@gmail.com, Visa
> -
>
> Brahma Reddy Battula, brahma@apache.org, Visa
>
>
>
> -
>
> Invite existing PPMC members to opt-in or otherwise go emeritus
> -
>
> Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in via
> Retirement DISCUSS thread [2])
>
>
>
> -
>
> Invite existing Committers to opt-out or otherwise continue
>
>
>
> -
>
> Establish Roadmap via follow up DISCUSS thread
> -
>
> Known Improvements from Forks which will need proposals and
> discussion:
> -
>
> Adding HA for Livy
> -
>
> Updating security capabilities (eg. kerberos for jdbc, fixing
> bugs in encryption)
> -
>
> Expanding the support for kubernetes
> -
>
> Responding to CVEs in dependencies (eg. log4j, thrift)
> -
>
> Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
> -
>
> Support multi Spark versions
> -
>
> Implemented a metrics system for Livy
> -
>
> Support customize batch/interactive session lifecycle event
> handler, default log event with log4j, very helpful for troubleshooting
> -
>
> Optimize log to track which session id the log message came
> from, also very helpful for troubleshooting
> -
>
> Support customize Spark config optimization rules, can be used
> to optimize config for users’ job
> -
>
> A set of command line tool which can be used to replace Spark’s
> spark-submit, pyspark, spark-sql but actually submit application in Livy
> -
>
> We are planning to implement a JDBC state store, and allow multi
> Livy Thrift sessions to share one backend Spark application in the next few
> months.
> -
>
> These items and others that are brought to community may need
> consolidation or multiple configurable options and will need to be part of
> the discussion
> -
>
> One-pager Livy Improvement Proposals (LIP) may make sense to
> drive these discussions and convergence
> -
>
> Feature Branch Strategy for large changes
> -
>
> Large features are hard to review we will need to define a
> strategy
> -
>
> Determine the Improvements to be delivered across first 3 Releases
> with Target Release Dates
>
>
>
> -
>
> Ensure CVE and Dependency management hygiene is in place
>
>
> The above approach will usher the community back to an active status with
> a Roadmap of 3 or more release plans and security hygiene in place.
> Development Practices
>
> The Livy project follows a review before commit philosophy. Every commit
> automatically runs through the unit tests and generates coverage reports
> presented as a pull request comment. Our experience with this process leads
> us to believe that it helps ease new contributors into the project. They
> get feedback quickly on common mistakes, lowering the burden on reviewers.
> Those same reviewers get to lead by example, showing the new contributors
> that we value feedback within our community even when changes are done by
> more experienced folks. Taken from the original Apache Livy Proposal [1],
> this should continue to be true. As mentioned, Livy is a mature project and
> as such RTC is the most appropriate for continued quality and awareness.
>
>
>
> 1.
>
> Original Apache Livy Proposal
> https://cwiki.apache.org/confluence/display/incubator/LivyProposal
> 2.
>
> Retirement DISCUSS thread
> https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
>
>
> On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <ma...@gmail.com>
> wrote:
>
>> Hi Larry,
>>
>> I'm an IPMC Member. madhawa30 at gmail dot com is my preferred email
>> address.
>> apache id: madhawa
>>
>> Thanks,
>> Madhawa
>>
>>
>> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <lm...@apache.org> wrote:
>>
>> > Hi Madhawa -
>> >
>> > That's awesome!
>> > Are you already a member of IPMC?
>> > If not, are you an ASF member?
>> > If you are an ASF member you can request that you be added as an IPMC
>> > member.
>> >
>> > Can you provide your company affiliation for the proposal and preferred
>> > email?
>> >
>> > thanks!
>> >
>> > --larry
>> >
>> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <madhawa30@gmail.com
>> >
>> > wrote:
>> >
>> > > Hi Larry,
>> > >
>> > > I'm interested in working with Livy and would like to join as a
>> Mentor.
>> > >
>> > > Thanks,
>> > > Madhawa
>> > >
>> > >
>> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <lm...@apache.org>
>> wrote:
>> > >
>> > > > Sorry, I missed commenting on this:
>> > > >
>> > > > "There is also no concept as an emeritus PPMC member at the ASF."
>> > > >
>> > > > I assume that we can remove PPMC members that do not opt-in
>> explicitly
>> > at
>> > > > this point.
>> > > > They will have every opportunity to rejoin.
>> > > >
>> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <lm...@apache.org>
>> > wrote:
>> > > >
>> > > > > I will ask in a separate thread, @Justin Mclean <
>> justinmclean@me.com
>> > >
>> > > -
>> > > > > thanks.
>> > > > > Adding JB adds another company and we are certainly open to anyone
>> > else
>> > > > > that would like to join as a mentor.
>> > > > > At the end of the day, the mentors are for instilling the Apache
>> Way
>> > > > > knowledge and steering toward graduation.
>> > > > > I feel that this diversity, while nice to have, is less important
>> > than
>> > > > > that of the PPMC and committers for the long term health of the
>> > > > community.
>> > > > >
>> > > > > We need to push this podling to graduation as quickly as possible
>> > since
>> > > > it
>> > > > > is rather mature and needs to get to the next level.
>> > > > >
>> > > > > Again, any potential Mentors that would like to join are more than
>> > > > welcome.
>> > > > >
>> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
>> > > justin@classsoftware.com
>> > > > >
>> > > > > wrote:
>> > > > >
>> > > > >> Hi,
>> > > > >>
>> > > > >> I’m sorry, but Imran Rashid can’t be a mentor for the project as
>> > they
>> > > > are
>> > > > >> not an IPMC member. Currently, both Sunil and Larry (as they are
>> ASF
>> > > > >> members) need to ask to join the IPMC and NOTICE sent to the ASF
>> > > board.
>> > > > I
>> > > > >> would also prefer that mentors come from different companies.
>> > > > >>
>> > > > >> There is also no concept as an emeritus PPMC member at the ASF.
>> > > > >>
>> > > > >> Kind Regards,
>> > > > >> Justin
>> > > > >>
>> > > > >>
>> > > > >>
>> > ---------------------------------------------------------------------
>> > > > >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> > > > >> For additional commands, e-mail:
>> general-help@incubator.apache.org
>> > > > >>
>> > > > >>
>> > > >
>> > >
>> >
>>
>
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
@Justin Mclean <ju...@me.com> - any insights on next steps here?
On Tue, Oct 18, 2022 at 5:44 PM larry mccay <lm...@apache.org> wrote:
> Very good, here is the latest revision with updated Mentors.
> Sunil and I have been added to the IPMC as well.
> Welcome Madhawa and thanks for stepping up as a Mentor for Livy!
>
> Abstract
>
> Livy is a web service that exposes a REST interface for managing long
> running Apache Spark contexts in your cluster. With Livy, new applications
> can be built on top of Apache Spark that require fine grained interaction
> with many Spark contexts [1].
>
> While this project has been well regarded and used in many contexts as the
> defacto standard API to Spark environments, it has been incubating for over
> 5 years without graduation to a TLP and it has become difficult to
> impossible for fixes and improvements to be contributed as the current
> community seems to have moved on.
>
> There has been discussion regarding retirement of this podling where there
> seems to be some increasing interest in joining and reviving the community
> [2].
>
> The intent of this proposal is to avoid retiring a well regarded, actively
> used and rather mature project by reviving the PPMC and community with new
> folks that have a vested interest in the project and health of the
> community.
> Proposal
>
> We propose to revive the PPMC with a set of contributors and maintainers
> as mentors, PPMC members and committers.
>
> The retirement DISCUSS thread [2] has shown a growing interest in
> providing new committers and bringing improvements and fixes from
> organization’s internally maintained forks back to a revived community.
>
> General Approach to Revival:
>
> -
>
> Add new Mentors
> -
>
> Larry McCay, lmccay@apache.org , Cloudera
> -
>
> Sunil Govindan, sunilg@apache.org, Cloudera
> -
>
> Jean-Baptiste Onofré, jbonofre@apache.org, Talend
> -
>
> Madhawa Gunasekara, madhawa30@gmail.com, Independent
>
>
>
> -
>
> Add new Committers/PPMC
> -
>
> Larry McCay, lmccay@apache.org, Cloudera
> -
>
> Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
> -
>
> Imran Rashid - irashid@apache.org, Cloudera
> -
>
> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
> -
>
> Wing Yew Poon, wypoon@cloudera.com, Cloudera
> -
>
> Xilang Yan, xilang.yan@gmail.com, Shopee
> -
>
> Jianzhen Wu, myjianzhen@gmail.com, Shopee
> -
>
> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
> -
>
> Pralab Kumar, pralkuma@visa.com, Visa
> -
>
> Prasad Shrikant, shrikant.ism@gmail.com, Visa
> -
>
> Brahma Reddy Battula, brahma@apache.org, Visa
>
>
>
> -
>
> Invite existing PPMC members to opt-in or otherwise go emeritus
> -
>
> Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in via
> Retirement DISCUSS thread [2])
>
>
>
> -
>
> Invite existing Committers to opt-out or otherwise continue
>
>
>
> -
>
> Establish Roadmap via follow up DISCUSS thread
> -
>
> Known Improvements from Forks which will need proposals and
> discussion:
> -
>
> Adding HA for Livy
> -
>
> Updating security capabilities (eg. kerberos for jdbc, fixing
> bugs in encryption)
> -
>
> Expanding the support for kubernetes
> -
>
> Responding to CVEs in dependencies (eg. log4j, thrift)
> -
>
> Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
> -
>
> Support multi Spark versions
> -
>
> Implemented a metrics system for Livy
> -
>
> Support customize batch/interactive session lifecycle event
> handler, default log event with log4j, very helpful for troubleshooting
> -
>
> Optimize log to track which session id the log message came
> from, also very helpful for troubleshooting
> -
>
> Support customize Spark config optimization rules, can be used
> to optimize config for users’ job
> -
>
> A set of command line tool which can be used to replace Spark’s
> spark-submit, pyspark, spark-sql but actually submit application in Livy
> -
>
> We are planning to implement a JDBC state store, and allow multi
> Livy Thrift sessions to share one backend Spark application in the next few
> months.
> -
>
> These items and others that are brought to community may need
> consolidation or multiple configurable options and will need to be part of
> the discussion
> -
>
> One-pager Livy Improvement Proposals (LIP) may make sense to
> drive these discussions and convergence
> -
>
> Feature Branch Strategy for large changes
> -
>
> Large features are hard to review we will need to define a
> strategy
> -
>
> Determine the Improvements to be delivered across first 3 Releases
> with Target Release Dates
>
>
>
> -
>
> Ensure CVE and Dependency management hygiene is in place
>
>
> The above approach will usher the community back to an active status with
> a Roadmap of 3 or more release plans and security hygiene in place.
> Development Practices
>
> The Livy project follows a review before commit philosophy. Every commit
> automatically runs through the unit tests and generates coverage reports
> presented as a pull request comment. Our experience with this process leads
> us to believe that it helps ease new contributors into the project. They
> get feedback quickly on common mistakes, lowering the burden on reviewers.
> Those same reviewers get to lead by example, showing the new contributors
> that we value feedback within our community even when changes are done by
> more experienced folks. Taken from the original Apache Livy Proposal [1],
> this should continue to be true. As mentioned, Livy is a mature project and
> as such RTC is the most appropriate for continued quality and awareness.
>
>
>
> 1.
>
> Original Apache Livy Proposal
> https://cwiki.apache.org/confluence/display/incubator/LivyProposal
> 2.
>
> Retirement DISCUSS thread
> https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
>
>
> On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <ma...@gmail.com>
> wrote:
>
>> Hi Larry,
>>
>> I'm an IPMC Member. madhawa30 at gmail dot com is my preferred email
>> address.
>> apache id: madhawa
>>
>> Thanks,
>> Madhawa
>>
>>
>> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <lm...@apache.org> wrote:
>>
>> > Hi Madhawa -
>> >
>> > That's awesome!
>> > Are you already a member of IPMC?
>> > If not, are you an ASF member?
>> > If you are an ASF member you can request that you be added as an IPMC
>> > member.
>> >
>> > Can you provide your company affiliation for the proposal and preferred
>> > email?
>> >
>> > thanks!
>> >
>> > --larry
>> >
>> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <madhawa30@gmail.com
>> >
>> > wrote:
>> >
>> > > Hi Larry,
>> > >
>> > > I'm interested in working with Livy and would like to join as a
>> Mentor.
>> > >
>> > > Thanks,
>> > > Madhawa
>> > >
>> > >
>> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <lm...@apache.org>
>> wrote:
>> > >
>> > > > Sorry, I missed commenting on this:
>> > > >
>> > > > "There is also no concept as an emeritus PPMC member at the ASF."
>> > > >
>> > > > I assume that we can remove PPMC members that do not opt-in
>> explicitly
>> > at
>> > > > this point.
>> > > > They will have every opportunity to rejoin.
>> > > >
>> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <lm...@apache.org>
>> > wrote:
>> > > >
>> > > > > I will ask in a separate thread, @Justin Mclean <
>> justinmclean@me.com
>> > >
>> > > -
>> > > > > thanks.
>> > > > > Adding JB adds another company and we are certainly open to anyone
>> > else
>> > > > > that would like to join as a mentor.
>> > > > > At the end of the day, the mentors are for instilling the Apache
>> Way
>> > > > > knowledge and steering toward graduation.
>> > > > > I feel that this diversity, while nice to have, is less important
>> > than
>> > > > > that of the PPMC and committers for the long term health of the
>> > > > community.
>> > > > >
>> > > > > We need to push this podling to graduation as quickly as possible
>> > since
>> > > > it
>> > > > > is rather mature and needs to get to the next level.
>> > > > >
>> > > > > Again, any potential Mentors that would like to join are more than
>> > > > welcome.
>> > > > >
>> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
>> > > justin@classsoftware.com
>> > > > >
>> > > > > wrote:
>> > > > >
>> > > > >> Hi,
>> > > > >>
>> > > > >> I’m sorry, but Imran Rashid can’t be a mentor for the project as
>> > they
>> > > > are
>> > > > >> not an IPMC member. Currently, both Sunil and Larry (as they are
>> ASF
>> > > > >> members) need to ask to join the IPMC and NOTICE sent to the ASF
>> > > board.
>> > > > I
>> > > > >> would also prefer that mentors come from different companies.
>> > > > >>
>> > > > >> There is also no concept as an emeritus PPMC member at the ASF.
>> > > > >>
>> > > > >> Kind Regards,
>> > > > >> Justin
>> > > > >>
>> > > > >>
>> > > > >>
>> > ---------------------------------------------------------------------
>> > > > >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> > > > >> For additional commands, e-mail:
>> general-help@incubator.apache.org
>> > > > >>
>> > > > >>
>> > > >
>> > >
>> >
>>
>
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
Very good, here is the latest revision with updated Mentors.
Sunil and I have been added to the IPMC as well.
Welcome Madhawa and thanks for stepping up as a Mentor for Livy!
Abstract
Livy is a web service that exposes a REST interface for managing long
running Apache Spark contexts in your cluster. With Livy, new applications
can be built on top of Apache Spark that require fine grained interaction
with many Spark contexts [1].
While this project has been well regarded and used in many contexts as the
defacto standard API to Spark environments, it has been incubating for over
5 years without graduation to a TLP and it has become difficult to
impossible for fixes and improvements to be contributed as the current
community seems to have moved on.
There has been discussion regarding retirement of this podling where there
seems to be some increasing interest in joining and reviving the community
[2].
The intent of this proposal is to avoid retiring a well regarded, actively
used and rather mature project by reviving the PPMC and community with new
folks that have a vested interest in the project and health of the
community.
Proposal
We propose to revive the PPMC with a set of contributors and maintainers as
mentors, PPMC members and committers.
The retirement DISCUSS thread [2] has shown a growing interest in providing
new committers and bringing improvements and fixes from organization’s
internally maintained forks back to a revived community.
General Approach to Revival:
-
Add new Mentors
-
Larry McCay, lmccay@apache.org , Cloudera
-
Sunil Govindan, sunilg@apache.org, Cloudera
-
Jean-Baptiste Onofré, jbonofre@apache.org, Talend
-
Madhawa Gunasekara, madhawa30@gmail.com, Independent
-
Add new Committers/PPMC
-
Larry McCay, lmccay@apache.org, Cloudera
-
Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
-
Imran Rashid - irashid@apache.org, Cloudera
-
Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
-
Wing Yew Poon, wypoon@cloudera.com, Cloudera
-
Xilang Yan, xilang.yan@gmail.com, Shopee
-
Jianzhen Wu, myjianzhen@gmail.com, Shopee
-
Nagella Jagadeewara Rao, jnagella@visa.com, Visa
-
Pralab Kumar, pralkuma@visa.com, Visa
-
Prasad Shrikant, shrikant.ism@gmail.com, Visa
-
Brahma Reddy Battula, brahma@apache.org, Visa
-
Invite existing PPMC members to opt-in or otherwise go emeritus
-
Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in via
Retirement DISCUSS thread [2])
-
Invite existing Committers to opt-out or otherwise continue
-
Establish Roadmap via follow up DISCUSS thread
-
Known Improvements from Forks which will need proposals and
discussion:
-
Adding HA for Livy
-
Updating security capabilities (eg. kerberos for jdbc, fixing bugs
in encryption)
-
Expanding the support for kubernetes
-
Responding to CVEs in dependencies (eg. log4j, thrift)
-
Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
-
Support multi Spark versions
-
Implemented a metrics system for Livy
-
Support customize batch/interactive session lifecycle event
handler, default log event with log4j, very helpful for troubleshooting
-
Optimize log to track which session id the log message came from,
also very helpful for troubleshooting
-
Support customize Spark config optimization rules, can be used to
optimize config for users’ job
-
A set of command line tool which can be used to replace Spark’s
spark-submit, pyspark, spark-sql but actually submit
application in Livy
-
We are planning to implement a JDBC state store, and allow multi
Livy Thrift sessions to share one backend Spark application
in the next few
months.
-
These items and others that are brought to community may need
consolidation or multiple configurable options and will need to
be part of
the discussion
-
One-pager Livy Improvement Proposals (LIP) may make sense to drive
these discussions and convergence
-
Feature Branch Strategy for large changes
-
Large features are hard to review we will need to define a
strategy
-
Determine the Improvements to be delivered across first 3 Releases
with Target Release Dates
-
Ensure CVE and Dependency management hygiene is in place
The above approach will usher the community back to an active status with a
Roadmap of 3 or more release plans and security hygiene in place.
Development Practices
The Livy project follows a review before commit philosophy. Every commit
automatically runs through the unit tests and generates coverage reports
presented as a pull request comment. Our experience with this process leads
us to believe that it helps ease new contributors into the project. They
get feedback quickly on common mistakes, lowering the burden on reviewers.
Those same reviewers get to lead by example, showing the new contributors
that we value feedback within our community even when changes are done by
more experienced folks. Taken from the original Apache Livy Proposal [1],
this should continue to be true. As mentioned, Livy is a mature project and
as such RTC is the most appropriate for continued quality and awareness.
1.
Original Apache Livy Proposal
https://cwiki.apache.org/confluence/display/incubator/LivyProposal
2.
Retirement DISCUSS thread
https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <ma...@gmail.com>
wrote:
> Hi Larry,
>
> I'm an IPMC Member. madhawa30 at gmail dot com is my preferred email
> address.
> apache id: madhawa
>
> Thanks,
> Madhawa
>
>
> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <lm...@apache.org> wrote:
>
> > Hi Madhawa -
> >
> > That's awesome!
> > Are you already a member of IPMC?
> > If not, are you an ASF member?
> > If you are an ASF member you can request that you be added as an IPMC
> > member.
> >
> > Can you provide your company affiliation for the proposal and preferred
> > email?
> >
> > thanks!
> >
> > --larry
> >
> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <ma...@gmail.com>
> > wrote:
> >
> > > Hi Larry,
> > >
> > > I'm interested in working with Livy and would like to join as a Mentor.
> > >
> > > Thanks,
> > > Madhawa
> > >
> > >
> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <lm...@apache.org> wrote:
> > >
> > > > Sorry, I missed commenting on this:
> > > >
> > > > "There is also no concept as an emeritus PPMC member at the ASF."
> > > >
> > > > I assume that we can remove PPMC members that do not opt-in
> explicitly
> > at
> > > > this point.
> > > > They will have every opportunity to rejoin.
> > > >
> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <lm...@apache.org>
> > wrote:
> > > >
> > > > > I will ask in a separate thread, @Justin Mclean <
> justinmclean@me.com
> > >
> > > -
> > > > > thanks.
> > > > > Adding JB adds another company and we are certainly open to anyone
> > else
> > > > > that would like to join as a mentor.
> > > > > At the end of the day, the mentors are for instilling the Apache
> Way
> > > > > knowledge and steering toward graduation.
> > > > > I feel that this diversity, while nice to have, is less important
> > than
> > > > > that of the PPMC and committers for the long term health of the
> > > > community.
> > > > >
> > > > > We need to push this podling to graduation as quickly as possible
> > since
> > > > it
> > > > > is rather mature and needs to get to the next level.
> > > > >
> > > > > Again, any potential Mentors that would like to join are more than
> > > > welcome.
> > > > >
> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
> > > justin@classsoftware.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> I’m sorry, but Imran Rashid can’t be a mentor for the project as
> > they
> > > > are
> > > > >> not an IPMC member. Currently, both Sunil and Larry (as they are
> ASF
> > > > >> members) need to ask to join the IPMC and NOTICE sent to the ASF
> > > board.
> > > > I
> > > > >> would also prefer that mentors come from different companies.
> > > > >>
> > > > >> There is also no concept as an emeritus PPMC member at the ASF.
> > > > >>
> > > > >> Kind Regards,
> > > > >> Justin
> > > > >>
> > > > >>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > >> For additional commands, e-mail:
> general-help@incubator.apache.org
> > > > >>
> > > > >>
> > > >
> > >
> >
>
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
Very good, here is the latest revision with updated Mentors.
Sunil and I have been added to the IPMC as well.
Welcome Madhawa and thanks for stepping up as a Mentor for Livy!
Abstract
Livy is a web service that exposes a REST interface for managing long
running Apache Spark contexts in your cluster. With Livy, new applications
can be built on top of Apache Spark that require fine grained interaction
with many Spark contexts [1].
While this project has been well regarded and used in many contexts as the
defacto standard API to Spark environments, it has been incubating for over
5 years without graduation to a TLP and it has become difficult to
impossible for fixes and improvements to be contributed as the current
community seems to have moved on.
There has been discussion regarding retirement of this podling where there
seems to be some increasing interest in joining and reviving the community
[2].
The intent of this proposal is to avoid retiring a well regarded, actively
used and rather mature project by reviving the PPMC and community with new
folks that have a vested interest in the project and health of the
community.
Proposal
We propose to revive the PPMC with a set of contributors and maintainers as
mentors, PPMC members and committers.
The retirement DISCUSS thread [2] has shown a growing interest in providing
new committers and bringing improvements and fixes from organization’s
internally maintained forks back to a revived community.
General Approach to Revival:
-
Add new Mentors
-
Larry McCay, lmccay@apache.org , Cloudera
-
Sunil Govindan, sunilg@apache.org, Cloudera
-
Jean-Baptiste Onofré, jbonofre@apache.org, Talend
-
Madhawa Gunasekara, madhawa30@gmail.com, Independent
-
Add new Committers/PPMC
-
Larry McCay, lmccay@apache.org, Cloudera
-
Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
-
Imran Rashid - irashid@apache.org, Cloudera
-
Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
-
Wing Yew Poon, wypoon@cloudera.com, Cloudera
-
Xilang Yan, xilang.yan@gmail.com, Shopee
-
Jianzhen Wu, myjianzhen@gmail.com, Shopee
-
Nagella Jagadeewara Rao, jnagella@visa.com, Visa
-
Pralab Kumar, pralkuma@visa.com, Visa
-
Prasad Shrikant, shrikant.ism@gmail.com, Visa
-
Brahma Reddy Battula, brahma@apache.org, Visa
-
Invite existing PPMC members to opt-in or otherwise go emeritus
-
Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in via
Retirement DISCUSS thread [2])
-
Invite existing Committers to opt-out or otherwise continue
-
Establish Roadmap via follow up DISCUSS thread
-
Known Improvements from Forks which will need proposals and
discussion:
-
Adding HA for Livy
-
Updating security capabilities (eg. kerberos for jdbc, fixing bugs
in encryption)
-
Expanding the support for kubernetes
-
Responding to CVEs in dependencies (eg. log4j, thrift)
-
Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
-
Support multi Spark versions
-
Implemented a metrics system for Livy
-
Support customize batch/interactive session lifecycle event
handler, default log event with log4j, very helpful for troubleshooting
-
Optimize log to track which session id the log message came from,
also very helpful for troubleshooting
-
Support customize Spark config optimization rules, can be used to
optimize config for users’ job
-
A set of command line tool which can be used to replace Spark’s
spark-submit, pyspark, spark-sql but actually submit
application in Livy
-
We are planning to implement a JDBC state store, and allow multi
Livy Thrift sessions to share one backend Spark application
in the next few
months.
-
These items and others that are brought to community may need
consolidation or multiple configurable options and will need to
be part of
the discussion
-
One-pager Livy Improvement Proposals (LIP) may make sense to drive
these discussions and convergence
-
Feature Branch Strategy for large changes
-
Large features are hard to review we will need to define a
strategy
-
Determine the Improvements to be delivered across first 3 Releases
with Target Release Dates
-
Ensure CVE and Dependency management hygiene is in place
The above approach will usher the community back to an active status with a
Roadmap of 3 or more release plans and security hygiene in place.
Development Practices
The Livy project follows a review before commit philosophy. Every commit
automatically runs through the unit tests and generates coverage reports
presented as a pull request comment. Our experience with this process leads
us to believe that it helps ease new contributors into the project. They
get feedback quickly on common mistakes, lowering the burden on reviewers.
Those same reviewers get to lead by example, showing the new contributors
that we value feedback within our community even when changes are done by
more experienced folks. Taken from the original Apache Livy Proposal [1],
this should continue to be true. As mentioned, Livy is a mature project and
as such RTC is the most appropriate for continued quality and awareness.
1.
Original Apache Livy Proposal
https://cwiki.apache.org/confluence/display/incubator/LivyProposal
2.
Retirement DISCUSS thread
https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
On Tue, Oct 18, 2022 at 5:05 PM Madhawa Gunasekara <ma...@gmail.com>
wrote:
> Hi Larry,
>
> I'm an IPMC Member. madhawa30 at gmail dot com is my preferred email
> address.
> apache id: madhawa
>
> Thanks,
> Madhawa
>
>
> On Tue, Oct 18, 2022 at 10:05 PM larry mccay <lm...@apache.org> wrote:
>
> > Hi Madhawa -
> >
> > That's awesome!
> > Are you already a member of IPMC?
> > If not, are you an ASF member?
> > If you are an ASF member you can request that you be added as an IPMC
> > member.
> >
> > Can you provide your company affiliation for the proposal and preferred
> > email?
> >
> > thanks!
> >
> > --larry
> >
> > On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <ma...@gmail.com>
> > wrote:
> >
> > > Hi Larry,
> > >
> > > I'm interested in working with Livy and would like to join as a Mentor.
> > >
> > > Thanks,
> > > Madhawa
> > >
> > >
> > > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <lm...@apache.org> wrote:
> > >
> > > > Sorry, I missed commenting on this:
> > > >
> > > > "There is also no concept as an emeritus PPMC member at the ASF."
> > > >
> > > > I assume that we can remove PPMC members that do not opt-in
> explicitly
> > at
> > > > this point.
> > > > They will have every opportunity to rejoin.
> > > >
> > > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <lm...@apache.org>
> > wrote:
> > > >
> > > > > I will ask in a separate thread, @Justin Mclean <
> justinmclean@me.com
> > >
> > > -
> > > > > thanks.
> > > > > Adding JB adds another company and we are certainly open to anyone
> > else
> > > > > that would like to join as a mentor.
> > > > > At the end of the day, the mentors are for instilling the Apache
> Way
> > > > > knowledge and steering toward graduation.
> > > > > I feel that this diversity, while nice to have, is less important
> > than
> > > > > that of the PPMC and committers for the long term health of the
> > > > community.
> > > > >
> > > > > We need to push this podling to graduation as quickly as possible
> > since
> > > > it
> > > > > is rather mature and needs to get to the next level.
> > > > >
> > > > > Again, any potential Mentors that would like to join are more than
> > > > welcome.
> > > > >
> > > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
> > > justin@classsoftware.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> I’m sorry, but Imran Rashid can’t be a mentor for the project as
> > they
> > > > are
> > > > >> not an IPMC member. Currently, both Sunil and Larry (as they are
> ASF
> > > > >> members) need to ask to join the IPMC and NOTICE sent to the ASF
> > > board.
> > > > I
> > > > >> would also prefer that mentors come from different companies.
> > > > >>
> > > > >> There is also no concept as an emeritus PPMC member at the ASF.
> > > > >>
> > > > >> Kind Regards,
> > > > >> Justin
> > > > >>
> > > > >>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > >> For additional commands, e-mail:
> general-help@incubator.apache.org
> > > > >>
> > > > >>
> > > >
> > >
> >
>
Re: Proposal to Revive Apache Livy Community
Posted by Madhawa Gunasekara <ma...@gmail.com>.
Hi Larry,
I'm an IPMC Member. madhawa30 at gmail dot com is my preferred email
address.
apache id: madhawa
Thanks,
Madhawa
On Tue, Oct 18, 2022 at 10:05 PM larry mccay <lm...@apache.org> wrote:
> Hi Madhawa -
>
> That's awesome!
> Are you already a member of IPMC?
> If not, are you an ASF member?
> If you are an ASF member you can request that you be added as an IPMC
> member.
>
> Can you provide your company affiliation for the proposal and preferred
> email?
>
> thanks!
>
> --larry
>
> On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <ma...@gmail.com>
> wrote:
>
> > Hi Larry,
> >
> > I'm interested in working with Livy and would like to join as a Mentor.
> >
> > Thanks,
> > Madhawa
> >
> >
> > On Tue, Oct 18, 2022 at 6:57 PM larry mccay <lm...@apache.org> wrote:
> >
> > > Sorry, I missed commenting on this:
> > >
> > > "There is also no concept as an emeritus PPMC member at the ASF."
> > >
> > > I assume that we can remove PPMC members that do not opt-in explicitly
> at
> > > this point.
> > > They will have every opportunity to rejoin.
> > >
> > > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <lm...@apache.org>
> wrote:
> > >
> > > > I will ask in a separate thread, @Justin Mclean <justinmclean@me.com
> >
> > -
> > > > thanks.
> > > > Adding JB adds another company and we are certainly open to anyone
> else
> > > > that would like to join as a mentor.
> > > > At the end of the day, the mentors are for instilling the Apache Way
> > > > knowledge and steering toward graduation.
> > > > I feel that this diversity, while nice to have, is less important
> than
> > > > that of the PPMC and committers for the long term health of the
> > > community.
> > > >
> > > > We need to push this podling to graduation as quickly as possible
> since
> > > it
> > > > is rather mature and needs to get to the next level.
> > > >
> > > > Again, any potential Mentors that would like to join are more than
> > > welcome.
> > > >
> > > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
> > justin@classsoftware.com
> > > >
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> I’m sorry, but Imran Rashid can’t be a mentor for the project as
> they
> > > are
> > > >> not an IPMC member. Currently, both Sunil and Larry (as they are ASF
> > > >> members) need to ask to join the IPMC and NOTICE sent to the ASF
> > board.
> > > I
> > > >> would also prefer that mentors come from different companies.
> > > >>
> > > >> There is also no concept as an emeritus PPMC member at the ASF.
> > > >>
> > > >> Kind Regards,
> > > >> Justin
> > > >>
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > >> For additional commands, e-mail: general-help@incubator.apache.org
> > > >>
> > > >>
> > >
> >
>
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
Hi Madhawa -
That's awesome!
Are you already a member of IPMC?
If not, are you an ASF member?
If you are an ASF member you can request that you be added as an IPMC
member.
Can you provide your company affiliation for the proposal and preferred
email?
thanks!
--larry
On Tue, Oct 18, 2022 at 2:18 PM Madhawa Gunasekara <ma...@gmail.com>
wrote:
> Hi Larry,
>
> I'm interested in working with Livy and would like to join as a Mentor.
>
> Thanks,
> Madhawa
>
>
> On Tue, Oct 18, 2022 at 6:57 PM larry mccay <lm...@apache.org> wrote:
>
> > Sorry, I missed commenting on this:
> >
> > "There is also no concept as an emeritus PPMC member at the ASF."
> >
> > I assume that we can remove PPMC members that do not opt-in explicitly at
> > this point.
> > They will have every opportunity to rejoin.
> >
> > On Tue, Oct 18, 2022 at 12:48 PM larry mccay <lm...@apache.org> wrote:
> >
> > > I will ask in a separate thread, @Justin Mclean <ju...@me.com>
> -
> > > thanks.
> > > Adding JB adds another company and we are certainly open to anyone else
> > > that would like to join as a mentor.
> > > At the end of the day, the mentors are for instilling the Apache Way
> > > knowledge and steering toward graduation.
> > > I feel that this diversity, while nice to have, is less important than
> > > that of the PPMC and committers for the long term health of the
> > community.
> > >
> > > We need to push this podling to graduation as quickly as possible since
> > it
> > > is rather mature and needs to get to the next level.
> > >
> > > Again, any potential Mentors that would like to join are more than
> > welcome.
> > >
> > > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <
> justin@classsoftware.com
> > >
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> I’m sorry, but Imran Rashid can’t be a mentor for the project as they
> > are
> > >> not an IPMC member. Currently, both Sunil and Larry (as they are ASF
> > >> members) need to ask to join the IPMC and NOTICE sent to the ASF
> board.
> > I
> > >> would also prefer that mentors come from different companies.
> > >>
> > >> There is also no concept as an emeritus PPMC member at the ASF.
> > >>
> > >> Kind Regards,
> > >> Justin
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > >> For additional commands, e-mail: general-help@incubator.apache.org
> > >>
> > >>
> >
>
Re: Proposal to Revive Apache Livy Community
Posted by Madhawa Gunasekara <ma...@gmail.com>.
Hi Larry,
I'm interested in working with Livy and would like to join as a Mentor.
Thanks,
Madhawa
On Tue, Oct 18, 2022 at 6:57 PM larry mccay <lm...@apache.org> wrote:
> Sorry, I missed commenting on this:
>
> "There is also no concept as an emeritus PPMC member at the ASF."
>
> I assume that we can remove PPMC members that do not opt-in explicitly at
> this point.
> They will have every opportunity to rejoin.
>
> On Tue, Oct 18, 2022 at 12:48 PM larry mccay <lm...@apache.org> wrote:
>
> > I will ask in a separate thread, @Justin Mclean <ju...@me.com> -
> > thanks.
> > Adding JB adds another company and we are certainly open to anyone else
> > that would like to join as a mentor.
> > At the end of the day, the mentors are for instilling the Apache Way
> > knowledge and steering toward graduation.
> > I feel that this diversity, while nice to have, is less important than
> > that of the PPMC and committers for the long term health of the
> community.
> >
> > We need to push this podling to graduation as quickly as possible since
> it
> > is rather mature and needs to get to the next level.
> >
> > Again, any potential Mentors that would like to join are more than
> welcome.
> >
> > On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <justin@classsoftware.com
> >
> > wrote:
> >
> >> Hi,
> >>
> >> I’m sorry, but Imran Rashid can’t be a mentor for the project as they
> are
> >> not an IPMC member. Currently, both Sunil and Larry (as they are ASF
> >> members) need to ask to join the IPMC and NOTICE sent to the ASF board.
> I
> >> would also prefer that mentors come from different companies.
> >>
> >> There is also no concept as an emeritus PPMC member at the ASF.
> >>
> >> Kind Regards,
> >> Justin
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: general-help@incubator.apache.org
> >>
> >>
>
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
Sorry, I missed commenting on this:
"There is also no concept as an emeritus PPMC member at the ASF."
I assume that we can remove PPMC members that do not opt-in explicitly at
this point.
They will have every opportunity to rejoin.
On Tue, Oct 18, 2022 at 12:48 PM larry mccay <lm...@apache.org> wrote:
> I will ask in a separate thread, @Justin Mclean <ju...@me.com> -
> thanks.
> Adding JB adds another company and we are certainly open to anyone else
> that would like to join as a mentor.
> At the end of the day, the mentors are for instilling the Apache Way
> knowledge and steering toward graduation.
> I feel that this diversity, while nice to have, is less important than
> that of the PPMC and committers for the long term health of the community.
>
> We need to push this podling to graduation as quickly as possible since it
> is rather mature and needs to get to the next level.
>
> Again, any potential Mentors that would like to join are more than welcome.
>
> On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <ju...@classsoftware.com>
> wrote:
>
>> Hi,
>>
>> I’m sorry, but Imran Rashid can’t be a mentor for the project as they are
>> not an IPMC member. Currently, both Sunil and Larry (as they are ASF
>> members) need to ask to join the IPMC and NOTICE sent to the ASF board. I
>> would also prefer that mentors come from different companies.
>>
>> There is also no concept as an emeritus PPMC member at the ASF.
>>
>> Kind Regards,
>> Justin
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
I will ask in a separate thread, @Justin Mclean <ju...@me.com> -
thanks.
Adding JB adds another company and we are certainly open to anyone else
that would like to join as a mentor.
At the end of the day, the mentors are for instilling the Apache Way
knowledge and steering toward graduation.
I feel that this diversity, while nice to have, is less important than that
of the PPMC and committers for the long term health of the community.
We need to push this podling to graduation as quickly as possible since it
is rather mature and needs to get to the next level.
Again, any potential Mentors that would like to join are more than welcome.
On Tue, Oct 18, 2022 at 12:38 PM Justin Mclean <ju...@classsoftware.com>
wrote:
> Hi,
>
> I’m sorry, but Imran Rashid can’t be a mentor for the project as they are
> not an IPMC member. Currently, both Sunil and Larry (as they are ASF
> members) need to ask to join the IPMC and NOTICE sent to the ASF board. I
> would also prefer that mentors come from different companies.
>
> There is also no concept as an emeritus PPMC member at the ASF.
>
> Kind Regards,
> Justin
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: Proposal to Revive Apache Livy Community
Posted by Justin Mclean <ju...@classsoftware.com>.
Hi,
I’m sorry, but Imran Rashid can’t be a mentor for the project as they are not an IPMC member. Currently, both Sunil and Larry (as they are ASF members) need to ask to join the IPMC and NOTICE sent to the ASF board. I would also prefer that mentors come from different companies.
There is also no concept as an emeritus PPMC member at the ASF.
Kind Regards,
Justin
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <la...@gmail.com>.
Revised Proposal with opt-out for committers and JB as a mentor which adds
more diversity to the company based mentors.
These were items suggested on this thread. Thanks again, Justin!
Abstract
Livy is a web service that exposes a REST interface for managing long
running Apache Spark contexts in your cluster. With Livy, new applications
can be built on top of Apache Spark that require fine grained interaction
with many Spark contexts [1].
While this project has been well regarded and used in many contexts as the
defacto standard API to Spark environments, it has been incubating for over
5 years without graduation to a TLP and it has become difficult to
impossible for fixes and improvements to be contributed as the current
community seems to have moved on.
There has been discussion regarding retirement of this podling where there
seems to be some increasing interest in joining and reviving the community
[2].
The intent of this proposal is to avoid retiring a well regarded, actively
used and rather mature project by reviving the PPMC and community with new
folks that have a vested interest in the project and health of the
community.
Proposal
We propose to revive the PPMC with a set of contributors and maintainers as
mentors, PPMC members and committers.
The retirement DISCUSS thread [2] has shown a growing interest in providing
new committers and bringing improvements and fixes from organization’s
internally maintained forks back to a revived community.
General Approach to Revival:
-
Add new Mentors
-
Larry McCay, lmccay@apache.org , Cloudera
-
Sunil Govindan, sunilg@apache.org, Cloudera
-
Imran Rashid - irashid@apache.org, Cloudera
-
Jean-Baptiste Onofré, jbonofre@apache.org, Talend
-
Add new Committers/PPMC
-
Larry McCay, lmccay@apache.org, Cloudera
-
Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
-
Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
-
Wing Yew Poon, wypoon@cloudera.com, Cloudera
-
Xilang Yan, xilang.yan@gmail.com, Shopee
-
Jianzhen Wu, myjianzhen@gmail.com, Shopee
-
Nagella Jagadeewara Rao, jnagella@visa.com, Visa
-
Pralab Kumar, pralkuma@visa.com, Visa
-
Prasad Shrikant, shrikant.ism@gmail.com, Visa
-
Brahma Reddy Battula, brahma@apache.org, Visa
-
Invite existing PPMC members to opt-in or otherwise go emeritus
-
Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in via
Retirement DISCUSS thread [2])
-
Invite existing Committers to opt-out or otherwise continue
-
Establish Roadmap via follow up DISCUSS thread
-
Known Improvements from Forks which will need proposals and
discussion:
-
Adding HA for Livy
-
Updating security capabilities (eg. kerberos for jdbc, fixing bugs
in encryption)
-
Expanding the support for kubernetes
-
Responding to CVEs in dependencies (eg. log4j, thrift)
-
Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
-
Support multi Spark versions
-
Implemented a metrics system for Livy
-
Support customize batch/interactive session lifecycle event
handler, default log event with log4j, very helpful for troubleshooting
-
Optimize log to track which session id the log message came from,
also very helpful for troubleshooting
-
Support customize Spark config optimization rules, can be used to
optimize config for users’ job
-
A set of command line tool which can be used to replace Spark’s
spark-submit, pyspark, spark-sql but actually submit
application in Livy
-
We are planning to implement a JDBC state store, and allow multi
Livy Thrift sessions to share one backend Spark application
in the next few
months.
-
These items and others that are brought to community may need
consolidation or multiple configurable options and will need to
be part of
the discussion
-
One-pager Livy Improvement Proposals (LIP) may make sense to drive
these discussions and convergence
-
Feature Branch Strategy for large changes
-
Large features are hard to review we will need to define a
strategy
-
CTR for feature branches and a we will define a walkthrough of
implementation details to aid in review for merge
-
Determine the Improvements to be delivered across first 3 Releases
with Target Release Dates
-
Ensure CVE and Dependency management hygiene is in place
The above approach will usher the community back to an active status with a
Roadmap of 3 or more release plans and security hygiene in place.
Development Practices
The Livy project follows a review before commit philosophy. Every commit
automatically runs through the unit tests and generates coverage reports
presented as a pull request comment. Our experience with this process leads
us to believe that it helps ease new contributors into the project. They
get feedback quickly on common mistakes, lowering the burden on reviewers.
Those same reviewers get to lead by example, showing the new contributors
that we value feedback within our community even when changes are done by
more experienced folks. Taken from the original Apache Livy Proposal [1],
this should continue to be true. As mentioned, Livy is a mature project and
as such RTC is the most appropriate for continued quality and awareness.
1.
Original Apache Livy Proposal
https://cwiki.apache.org/confluence/display/incubator/LivyProposal
2.
Retirement DISCUSS thread
https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
On Sun, Oct 16, 2022 at 5:14 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Larry,
>
> thanks for this !
>
> I'm still interested to work on Livy. Can you please add me as mentor ?
>
> Thanks !
> Regards
> JB
>
> On Fri, Oct 14, 2022 at 8:38 PM larry mccay <la...@gmail.com> wrote:
> >
> > Abstract
> >
> > Livy is a web service that exposes a REST interface for managing long
> > running Apache Spark contexts in your cluster. With Livy, new
> applications
> > can be built on top of Apache Spark that require fine grained interaction
> > with many Spark contexts [1].
> >
> > While this project has been well regarded and used in many contexts as
> the
> > defacto standard API to Spark environments, it has been incubating for
> over
> > 5 years without graduation to a TLP and it has become difficult to
> > impossible for fixes and improvements to be contributed as the current
> > community seems to have moved on.
> >
> > There has been discussion regarding retirement of this podling where
> there
> > seems to be some increasing interest in joining and reviving the
> community
> > [2].
> >
> > The intent of this proposal is to avoid retiring a well regarded,
> actively
> > used and rather mature project by reviving the PPMC and community with
> new
> > folks that have a vested interest in the project and health of the
> > community.
> > Proposal
> >
> > We propose to revive the PPMC with a set of contributors and maintainers
> as
> > mentors, PPMC members and committers.
> >
> > The retirement DISCUSS thread [2] has shown a growing interest in
> providing
> > new committers and bringing improvements and fixes from organization’s
> > internally maintained forks back to a revived community.
> >
> > General Approach to Revival:
> >
> > -
> >
> > Add new Mentors
> > -
> >
> > Larry McCay, lmccay@apache.org , Cloudera
> > -
> >
> > Sunil Govindan, sunilg@apache.org, Cloudera
> > -
> >
> > Imran Rashid - irashid@apache.org, Cloudera
> >
> >
> >
> > -
> >
> > Add new Committers/PPMC
> > -
> >
> > Larry McCay, lmccay@apache.org, Cloudera
> > -
> >
> > Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
> > -
> >
> > Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
> > -
> >
> > Wing Yew Poon, wypoon@cloudera.com, Cloudera
> > -
> >
> > Xilang Yan, xilang.yan@gmail.com, Shopee
> > -
> >
> > Jianzhen Wu, myjianzhen@gmail.com, Shopee
> > -
> >
> > Nagella Jagadeewara Rao, jnagella@visa.com, Visa
> > -
> >
> > Pralab Kumar, pralkuma@visa.com, Visa
> > -
> >
> > Prasad Shrikant, shrikant.ism@gmail.com, Visa
> > -
> >
> > Brahma Reddy Battula, brahma@apache.org, Visa
> >
> >
> >
> > -
> >
> > Invite existing PPMC members to opt-in or otherwise go emeritus
> > -
> >
> > Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in via
> > Retirement DISCUSS thread [2])
> >
> >
> >
> > -
> >
> > Invite existing Committers/PPMC members to opt-in or otherwise go
> > emeritus
> >
> >
> >
> > -
> >
> > Establish Roadmap via follow up DISCUSS thread
> > -
> >
> > Known Improvements from Forks which will need proposals and
> > discussion:
> > -
> >
> > Adding HA for Livy
> > -
> >
> > Updating security capabilities (eg. kerberos for jdbc, fixing
> bugs
> > in encryption)
> > -
> >
> > Expanding the support for kubernetes
> > -
> >
> > Responding to CVEs in dependencies (eg. log4j, thrift)
> > -
> >
> > Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
> > -
> >
> > Support multi Spark versions
> > -
> >
> > Implemented a metrics system for Livy
> > -
> >
> > Support customize batch/interactive session lifecycle event
> > handler, default log event with log4j, very helpful for
> > trouble shooting
> > -
> >
> > Optimize log to track which session id the log message came
> from,
> > also very helpful for trouble shooting
> > -
> >
> > Support customize Spark config optimization rules, can be used
> to
> > optimize config for users’ job
> > -
> >
> > A set of command line tool which can be used to replace Spark’s
> > spark-submit, pyspark, spark-sql but actually submit
> > application in Livy
> > -
> >
> > We are planning to implement a JDBC state store, and allow multi
> > Livy Thrift sessions to share one backend Spark application
> > in the next few
> > months.
> > -
> >
> > These items and others that are brought to community may need
> > consolidation or multiple configurable options and will need to
> > be part of
> > the discussion
> > -
> >
> > One-pager Livy Improvement Proposals (LIP) may make sense to
> drive
> > these discussions and convergence
> > -
> >
> > Feature Branch Strategy for large changes
> > -
> >
> > Large features are hard to review we will need to define a
> > strategy
> > -
> >
> > Determine the Improvements to be delivered across first 3 Releases
> > with Target Release Dates
> >
> >
> >
> > -
> >
> > Ensure CVE and Dependency management hygiene is in place
> >
> >
> > The above approach will usher the community back to an active status
> with a
> > Roadmap of 3 or more release plans and security hygiene in place.
> > Development Practices
> >
> > The Livy project follows a review before commit philosophy. Every commit
> > automatically runs through the unit tests and generates coverage reports
> > presented as a pull request comment. Our experience with this process
> leads
> > us to believe that it helps ease new contributors into the project. They
> > get feedback quickly on common mistakes, lowering the burden on
> reviewers.
> > Those same reviewers get to lead by example, showing the new contributors
> > that we value feedback within our community even when changes are done by
> > more experienced folks. Taken from the original Apache Livy Proposal [1],
> > this should continue to be true. As mentioned, Livy is a mature project
> and
> > as such RTC is the most appropriate for continued quality and awareness.
> >
> >
> >
> > 1.
> >
> > Original Apache Livy Proposal
> > https://cwiki.apache.org/confluence/display/incubator/LivyProposal
> > 2.
> >
> > Retirement DISCUSS thread
> > https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <la...@gmail.com>.
Revised Proposal with opt-out for committers and JB as a mentor which adds
more diversity to the company based mentors.
These were items suggested on this thread. Thanks again, Justin!
Abstract
Livy is a web service that exposes a REST interface for managing long
running Apache Spark contexts in your cluster. With Livy, new applications
can be built on top of Apache Spark that require fine grained interaction
with many Spark contexts [1].
While this project has been well regarded and used in many contexts as the
defacto standard API to Spark environments, it has been incubating for over
5 years without graduation to a TLP and it has become difficult to
impossible for fixes and improvements to be contributed as the current
community seems to have moved on.
There has been discussion regarding retirement of this podling where there
seems to be some increasing interest in joining and reviving the community
[2].
The intent of this proposal is to avoid retiring a well regarded, actively
used and rather mature project by reviving the PPMC and community with new
folks that have a vested interest in the project and health of the
community.
Proposal
We propose to revive the PPMC with a set of contributors and maintainers as
mentors, PPMC members and committers.
The retirement DISCUSS thread [2] has shown a growing interest in providing
new committers and bringing improvements and fixes from organization’s
internally maintained forks back to a revived community.
General Approach to Revival:
-
Add new Mentors
-
Larry McCay, lmccay@apache.org , Cloudera
-
Sunil Govindan, sunilg@apache.org, Cloudera
-
Imran Rashid - irashid@apache.org, Cloudera
-
Jean-Baptiste Onofré, jbonofre@apache.org, Talend
-
Add new Committers/PPMC
-
Larry McCay, lmccay@apache.org, Cloudera
-
Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
-
Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
-
Wing Yew Poon, wypoon@cloudera.com, Cloudera
-
Xilang Yan, xilang.yan@gmail.com, Shopee
-
Jianzhen Wu, myjianzhen@gmail.com, Shopee
-
Nagella Jagadeewara Rao, jnagella@visa.com, Visa
-
Pralab Kumar, pralkuma@visa.com, Visa
-
Prasad Shrikant, shrikant.ism@gmail.com, Visa
-
Brahma Reddy Battula, brahma@apache.org, Visa
-
Invite existing PPMC members to opt-in or otherwise go emeritus
-
Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in via
Retirement DISCUSS thread [2])
-
Invite existing Committers to opt-out or otherwise continue
-
Establish Roadmap via follow up DISCUSS thread
-
Known Improvements from Forks which will need proposals and
discussion:
-
Adding HA for Livy
-
Updating security capabilities (eg. kerberos for jdbc, fixing bugs
in encryption)
-
Expanding the support for kubernetes
-
Responding to CVEs in dependencies (eg. log4j, thrift)
-
Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
-
Support multi Spark versions
-
Implemented a metrics system for Livy
-
Support customize batch/interactive session lifecycle event
handler, default log event with log4j, very helpful for troubleshooting
-
Optimize log to track which session id the log message came from,
also very helpful for troubleshooting
-
Support customize Spark config optimization rules, can be used to
optimize config for users’ job
-
A set of command line tool which can be used to replace Spark’s
spark-submit, pyspark, spark-sql but actually submit
application in Livy
-
We are planning to implement a JDBC state store, and allow multi
Livy Thrift sessions to share one backend Spark application
in the next few
months.
-
These items and others that are brought to community may need
consolidation or multiple configurable options and will need to
be part of
the discussion
-
One-pager Livy Improvement Proposals (LIP) may make sense to drive
these discussions and convergence
-
Feature Branch Strategy for large changes
-
Large features are hard to review we will need to define a
strategy
-
CTR for feature branches and a we will define a walkthrough of
implementation details to aid in review for merge
-
Determine the Improvements to be delivered across first 3 Releases
with Target Release Dates
-
Ensure CVE and Dependency management hygiene is in place
The above approach will usher the community back to an active status with a
Roadmap of 3 or more release plans and security hygiene in place.
Development Practices
The Livy project follows a review before commit philosophy. Every commit
automatically runs through the unit tests and generates coverage reports
presented as a pull request comment. Our experience with this process leads
us to believe that it helps ease new contributors into the project. They
get feedback quickly on common mistakes, lowering the burden on reviewers.
Those same reviewers get to lead by example, showing the new contributors
that we value feedback within our community even when changes are done by
more experienced folks. Taken from the original Apache Livy Proposal [1],
this should continue to be true. As mentioned, Livy is a mature project and
as such RTC is the most appropriate for continued quality and awareness.
1.
Original Apache Livy Proposal
https://cwiki.apache.org/confluence/display/incubator/LivyProposal
2.
Retirement DISCUSS thread
https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
On Sun, Oct 16, 2022 at 5:14 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Larry,
>
> thanks for this !
>
> I'm still interested to work on Livy. Can you please add me as mentor ?
>
> Thanks !
> Regards
> JB
>
> On Fri, Oct 14, 2022 at 8:38 PM larry mccay <la...@gmail.com> wrote:
> >
> > Abstract
> >
> > Livy is a web service that exposes a REST interface for managing long
> > running Apache Spark contexts in your cluster. With Livy, new
> applications
> > can be built on top of Apache Spark that require fine grained interaction
> > with many Spark contexts [1].
> >
> > While this project has been well regarded and used in many contexts as
> the
> > defacto standard API to Spark environments, it has been incubating for
> over
> > 5 years without graduation to a TLP and it has become difficult to
> > impossible for fixes and improvements to be contributed as the current
> > community seems to have moved on.
> >
> > There has been discussion regarding retirement of this podling where
> there
> > seems to be some increasing interest in joining and reviving the
> community
> > [2].
> >
> > The intent of this proposal is to avoid retiring a well regarded,
> actively
> > used and rather mature project by reviving the PPMC and community with
> new
> > folks that have a vested interest in the project and health of the
> > community.
> > Proposal
> >
> > We propose to revive the PPMC with a set of contributors and maintainers
> as
> > mentors, PPMC members and committers.
> >
> > The retirement DISCUSS thread [2] has shown a growing interest in
> providing
> > new committers and bringing improvements and fixes from organization’s
> > internally maintained forks back to a revived community.
> >
> > General Approach to Revival:
> >
> > -
> >
> > Add new Mentors
> > -
> >
> > Larry McCay, lmccay@apache.org , Cloudera
> > -
> >
> > Sunil Govindan, sunilg@apache.org, Cloudera
> > -
> >
> > Imran Rashid - irashid@apache.org, Cloudera
> >
> >
> >
> > -
> >
> > Add new Committers/PPMC
> > -
> >
> > Larry McCay, lmccay@apache.org, Cloudera
> > -
> >
> > Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
> > -
> >
> > Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
> > -
> >
> > Wing Yew Poon, wypoon@cloudera.com, Cloudera
> > -
> >
> > Xilang Yan, xilang.yan@gmail.com, Shopee
> > -
> >
> > Jianzhen Wu, myjianzhen@gmail.com, Shopee
> > -
> >
> > Nagella Jagadeewara Rao, jnagella@visa.com, Visa
> > -
> >
> > Pralab Kumar, pralkuma@visa.com, Visa
> > -
> >
> > Prasad Shrikant, shrikant.ism@gmail.com, Visa
> > -
> >
> > Brahma Reddy Battula, brahma@apache.org, Visa
> >
> >
> >
> > -
> >
> > Invite existing PPMC members to opt-in or otherwise go emeritus
> > -
> >
> > Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in via
> > Retirement DISCUSS thread [2])
> >
> >
> >
> > -
> >
> > Invite existing Committers/PPMC members to opt-in or otherwise go
> > emeritus
> >
> >
> >
> > -
> >
> > Establish Roadmap via follow up DISCUSS thread
> > -
> >
> > Known Improvements from Forks which will need proposals and
> > discussion:
> > -
> >
> > Adding HA for Livy
> > -
> >
> > Updating security capabilities (eg. kerberos for jdbc, fixing
> bugs
> > in encryption)
> > -
> >
> > Expanding the support for kubernetes
> > -
> >
> > Responding to CVEs in dependencies (eg. log4j, thrift)
> > -
> >
> > Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
> > -
> >
> > Support multi Spark versions
> > -
> >
> > Implemented a metrics system for Livy
> > -
> >
> > Support customize batch/interactive session lifecycle event
> > handler, default log event with log4j, very helpful for
> > trouble shooting
> > -
> >
> > Optimize log to track which session id the log message came
> from,
> > also very helpful for trouble shooting
> > -
> >
> > Support customize Spark config optimization rules, can be used
> to
> > optimize config for users’ job
> > -
> >
> > A set of command line tool which can be used to replace Spark’s
> > spark-submit, pyspark, spark-sql but actually submit
> > application in Livy
> > -
> >
> > We are planning to implement a JDBC state store, and allow multi
> > Livy Thrift sessions to share one backend Spark application
> > in the next few
> > months.
> > -
> >
> > These items and others that are brought to community may need
> > consolidation or multiple configurable options and will need to
> > be part of
> > the discussion
> > -
> >
> > One-pager Livy Improvement Proposals (LIP) may make sense to
> drive
> > these discussions and convergence
> > -
> >
> > Feature Branch Strategy for large changes
> > -
> >
> > Large features are hard to review we will need to define a
> > strategy
> > -
> >
> > Determine the Improvements to be delivered across first 3 Releases
> > with Target Release Dates
> >
> >
> >
> > -
> >
> > Ensure CVE and Dependency management hygiene is in place
> >
> >
> > The above approach will usher the community back to an active status
> with a
> > Roadmap of 3 or more release plans and security hygiene in place.
> > Development Practices
> >
> > The Livy project follows a review before commit philosophy. Every commit
> > automatically runs through the unit tests and generates coverage reports
> > presented as a pull request comment. Our experience with this process
> leads
> > us to believe that it helps ease new contributors into the project. They
> > get feedback quickly on common mistakes, lowering the burden on
> reviewers.
> > Those same reviewers get to lead by example, showing the new contributors
> > that we value feedback within our community even when changes are done by
> > more experienced folks. Taken from the original Apache Livy Proposal [1],
> > this should continue to be true. As mentioned, Livy is a mature project
> and
> > as such RTC is the most appropriate for continued quality and awareness.
> >
> >
> >
> > 1.
> >
> > Original Apache Livy Proposal
> > https://cwiki.apache.org/confluence/display/incubator/LivyProposal
> > 2.
> >
> > Retirement DISCUSS thread
> > https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: Proposal to Revive Apache Livy Community
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Larry,
thanks for this !
I'm still interested to work on Livy. Can you please add me as mentor ?
Thanks !
Regards
JB
On Fri, Oct 14, 2022 at 8:38 PM larry mccay <la...@gmail.com> wrote:
>
> Abstract
>
> Livy is a web service that exposes a REST interface for managing long
> running Apache Spark contexts in your cluster. With Livy, new applications
> can be built on top of Apache Spark that require fine grained interaction
> with many Spark contexts [1].
>
> While this project has been well regarded and used in many contexts as the
> defacto standard API to Spark environments, it has been incubating for over
> 5 years without graduation to a TLP and it has become difficult to
> impossible for fixes and improvements to be contributed as the current
> community seems to have moved on.
>
> There has been discussion regarding retirement of this podling where there
> seems to be some increasing interest in joining and reviving the community
> [2].
>
> The intent of this proposal is to avoid retiring a well regarded, actively
> used and rather mature project by reviving the PPMC and community with new
> folks that have a vested interest in the project and health of the
> community.
> Proposal
>
> We propose to revive the PPMC with a set of contributors and maintainers as
> mentors, PPMC members and committers.
>
> The retirement DISCUSS thread [2] has shown a growing interest in providing
> new committers and bringing improvements and fixes from organization’s
> internally maintained forks back to a revived community.
>
> General Approach to Revival:
>
> -
>
> Add new Mentors
> -
>
> Larry McCay, lmccay@apache.org , Cloudera
> -
>
> Sunil Govindan, sunilg@apache.org, Cloudera
> -
>
> Imran Rashid - irashid@apache.org, Cloudera
>
>
>
> -
>
> Add new Committers/PPMC
> -
>
> Larry McCay, lmccay@apache.org, Cloudera
> -
>
> Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
> -
>
> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
> -
>
> Wing Yew Poon, wypoon@cloudera.com, Cloudera
> -
>
> Xilang Yan, xilang.yan@gmail.com, Shopee
> -
>
> Jianzhen Wu, myjianzhen@gmail.com, Shopee
> -
>
> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
> -
>
> Pralab Kumar, pralkuma@visa.com, Visa
> -
>
> Prasad Shrikant, shrikant.ism@gmail.com, Visa
> -
>
> Brahma Reddy Battula, brahma@apache.org, Visa
>
>
>
> -
>
> Invite existing PPMC members to opt-in or otherwise go emeritus
> -
>
> Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in via
> Retirement DISCUSS thread [2])
>
>
>
> -
>
> Invite existing Committers/PPMC members to opt-in or otherwise go
> emeritus
>
>
>
> -
>
> Establish Roadmap via follow up DISCUSS thread
> -
>
> Known Improvements from Forks which will need proposals and
> discussion:
> -
>
> Adding HA for Livy
> -
>
> Updating security capabilities (eg. kerberos for jdbc, fixing bugs
> in encryption)
> -
>
> Expanding the support for kubernetes
> -
>
> Responding to CVEs in dependencies (eg. log4j, thrift)
> -
>
> Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
> -
>
> Support multi Spark versions
> -
>
> Implemented a metrics system for Livy
> -
>
> Support customize batch/interactive session lifecycle event
> handler, default log event with log4j, very helpful for
> trouble shooting
> -
>
> Optimize log to track which session id the log message came from,
> also very helpful for trouble shooting
> -
>
> Support customize Spark config optimization rules, can be used to
> optimize config for users’ job
> -
>
> A set of command line tool which can be used to replace Spark’s
> spark-submit, pyspark, spark-sql but actually submit
> application in Livy
> -
>
> We are planning to implement a JDBC state store, and allow multi
> Livy Thrift sessions to share one backend Spark application
> in the next few
> months.
> -
>
> These items and others that are brought to community may need
> consolidation or multiple configurable options and will need to
> be part of
> the discussion
> -
>
> One-pager Livy Improvement Proposals (LIP) may make sense to drive
> these discussions and convergence
> -
>
> Feature Branch Strategy for large changes
> -
>
> Large features are hard to review we will need to define a
> strategy
> -
>
> Determine the Improvements to be delivered across first 3 Releases
> with Target Release Dates
>
>
>
> -
>
> Ensure CVE and Dependency management hygiene is in place
>
>
> The above approach will usher the community back to an active status with a
> Roadmap of 3 or more release plans and security hygiene in place.
> Development Practices
>
> The Livy project follows a review before commit philosophy. Every commit
> automatically runs through the unit tests and generates coverage reports
> presented as a pull request comment. Our experience with this process leads
> us to believe that it helps ease new contributors into the project. They
> get feedback quickly on common mistakes, lowering the burden on reviewers.
> Those same reviewers get to lead by example, showing the new contributors
> that we value feedback within our community even when changes are done by
> more experienced folks. Taken from the original Apache Livy Proposal [1],
> this should continue to be true. As mentioned, Livy is a mature project and
> as such RTC is the most appropriate for continued quality and awareness.
>
>
>
> 1.
>
> Original Apache Livy Proposal
> https://cwiki.apache.org/confluence/display/incubator/LivyProposal
> 2.
>
> Retirement DISCUSS thread
> https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <la...@gmail.com>.
+dev@livy.incubator.apache.org <de...@livy.incubator.apache.org>
On Fri, Oct 14, 2022 at 2:38 PM larry mccay <la...@gmail.com> wrote:
> Abstract
>
> Livy is a web service that exposes a REST interface for managing long
> running Apache Spark contexts in your cluster. With Livy, new applications
> can be built on top of Apache Spark that require fine grained interaction
> with many Spark contexts [1].
>
> While this project has been well regarded and used in many contexts as the
> defacto standard API to Spark environments, it has been incubating for over
> 5 years without graduation to a TLP and it has become difficult to
> impossible for fixes and improvements to be contributed as the current
> community seems to have moved on.
>
> There has been discussion regarding retirement of this podling where there
> seems to be some increasing interest in joining and reviving the community
> [2].
>
> The intent of this proposal is to avoid retiring a well regarded, actively
> used and rather mature project by reviving the PPMC and community with new
> folks that have a vested interest in the project and health of the
> community.
> Proposal
>
> We propose to revive the PPMC with a set of contributors and maintainers
> as mentors, PPMC members and committers.
>
> The retirement DISCUSS thread [2] has shown a growing interest in
> providing new committers and bringing improvements and fixes from
> organization’s internally maintained forks back to a revived community.
>
> General Approach to Revival:
>
> -
>
> Add new Mentors
> -
>
> Larry McCay, lmccay@apache.org , Cloudera
> -
>
> Sunil Govindan, sunilg@apache.org, Cloudera
> -
>
> Imran Rashid - irashid@apache.org, Cloudera
>
>
>
> -
>
> Add new Committers/PPMC
> -
>
> Larry McCay, lmccay@apache.org, Cloudera
> -
>
> Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
> -
>
> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
> -
>
> Wing Yew Poon, wypoon@cloudera.com, Cloudera
> -
>
> Xilang Yan, xilang.yan@gmail.com, Shopee
> -
>
> Jianzhen Wu, myjianzhen@gmail.com, Shopee
> -
>
> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
> -
>
> Pralab Kumar, pralkuma@visa.com, Visa
> -
>
> Prasad Shrikant, shrikant.ism@gmail.com, Visa
> -
>
> Brahma Reddy Battula, brahma@apache.org, Visa
>
>
>
> -
>
> Invite existing PPMC members to opt-in or otherwise go emeritus
> -
>
> Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in via
> Retirement DISCUSS thread [2])
>
>
>
> -
>
> Invite existing Committers/PPMC members to opt-in or otherwise go
> emeritus
>
>
>
> -
>
> Establish Roadmap via follow up DISCUSS thread
> -
>
> Known Improvements from Forks which will need proposals and
> discussion:
> -
>
> Adding HA for Livy
> -
>
> Updating security capabilities (eg. kerberos for jdbc, fixing
> bugs in encryption)
> -
>
> Expanding the support for kubernetes
> -
>
> Responding to CVEs in dependencies (eg. log4j, thrift)
> -
>
> Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
> -
>
> Support multi Spark versions
> -
>
> Implemented a metrics system for Livy
> -
>
> Support customize batch/interactive session lifecycle event
> handler, default log event with log4j, very helpful for trouble shooting
> -
>
> Optimize log to track which session id the log message came
> from, also very helpful for trouble shooting
> -
>
> Support customize Spark config optimization rules, can be used
> to optimize config for users’ job
> -
>
> A set of command line tool which can be used to replace Spark’s
> spark-submit, pyspark, spark-sql but actually submit application in Livy
> -
>
> We are planning to implement a JDBC state store, and allow multi
> Livy Thrift sessions to share one backend Spark application in the next few
> months.
> -
>
> These items and others that are brought to community may need
> consolidation or multiple configurable options and will need to be part of
> the discussion
> -
>
> One-pager Livy Improvement Proposals (LIP) may make sense to
> drive these discussions and convergence
> -
>
> Feature Branch Strategy for large changes
> -
>
> Large features are hard to review we will need to define a
> strategy
> -
>
> Determine the Improvements to be delivered across first 3 Releases
> with Target Release Dates
>
>
>
> -
>
> Ensure CVE and Dependency management hygiene is in place
>
>
> The above approach will usher the community back to an active status with
> a Roadmap of 3 or more release plans and security hygiene in place.
> Development Practices
>
> The Livy project follows a review before commit philosophy. Every commit
> automatically runs through the unit tests and generates coverage reports
> presented as a pull request comment. Our experience with this process leads
> us to believe that it helps ease new contributors into the project. They
> get feedback quickly on common mistakes, lowering the burden on reviewers.
> Those same reviewers get to lead by example, showing the new contributors
> that we value feedback within our community even when changes are done by
> more experienced folks. Taken from the original Apache Livy Proposal [1],
> this should continue to be true. As mentioned, Livy is a mature project and
> as such RTC is the most appropriate for continued quality and awareness.
>
>
>
> 1.
>
> Original Apache Livy Proposal
> https://cwiki.apache.org/confluence/display/incubator/LivyProposal
> 2.
>
> Retirement DISCUSS thread
> https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
>
>
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
Awesome - thanks, Justin!
On Sat, Oct 15, 2022, 11:51 AM Justin Mclean <ju...@classsoftware.com>
wrote:
> Hi,
>
> The project is free to choose RTC or RTC. I just wanted to check if it was
> considered. I’ve seen in some cases, CTR tends to put a lot of work onto
> existing committers and cause frustration when contributions are not
> reviewed in a timely way.
>
> Another thing to consider is what someone would have to do to become a
> committer. A RTC with a low committer bar would probably work better than a
> RTC with a high committer bar.
>
> You are not an IPMC member but can ask to become one as you are an ASF
> member.
>
> I started a roll call on the Livy private list that may get the attention
> of any existing PMC members.
>
> Kind Regards,
> Justin
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: Proposal to Revive Apache Livy Community
Posted by Justin Mclean <ju...@classsoftware.com>.
Hi,
The project is free to choose RTC or RTC. I just wanted to check if it was considered. I’ve seen in some cases, CTR tends to put a lot of work onto existing committers and cause frustration when contributions are not reviewed in a timely way.
Another thing to consider is what someone would have to do to become a committer. A RTC with a low committer bar would probably work better than a RTC with a high committer bar.
You are not an IPMC member but can ask to become one as you are an ASF member.
I started a roll call on the Livy private list that may get the attention of any existing PMC members.
Kind Regards,
Justin
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
Hi Justin -
Thanks for the feedback.
I can buy all of that other than the dev practices suggestion.
The project originally used RTC as per the original proposal and being a
more mature podling, I don't think we need to move that fast.
With new committers, it will also be good to ensure others are familiar
with the code going in.
For pulling in large features from forks, we can have feature branches that
are CTR and define merge criteria.
I believe that I was an IPMC member at some point and have been a mentor on
other podlings (still am???) but maybe somehow that went away?
We would certainly welcome any other mentors as well.
JB is currently listed as a Mentor on the site - I added him as a committer.
@JB do you intend to be a Committer or Mentor going forward?
The proposal was sent to the dev@livy.incubator.org list as well in order
to get their thoughts.
There have also been public threads about retiring it where this has been
discussed as you know.
Do you have further suggestions for how to reach out and/or how long to
wait?
Thanks again!
--larry
On Sat, Oct 15, 2022 at 12:24 AM Justin Mclean <ju...@classsoftware.com>
wrote:
> Hi,
>
> A couple of things stand out here to me:
> - Your mentors need to be IPMC members.
> - It would be helpful to include mentors who are not all affiliated with
> one company
> - It would be helpful to know what the existing PMC and the wider
> community think about this reboot. It may be there’s no one left, and no
> objections from those who are still about, but we should give them some
> time to speak up.
> - RTC can slow development and, in some cases, limit contributions,
> particularly when you have a less active project. Have you considered CTR?
> - While asking existing PPMC members to opt-in is probably fine, I think
> you should leave existing committers as they are and not remove anyone
> unless they asked to be removed.
> - Existing PPMC members who don’t explicitly opt-out should be able to
> rejoin the PPMC later if they ask.
>
> Kind Regards,
> Justin
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <lm...@apache.org>.
Hi Justin -
Thanks for the feedback.
I can buy all of that other than the dev practices suggestion.
The project originally used RTC as per the original proposal and being a
more mature podling, I don't think we need to move that fast.
With new committers, it will also be good to ensure others are familiar
with the code going in.
For pulling in large features from forks, we can have feature branches that
are CTR and define merge criteria.
I believe that I was an IPMC member at some point and have been a mentor on
other podlings (still am???) but maybe somehow that went away?
We would certainly welcome any other mentors as well.
JB is currently listed as a Mentor on the site - I added him as a committer.
@JB do you intend to be a Committer or Mentor going forward?
The proposal was sent to the dev@livy.incubator.org list as well in order
to get their thoughts.
There have also been public threads about retiring it where this has been
discussed as you know.
Do you have further suggestions for how to reach out and/or how long to
wait?
Thanks again!
--larry
On Sat, Oct 15, 2022 at 12:24 AM Justin Mclean <ju...@classsoftware.com>
wrote:
> Hi,
>
> A couple of things stand out here to me:
> - Your mentors need to be IPMC members.
> - It would be helpful to include mentors who are not all affiliated with
> one company
> - It would be helpful to know what the existing PMC and the wider
> community think about this reboot. It may be there’s no one left, and no
> objections from those who are still about, but we should give them some
> time to speak up.
> - RTC can slow development and, in some cases, limit contributions,
> particularly when you have a less active project. Have you considered CTR?
> - While asking existing PPMC members to opt-in is probably fine, I think
> you should leave existing committers as they are and not remove anyone
> unless they asked to be removed.
> - Existing PPMC members who don’t explicitly opt-out should be able to
> rejoin the PPMC later if they ask.
>
> Kind Regards,
> Justin
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: Proposal to Revive Apache Livy Community
Posted by Justin Mclean <ju...@classsoftware.com>.
Hi,
A couple of things stand out here to me:
- Your mentors need to be IPMC members.
- It would be helpful to include mentors who are not all affiliated with one company
- It would be helpful to know what the existing PMC and the wider community think about this reboot. It may be there’s no one left, and no objections from those who are still about, but we should give them some time to speak up.
- RTC can slow development and, in some cases, limit contributions, particularly when you have a less active project. Have you considered CTR?
- While asking existing PPMC members to opt-in is probably fine, I think you should leave existing committers as they are and not remove anyone unless they asked to be removed.
- Existing PPMC members who don’t explicitly opt-out should be able to rejoin the PPMC later if they ask.
Kind Regards,
Justin
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Proposal to Revive Apache Livy Community
Posted by larry mccay <la...@gmail.com>.
+dev@livy.incubator.apache.org <de...@livy.incubator.apache.org>
On Fri, Oct 14, 2022 at 2:38 PM larry mccay <la...@gmail.com> wrote:
> Abstract
>
> Livy is a web service that exposes a REST interface for managing long
> running Apache Spark contexts in your cluster. With Livy, new applications
> can be built on top of Apache Spark that require fine grained interaction
> with many Spark contexts [1].
>
> While this project has been well regarded and used in many contexts as the
> defacto standard API to Spark environments, it has been incubating for over
> 5 years without graduation to a TLP and it has become difficult to
> impossible for fixes and improvements to be contributed as the current
> community seems to have moved on.
>
> There has been discussion regarding retirement of this podling where there
> seems to be some increasing interest in joining and reviving the community
> [2].
>
> The intent of this proposal is to avoid retiring a well regarded, actively
> used and rather mature project by reviving the PPMC and community with new
> folks that have a vested interest in the project and health of the
> community.
> Proposal
>
> We propose to revive the PPMC with a set of contributors and maintainers
> as mentors, PPMC members and committers.
>
> The retirement DISCUSS thread [2] has shown a growing interest in
> providing new committers and bringing improvements and fixes from
> organization’s internally maintained forks back to a revived community.
>
> General Approach to Revival:
>
> -
>
> Add new Mentors
> -
>
> Larry McCay, lmccay@apache.org , Cloudera
> -
>
> Sunil Govindan, sunilg@apache.org, Cloudera
> -
>
> Imran Rashid - irashid@apache.org, Cloudera
>
>
>
> -
>
> Add new Committers/PPMC
> -
>
> Larry McCay, lmccay@apache.org, Cloudera
> -
>
> Vinod Kumar Vavilapalli, vinodkv@cloudera.com, Cloudera
> -
>
> Gyorgy Gal, ggal ,gal.gyorgy@gmail.com, Cloudera
> -
>
> Wing Yew Poon, wypoon@cloudera.com, Cloudera
> -
>
> Xilang Yan, xilang.yan@gmail.com, Shopee
> -
>
> Jianzhen Wu, myjianzhen@gmail.com, Shopee
> -
>
> Nagella Jagadeewara Rao, jnagella@visa.com, Visa
> -
>
> Pralab Kumar, pralkuma@visa.com, Visa
> -
>
> Prasad Shrikant, shrikant.ism@gmail.com, Visa
> -
>
> Brahma Reddy Battula, brahma@apache.org, Visa
>
>
>
> -
>
> Invite existing PPMC members to opt-in or otherwise go emeritus
> -
>
> Jean-Baptiste Onofré, jbonofre@apache.org, Talend (opted-in via
> Retirement DISCUSS thread [2])
>
>
>
> -
>
> Invite existing Committers/PPMC members to opt-in or otherwise go
> emeritus
>
>
>
> -
>
> Establish Roadmap via follow up DISCUSS thread
> -
>
> Known Improvements from Forks which will need proposals and
> discussion:
> -
>
> Adding HA for Livy
> -
>
> Updating security capabilities (eg. kerberos for jdbc, fixing
> bugs in encryption)
> -
>
> Expanding the support for kubernetes
> -
>
> Responding to CVEs in dependencies (eg. log4j, thrift)
> -
>
> Livy rest cluster - IS THIS SAME AS HA for Livy ABOVE?
> -
>
> Support multi Spark versions
> -
>
> Implemented a metrics system for Livy
> -
>
> Support customize batch/interactive session lifecycle event
> handler, default log event with log4j, very helpful for trouble shooting
> -
>
> Optimize log to track which session id the log message came
> from, also very helpful for trouble shooting
> -
>
> Support customize Spark config optimization rules, can be used
> to optimize config for users’ job
> -
>
> A set of command line tool which can be used to replace Spark’s
> spark-submit, pyspark, spark-sql but actually submit application in Livy
> -
>
> We are planning to implement a JDBC state store, and allow multi
> Livy Thrift sessions to share one backend Spark application in the next few
> months.
> -
>
> These items and others that are brought to community may need
> consolidation or multiple configurable options and will need to be part of
> the discussion
> -
>
> One-pager Livy Improvement Proposals (LIP) may make sense to
> drive these discussions and convergence
> -
>
> Feature Branch Strategy for large changes
> -
>
> Large features are hard to review we will need to define a
> strategy
> -
>
> Determine the Improvements to be delivered across first 3 Releases
> with Target Release Dates
>
>
>
> -
>
> Ensure CVE and Dependency management hygiene is in place
>
>
> The above approach will usher the community back to an active status with
> a Roadmap of 3 or more release plans and security hygiene in place.
> Development Practices
>
> The Livy project follows a review before commit philosophy. Every commit
> automatically runs through the unit tests and generates coverage reports
> presented as a pull request comment. Our experience with this process leads
> us to believe that it helps ease new contributors into the project. They
> get feedback quickly on common mistakes, lowering the burden on reviewers.
> Those same reviewers get to lead by example, showing the new contributors
> that we value feedback within our community even when changes are done by
> more experienced folks. Taken from the original Apache Livy Proposal [1],
> this should continue to be true. As mentioned, Livy is a mature project and
> as such RTC is the most appropriate for continued quality and awareness.
>
>
>
> 1.
>
> Original Apache Livy Proposal
> https://cwiki.apache.org/confluence/display/incubator/LivyProposal
> 2.
>
> Retirement DISCUSS thread
> https://lists.apache.org/thread/gcstsrhbp91c5mm55htqn1l3djv8m7o0
>
>