[apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Siyuan Feng <no...@github.com.INVALID>]

In this process RFC, We'd like to propose a process to encourage scoped modules and set expectations about what we anticipate in such inclusion.

[rendered] (https://github.com/Hzfengsy/tvm-rfcs/blob/empowering-new-scoped-module/rfcs/0095-empowering-new-scoped-module.md)
[discuss thread](https://discuss.tvm.apache.org/t/process-rfc-empowering-new-scoped-module-to-the-project/13617)

You can view, comment on, or merge this pull request online at:

  https://github.com/apache/tvm-rfcs/pull/95

-- Commit Summary --

  * [Process RFC] Empowering New Scoped Module to the Project

-- File Changes --

    A rfcs/0095-empowering-new-scoped-module.md (74)

-- Patch Links --

https://github.com/apache/tvm-rfcs/pull/95.patch
https://github.com/apache/tvm-rfcs/pull/95.diff

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95
You are receiving this because you are subscribed to this thread.

Message ID: <apache/tvm-rfcs/pull/95@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Siyuan Feng <no...@github.com.INVALID>]

@mbaret 

> I don't think it's fair or accurate to dismiss legitimate concerns of community contributors as 'subjective'. @areusch has already enumerated in some detail an 'objective' list of impacts that an S0 module can have on the wider project. I think at a minimum we should be addressing those points before implying that objecting to S0 modules is somehow improper (which of course it is not).

I don't want to dismiss every concern and comment in the discussion (also other feedback from the community). It spent me some time to reply to @areusch as the comments are a bit long and I'm a non-native speaker, which costs me a bit more time to organize my words. 

> Let's keep it that way and make sure everyone feels welcomed, recognising that constructive debate (like that happening on this very RFC) benefits us all.

Please directly point it out if I miss any public voice. And please let me know if you feel un-welcomed during the discussion and the reasoning. I'm not a native speaker, so maybe I will make mistakes in wording and may make you uncomfortable. If so, please also let me know, and I will apologize.

Again, we are working on the points and trying to address them before voting, which will absolutely benefit TVM.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1288803485
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Siyuan Feng <no...@github.com.INVALID>]

Closed #95.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#event-10194035435
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Tianqi Chen <no...@github.com.INVALID>]

Thank you, everyone, for the discussion so far, and thank you, @Hzfengsy  for driving the proposal. After reviewing the thread and watching all the relevant conversations. I would like to come out and share my support for the proposal. 

I am making this statement [wearing Apache TVM hat](https://tvm.apache.org/docs/contribute/committer_guide.html?highlight=apache%20tvm%20hat#independent-project-management).

I would like to summarize a few important perspectives from our community members here as guidelines for community members:

##  How should we operate as a community?

When considering proposals, there are many factors being considered by members. A lot of them have to do with the impact to code:

- Impact to Code: cohesiveness to the overall vision,  testing, clarity among modules, quality, stability, the overall complexity of the codebase, and presentation to the users.

This is what we are getting used to in normal software development settings. There are even trade-off factors here. On the one hand, indeed, new modules bring different levels of impact on these perspectives depending on their scope. However, in many cases, new modules do not bring impact in a purely negative way. 

MLC(machine learning compilation) is a fast-evolving field. The ability to reinvent ourselves and keep up with innovation while keeping the main component stable is the key to survival in the ML landscape. PyTorch is exemplary in terms of development philosophy on bringing in related modules. In many ML frameworks such as PyTorch, allowing space for multiple relevant solutions(e.g. Dynamo, TorchScript, LazyTensor) to coexist and evolve is critical for success. It is also important to allow flexibility of growth rather than plan everything out in the beginning. When TorchFX first came out, there was no clear prediction of Dynamo being relevant. Even today, there is no detailed plan of replacing TorchScript with Dynamo. Nevertheless, the community members continue to use the relevant modules and have an evolving conversation. ML is fast evolving and we should never expect to lock down every aspect of a plan in the beginning. Instead, open doors for diverse community members to come and participate and contribute their perspectives. Such open philosophy is what the TVM community(or every successful ML OSS project) should aspire to be.

This brings us to another **very important** factor, which is OK to ignore in software development settings but should always but put first in Apache projects:

- Impact to the community: bringing in new community members and growing the community. A lot of that ties back into the "community over code principle".
When evaluating modules, we should certainly consider how the proposal empowers new users/developers. Can the proposal welcome new developers? Can they solve different needs of the community? The TVM project exists because of the community. 

That means we need to empower each member with different needs, and sometimes that could mean that we would do something different than our normal development rationales to empower the community members – especially when many, or even the majority of community members, shared their voice and support. This does not mean our original rationale is wrong, but we are more open-minded and bring that community empowerment as the first principle.

## How can we empower each other

Being in the same community means we share the same burdens(of maintenance, advocations) and joys together. Any process or system is not perfect, and the intention was never to assume people as adversarial. 

The first principle of the Apache community is community over code, which means we empower each other. Some in the discussion thread raised the question about what if a person operates in an outright stubborn manner – like declaring outright rejection of later S1/S2 proposals or code PRs without reviewing the additional context being brought up, and not having a constructive conversation with additional context. Such behavior would certainly be frowned upon in any OSS community,  and this is not a behavior I would expect from any committer/reviewers. 

This also comes from the exclusive mindset and inclusive mindset we have in the community. Indeed it is an additional burden for us to help maintain, and development the code that is less relevant to our interest. One could outright dislike code that causes regressions irrelevant to our own customer use cases. It is harder to directly settle a lot of those conversations through a simple technical debate since many can have disagreements about what are important parts – to some graphs IR and executor are important, to many other TVM users, they might only care about FFI and TensorIR while building their own graph integration. 

When in confusion, always remember to come back to the community. Remember that we empower each other. These are codes that are developed by members who also volunteer their time reviewing our code, writing test cases, and contributing to common foundation components such as codegen, FFI. While the need perhaps is less urgent to ourselves, empowering them would help the community to grow stronger.

I can give examples of when we do so. For example, the AOT proposal comes with quite a few complexities and burdens that not all of us fully agree to – a divergent ABI that could have been optimized through C Packed, an IR node that later caused regression in auto-scheduling. Extra code path that only supports a limited set of models and dynamism due to (reasonable) restrictions. To many of us, it is also not necessarily an executor that we need in our daily applications.

One could have forced the AOT proposal to always take C Packed that is consistent with other server-level modules, or block the module because they contain overlapping features that are implemented differently nor support all models.   We did not choose to do that; instead, we welcomed the modules because we knew that community members would need them and would benefit from them in a timely module establishment, even if some of the detailed plans still needed to be flushed out. We know that they will empower additional community members. A proposal for graph IR, or auto-scheduling layer is not that different in nature. While each module might need a different level of technical deliberation, community empowerment should always be an important factor. 

Time is also an important consideration – as we are facing competition, and time to market is important to ML. When reviewing proposals that have a lot of community support, we would always try to keep a constructive conversation going and facilitate the establishment.

Being inclusive and empowering each other is an important factor in bringing a welcoming and productive community. This does mean we need to take burdens, and we also openly contribute to each different perspective of the community that supports each other. 

I and many other PMC members have personally done so in evaluating past proposals (like the above example) and would expect our colleagues to do the same.


## How shall we handle disagreements 

Being in a big community means we have a diverse view. When one says, "this code produces this bug", likely no one will disagree. But as we evaluate code and community factors, each of us could bring disagreements because we might have our own opinion on how things should be done considering the factors.

Shall we always disallow new modules that have relevant features to coexist with existing ones? Should we outright reject the phased approach and always ask for a detailed plan before accepting a module? After considering the arguments, shall we empower community members so some of them can enjoy the new module, and help us to grow our developer community?

First of all, it is important to acknowledge that each of our views is both valid in many cases. It is very important to understand each others' view, and re-evaluate the proposal under such a context. Perhaps we should consider more impact to test in this case; perhaps we should put more weight on community empowerment. After acknowledging and understanding the views, having a conversation around the cases, there can still be differences of opinion, how do we as a community proceed? 

Stalling is not the right option because that would implicitly ignore the voices of those who want to move forward, especially when the majority of the community wants to do so. We also need agility and support for many who value innovation while maintaining high-quality code.




As a result, after acknowledging each others' views and having a constructive conversation, I recommend listening to the majority community's voices -- as it is a decision of the community, by the community, and for the community.

When in doubt, we choose community over code. That is why I support this proposal.


-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1336463901
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Siyuan Feng <no...@github.com.INVALID>]

## S1-level module

There are a few suggestions for clarifying the S1-level module. An S1-level module is a 
module that does not follow the restrictions outlined in the S0. Specifically, an S1-level module is usually used as dependencies by other major modules in the project. 

The considerations that go into the S1-module are also not about how good the code is. As a matter of fact, if it is possible to maintain clearly isolated components while being useful in general to a broader set of users. Keeping modules in S0-level scoping is even desirable. When there is a need to affect a broader set of modules, however, certainly such broader scope would trigger more careful discussions among more people.  It again boils down to the level of open-mindedness due to the different scopes of impacts.


> Requiring 3 endorsing committers who agree to own/maintain is a very high bar and would be exclusionary to newer contributors.

We need a clear set of people who can review/maintain the code and maintain the overall review quality bar following the same process of all modules, so at least one is necessary. The volunteers also do not have to write code, but need to be responsible for code reviews and follow up actions, and contribute their time on a part-time basis. Likely the supporting PMC members would be able to serve that role initially. This being said, we anticipate new contributors being bought in as committers. The goal is to grow a set of contributions rather than restricting them. 


> Is there any guide about how to get this PMC member, assuming the S0 RFC is not proposed by a PMC member? Also what should we do if the champion doesn't response on time during the upstreaming process? Should the owner find another PMC/committer for help?

Great question. We anticipate the champion to signal such intent during the pre-RFC stage (at the forum). Normally we expect champions to be able to land a commitment of their time. A new champion is certainly possible if it is not the case. 


-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1285388768
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Andrew Reusch <no...@github.com.INVALID>]

@Hzfengsy 

> Note that the reviews of each PR are brought to their own context, and we anticipate grounded constructive actionable comments, such as adding test coverage here/there. The module establishment part can be into subjective discussions that are less grounded, and community empowerment becomes more critical in such cases.

Taking a step back here: why do S0 modules need to live in `main`? All of our development actually happens on forks (i.e. you fork the repo on GitHub)--the question is: how much development goes on outside of `main` before we merge? Forks could technically live for years outside of `main` (as Relax has). So what are the benefits of merging:

B0. Joins development of the S0 module to TVM's S1 and S2 modules, distributing the need to rebase and handle merge conflicts throughout the TVM community.
B1. Solicits a wider audience to interact with the module by publicizing it as part of TVM's tree.
B2. Fosters further development of the module inside of TVM's core forums. This also helps some community members to better engage by potentially removing hurdles (i.e. engineers at industry companies may be restricted by company policy to engaging with only specified OSS communities).
B3. Grants access to TVM's CI resources and runs tests in the mainline regression suite.
B4. Permits development of tighter integrations with the remainder of the project by removing merge conflicts.

There are also some drawbacks:

D0. Increases the load on TVM's CI.
D1. Increases the scope of TVM's codebase, potentially making it harder to maintain. For example, changes to core components such as the test infrastructure, the runtime, the object system may require sweeping knowledge of the project. As the number of components in the project increases, the number of engineers with that knowledge decreases relative to the size of the codebase. If that number drops to 0, the codebase will rot.
D2. Could contribute to effectively splitting the community by creating multiple arenas for development. While there are benefits to this in terms of exploring new techniques, allowing this as a permanent state means that effectively TVM is many projects with sub-communities rather than one. This may be a reasonable thing to strive for, but for D1 reason it is not clear that those projects should be in a monorepo.

A theme across these is that some aspects are community-related (i.e. publicizing a flow, consolidating discussions, allowing us to explore multiple avenues to solving a problem) and some of these are essentially technical/logistical problems (i.e. maintaining a CI, maintaining the core infrastructure, etc). Weighing the community aspects of these benefits and drawbacks, I believe that the benefits are worth the drawbacks and I am in favor of allowing new contributions and approaches in TVM.

However, I feel we are still not really addressing the technical side of things: there is no such thing as a truly S0 module, because we state it is
> No disruptive change to the rest of the codebase

The problem is that because we are in a monorepo, there are technical/logistical aspects to integrating a module where "a failure in one module is a failure for all of TVM," the most obvious example being CI:
- A flaky test is checked in; someone is trying to make a change to an S1 module, but the CI fails sporadically due to the flaky test to impede progress.
- Someone is trying to improve the way we shard tests with pytest; in the S1 modules, we have e.g. properly parameterized the tests, but in some S0 module, the parameterization was not properly followed. Now that engineer needs to go and learn about how the S0 module works, fix its tests, and find a reviewer for those improvements before they can proceed.
- Someone adds a new CI image to test the S1 modules on a new hardware backend. The S0 module is excluded from that backend. Now a user comes along and tries to use the S0 module with the backend, and files a bug. Can the owner of the hardware backend truly claim they "support TVM" if they don't fix the S0 module bug? What mechanism have we put in place to ensure that users approaching TVM understand that they are using an S0 module?
- Someone is modifying a core TVM API; an S0 module uses it in a novel way. Now the modifier has to go and understand how the S0 module works so that they can modify the API in a way that doesn't break it.

While i agree with you that
> The module establishment part can be into subjective discussions that are less grounded, and community empowerment becomes more critical in such cases.

I am concerned that the situations I elaborated above could similarly lead to people leaving the community if it becomes too fragmented. I also feel a bit like the perspective of maintaining the core APIs and test infrastructure is not adequately represented in the discourse so far, and may not be represented in the proposed voting mechanism.

There is a lot of detail here about the establishment of an S0 module, down to even introducing a new voting process. When it comes to transitions between those states, there is really not much detail:

> - S0 -> deprecation: When a S0 module no longer has an active set of maintainers, the module will be deprecated. The removal of the module is easy as they are contained in the respective folders, with no modules that come and depend on them.
> - S0 -> S1: When developers propose to incorporate a S0 module broadly into existing flows/components. Each such composition would require its own RFC and following the existing RFC process.

For instance, suppose that yes there is a maintainer for an S0 module, but tests keep breaking or they are relatively unresponsive, dragging down any of the core maintenance tasks above? What is the bar after which it's reasonable for a core maintainer to propose deprecation? Proposing deprecation is a lot more unwelcoming than requesting clarifications to new code or saying "no for now," and is a lot more challenging because it might involve finding a new home for code that is already being used by some members of the TVM community, but for which we are finding it hard to keep maintained.

I am not trying to be unwelcoming to new code here. But these considerations can often be caught at the RFC level, when questions about test strategy, modularization, and documentation can be discussed at a level that might be accessible to a broader section of the community. Developing the same level of understanding through reading PRs both takes considerably more time and energy than reading RFCs, and may come at a time when a module is half-landed. For this reason, it is not clear to me why we should relax the voting consensus requirements on S0 proposals only to land in a half-baked state that may deadlock on a committer.

I would summarize my concern here in two areas:
1. There isn't a clear boundary in TVM between what is "S1" and what is "S0" right now. It may not be clear to core maintainers where their responsibilities end and the S0 maintainers' responsibilities begin. Modularizing TVM could help here, but we haven't done that yet.
2. Already today, "supported by TVM" is quite ambiguous--which flow, on which hardware, with which model? S1 modules here are simply those that are "not S0." How should a user know what these are? Allowing S0 modules is good, but there is currently no distinction in how they're represented to users.

I'm concerned that this proposal could use more detail given that we are a monorepo with the problems I mentioned above. I don't know these concerns necessarily need to get addressed in this specific proposal (e.g. more clearly delineating the S1 modules), but I also don't feel that this proposal has adequate detail right now given the state of the project.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1286821762
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Sunghyun Park <no...@github.com.INVALID>]

Thank you for sharing your thoughts, @areusch. I think it might be helpful for better discussion if you can clarify a little further especially about your main concern (2nd one). Particularly, I want to hear more about your thoughts on the following things:
1. Are you suggesting the consensus is the right way to go for this optional module? If so, I'm wondering how you define the consensus. IMHO, reaching unanimous might be practically impossible for the large community like this and it does not seem to fit for S0: by definition, S0 is to satisfy clear needs from some users in the community, not the entire users. I'm afraid if too high bar for optional module may discourage contributors. 
2. This quote below sounds more about the standard process of how we digest a giant commit, which could be orthogonal to voting rules. IIUC, this seems like the process of integration after the vote of deciding whether community members want to accept the proposed optional module or not. Personally, I believe voting should be based on the idea or proposal rather than the code size. 
> They might create a giant merge commit, or a 20,000-line commit that isn't reviewed according to the project's standards. Such large changes are often so disruptive to a project as to influence the project's overall direction and, in turn, its community. 

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1284437870
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Andrew Reusch <no...@github.com.INVALID>]

I appreciate you guys bringing this back up, and I appreciate the goal here of establishing a project policy to ensure we're welcoming to new projects as the MLC landscape changes. I still have some reservations about this proposal in its current form, but I would support it if a small number of changes were made. Here is what I think it needs:

1. **Approval mechanism**. I disagree with @Hzfengsy's statement that "establishing a scoped module is not a typical code change." Because the typical S0 module would reside inside `apache/tvm` on `main`, a subproject at the moment is essentially several PRs that add code to the codebase. This is a typical code change, so I don't see a reason to adopt Lazy Majority consensus here. I appreciate that Hive uses Lazy Majority, but Hive has [52 PMCs and 104 committers](https://projects.apache.org/committee.html?hive), and also has a process by which committers and PMCs automatically gain [emeritus status](https://cwiki.apache.org/confluence/display/Hive//Bylaws#Bylaws-Committers) after 6 months of inactivity. TVM has a much smaller PMC. I would support this proposal if Lazy Consensus was the voting mechanism.

    One of my practical concerns here is that we lose recourse for preventing overload of the CI. And while I do want us to be welcoming of new projects, I could see the following situation arising that would place committers in an awkward position:
    1. A new S0 module is proposed and a majority of PMC members approve. The S0 module is accepted.
    2. PRs for the new module begin to merge.
    3. At some point, the integration test suite is added, but this involves regression tests that add 1 hour to the CI. The proposer either doesn't have time to rewrite the tests, or it's not practical to do so. Meanwhile a committer more familiar with the CI flags this, but the proposer cites acceptance of the RFC as grounds that the project has chosen to merge the new S0 module and its tests.

    CI is a bit tricky since it's complex enough that not everyone understands it, but it's a shared community resource that underpins the whole project. I think that choosing community over code means that we don't overextend the project so that we can ensure that such shared resources are available to everyone to work towards the project's goals. While I agree the community has been generally involved with keeping the CI green, only 4 of 20 PMCs have made or been involved with [changes](https://github.com/apache/tvm/commits/main/Jenkinsfile) to `Jenkinsfile` in the past year (and of those, only 2 made changes other than updating a CI image). Changing Jenkinsfile is the primary way we do things like resharding tests to reduce apparent CI runtime. I'm not sure Lazy majority is the right tool to evaluate this aspect of the decision, considering that adding S0 modules may involve significant changes to the CI.

    To conjure another situation at the other end of the spectrum: Someone new to the project proposes that we add an S0 module, but although they get a few supporters, they don't manage to get majority PMC support. Perhaps they don't know many PMCs, or the PMC doesn't know much about the relevant subject area. Is the proposal rejected? Majority seems like it could also be a high bar here, and if we intentionally adopt a different voting mechanism to accept new projects, the rules seem more confusing for newcomers.

   It doesn't make sense to me to have two different voting mechanisms involved with adding code (one for PRs, and one for RFCs that propose a bunch of PRs to add isolated code). There are many situations today in which committers and PMCs can block changes to the project, and we rely on a sense of community to avoid such deadlock. I continue to think that fostering consensus in the community is the best way to choose community over code. Should we come to a point where the community needs to choose between two paths forward, such as when making an S1 -> S2 change, and consensus cannot be reached, then that is a case where I think Lazy Majority may be a reasonable path to making forward progress.

2. **S1 and S2 module lifecycle**. This RFC talks a lot about S0 proposals, and it is great that we are creating a welcoming stance towards new contributions. However, I would like to see the proposal rounded out with discussion of how an S0 module should become S1 and how an S1 module can deprecate others (i.e. S1 -> S2). I agree that some of this is covered in the RFC today, but because the RFC doesn't include complete discussions around the S1 and S2 phases, I think it's hard to understand the S1 and S2 concepts. Here's a specific example:

    > There can be follow-up steps (S1), such as making a dialect broadly accessible in existing compilation flows.
    
    This sentence, inside "Background," discusses the existence of S1 steps, giving one example. However, the way it's worded makes it sound like others may exist; however those aren't enumerated there. It doesn't really make sense to enumerate them in the Background section, but without a full S1 section, there isn't a good place to put that info. Here I am thinking of new contributors and wondering what a new contributor would do should they want to understand the full process. Where could they look?

    Some specific questions I can imagine readers having that I think should be answered in the same place as theSS0 proposal:
    - How should someone propose to make a module S1? Is it an RFC? What voting mechanism is used?
    - Are there additional roadmap or documentation requirements placed on modules that become S1?
    - At what point do we decide to deprecate modules (i.e. move other modules to S2 state)? What voting mechanism is used for that decision? How long is the typical notice period?

5. **What is "stable" in TVM and how long is it stable for?** I'm asking the specific questions in the previous point because industry contributors need to have an understanding of such things for project planning purposes. Is [`GraphExecutor`](https://github.com/apache/tvm/blob/main/python/tvm/contrib/graph_executor.py) stable? It's still in `contrib` years after being established as the reference for many unittests. 
 
    I would like to see us be okay with making it explicit which modules are S0, S1, and S2 in the TVM codebase. Doing this is not intended to make any judgement on such code, nor is it intended to imply the quality is poor--it is merely intended to guide newcomers so that they understand which paths have been well-established in TVM and which are under development.

I'd be happy to discuss any of this further. I think these are the same three points I've been asking for in the Discuss post and initially on this review thread, and I appreciate that you guys have been iterating on the RFC. Hopefully this clarifies my sticking points.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1338415060
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Tianqi Chen <no...@github.com.INVALID>]

Thank you, everyone, for the discussions in this thread. To help us build future constructive conversations and avoid mixing conversations in different threads, I suggest that we only apply the proposed process to future RFCs to be opened. Thank you, everyone, and let us continue to have civil and constructive discussions.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1287543981
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Tianqi Chen <no...@github.com.INVALID>]

I want to come back and encourage that we take fine-grained steps. As @Hzfengsy  mentioned, many community members, including himself, had to overcome a large language barrier to be able to come out and help facilitate this process. Having a dragging conversation simply due to bundling reduces the ability for volunteers to participate, which goes against core one principle of ASF – which is empowering everyone. We should reduce such burden by having well scoped proposals, knowing that with a growing community, the conversation always continues. 

One observation is that there can be a disagreement on how much bundling we have in making decisions. I would like to come back and acknowledge what you stated, @areusch  have clearly made  points– requiring all approvals, requiring S1 and S2 discussions, and bundling stability discussion to be addressed is a valid position on how we do things to establish this process.

It is also clear that many others had a different position, that emphasizes on empowering the majority of the community, more fine-grained decision making so we don’t have to bundle all these conversations together and continue to empower each other.

They are different valid opinions on how we shall operate. After acknowledging each other’s views, take these into context, and have a constructive conversation, I recommend listening to the majority community's voices and empowering the community.


-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1338702897
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

["Steven S. Lyubomirsky" <no...@github.com.INVALID>]

I wonder about a situation where some parties indicate up front that they are resolutely opposed to ever permitting an S0 module to become S1. Even if this process permits the module to be merged as S0, it would essentially be known in advance that it is unlikely ever to be fully integrated into the project. Is having such a module remain indefinitely in the optional state an acceptable outcome? The text of the RFC makes it sound like the S0 state is intended to be temporary until the module is either made permanent or deprecated; is that a reasonable inference?

In particular, the point about discussing future plans for a module may be worth further consideration:
>There should be discussions about how the proposal fits into the project to bring clarity. We also acknowledge that not all S1, S2 level decisions can be made at the beginning. Additionally, an S0-module should show a clear positive fit to some(but not all) aspects of the project and clear cohesion to some of the existing modules. As the development evolves, more discussions will happen in future RFCs with additional evidence that help us to make informed decisions.

If some community members indicate their opposition to an eventual RFC, that would render the future roadmap moot, no?

I guess this is a question about having the hard discussion up front or later.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1284610277
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Siyuan Feng <no...@github.com.INVALID>]

Hi all, as suggested in the thread, we held this thread for a while. And now it can be a good time to come back.

Let me summarize the previous discussion here:

- Scoped module
  A scoped module (S0-module) can be:
  > - Clearly isolated in its own namespace.
  > - Clearly needed by some users in the community.
  > - No disruptive change to the rest of the codebase
  > - Can be easily deprecated by removing the related namespaces
  > - Can be turned off through a feature toggle to contain the overall dependency from the rest of the modules.

- voting mechanism
  Establishing a scoped module is not a typical code change, which needs to get majority support from PMC. This kind of voting mechanism is also used in other Apache Projects (e.g., [Apache Hadoop process](https://hadoop.apache.org/bylaws.html) and [Apache Hive Bylaws](https://cwiki.apache.org/confluence/display/Hive//Bylaws)

- The community should evaluate the scoped module with a variety of factors, wearing the project hat:
  > - Fit into the overall project and rest of the modules and project.
  > - Test strategy, modularization, and documentation.
  > - The scope of impact of the added module, and levels of open-mindedness.
  > - Competitive landscape of the overall MLC space and enablement of the project towards goals that are not supported atm.
  > - Community empowerment in general: e.g. contributors who would become an added force of offset development complexity, and also in a lot of cases contribute to other existing modules.

- Scoped module does not mean to be a low-quality module. All code changes for scoped modules are in the same code review mechanism

Please let me know if I miss any public voice and considerations. And let's continue on discussion and finalize the RFC.


-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1336400761
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Matthew Barrett <no...@github.com.INVALID>]

@Hzfengsy 

I don't think it's fair or accurate to dismiss legitimate concerns of community contributors as 'subjective'. @areusch has already enumerated in some detail an 'objective' list of impacts that an S0 module can have on the wider project. I think at a minimum we should be addressing those points before implying that objecting to S0 modules is somehow improper (which of course it is not).

What is true is that in a diverse community there are contributors who value things differently. For instance, there may be community members who put a much higher value on moving fast compared to others whose primary interest is predictability and stability. You can construct perfectly objective arguments in favour of either of these philosophies, and having both voices in the community is an enormous benefit to continuing development and success of TVM. Let's keep it that way and make sure everyone feels welcomed, recognising that constructive debate (like that happening on this very RFC) benefits us all.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1288780802
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Siyuan Feng <no...@github.com.INVALID>]

Thanks for the input and feedback from the community. Here I'd like to clarify some questions.

For @areusch 
>  As the RFC stands now, a committer could simply go and -1 each following PR if they wanted to

Note that the reviews of each PR are brought to their own context, and we anticipate grounded constructive actionable comments, such as adding test coverage here/there. The module establishment part can be into subjective discussions that are less grounded, and community empowerment becomes more critical in such cases.

An S0 module only has things to do with its scope. We can expect a well-scoped module to benefit from more open-mindedness in evaluating module establishment. S0 modules are not temporal because every module is subject to depreciation with different levels of consideration. We also anticipate that if the S0 module continues to be helpful to the users in its well-scoped fashion, it is totally fine to remain at S0 if changing to S1 is not necessary for the use cases in mind. The change to S1 only has things to do with changes in the scope of impact but not how functional the module is, etc.

For @slyubomirsky 
> some parties indicate up front that they are resolutely opposed to ever permitting an S0 module to become S1

We also like to come back to the fact that it is impossible to cover all aspects in the beginning. e.g., TorchFX, as of now, does not anticipate the usage of TorchDynamo and other cases, and that was the reasoning. We value grounded discussions with technical reasonings. Stating that blank statement upfront is unlikely a welcoming move to the overall community, such a hypothetical statement also does not align with the community over code principle (and blank -1 that is not grounded does not carry weight). Additionally, everyone is supposed to [wear the project hat](https://tvm.apache.org/docs/contribute/committer_guide.html?highlight=committer#independent-project-management). A statement that is based on the interest of a party (and not directly the project), although essential to consider, does not carry binding weight. Having the S0 module evolve provides opportunities for community members to have more grounded cases and help the community collectively work towards goals that would help S1 conversations – while continuing to provide value already through S0-level empowerment. Everyone will evaluate the conversations at the point of S1 proposal and provide grounded discussions.

> I guess this is a question about having the hard discussion up front or later.

In the OSS project, we all know that it is not possible to get evidence of all perspectives in the very beginning. For example, nobody proposed TorchFX to become the backbone of PyTorch compilation when it was introduced. It is also impossible to have grounded conversations to sort everything out in that first stage. Different decisions are made at different time points with different sets of evidence at that time point. 


@mbaret 
> Again I don't think the Torch-MLIR/TOSA example is relevant here, they are different projects under separate governance.

Great point. We mistake the TorchMLIR here. There are similar sets of in-tree computational graph dialects that we intended to refer to – both TOSA and Linalg for example are in the MLIR tree and serve as computational graph dialects.

> This has been pointed out by others already, but this process does create a risk of modules getting 'stuck' in S0 should they subsequently get blocked on S1 transition (or even stuck in S1 if we can't get consensus for S2).

Note that these things happen at different time points and certainly have different considerations. This is also common practice in many ML projects. Take PyTorch as an example, the criteria for accepting TorchFX for example, is certainly much higher than deprecating TorchScript in favor of TorchFX – as of now there is no deprecation of TorchScript but nevertheless FX thrives as part of the core PyTorch compiler. 

We should also note that we have different sets of evidence at different points due to the nature of developments, and everyone in the community is bound to evaluate this updated evidence accordingly. For example, when TorchFX got started it was not intended for the main torch compilation flow, but as the project evolves and Dynamo is built up, it is now clear that it helps as part of the core PyTorch compiler stack.

In short, it is common practice in OSS projects to bring an open perspective to different scopes of impact of a module and evaluate them accordingly.

> There are other OSS compiler projects which maintain a high-bar to merge new modules directly (LLVM/MLIR comes to mind) so it'd be good to explain why we're taking an alternative (but still valid) approach.

The overall suggestion over S0 is not about lowering the bar to code itself. In that regard, we still come up with the statement of "an S0-module should show a clear positive fit to some(but not all) aspects of the project and clear cohesion to some of the existing modules." It mainly comes back to the scope of impact and level of open-mindedness when evaluating the related modules.

The goal of the RFC process is to clarify that and follow this practice with motivations also outlined in G1 and also empower the community members who clearly voiced their needs for such empowerment. 

> This is a lower threshold than I've seen in similar projects (which typically adopt majority or 2/3rds majority). Do we have a particular justification for this number?

Thank you for bringing this up. 2/3 majority decision normally applies to S1/S2 level decisions that have a bigger scope of impact. This is mainly considering different levels of open-mindedness people normally apply to decisions due to the difference in terms of scope of impact.


-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1285347175
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Siyuan Feng <no...@github.com.INVALID>]

I withdraw this proposal in favor of the simpler and better process in #102

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1693505583
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Matthew Barrett <no...@github.com.INVALID>]

> Having a dragging conversation simply due to bundling reduces the ability for volunteers to participate

I'm sorry but having a 'dragging conversation' is entirely the point of a RFC. It's not a rubber-stamping process, nor should it be. And this RFC fundamentally changes the governance model of the project.

> I recommend listening to the majority community's voices and empowering the community.

Firstly, there aren't many community members even present commenting here, let alone a majority, so I'm not sure what's meant by this. But more importantly, and with my Apache hat on, I fundamentally disagree with the characterisation of those who are not unconditionally supportive of this RFC as 'anti-community'. In my view this is both unnecessarily antagonistic and indeed wrong.

There are multiple different stakeholders that make up our community, I'll enumerate 3 types below:
- **Feature developers**
    They contribute new capabilities to TVM to ensure it keeps pace with ML landscape.
- **Maintainers**
  They maintain the code to ensure existing capabilities don't regress and fix bugs when they arise.
- **Users**
  They actually apply TVM to real problems so that the project provides value to the world.

As drafted, S0 modules seek only to empower feature developers at the cost of maintainers. Users wouldn't even see significant benefits as S0 modules would necessarily need to be hidden from the majority of users, otherwise they'd become de facto S1 modules. I therefore consider this proposal, in its current form, as hostile towards those wishing to contribute towards the maintenance of TVM.

I would further question whether we have substantial evidence that S0 modules are routinely blocked. We have 3 different auto-tuners, 3 executors, 12 backend targets, 14 BYOC targets and 11 frontends too. The vast majority of RFCs are approved, as are the vast majority of pull requests. These all point towards a community that is already extremely accommodating of new features, perhaps to a fault given our limited capacity to maintain these different components.

However, the S1 and S2 mechanisms would be of enormous value to maintainers (and users!) by strictly defining the maintenance surface (S1) and allowing for that maintenance surface to shrink as well as grow (S2). This would finally allow us to start reducing the complexity in the code base and aim to provide a rock-solid user experience where TVM could compete with PyTorch/TensorFlow/ONNX on robustness.

Could we therefore please stop pretending this RFC 'empowers the majority of the community' and say what we mean: it empowers the PMC - the composition of which currently doesn't closely reflect the active contributors in the community. There's a clear and obvious route to evolving this proposal into something that benefits everyone via S1/S2 processes, but for an unknown reason that's being refused.


-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1339463770
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Tianqi Chen <no...@github.com.INVALID>]

@mbaret I want to apologize about my wording. Indeed constructive conversation is important -- that is why we are having this conversation for about several months now.  I would like to point out the importance of unbundling -- i removed my wording on dragging. 

I do would like to point out the language barrier factor, the needs to empower our community members, and so.

There is no intention to indicate that that those who speaks against the RFC as "anti-community", I want to apologize if the text was interpreted in that way. Instead, we summarizes the case there is a disagreement on hows, and disagreement on how usually are addressed through procedural approaches.

> Firstly, there aren't many community members even present commenting here

We have observed at least community contributors from more than eight organizations supporting the proposal in this RFC and the original [thread](https://discuss.tvm.apache.org/t/process-rfc-empowering-new-scoped-module-to-the-project/13617/29). It is very rare to see vocal support of such proposals 

I agree with you on multiple personas. And i would like to remind us that every one of us are wearing multiple hats. I personally contribute to features, maintain the modules, fixing tests. 

It is unfair to simply categorize a S0 proposer as "feature developers". Many of them are maintainers of the modules, fixing bugs more often. Additionally, S0 as being defined only have to do with its "scope of impact". There is nothing to do with the reach-ness to the users, or whether people needs them.

Imagine a case when a module is being needed by more than eight organizations -- many of them are industry players, but not all other members. Will we count that module as "need to be hidden from the majority of users"? Of course not, we should present these modules to the user. A module that have limited scope of impact, while benefit a lot users is a great thing. The factor of inclusion should consider, testing, maintenance, documentations and others as being listed in the proposal. 

 
> However, the S1 and S2 mechanisms would be of enormous value to maintainers

There is no question that these are important and we should do them as well in other conversations.


> Could we therefore please stop pretending this RFC 'empowers the majority of the community' and say what we mean: it empowers the PMC - the composition of which currently doesn't closely reflect the active contributors in the community.

We all contribute in different means, some of them are code, some of them project management, directions. Apache principle means that we encourage volunteer contributions where our contribution past or recent are equally valued. I would value that our PMC members places the best interest  in mind in having such conversations.

Additionally, empowering and bringing people along means we will have more active contributors in the community overall.   







-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1339499093
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Andrew Reusch <no...@github.com.INVALID>]

 thanks for the replies here, everyone.

@comaniac

>-1 is totally fine and I don't think this Process RFC forbids this. On the other hand, if an RFC is suspensive for a while but we still cannot make every -1 voter happy, it doesn't make sense to me to reject the RFC (we don't actually "reject" it, but prevent it from merging forever is basically the same as reject).

I guess my point here though is: we have this problem for PRs too. We trust committers and PMC members not to blockade PRs so long as everyone is collaborating. For ordinary RFCs, we have the same situation. We achieve lazy consensus through discussion. In cases where there is a disagreement that cannot be resolved, and the argument has run its course, I agree it may become necessary to relax the requirement of lazy consensus in the name of progress. I am not aware of such a case in the project right now. 

To be explicit: in the case of the [Relax RFC](https://github.com/apache/tvm-rfcs/pull/89), the authors have not answered some key questions that contribute to "discussions about how the proposal fits into the project." My understanding is that those answers are coming so the argument is not deadlocked.

@sunggg
> Are you suggesting the consensus is the right way to go for this optional module? If so, I'm wondering how you define the consensus. IMHO, reaching unanimous might be practically impossible for the large community like this

Sorry--to clarify my original post, I am suggesting we adopt the same voting mechanism as we do for PRs and ordinary RFCs. 

> it does not seem to fit for S0: by definition, S0 is to satisfy clear needs from some users in the community, not the entire users. I'm afraid if too high bar for optional module may discourage contributors.

It seems like 3 PMC approvers is a higher bar for newcomers than merely getting a +1 from a committer.

> This quote below sounds more about the standard process of how we digest a giant commit, which could be orthogonal to voting rules.

This was meant to serve as an example of the effect of adopting a resolution such as replacing the implementation of a large component or adopting a submodule could wind up impacting the project in ways that materially determine its future. It is common for projects to fork when two different factions cannot reach consensus on a change of this magnitude. In such cases, I could see the need to relax the consensus requirement in order to decide which way the original project will go.

By contrast, an S0 proposal clearly limits its blast radius and should not impact a project to the degree of forking. I don't see a clear reason why we cannot use the same voting mechanism we do for PRs and RFCs here.


-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1285017979
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Yizhi Liu <no...@github.com.INVALID>]

@mbaret I think "incubator process" needs to make sure the incubated project gets support from the community. That's what they mentioned about "LLVM umbrella" or "Apache umbrella". LLVM and Apache are large enough organizations that they have dedicated mentors and more importantly, they attract outside contributors and users even for an small, incubating project sitting outside of main branch. If we are able to do the same I'm OK with the "incubator process", but I think at this stage what @Hzfengsy proposed in this RFC is the best trade-off.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1285985206
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Tianqi Chen <no...@github.com.INVALID>]

Thank you @areusch . 

First of all, I think @Hzfengsy  has already addressed many points you bought up in the past, including:
- Putting the CI/testing as part of the important factors to consider
- Clarifying the definition of S0
- Clarifyng n how deprecation of S0 happens 

I would like to acknowledge that you have made your point clearly in your last point. Most of those points are disagreement on how to do things. In light of such disagreement, it is important to acknowledge each others’ view, reevaluate, and come back to also put the community's voices into consideration.

Let me first try to address some of the cases you bought up:

My read is that the decision is around “module-establishment” but not blank approval of all code that comes. All the subsequent code changes are still subject to the normal review process.  

CI is indeed one important area of the codebase. It is also equally important in other areas, such as community member contributing code, document, importer, different modules. No area is certain superior to another one. We are working together to empower each other.

One thing to keep in mind though is that CI/testing is only good if we have community members who come and use them. It is a great thing to have comprehensive tests, and we work collectively to resolve issues in testing, CI, system design, documentation and other places.

Remember we are fostering a culture that empowers each other. Let us see how the two scenarios you mentioned will play out in the empowering culture (which I believe we all strive to do):

Imagine at some point, the integration test suite is added, but this involves regression tests that add 1 hour to the CI.  That particular code is receiving the normal review process, and will be blocked, which requires the contributing team to be mindful (either move integration, or come and help work CI to be able to support such cases):
- When some regression happens after the code gets merged, open-minded team members can always come and revert the tests, and work together to improve them. 
- If it gets down to fixing Jenkinsfile, people volunteer to help, just like in other areas. I have personally helped maintain Jenkinsfile for the first few years and would be more than happy to help improve Jenkinsfile for a module that gets supported by the majority of community members. My belief is other members will also do so.
-  Bottom-line, despite some hiccups, we bring a module that welcomes more community members, while still keeping other modules to support other community members. 

Remember that problems do arise in the community and we work collectively to address them. We also have tools in our hands and multiple decision points(including reverting) to iteratively improve. If we simply block the gate, we simply will not have those troubles, nor will we empower or even have those community members around who would otherwise not be part of the community. 

It is great to us collaboratively working together iteratively – which means indeed a few disruptions, but we also welcome more people to work together to bring in improvements. We all pay our credits to the community in one way or another. While not everyone might help in CI, and reverting some of their PR might cause a bit of burden on others, they help reviewing code in other modules, fixing regression tests, writing docs, and bringing in architectures that reduces burdens. 

In the case of someone proposing a new S0 module and not getting majority support. 

That would entail that there is at least one -1 from the PMC members. Which also means that it would get rejected at any other possible mechanisms. Not knowing PMC won’t be a problem at all because many of us are eager to review, and empower good proposals from community members. I also do not think relevant subject areas would be a problem since we have some of the best experts in ML compilation. Again you will find how much of an empowerment mindset we have in here. Establishing a module indeed has a lot of factors to consider, there are more negative considerations from the PMC, likely they have already considered the code and community factors. In such cases, I would still expect us to see constructive conversation and reasoning.

**OSS development is not a single bundled thing**.
- We establish module
- We add PR changes
- We revert patches that brings regression, CI issue or other cases

Each step has **their own contexts**, set of stakeholders. They can come from different people. Some volunteers can take over others. While it is important to set marks through RFC, roadmap, signaling, and strategy. It is also important to be able to facilitate and empower each other through each of those steps.

The same thing applies to process RFCs. We bring in the process RFC to address things one step at a time. While clarifying S1, S2 is indeed important, which can be a topic of future discussions. Same would go for improving the stability of TVM overall – which I believe we all are working hard as a community. Sometimes bringing new modules is part of that process.

I don’t think S0 has to do with stability btw, it only has things to do with the impact scope of the project – one module can be very stable and remains in S0. On the other hand, we might choose to refactor a S1 module that touches many areas (in that sense S1 might become unstable). They all go through the same process of code changes, documentation, release notes that signal changes. 

It is natural that some parts of the codebase will be under more active development than others, which also comes with clear documentation, release notes that signal these changes. There will also be roadmaps as part of the RFC that tracks what they propose to do, as well as follow ups RFCs(and different conversations) that track additional changes. All of this information will help to inform our community members – academic and industry alike. 

One thing to remember though is that being in OSS development means we cannot simply bend to the development needs of a single (industry or academic) player. We are empowering each other in this journey together. That means we need to also accommodate other industry players who need features in newer modules(while not disrupting old ones), academic contributors who contribute novel modules that are then taken over by industry members who help(and are willing) to maintain them. Having a few different modules helps here, where those who favor stability will certainly lean more heavily over longer established ones, while other industry players might benefit from the latest one. We see that happen for example in the PyTorch ecosystem and other ML ecosystems.


-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1338700949
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Siyuan Feng <no...@github.com.INVALID>]

Hi, @areusch 

Thank you, for posting the analysis of the benefits and drawbacks of merging a module. I would like to point out that there are a few critical pieces that are missing (mainly on the community side):
- Welcome new contributors who would become an added force, these contributors also contribute normally not only contributes to the particular S0 module but also to other modules.
- Enable TVM to solve problems that it was not able to atm and empower new users/developers who need them.

Thank you also for bringing up some of the factors to put into consideration in the RFC process:
- Discussions of test strategy, modularization, and documentation during the RFC process and not land half-baked code.
- Consider the relations among S0 modules to other modules, mainly on implicit impositions that S0 made on, say existing S1 APIs.
- Consider the overall complexity of the codebase and presentation to the users.
- Ease of maintenance overall for some of the core infra.

These are great goals that we all carry when evaluating new modules(including S0) and I also support these goals. They all align with G0. 

The proposal text does not imply when an S0 module gets established, it will lack these considerations throughout the process. As a matter of fact, we would also take these goals into account when evaluating any proposals(including S0).

One possible point of confusion here is connecting S0/S1 to importance, or how good the code is. The proposal intentionally avoids that, an earlier version of the proposal used the term "optional" which causes confusion, and we switched to a clear definition based on "scope of impact" rather than other factors such as importance, or how good the code test coverages are. 

An S0 module can become more important to some of our users than some of the S1 modules or have better code/tests coverage. The fact that they are more well-scoped simply means it is easier to remove them if needed, and there is less refactoring needed when refactoring other parts of the code.

Admittedly, this does not mean that S0 can simply be ignored when working on other modules. It is naturally true that there are considerations around testing and API changes when working on some of the more central modules, they should be considered during RFC and follow-up actions.  We want to come back and also acknowledge that the scope of impact is not binary – an S0 module in nature will have non-zero, but (sometimes significantly) less impact compared to non-S0, when it comes to changes in other modules and deprecations.

This proposal calls for putting the non-binary scope of impact and community into consideration for module establishment. Of course, such consideration should be put together with other considerations, including tests, documentation, modularization, fit into the overall project, etc. 

All the members are expected to take these factors into account when evaluating an S0-RFC and make collective decisions for module establishment. This comes to the decision process. There are many discussions that can be made during the RFC process, which is very helpful, for example. Making sure that the proposal is technically correct, providing suggestions to testing coverage, documents, etc. All of us are taking those into account when making that collective decision. 

Then there comes about reasonings that are relatively subjective reasonings in module establishment.
- Say we are introducing a new module to the codebase, majority of core devs think that the module provides new functionality that merits the module establishment, but some think that the new module has overlapped functionality(think about TorchFX and TorchScript case) that does not merit a module establishment.
- In some cases, the majority of core devs think that the establishment of a new module to support a not yet supported by the framework outweighs the cost of extra flow maintenance while some do not.

These are binary decisions that need to be made collectively by the community, they are likely subjective and different people have different pov and it is natural that people can disagree. Note that everyone also takes these factors into account. The proposal aims to provide guidelines and highlight different levels of open-mindedness. 

The proposal also helps to clarify the case for deprecation so we can get a cleaner codebase. Previously there wasn’t any text. 

Indeed we made the “S0 -> deprecation” part to be broad enough so we can handle various cases and we can talk about deprecation openly without having hard feelings about them. For example, in the case of “tests keep breaking or maintainers are relatively unresponsive”, would match this clause to start a discussion of deprecation without hard feelings. Such clarification is a positive step towards a more maintainable codebase overall.  Actually having such broadness in the proposal would help, so we can welcome contributions and not have hard feelings when talking about deprecations broadly. 

As a community we operate by empowering each other. Indeed it is hard to have a 100% clean cut of responsibility boundaries due to some factors being discussed. The scope of impact on the other hand is also not binary. For example, many of us volunteered to review, contribute to core modules, documents, advocations, writing tests and welcoming new members across our “boundaries”.

The proposal of course does not mean to indicate that S0 has zero impact on each of us. It certainly has a smaller scope of impact which is worth some different levels of open-mindedness. Remember the contributors/supporters of those modules also are contributing in different ways. They are all valuable and that is what makes us together as a community. 


-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1288784796
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Matthew Barrett <no...@github.com.INVALID>]

@Hzfengsy 
> There are similar sets of in-tree computational graph dialects that we intended to refer to – both TOSA and Linalg for example are in the MLIR tree and serve as computational graph dialects.

TOSA/Linalg are both graph dialects, but they don't fulfill the same function - TOSA is a higher-level representation than Linalg. As an example, IREE implements a lowering from TOSA to Linalg as part of its [TFLite support](https://iree-org.github.io/iree/blog/2021-07-19-tflite-tosa/#compiling-models). The closest example I can think of is [TCP](https://discourse.llvm.org/t/rfc-incubation-request-for-incubating-tcp-dialect-for-mlir/64883) vs. TOSA which was prompted by this lengthy [thread](https://discourse.llvm.org/t/rfc-proposal-for-a-high-level-ml-dialect-in-mlir/64249). However, that caused a huge amount of debate as to whether it was correct to have them both in the repo (similar to what we're seeing with Relax).

The solution in the LLVM community is to have TCP first exist as an [LLVM incubated project](https://lists.llvm.org/pipermail/llvm-dev/2020-June/142548.html) outside of the mono-repo. To quote the [LLVM developer policy](https://llvm.org/docs/DeveloperPolicy.html#incubating-new-projects) on why they have such a process:

> The burden to add a new project to the LLVM monorepo is intentionally very high, but that can have a chilling effect on new and innovative projects. To help foster these sorts of projects, LLVM supports an “incubator” process that is much easier to get started with. It provides space for potentially valuable, new top-level and sub-projects to reach a critical mass before they have enough code to prove their utility and grow a community. This also allows collaboration between teams that already have permissions to make contributions to projects under the LLVM umbrella.

This sounds extremely similar to the problem we're tackling with this RFC, just with an alternative solution. I think we need to be clear why we're arguing not for an incubation process (the MLIR/LLVM solution), but instead for this expedited approval process. It would also be good to include that argument in the RFC to make it clear that other processes have been considered.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1285672252
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Matthew Barrett <no...@github.com.INVALID>]

@yzhliu To clarify, I'm not suggesting we adopt an incubation process for TVM. I'm just explaining that we can't reference MLIR as an example of a community which uses the process described in this RFC, because it in fact uses a completely different process.

> In the S0-module proposal, majority of community supported in-tree dev

This process explicitly allows a module to be merged _without_ majority support - only 3 PMC members are required. In principle that means a '3 for / 22 against' vote on the PMC would actually _pass_ in favour of merging a new module which seems highly unusual to me.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1286036983
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Jared Roesch <no...@github.com.INVALID>]

While discussion continues I am asking that we hold off on voting so that all active contributors have a chance weigh in and comment. I know of at least two PMC members with perspectives to share that cannot comment until mid November. Holding off a few weeks seems like the right tradeoff for maintaining our shared values around community and community ownership at the cost of a little patience.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1287563004
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Siyuan Feng <no...@github.com.INVALID>]

@mbaret 

> TOSA/Linalg are both graph dialects, but they don't fulfill the same function

The definition of "same" is subjective; of course, different people can have different opinions that are less grounded. For example, what if many, or even the majority of people think a proposal contains sufficient differences(e.g., in the sense of TOSA/Linalg or TorchScript/TorchFX) that also serve community needs but some don't, does that still count as "same"?

The current RFC provides a guideline, considering that "An S0-module should show a clear positive fit to some(but not all) aspects of the project and clear cohesion to some of the existing modules."

The "clear positive fit to some" indicates the complementary aspect(that S0-module does something other modules do not do, as a result not the same). Note such evaluation can usually be subjective. 

We are also not singling out MLIR as a single case here. The TorchScript/TorchFX example serves as a better example in illustrating the situations we are facing here. 

> The closest example I can think of is [TCP](https://discourse.llvm.org/t/rfc-incubation-request-for-incubating-tcp-dialect-for-mlir/64883) vs. TOSA which was prompted by this lengthy [thread](https://discourse.llvm.org/t/rfc-proposal-for-a-high-level-ml-dialect-in-mlir/64249). However, that caused a huge amount of debate as to whether it was correct to have them both in the repo (similar to what we're seeing with Relax).
>
> The solution in the LLVM community is to have TCP first exist as an [LLVM incubated project](https://lists.llvm.org/pipermail/llvm-dev/2020-June/142548.html) outside of the mono-repo. To quote the [LLVM developer policy](https://llvm.org/docs/DeveloperPolicy.html#incubating-new-projects) on why they have such a process:

Thanks for bringing this up as a reference. Reading through the TCP thread, it is clear that the TCP proposal would not even get majority support from the MLIR core devs (most of the core devs signaled preference of not in-tree devs in that particular thread). An incubation process helps to incubate a module where the majority of the community is still unsure. We do anticipate different fork development methods will be helpful before the S0 stage.

We are solving a different problem. Specifically, this process RFC outlines the guideline in situations when the majority agree that in-tree dev is helpful and the discussions of module establishment become subjective.

In such cases, considering the scope of impact, users' needs and, more importantly, community empowerment in general. We think the call for the community over code and open-mindedness in the current proposal is appropriate. It also reflects G1 and aligns with the case such as TorchFX starts by offering clear positives to some aspects and continues to offer more aspects to the project.  

The open-mindedness of PyTorch in allowing modules(such as TorchScript and TorchFX) to co-exist contributes positively to the growth of the PyTorch ML ecosystem. As an ML project in nature and acknowledging G1 and community being critical, this proposal aims to bring that practice formally. 

One of the last factors is that many of us (considering many who supported the proposal) already were evaluating some of the previous RFCs following the open-mindedness principle already and pushed for community empowerment – which led to some of the RFC acceptance as well as broad inclusion of the community. If we all adopted a stronger subjective opinion and put away consideration of community, many past RFCs would have been rejected or delayed in the subjective debates, including RFCs from those who had stronger subjective opinions. We would hope to continue pushing the community over code principles. While also clarifying the clear transitions and scope of impact.



-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1288633361
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Tianqi Chen <no...@github.com.INVALID>]

Thanks @mbaret. First of all, I have updated my original post and please read it through again. 

> In such a situation, I would expect that with the combined resources of those eight organizations, they could address any concerns raised by the community through the standard process.  they'd first and foremost prioritize listening to and respecting the existing community.

These people **are part of** the existing community. And existing community as a whole should work on listening and empowering each other. People contribute in different ways, code, documents, advocation, no one means would be superior to another one.

> Resources and ability to go through process

A lot of our discussions are about how we should operate as a community.

Shall we always disallow new modules that have relevant features to coexist with existing ones? Should we outright reject the phased approach and always ask for a detailed plan before accepting a module? After considering the arguments, shall we empower community members so some of them can enjoy the new module, and help us to grow our developer community?

There is no single perfect answer here, and we all need to evaluate things by considering all the factors. First of all, it is important to acknowledge that each of our views is both valid in many cases. It is very important to understand each others' view, and re-evaluate the proposal under such a context. 

So while some of the concerns or ask from some of our members might seem valid, and it could be one way on how we operate things -- we should totally acknowledge that. The other ways of operating things are also valid. Simply stall in one case would implicitly mean that we reject the other way of operations that encourages moving things forward. 
 
And we needs to lean into the community when choosing hows. There is a reason why ASF have tools to do enable procedural votes and our community brings up the process RFC. We are following that process.



-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1339754681
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Matthew Barrett <no...@github.com.INVALID>]

> First of all, I have updated my last post and please read it through again.

Extensively editing your reply after people reply to it creates a discussion that's essentially impossible to follow. If you want to raise new points, and in the interest of maintaining the public record of our discussions, I'd appreciate a separate reply. Editing should be for making small corrections or minor additions, not completely re-drafting a response.

> Shall we always disallow new modules that have relevant features to coexist with existing ones?

It's getting a bit tiresome pretending this RFC isn't in direct response to what's happened with Relax, as no concrete examples are being cited and the mysterious 'hypothetical' happens to coincide precisely with the Relax situation.

If the request on the Relax RFC was 'scrap this entirely, rewrite it some other way', I'd have absolute sympathy. But let's recall what was _actually_ requested on that RFC:
- Explanation of why the technical approach taken we chosen
- Roadmap of features that were going to implemented (not explicit timelines)
- Roadmap of how someone currently using Relay could migrate to Relax

These were questions asked in good faith by an organisation that actually _wanted to migrate to Relax_, so it's unbelievable to me how they were treated. I personally spent many many hours of my own time trying to find a resolution to that RFC so that Relax could progress. And what I concluded was that positive, constructive and polite engagement would have entirely avoided the situation we're now in, it's completely unrelated to process. If we really want to improve our community, we should all reflect on how that RFC (and the following vote) was handled and what lessons were learnt.

I'm concerned that instead of being genuinely interested in improving the contribution lifecycle in TVM (an issue I care about and am very interested in), this RFC is instead just about amending the rules to override a specific veto. Could you reaffirm you're original commitment that this new process will not apply retroactively, including to Relax?

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1339922203
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Matthew Barrett <no...@github.com.INVALID>]

> Imagine a case when a module is being needed by more than eight organizations -- many of them are industry players, but not all other members.

In such a situation, I would expect that with the combined resources of those eight organizations, they could address any concerns raised by the community through the standard process. That's the behaviour I'd anticpate of any new organization looking to contribute to an OSS project - they'd first and foremost prioritize listening to and respecting the existing community.

I'd expect this S0 process to more commonly apply to smaller contributions where the contributor doesn't necessarily have the bandwidth to make long term commitments nor to continue parallel (forked) development. In those cases, I can see that a higher barrier to making a contribution (like requiring roadmaps, further technical documentation, tutorials, and extensive review) might exceed what is reasonable to ask of an individual/small team. Industrial contributors, however, should easily have the resource to clear that barrier, as it would almost certainly already form part of their internal process.

The only exception I can see to this would be where a proposal is being maliciously vetoed (e.g. with entirely unreasonable requests or no justification at all). I've personally never seen this happen in the TVM community, and hope things remain that way.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1339729885
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [Process RFC] Empowering New Scoped Module to the Project (PR #95)

[Tianqi Chen <no...@github.com.INVALID>]

>  I'd appreciate a separate reply. 

Here they are:

Thank you for laying out the personas. I agree with some of those categorizations. I would like to point out however, that almost every one of us in the community are wearing multiple hats. I personally contributed features, maintained the modules myself and others contributed, fixes test-cases, bring refactoring, write documents.

It is **unfair to use simplified arguments to categorize any S0 proposer as only feature developer**. This is some form of labeling that some of us unconsciously apply in our conversations, and they are not helpful for our conversations nor being fair to many community members here, please stop doing so.

Let's come back to think about bringing people along on the journey. Please remember that a set of “maintainers” are not fixed in the community.

Bringing additional contributors along would mean we would get more people maintaining the code together. This has been the case in a lot of our past empowerment. The proposal text explicitly suggested that the proposer would need to maintain the module.
Additionally, we enable a broader set of people who will help contribute to maintenance of existing modules. These are the people who otherwise will not be given opportunity or motivation to contribute.
It is indeed true that there will always be part of impact, and everyone in the community is supposed to take these into consideration when evaluating the factors.

In each proposal however, we would certainly evaluate the scenarios, and if such cases do occur – where the proposer ignores the maintenance duty, please let us know and we would certainly take these into strong consideration and resolve the case. The proposal also have an explicit deprecation trigger that makes suh case clear.

> Relax RFC treatment

I would suggest we move the conversation of relax RFC treatment to the particular thread. But to elaborate a bit, there are indeed constructive explainations of technical approach were taken, features that is going to be implemented. The questions were being genuinely answered through that thread. 

There was a disagreement on whether or not deprecation roadmap should be included as a gating criteria of that particular RFC that proposes module co-exist -- this is again a disagreement on how we should do things -- which should be resolved through community conversations. I would expect continued conversation, and will work wearing my Apache hat to help resolve that conversation separately from this process RFC and empower the community members. Please share followup thoughts on that thread as well.

> Could you reaffirm you're original commitment that this new process will not apply retroactively, including to Relax?

I have stated that and I am reaffirming that the new process will not apply retroactively to existing RFCs, that would include the current relax RFC. 

It is more important on how the community should move forward in the future. I would love us to be in a world where community's voices are being heard, and we **collectively working** to decide on how we do things through the defined process .  

I would love us to be in a world where community's voices are being considered as an important factor besides code. As someone who created the project and deeply invested into it, I am very concerned about the needs of community empowerment. Without such empowerment and consideration, we could be very successfull maybe even merging relax, doing very well on the code, but we will simply not get there as a community that empowers each other.

Thank you for having this conversation. I know that we have disagreements on hows. I would like to acknowledge again:

- The approaches(on must laying out X, Y Z) being described in your post are completely valid and possible approaches on how to do things in Apache TVM. There is nothing wrong in stating these positions. There is nothing wrong for you to make these suggestions on how we should do things.
- There are also different valid opinions on how to do do things. Which are also not wrong. Especially when it comes to putting community factor into consideration. Please look into those messages and re-evaluate under these context.

In the end, after  having conversations constructively, acknowledging different approaches on how to do things. We need to collectively work together on which approach to go as a community.





-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/95#issuecomment-1339974580
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>