[GitHub] [tvm-rfcs] JosephTheOctonaut commented on pull request #80: Asynchronous stage in software pipeline

[GitBox <gi...@apache.org>]

JosephTheOctonaut commented on PR #80:
URL: https://github.com/apache/tvm-rfcs/pull/80#issuecomment-1162481071

   Awesome stuff, @masahi !
   
   Specific questions:
   
   - I don't think I saw it explicitly mentioned, but it is assumed/required that the backend executes committed chunks in order, right? E.g., if the program commits a chunk for i=0 to i=5 before hitting a `wait`, the hardware can't decide to execute the i=5 chunk first.
   - The index `"software_pipeline_async_stages"` refers to elements of the `"software_pipeline_stage"` list, correct? I found this a bit confusing, because the asynchronous "stages" are not the same as the pipeline "stages". (e.g., a pipeline with `"software_pipeline_stage", [0,0,0,1]` could have `"software_pipeline_async_stages, [2,3]"` when there are no pipeline stages 2 and 3)
   - The definition of `async_commit_stage(i)` is to "Group one or more invocation of async operations...". Is the model here that all uncommitted async operations at that point in the program are committed? Furthermore, are "async operations" defined as any statement in a pipeline stage that was identified as async?
     - I initially found it confusing that `async_commit_stage(i)` could appear inside and outside of an `async_scope:` section. It makes sense that it could be both, because (assuming my above assumptions are correct) it doesn't have to do with any lexical scoping, operating instead on the "hidden state" of whatever async operations are uncommitted. No specific question here, but maybe a more intuitive grouping is possible that would be syntactically visible.
   - Is it correct that `async_wait_stage` will only block the "main" thread of execution? For example, if (on a particular backend) an async pipeline stage is executed by another thread, and `async_wait_stage` appeared in that stage, which thread would it block? I'm assuming the answer is "whatever the backend lowering does," but I was curious if there was an intended semantics.
   
   Small doc nits for clarity:
   
   - `for k0 in range(125):` I think the index should be `i` here
   - `B_local[0] <- B_shared[(I + 1) % 4, ...]` capital `I` used instead of `i`
   - `async_scoppe` instead of `async_scope`
   
   
   Higher-level thoughts:
   
   One concern is moving an async system into TIR that is too narrow. You discuss this quite a bit, and make a lot of good points. This system clearly works very well for CUDA, but if there isn't a good transform to token-based systems, do we end up with two async systems in TIR? Not trying to suggest this should not be incorporated, just genuinely curious what the long-term plans might be.
   
   >Importantly, we are not trying to propose a “general async semantics to TIR”. Rather the goal is to come up with an async design specifically for the TIR software pipeline transform.
   
   This can be a bit difficult, as it would make sense for future systems to try to use the existing async system to whatever extent possible. Again, not trying to suggest a specific course of action, just pointing out that there's a difficult general-vs-specific trade-off here.
   
   Regarding interopability with a token-based model: you make an excellent point that a token system has expressibility issues in a TIR program, because it has to refer to a specific statement at a specific loop iteration. But, it's also more intuitive. This contrasts with the implicit system here, which feels unintuitive, but is quite succinct and easily expressible. 
   
   I'm still working through what a translation to/from a token system might look like, but I'm currently thinking that they're much closer than I initially thought. In either case, what we want (at a high level) is a way to refer to a specific chunk, blocking execution until that chunk has finished. Your comment about keeping a buffer in the token case made me realize that it ends up pretty similar: a token system waiting for chunk _C_ from _N_ iterations previous might use `wait(C, i-N)`, which looks pretty similar to the `wait(C, N)` we would use here. (Just my preliminary thoughts; I'm not sure if there's a nicer abstraction to use in TIR that caters to both systems.)
   
   It's interesting that you mention that there's no easy translation from tokens to counting (based on MLIR not having implemented one?), but you suspect the reverse could be simple. Does this suggest that the token system has less information encoded than the counting system? (I.e., we can go counting --> tokens but not the reverse because we lost information in the transformation.) Or is it just specifics of a PTX-like system, not a "counting system" in general, that make the translation to it hard?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org