You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by "junrushao (via GitHub)" <gi...@apache.org> on 2023/09/03 23:05:07 UTC

[GitHub] [tvm] junrushao opened a new pull request, #15662: [Unity][Frontend][NN] Add nn.MultiLinear

junrushao opened a new pull request, #15662:
URL: https://github.com/apache/tvm/pull/15662

   This PR introduces `nn.MultiLinear` as an extension to `nn.Linear` which is arithmetically equivalent to applying multiple `nn.Linear` to the same input.
   
   For example, QKV projection in attention, represented in `nn.Linear` being:
   
   ```python
   q_proj = nn.Linear(hidden_dim, ...)
   k_proj = nn.Linear(hidden_dim, ...)
   v_proj = nn.Linear(hidden_dim, ...)
   
   q = q_proj(x)
   k = k_proj(x)
   v = v_proj(x)
   ```
   
   could be simplified using `nn.MultiLinear` as:
   
   ```python
   qkv_proj = nn.MultiLinear((hidden_dim, hidden_dim, hidden_dim), ...)
   q, k, v = qkv_proj(x)
   ```
   
   The underlying implementation, by default, will fuse all the projection matrices into a single one, to allow potentially larger room for optimization.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] junrushao commented on pull request #15662: [Unity][Frontend][NN] Add nn.MultiLinear

Posted by "junrushao (via GitHub)" <gi...@apache.org>.
junrushao commented on PR #15662:
URL: https://github.com/apache/tvm/pull/15662#issuecomment-1704429272

   CC: @MasterJH5574 @jwfromm 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] junrushao merged pull request #15662: [Unity][Frontend][NN] Add nn.MultiLinear

Posted by "junrushao (via GitHub)" <gi...@apache.org>.
junrushao merged PR #15662:
URL: https://github.com/apache/tvm/pull/15662


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org