You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2020/10/26 21:42:48 UTC

[GitHub] [incubator-tvm] altanh opened a new pull request #6767: [Relay][Training] Add more missing gradients

altanh opened a new pull request #6767:
URL: https://github.com/apache/incubator-tvm/pull/6767


   Added the following gradients:
   - `take`
   - `reverse_reshape`
   - `stack`
   - `squeeze`
   - `expand_dims`
   - `arange`
   
   Also fixed a typo in type solver diagnostics. I had to use a Relay loop in `take_grad` to support Any size in indices.
   
   cc @t-vi @SWu @MarisaKirisame 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-tvm] antinucleon commented on pull request #6767: [Relay][Training] Add more missing gradients

Posted by GitBox <gi...@apache.org>.
antinucleon commented on pull request #6767:
URL: https://github.com/apache/incubator-tvm/pull/6767#issuecomment-716969832


   Please try to pass lint locally first~


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-tvm] jroesch commented on pull request #6767: [Relay][Training] Add more missing gradients

Posted by GitBox <gi...@apache.org>.
jroesch commented on pull request #6767:
URL: https://github.com/apache/incubator-tvm/pull/6767#issuecomment-716843558


   @antinucleon 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-tvm] MarisaKirisame commented on a change in pull request #6767: [Relay][Training] Add more missing gradients

Posted by GitBox <gi...@apache.org>.
MarisaKirisame commented on a change in pull request #6767:
URL: https://github.com/apache/incubator-tvm/pull/6767#discussion_r512425339



##########
File path: python/tvm/relay/op/_tensor_grad.py
##########
@@ -665,3 +673,115 @@ def cross_entropy_with_logits_grad(orig, grad):
     batch_size = take(shape, const(0, dtype="int32"), axis=0)
     grad = grad / batch_size.astype(x.checked_type.dtype)
     return [-grad * y, -grad * x]
+
+
+@register_gradient("take")
+def take_grad(orig, grad):

Review comment:
       you can get by by defining a 'put' operator, that put a scalar into an index of a tensor, and leave other palces unchanged. put and take has some classic property which I assume will be better for the optimizer. It also allow other optimization (e.g. put and reduce_sum, using grad + (put vala at idxa in 0_array) + (put valb at idxb in 0_array) will be collapsed into a long chain of put on grad, allowing COW to kick in and all take grad mutation update (instead of creating another tensor).

##########
File path: python/tvm/relay/op/_tensor_grad.py
##########
@@ -665,3 +673,115 @@ def cross_entropy_with_logits_grad(orig, grad):
     batch_size = take(shape, const(0, dtype="int32"), axis=0)
     grad = grad / batch_size.astype(x.checked_type.dtype)
     return [-grad * y, -grad * x]
+
+
+@register_gradient("take")
+def take_grad(orig, grad):

Review comment:
       @jroesch please look and comment as well.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-tvm] altanh commented on a change in pull request #6767: [Relay][Training] Add more missing gradients

Posted by GitBox <gi...@apache.org>.
altanh commented on a change in pull request #6767:
URL: https://github.com/apache/incubator-tvm/pull/6767#discussion_r512887971



##########
File path: python/tvm/relay/op/_tensor_grad.py
##########
@@ -665,3 +673,115 @@ def cross_entropy_with_logits_grad(orig, grad):
     batch_size = take(shape, const(0, dtype="int32"), axis=0)
     grad = grad / batch_size.astype(x.checked_type.dtype)
     return [-grad * y, -grad * x]
+
+
+@register_gradient("take")
+def take_grad(orig, grad):

Review comment:
       This is a good point that I was wondering about. The loop is basically just implementing a `put` operation (like I described in the comment), so it would make sense to have it be a separate op since I imagine it will be useful in general. Should I remove this gradient for now, or keep it and replace it with `put` once I implement it?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-tvm] tqchen commented on pull request #6767: [Relay][Training] Add more missing gradients

Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #6767:
URL: https://github.com/apache/incubator-tvm/pull/6767#issuecomment-718218947


   merging per reviews, feel free to send followup improvements per @MarisaKirisame 's comment
   
   Thanks @jroesch @altanh @antinucleon @MarisaKirisame 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-tvm] tqchen merged pull request #6767: [Relay][Training] Add more missing gradients

Posted by GitBox <gi...@apache.org>.
tqchen merged pull request #6767:
URL: https://github.com/apache/incubator-tvm/pull/6767


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-tvm] MarisaKirisame commented on a change in pull request #6767: [Relay][Training] Add more missing gradients

Posted by GitBox <gi...@apache.org>.
MarisaKirisame commented on a change in pull request #6767:
URL: https://github.com/apache/incubator-tvm/pull/6767#discussion_r513029211



##########
File path: python/tvm/relay/op/_tensor_grad.py
##########
@@ -665,3 +673,115 @@ def cross_entropy_with_logits_grad(orig, grad):
     batch_size = take(shape, const(0, dtype="int32"), axis=0)
     grad = grad / batch_size.astype(x.checked_type.dtype)
     return [-grad * y, -grad * x]
+
+
+@register_gradient("take")
+def take_grad(orig, grad):

Review comment:
       Both are fine.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org