You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@tvm.apache.org by AndrewZhaoLuo <no...@github.com.INVALID> on 2021/06/09 04:06:59 UTC

[apache/tvm-rfcs] Automatic Mixed Precision Pass RFC (#6)

You can view, comment on, or merge this pull request online at:

  https://github.com/apache/tvm-rfcs/pull/6

-- Commit Summary --

  * draft v1

-- File Changes --

    A rfcs/0001-AMP_pass.md (128)

-- Patch Links --

https://github.com/apache/tvm-rfcs/pull/6.patch
https://github.com/apache/tvm-rfcs/pull/6.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6

Re: [apache/tvm-rfcs] Automatic Mixed Precision Pass RFC (#6)

Posted by Cody Yu <no...@github.com.INVALID>.

Thanks for the answers. I'll review the PR to get more implementation details.
One more question regarding the extensibility: can this be extended easily to support bfloat16?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-857914672

Re: [apache/tvm-rfcs] [RFC] [Relay] Automatic Mixed Precision Pass (#6)

Posted by Gayatri P K <no...@github.com.INVALID>.

@AndrewZhaoLuo, When we enable mixed precision arithmetic, the output types of a model depends on the last operation in the relay IR, ie, if it convolution, it will be in float16 and for the case argmax it will be int32 and in the case of softmax it will be float32. So, is this expected? Or should the nodes that are marked as output nodes should be treated differently so that they return the expected specific out_type irrespective of the optimization(mixed precision pass) happening? Please let me know your thoughts on this.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-1024374936
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [RFC] [Relay] Automatic Mixed Precision Pass (#6)

Posted by Cody Yu <no...@github.com.INVALID>.

btw, according to #17, please update the RFC number on the file name to align with this PR number.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-893007973

Re: [apache/tvm-rfcs] [RFC] [Relay] Automatic Mixed Precision Pass (#6)

Posted by AndrewZhaoLuo <no...@github.com.INVALID>.

Thanks for driving this review @comaniac. I'll get to this later in the week.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-887889315

Re: [apache/tvm-rfcs] Automatic Mixed Precision Pass RFC (#6)

Posted by Cody Yu <no...@github.com.INVALID>.

Thanks for the RFC. I have two questions:
1. How to mark/set the color (i.e., attribute) of every operator?
2. It seems to me that if we register a casting checker instead of just a label (color), then we can simplify the algorithm a lot. Taking the case `A(green) - B(gray) - C(green)` as an example, if we could register a casting rule of B as follows, then we just need one traverse to know if we need cast around B:

    ```
    def amp_B(expr, args):
        a = args[0]
        if (a.dtype is float16):
          return fp16
        return fp32
    ```

    After all, we only need the previous nodes to determine 1) whether to use FP16 implementation, and 2) whether to insert casts. It seems to me that this pass is similar to the layout conversion pass, which uses one traverse to finish everything, so it might be possible for AMP too.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-857889774

Re: [apache/tvm-rfcs] Automatic Mixed Precision Pass RFC (#6)

Posted by AndrewZhaoLuo <no...@github.com.INVALID>.

> Thanks for the answers. I'll review the PR to get more implementation details.
> One more question regarding the extensibility: can this be extended easily to support bfloat16?

It should be trivial (hope I don't eat my words). I'm not 100% sure of the support for bfloat16 in current relay ops however.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-857922283

Re: [apache/tvm-rfcs] [RFC] [Relay] Automatic Mixed Precision Pass (#6)

Posted by Cody Yu <no...@github.com.INVALID>.

Thanks @AndrewZhaoLuo

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-904813648

Re: [apache/tvm-rfcs] Automatic Mixed Precision Pass RFC (#6)

Posted by AndrewZhaoLuo <no...@github.com.INVALID>.

> Thanks for the RFC. I have two questions:
> 
> 1. How to mark/set the color (i.e., attribute) of every operator?
> 2. It seems to me that if we register a casting checker instead of just a label (color), then we can simplify the algorithm a lot. Taking the case `A(green) - B(gray) - C(green)` as an example, if we could register a casting rule of B as follows, then we just need one traverse to know if we need cast around B:
>    ```
>    def amp_B(expr, args):
>        a = args[0]
>        if (a.dtype is float16):
>          return fp16
>        return fp32
>    ```
>    
>    
>        
>          
>        
>    
>          
>        
>    
>        
>      
>    After all, we only need the previous nodes to determine 1) whether to use FP16 implementation, and 2) whether to insert casts. It seems to me that this pass is similar to the layout conversion pass, which uses one traverse to finish everything, so it might be possible for AMP too.

Yep that is correct. This RFC has an initial PR here: https://github.com/apache/tvm/pull/8069.

To answer your questions:
1.  src/relay/transforms/fp32_to_fp16.h -- DefaultFP16Colorer is the default way. But the only thing we need is a callable with type CallNode*(Color). So you could write your own colorer that does arbitrary stuff when only looking at a single node at a time.

2. This is functionally what is done in the PR I link.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-857903539

Re: [apache/tvm-rfcs] [RFC] [Relay] Automatic Mixed Precision Pass (#6)

Posted by MeJerry215 <no...@github.com.INVALID>.

@AndrewZhaoLuo will it remove cast weight to float16 from graph? and make weight as float16 when build lib. 
in my opinion, it will reduce the bandwidth.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-1002905618
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>

Re: [apache/tvm-rfcs] [RFC] [Relay] Automatic Mixed Precision Pass (#6)

Posted by AndrewZhaoLuo <no...@github.com.INVALID>.

@comaniac, I'll be talking about this at the TVM community meeting tomorrow so put off merging until after.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-901318475

Re: [apache/tvm-rfcs] [RFC] [Relay] Automatic Mixed Precision Pass (#6)

Posted by Cody Yu <no...@github.com.INVALID>.

Merged #6 into main.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#event-5201897255

Re: [apache/tvm-rfcs] [RFC] [Relay] Automatic Mixed Precision Pass (#6)

Posted by AndrewZhaoLuo <no...@github.com.INVALID>.

So the associated PR is getting closer to a mergeable state. Is this RFC ready to merge?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-861736780

Re: [apache/tvm-rfcs] Automatic Mixed Precision Pass RFC (#6)

Posted by Thierry Moreau <no...@github.com.INVALID>.

CCing @csullivan

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-857933603

Re: [apache/tvm-rfcs] [RFC] [Relay] Automatic Mixed Precision Pass (#6)

Posted by Cody Yu <no...@github.com.INVALID>.

Took a quick pass to the updated RFC. I think it's almost ready to merge as long as the last 3 comments are resolved.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-899816190

Re: [apache/tvm-rfcs] [RFC] [Relay] Automatic Mixed Precision Pass (#6)

Posted by AndrewZhaoLuo <no...@github.com.INVALID>.

If there is not other objections, this will be merged on monday.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-902192178

Re: [apache/tvm-rfcs] [RFC] [Relay] Automatic Mixed Precision Pass (#6)

Posted by Tianqi Chen <no...@github.com.INVALID>.

cc @comaniac would be great if you can help shepherd this RFC

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-886048408

Re: [apache/tvm-rfcs] [RFC] [Relay] Automatic Mixed Precision Pass (#6)

Posted by AndrewZhaoLuo <no...@github.com.INVALID>.

PTAL @comaniac 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-900596242

Re: [apache/tvm-rfcs] Automatic Mixed Precision Pass RFC (#6)

Posted by Cody Yu <no...@github.com.INVALID>.

> > Thanks for the answers. I'll review the PR to get more implementation details.
> > One more question regarding the extensibility: can this be extended easily to support bfloat16?
> 
> It should be trivial (hope I don't eat my words). I'm not 100% sure of the support for bfloat16 in current relay ops however.

TVM has limited bfloat16 support now but it's on the way, so it would be better for this RFC to also consider this case, even the initial version may not cover it.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-857950295

Re: [apache/tvm-rfcs] Automatic Mixed Precision Pass RFC (#6)

Posted by AndrewZhaoLuo <no...@github.com.INVALID>.

I don't know Chris Sullivan's github handle so if someone could cc him too that would be great.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-857931228

Re: [apache/tvm-rfcs] [RFC] [Relay] Automatic Mixed Precision Pass (#6)

Posted by AndrewZhaoLuo <no...@github.com.INVALID>.

Going to get to this tomorrow 😬. Promise 🤞

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-893004650

Re: [apache/tvm-rfcs] [RFC] [Relay] Automatic Mixed Precision Pass (#6)

Posted by masahi <no...@github.com.INVALID>.

@MeJerry215 Yes, casting of weight to fp16 is done at compile time by `FoldConstant` pass, so weights will be in fp16 at deploy time. 

-- 
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm-rfcs/pull/6#issuecomment-1002908086
You are receiving this because you are subscribed to this thread.

Message ID: <ap...@github.com>