You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/11/18 17:08:45 UTC

[GitHub] [incubator-mxnet] bartekkuncer opened a new pull request #19562: Integrate oneDNN layer normalization implementation

bartekkuncer opened a new pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562


   Opperf:
   
     | forward |   |   | backward |   |  
   -- | -- | -- | -- | -- | -- | --
   size | w/o | w/ | time diff in % | w/o | w/ | time diff in %
   1000x5 | 0,0203 | 0,0338 | 66,50246 | 0,1402 | 0,137 | -2,28245364
   1000x100 | 0,0394 | 0,0375 | -4,82234 | 0,6158 | 0,1037 | -83,1601169
   300x512 | 0,0428 | 0,0398 | -7,00935 | 0,6597 | 0,071 | -89,2375322
   500x512 | 0,0737 | 0,042 | -43,0122 | 1,0916 | 0,0765 | -92,9919384
   1000x2048 | 0,2187 | 0,0696 | -68,1756 | 8,6593 | 0,1659 | -98,0841408
   1000x3 | 0,02 | 0,0312 | 56 | 0,1145 | 0,1221 | 6,63755459
   45x512 | 0,0267 | 0,033 | 23,59551 | 0,2213 | 0,0632 | -71,4414822
   1000x5x100 | 0,0968 | 0,0504 | -47,9339 | 2,2083 | 0,1869 | -91,536476
   1000x8x100 | 0,1273 | 0,0693 | -45,5617 | 3,4197 | 0,314 | -90,817908
   300x512x512 | 6,947 | 5,559 | -19,9798 | 583,0516 | 12,5954 | -97,8397452
   500x512x10 | 0,2469 | 0,471 | 90,76549 | 7,883 | 9,5545 | 21,2038564
   1000x5x2048 | 1,1808 | 0,4726 | -59,9763 | 69,3604 | 1,5419 | -97,7769736
   1000x2048x3 | 0,8903 | 2,3565 | 164,6861 | 23,2045 | 179,1877 | 672,210994
   45x512x512 | 1,267 | 0,4315 | -65,9432 | 79,5693 | 1,4785 | -98,1418713
   1000x5x30x200 | 2,9532 | 1,8339 | -37,9013 | 190,4334 | 6,9446 | -96,3532658
   100x100x10x300 | 2,9831 | 1,83 | -38,6544 | 189,7882 | 6,9442 | -96,3410792
   300x512x45x20 | 11,6564 | 13,2278 | 13,48101 | 441,5131 | 214,6873 | -51,3746478
   50x512x40x30 | 3,0377 | 2,4979 | -17,77 | 193,4521 | 36,4343 | -81,1662422
   100x2048x10x10 | 2,2314 | 3,1573 | 41,49413 | 64,9861 | 85,1543 | 31,0346366
   1000x3x10x200 | 0,793 | 0,1644 | -79,2686 | 28,1187 | 1,0968 | -96,0993929
   45x52x300x45 | 3,1359 | 2,2834 | -27,1852 | 221,7148 | 21,7037 | -90,2109828
   100x5x30x20x10 | 0,286 | 0,4941 | 72,76224 | 9,1756 | 13,4736 | 46,8416234
   100x4x100x10x30 | 1,4714 | 1,0342 | -29,7132 | 67,6208 | 10,4669 | -84,5211828
   300x52x2x45x20 | 2,7541 | 2,7491 | -0,18155 | 93,0693 | 53,0185 | -43,0333096
   500x52x3x40x30 | 8,1896 | 7,7518 | -5,3458 | 602,2581 | 101,6845 | -83,1161258
   100x28x10x10x10 | 0,2706 | 0,4627 | 70,99039 | 8,58 | 11,4651 | 33,6258741
   100x3x10x18x200 | 1,2413 | 0,4217 | -66,0276 | 53,8475 | 2,4962 | -95,3643159
   45x512x30x4x45 | 10,9092 | 9,5348 | -12,5985 | 947,7691 | 79,3489 | -91,6278237
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] pengzhao-intel commented on pull request #19562: Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
pengzhao-intel commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-732940638


   @bartekkuncer could you add more description in the PR to avoid confusion for the reviewers?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19562: [operator] Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-902229641


   Jenkins CI successfully triggered : [centos-gpu]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-758478282


   @mxnet-bot run ci [unix-gpu, windows-gpu, windows-cpu]


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] kpuatamazon commented on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
kpuatamazon commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-744892721


   I made the sizing more systematic.  
   
   AVX512 means `__attribute__((target("avx512f,avx512bw,avx512cd,avx512dq,avx512vnni")))` 
   
   Inverse means the 1.f/std is computed in advance rather than dividing by std in the loop.  
   
   Overall, the Marian implementation seems to win on smaller problem sizes, including the x512 sizes from @fhieber but lose on larger problem sizes.  
   
   Of course there are edge cases when the width is not a multiple of 16 and gcc is testing for those edge cases every time, so I see how that could be optimized.  
   
   | Shape | Marian | + AVX512 | + AVX512 + inverse | oneDNN |
   |--|--|--|--|--|
   |    1x    3 | 0.0000363| 0.0000370| 0.0000364| 0.0000453|
   |    5x    3 | 0.0000337| 0.0000355| 0.0000348| 0.0000426|
   |   10x    3 | 0.0000346| 0.0000352| 0.0000344| 0.0000434|
   |   20x    3 | 0.0000337| 0.0000342| 0.0000349| 0.0000438|
   |   30x    3 | 0.0000354| 0.0000376| 0.0000371| 0.0000434|
   |   40x    3 | 0.0000373| 0.0000396| 0.0000382| 0.0000431|
   |   50x    3 | 0.0000381| 0.0000405| 0.0000393| 0.0000436|
   |   60x    3 | 0.0000390| 0.0000408| 0.0000403| 0.0000447|
   |   70x    3 | 0.0000391| 0.0000376| 0.0000411| 0.0000440|
   |   80x    3 | 0.0000403| 0.0000400| 0.0000414| 0.0000438|
   |   90x    3 | 0.0000399| 0.0000399| 0.0000392| 0.0000446|
   |  100x    3 | 0.0000378| 0.0000409| 0.0000399| 0.0000451|
   |  110x    3 | 0.0000385| 0.0000414| 0.0000398| 0.0000446|
   |  120x    3 | 0.0000390| 0.0000413| 0.0000417| 0.0000454|
   |  130x    3 | 0.0000389| 0.0000429| 0.0000420| 0.0000457|
   |  140x    3 | 0.0000402| 0.0000437| 0.0000436| 0.0000452|
   |  150x    3 | 0.0000403| 0.0000442| 0.0000432| 0.0000461|
   |  200x    3 | 0.0000432| 0.0000480| 0.0000466| 0.0000476|
   |  300x    3 | 0.0000476| 0.0000553| 0.0000533| 0.0000506|
   |  400x    3 | 0.0000532| 0.0000617| 0.0000604| 0.0000533|
   |  500x    3 | 0.0000575| 0.0000694| 0.0000670| 0.0000570|
   | 1000x    3 | 0.0000826| 0.0001037| 0.0001029| 0.0000713|
   | 2000x    3 | 0.0001340| 0.0001730| 0.0001706| 0.0000988|
   | 3000x    3 | 0.0001818| 0.0002431| 0.0002402| 0.0001275|
   | 4000x    3 | 0.0002298| 0.0003116| 0.0003092| 0.0001554|
   | 5000x    3 | 0.0002777| 0.0003804| 0.0003814| 0.0001832|
   |16384x    3 | 0.0008319| 0.0011868| 0.0012013| 0.0005041|
   |    1x   10 | 0.0000339| 0.0000387| 0.0000348| 0.0000412|
   |    5x   10 | 0.0000336| 0.0000346| 0.0000345| 0.0000423|
   |   10x   10 | 0.0000349| 0.0000347| 0.0000343| 0.0000426|
   |   20x   10 | 0.0000339| 0.0000369| 0.0000374| 0.0000423|
   |   30x   10 | 0.0000371| 0.0000391| 0.0000378| 0.0000439|
   |   40x   10 | 0.0000378| 0.0000397| 0.0000395| 0.0000433|
   |   50x   10 | 0.0000393| 0.0000407| 0.0000408| 0.0000440|
   |   60x   10 | 0.0000398| 0.0000411| 0.0000423| 0.0000444|
   |   70x   10 | 0.0000403| 0.0000399| 0.0000416| 0.0000445|
   |   80x   10 | 0.0000411| 0.0000414| 0.0000398| 0.0000449|
   |   90x   10 | 0.0000410| 0.0000417| 0.0000406| 0.0000452|
   |  100x   10 | 0.0000391| 0.0000424| 0.0000416| 0.0000457|
   |  110x   10 | 0.0000403| 0.0000433| 0.0000425| 0.0000460|
   |  120x   10 | 0.0000404| 0.0000446| 0.0000433| 0.0000465|
   |  130x   10 | 0.0000414| 0.0000450| 0.0000440| 0.0000463|
   |  140x   10 | 0.0000413| 0.0000460| 0.0000455| 0.0000467|
   |  150x   10 | 0.0000420| 0.0000474| 0.0000455| 0.0000471|
   |  200x   10 | 0.0000457| 0.0000511| 0.0000497| 0.0000496|
   |  300x   10 | 0.0000517| 0.0000598| 0.0000579| 0.0000522|
   |  400x   10 | 0.0000584| 0.0000697| 0.0000660| 0.0000560|
   |  500x   10 | 0.0000646| 0.0000772| 0.0000759| 0.0000592|
   | 1000x   10 | 0.0000956| 0.0001229| 0.0001169| 0.0000773|
   | 2000x   10 | 0.0001586| 0.0002063| 0.0002010| 0.0001119|
   | 3000x   10 | 0.0002192| 0.0002946| 0.0002856| 0.0001441|
   | 4000x   10 | 0.0002827| 0.0003824| 0.0003686| 0.0001793|
   | 5000x   10 | 0.0003435| 0.0004679| 0.0004548| 0.0002134|
   |16384x   10 | 0.0010506| 0.0015780| 0.0015417| 0.0006026|
   |    1x  100 | 0.0000348| 0.0000379| 0.0000405| 0.0000427|
   |    5x  100 | 0.0000348| 0.0000346| 0.0000343| 0.0000433|
   |   10x  100 | 0.0000337| 0.0000350| 0.0000367| 0.0000436|
   |   20x  100 | 0.0000401| 0.0000388| 0.0000395| 0.0000445|
   |   30x  100 | 0.0000411| 0.0000406| 0.0000393| 0.0000452|
   |   40x  100 | 0.0000379| 0.0000381| 0.0000375| 0.0000462|
   |   50x  100 | 0.0000391| 0.0000399| 0.0000384| 0.0000466|
   |   60x  100 | 0.0000394| 0.0000412| 0.0000393| 0.0000474|
   |   70x  100 | 0.0000422| 0.0000428| 0.0000408| 0.0000489|
   |   80x  100 | 0.0000433| 0.0000439| 0.0000408| 0.0000492|
   |   90x  100 | 0.0000436| 0.0000445| 0.0000425| 0.0000500|
   |  100x  100 | 0.0000448| 0.0000458| 0.0000435| 0.0000510|
   |  110x  100 | 0.0000467| 0.0000476| 0.0000448| 0.0000514|
   |  120x  100 | 0.0000466| 0.0000481| 0.0000456| 0.0000527|
   |  130x  100 | 0.0000487| 0.0000501| 0.0000469| 0.0000529|
   |  140x  100 | 0.0000501| 0.0000515| 0.0000479| 0.0000538|
   |  150x  100 | 0.0000503| 0.0000517| 0.0000494| 0.0000550|
   |  200x  100 | 0.0000556| 0.0000595| 0.0000539| 0.0000592|
   |  300x  100 | 0.0000670| 0.0000708| 0.0000639| 0.0000678|
   |  400x  100 | 0.0000782| 0.0000825| 0.0000748| 0.0000727|
   |  500x  100 | 0.0000898| 0.0000946| 0.0000857| 0.0000824|
   | 1000x  100 | 0.0001492| 0.0001591| 0.0001409| 0.0001263|
   | 2000x  100 | 0.0002653| 0.0002819| 0.0002536| 0.0002101|
   | 3000x  100 | 0.0003822| 0.0004043| 0.0003598| 0.0002898|
   | 4000x  100 | 0.0004926| 0.0005266| 0.0004686| 0.0003655|
   | 5000x  100 | 0.0006051| 0.0006524| 0.0005765| 0.0004505|
   |16384x  100 | 0.0020228| 0.0021531| 0.0019262| 0.0014567|
   |    1x  256 | 0.0000374| 0.0000397| 0.0000358| 0.0000434|
   |    5x  256 | 0.0000336| 0.0000409| 0.0000335| 0.0000434|
   |   10x  256 | 0.0000399| 0.0000370| 0.0000404| 0.0000436|
   |   20x  256 | 0.0000423| 0.0000408| 0.0000400| 0.0000450|
   |   30x  256 | 0.0000383| 0.0000371| 0.0000373| 0.0000463|
   |   40x  256 | 0.0000411| 0.0000394| 0.0000384| 0.0000469|
   |   50x  256 | 0.0000418| 0.0000411| 0.0000386| 0.0000476|
   |   60x  256 | 0.0000431| 0.0000424| 0.0000407| 0.0000481|
   |   70x  256 | 0.0000455| 0.0000441| 0.0000419| 0.0000495|
   |   80x  256 | 0.0000465| 0.0000456| 0.0000433| 0.0000496|
   |   90x  256 | 0.0000493| 0.0000476| 0.0000445| 0.0000510|
   |  100x  256 | 0.0000502| 0.0000495| 0.0000457| 0.0000522|
   |  110x  256 | 0.0000524| 0.0000500| 0.0000467| 0.0000534|
   |  120x  256 | 0.0000535| 0.0000517| 0.0000475| 0.0000535|
   |  130x  256 | 0.0000554| 0.0000534| 0.0000493| 0.0000549|
   |  140x  256 | 0.0000573| 0.0000551| 0.0000512| 0.0000553|
   |  150x  256 | 0.0000597| 0.0000570| 0.0000521| 0.0000568|
   |  200x  256 | 0.0000679| 0.0000639| 0.0000581| 0.0000631|
   |  300x  256 | 0.0000850| 0.0000826| 0.0000713| 0.0000709|
   |  400x  256 | 0.0001040| 0.0000962| 0.0000854| 0.0000832|
   |  500x  256 | 0.0001231| 0.0001130| 0.0001000| 0.0000967|
   | 1000x  256 | 0.0002105| 0.0001881| 0.0001694| 0.0001590|
   | 2000x  256 | 0.0003913| 0.0003506| 0.0003000| 0.0002847|
   | 3000x  256 | 0.0005685| 0.0005093| 0.0004264| 0.0004101|
   | 4000x  256 | 0.0007300| 0.0006509| 0.0005445| 0.0005121|
   | 5000x  256 | 0.0009223| 0.0008244| 0.0006968| 0.0006512|
   |16384x  256 | 0.0036732| 0.0032559| 0.0029458| 0.0025328|
   |    1x  512 | 0.0000366| 0.0000392| 0.0000344| 0.0000438|
   |    5x  512 | 0.0000394| 0.0000354| 0.0000346| 0.0000441|
   |   10x  512 | 0.0000408| 0.0000401| 0.0000381| 0.0000448|
   |   20x  512 | 0.0000444| 0.0000435| 0.0000389| 0.0000453|
   |   30x  512 | 0.0000410| 0.0000403| 0.0000386| 0.0000474|
   |   40x  512 | 0.0000446| 0.0000429| 0.0000403| 0.0000481|
   |   50x  512 | 0.0000476| 0.0000463| 0.0000428| 0.0000496|
   |   60x  512 | 0.0000507| 0.0000478| 0.0000440| 0.0000510|
   |   70x  512 | 0.0000539| 0.0000502| 0.0000458| 0.0000518|
   |   80x  512 | 0.0000577| 0.0000538| 0.0000476| 0.0000537|
   |   90x  512 | 0.0000602| 0.0000558| 0.0000504| 0.0000572|
   |  100x  512 | 0.0000616| 0.0000582| 0.0000518| 0.0000565|
   |  110x  512 | 0.0000667| 0.0000607| 0.0000556| 0.0000600|
   |  120x  512 | 0.0000689| 0.0000642| 0.0000576| 0.0000612|
   |  130x  512 | 0.0000735| 0.0000654| 0.0000587| 0.0000616|
   |  140x  512 | 0.0000744| 0.0000695| 0.0000607| 0.0000661|
   |  150x  512 | 0.0000759| 0.0000695| 0.0000636| 0.0000653|
   |  200x  512 | 0.0000913| 0.0000831| 0.0000738| 0.0000793|
   |  300x  512 | 0.0001299| 0.0001136| 0.0001037| 0.0001073|
   |  400x  512 | 0.0001585| 0.0001409| 0.0001313| 0.0001338|
   |  500x  512 | 0.0001883| 0.0001598| 0.0001498| 0.0001580|
   | 1000x  512 | 0.0003369| 0.0002909| 0.0002665| 0.0002632|
   | 2000x  512 | 0.0006396| 0.0005457| 0.0004962| 0.0004790|
   | 3000x  512 | 0.0009528| 0.0008094| 0.0007397| 0.0006964|
   | 4000x  512 | 0.0013894| 0.0011007| 0.0010205| 0.0009181|
   | 5000x  512 | 0.0018264| 0.0015710| 0.0015030| 0.0012737|
   |16384x  512 | 0.0074474| 0.0067132| 0.0065738| 0.0060986|
   |    1x 1024 | 0.0000347| 0.0000357| 0.0000356| 0.0000445|
   |    5x 1024 | 0.0000391| 0.0000385| 0.0000362| 0.0000450|
   |   10x 1024 | 0.0000383| 0.0000410| 0.0000393| 0.0000459|
   |   20x 1024 | 0.0000439| 0.0000408| 0.0000384| 0.0000493|
   |   30x 1024 | 0.0000489| 0.0000458| 0.0000417| 0.0000504|
   |   40x 1024 | 0.0000557| 0.0000516| 0.0000448| 0.0000527|
   |   50x 1024 | 0.0000587| 0.0000545| 0.0000490| 0.0000546|
   |   60x 1024 | 0.0000650| 0.0000586| 0.0000516| 0.0000610|
   |   70x 1024 | 0.0000707| 0.0000617| 0.0000563| 0.0000616|
   |   80x 1024 | 0.0000768| 0.0000683| 0.0000618| 0.0000659|
   |   90x 1024 | 0.0000834| 0.0000735| 0.0000695| 0.0000716|
   |  100x 1024 | 0.0000886| 0.0000759| 0.0000708| 0.0000757|
   |  110x 1024 | 0.0000949| 0.0000833| 0.0000779| 0.0000826|
   |  120x 1024 | 0.0001031| 0.0000882| 0.0000841| 0.0000858|
   |  130x 1024 | 0.0001088| 0.0000956| 0.0000887| 0.0000903|
   |  140x 1024 | 0.0001156| 0.0001010| 0.0000923| 0.0000969|
   |  150x 1024 | 0.0001152| 0.0001057| 0.0000978| 0.0001028|
   |  200x 1024 | 0.0001450| 0.0001328| 0.0001257| 0.0001313|
   |  300x 1024 | 0.0002082| 0.0001793| 0.0001721| 0.0001762|
   |  400x 1024 | 0.0002650| 0.0002286| 0.0002192| 0.0002239|
   |  500x 1024 | 0.0003157| 0.0002699| 0.0002596| 0.0002592|
   | 1000x 1024 | 0.0005968| 0.0005075| 0.0004867| 0.0004764|
   | 2000x 1024 | 0.0012529| 0.0010099| 0.0009831| 0.0009186|
   | 3000x 1024 | 0.0021209| 0.0017632| 0.0018832| 0.0017245|
   | 4000x 1024 | 0.0029924| 0.0025091| 0.0027886| 0.0024449|
   | 5000x 1024 | 0.0040340| 0.0034140| 0.0037645| 0.0033816|
   |16384x 1024 | 0.0149200| 0.0130993| 0.0142042| 0.0132403|
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] szha commented on pull request #19562: [operator] Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
szha commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-903205489


   @mxnet-bot run ci [website]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] kpuatamazon commented on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
kpuatamazon commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-744344661


   I've been using a c5.12xlarge `Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz`.  Assume these are some sort of seconds?  
   
   We should at least do `-march=native` to see if it's just a matter of CPU support i.e. MXNet doesn't seem to enable AVX512 by default and one could add CPUID dispatch.  
   
   Might as well reshape to two dimensions with the axis preserved and everything else multiplied right?  
   
   Also, I feel like the optimal assembly implementation would benefit from a different ordering of the input tensor to allow for pure vertical adds whereas layer normalization is currently setup for horizontal adds.  


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19562: Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-758478322


   Jenkins CI successfully triggered : [windows-gpu, windows-cpu, unix-gpu]


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] bartekkuncer edited a comment on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
bartekkuncer edited a comment on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-742603472


   > Hi, can we compare https://github.com/apache/incubator-mxnet/pull/19601 ?
   
   @kpuatamazon Sorry for the late response, was waiting for the layer norm optimization in oneDNN. Below are the results I got using marian and my onednn implementation. It looks like oneDNN is faster in most cases. Can you tell me what CPU are you using?
   
   
   threads amount | 28 threads | 28 threads | 4 threads | 4 threads
   -- | -- | -- | -- | --
   execution | marian | mx w/ onednn | marian | mx w/ onednn
   1000x5 | 0,0213 | 0,032 | 0,0272 | 0,029
   1000x100 | 0,03 | 0,0859 | 0,0664 | 0,0421
   300x512 | 0,0255 | 0,1505 | 0,0667 | 0,073
   500x512 | 0,0305 | 0,2436 | 0,0859 | 0,0449
   1000x2048 | 0,1396 | 0,0616 | 0,4103 | 0,2594
   1000x3 | 0,0156 | 0,0239 | 0,018 | 0,0242
   45x512 | 0,0152 | 0,038 | 0,0252 | 0,0377
   1000x5x100 | 0,0652 | 0,0457 | 0,2194 | 0,117
   1000x8x100 | 0,0963 | 0,0538 | 0,3147 | 0,1594
   300x512x512 | 8,0384 | 7,8368 | 31,491 | 28,0296
   500x512x10 | 0,6231 | 0,3557 | 3,9009 | 1,0595
   1000x5x2048 | 2,1114 | 1,574 | 4,9501 | 3,8672
   1000x2048x3 | 2,6274 | 1,3618 | 12,9816 | 5,8291
   45x512x512 | 2,2648 | 1,7278 | 5,4215 | 4,6426
   1000x5x30x200 | 3,6479 | 3,5311 | 13,3026 | 11,3592
   100x100x10x300 | 3,6913 | 3,9369 | 13,4574 | 10,93
   300x512x45x20 | 24,2647 | 16,6294 | 146,4946 | 76,6008
   50x512x40x30 | 5,3653 | 4,0195 | 27,1334 | 15,8884
   100x2048x10x10 | 6,2846 | 3,4051 | 34,9686 | 13,3237
   1000x3x10x200 | 0,9976 | 0,5507 | 2,1544 | 1,0273
   45x52x300x45 | 4,7193 | 3,9909 | 22,6179 | 14,0623
   100x5x30x20x10 | 0,7225 | 0,3956 | 4,5843 | 1,3655
   100x4x100x10x30 | 2,7491 | 1,9519 | 10,9089 | 6,3355
   300x52x2x45x20 | 5,7044 | 3,9567 | 30,7142 | 15,74
   500x52x3x40x30 | 13,8315 | 10,8504 | 80,604 | 46,8657
   100x28x10x10x10 | 0,6829 | 0,4126 | 4,3192 | 1,2705
   100x3x10x18x200 | 2,1217 | 1,752 | 5,232 | 4,2747
   45x512x30x4x45 | 15,4316 | 12,9341 | 86,8337 | 54,7685
   
   I built mxnet using:
   cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DUSE_BLAS=mkl -DUSE_CUDA=0 -DUSE_LAPACK=0 -DUSE_GPERFTOOLS=0 -DUSE_OPENCV=0 ..
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] kpuatamazon commented on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
kpuatamazon commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-746568786


   I used whatever was in your pull request: c4b6bcebeed0c5a52383f0317ab0a3b2841b8b6b


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] kpuatamazon commented on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
kpuatamazon commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-735953653


   Hi, can we compare #19601 ?  


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] kpuatamazon edited a comment on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
kpuatamazon edited a comment on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-744344661


   I've been using a c5.12xlarge `Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz`.  Assume these are some sort of seconds?  
   
   We should at least do `-march=native` to see if it's just a matter of CPU support i.e. MXNet doesn't seem to enable AVX512 by default and one could add CPUID dispatch.  
   
   Might as well reshape to two dimensions with the axis preserved and everything else multiplied.  The problem is identical for e.g. 100x28x10x10x10 and 280000x10.  Also, those are some really small channels to layer normalize over.  
   
   Also, I feel like the optimal assembly implementation would benefit from a different ordering of the input tensor to allow for pure vertical adds whereas layer normalization is currently setup for horizontal adds.  I can certainly see how a JIT will do better at e.g. 1000x3 where multiple problems share the same vector.  


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] anko-intel commented on a change in pull request #19562: [operator] Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
anko-intel commented on a change in pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#discussion_r692819028



##########
File path: src/operator/nn/mkldnn/mkldnn_layer_norm.cc
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file mkldnn_layer_norm.cc
+ */
+
+#if MXNET_USE_ONEDNN == 1
+
+#include "./mkldnn_layer_norm-inl.h"
+
+namespace mxnet {
+namespace op {
+
+bool SupportMKLDNNLayerNorm(const LayerNormParam& param, const std::vector<NDArray>& inputs) {
+  const mxnet::TShape& shape = inputs[layernorm::kData].shape();
+
+  // Native implementation (which can be found in function LayerNormCPU) is faster than oneDNN's one
+  // for small tensors. Below is the heuristic based on measurements on clx machine deciding whether
+  // the shape is better for oneDNN or native implementation.
+  auto ShapeBetterForMKLDNN = [](const mxnet::TShape& shape) {
+    constexpr size_t shapeLimit = 1024;
+    return shape.Size() / shape[0] >= shapeLimit && shape[0] >= shapeLimit;
+  };
+
+  return (ShapeBetterForMKLDNN(shape) &&
+          (GetRealAxis(param.axis, shape.ndim()) == shape.ndim() - 1) && (shape.ndim() >= 2) &&
+          (shape.ndim() <= 5) &&
+          (inputs[layernorm::kData].dtype() == mshadow::kFloat32 ||
+           inputs[layernorm::kData].dtype() == mshadow::kBfloat16) &&
+          inputs[layernorm::kGamma].dtype() == mshadow::kFloat32 &&
+          inputs[layernorm::kBeta].dtype() == mshadow::kFloat32);
+}
+
+void MKLDNNLayerNormForward(const nnvm::NodeAttrs& attrs,
+                            const OpContext& ctx,
+                            const std::vector<NDArray>& inputs,
+                            const std::vector<OpReqType>& req,
+                            const std::vector<NDArray>& outputs) {
+  const LayerNormParam& param = nnvm::get<LayerNormParam>(attrs.parsed);
+  const auto& fwd             = MKLDNNLayerNormFwd::GetCached(param, ctx, inputs[layernorm::kData]);
+  fwd.Execute(param, ctx, inputs, req[layernorm::kOut], outputs);
+}
+
+MKLDNNLayerNormFwd& MKLDNNLayerNormFwd::GetCached(const LayerNormParam& param,
+                                                  const OpContext& ctx,
+                                                  const NDArray& data) {
+  using layernorm_fwd_map = std::unordered_map<LayerNormSignature, MKLDNNLayerNormFwd, OpHash>;
+#if DMLC_CXX11_THREAD_LOCAL
+  static thread_local layernorm_fwd_map layer_norm_fwds;
+#else
+  static MX_THREAD_LOCAL layernorm_fwd_map layer_norm_fwds;
+#endif
+
+  LayerNormSignature key(param);
+  key.AddSign(data);
+  key.AddSign(param.eps);
+
+  auto it = layer_norm_fwds.find(key);
+  if (it == layer_norm_fwds.end()) {
+    MKLDNNLayerNormFwd fwd(param, data);
+    it = AddToCache(&layer_norm_fwds, key, fwd);
+  }
+  return it->second;
+}
+
+MKLDNNLayerNormFwd::MKLDNNLayerNormFwd(const LayerNormParam& param, const NDArray& data) {
+  const mkldnn::memory::desc data_md = data.GetMKLDNNData()->get_desc();
+  fwd_pd                             = CreatePrimitiveDesc(param, data_md);
+  fwd = std::make_shared<mkldnn::layer_normalization_forward>(*fwd_pd);
+}
+
+std::shared_ptr<layernorm_fwd_pd_t> MKLDNNLayerNormFwd::CreatePrimitiveDesc(
+    const LayerNormParam& param,
+    const mkldnn::memory::desc& src_md) {
+  layernorm_fwd_t::desc fwd_desc(mkldnn::prop_kind::forward_training,
+                                 src_md,
+                                 param.eps,
+                                 dnnl::normalization_flags::use_scale_shift);
+  mkldnn::engine& engine = CpuEngine::Get()->get_engine();
+  return std::make_shared<layernorm_fwd_pd_t>(fwd_desc, engine);
+}
+
+inline mkldnn::memory::desc GetMeanVarDesc(const mkldnn::memory::data_type& dtype,
+                                           const mxnet::TShape& _shape) {
+  const auto ndim = _shape.ndim();
+
+  mkldnn::memory::dims shape(ndim, 1), strides(ndim, 1);
+  shape[0] = _shape[0];
+  for (int i = ndim - 1; i > 0; --i) {
+    shape[i]       = _shape[i];
+    strides[i - 1] = strides[i] * shape[i];
+  }
+
+  return mkldnn::memory::desc{shape, dtype, strides};
+}
+
+inline mkldnn::memory GetScaleShiftMem(const NDArray& gamma, const NDArray& beta) {
+  // OneDNN takes gamma and beta as one SCALE_SHIFT tensor when both scale and shift are used. In
+  // mxnet scale is called gamma and shift is called beta.
+  constexpr size_t gammaAndBeta = 2;
+  CHECK_EQ(gamma.shape()[0], beta.shape()[0]);
+  const mkldnn::memory::desc scale_shift_md(mkldnn::memory::dims{gammaAndBeta, gamma.shape()[0]},
+                                            get_mkldnn_type(gamma.dtype()),
+                                            mkldnn::memory::format_tag::nc);
+  auto scale_shift_mem = mkldnn::memory(scale_shift_md, CpuEngine::Get()->get_engine());
+  char* ptr            = reinterpret_cast<char*>(scale_shift_mem.get_data_handle());
+  const size_t bytes   = scale_shift_md.get_size() / gammaAndBeta;
+  memcpy(ptr, gamma.data().dptr_, bytes);
+  memcpy(ptr + bytes, beta.data().dptr_, bytes);
+  return scale_shift_mem;
+}
+
+void MKLDNNLayerNormFwd::Execute(const LayerNormParam& param,
+                                 const OpContext& ctx,
+                                 const std::vector<NDArray>& inputs,
+                                 const OpReqType& req,
+                                 const std::vector<NDArray>& outputs) const {
+  auto mean_var_md = GetMeanVarDesc(get_mkldnn_type(outputs[layernorm::kMean].dtype()),
+                                    outputs[layernorm::kMean].shape());
+  auto mean_mem    = mkldnn_output_t(
+      OutDataOp::Noop,
+      const_cast<NDArray&>(outputs[layernorm::kMean]).CreateMKLDNNData(mean_var_md));
+  auto variance_mem =
+      mkldnn_output_t(OutDataOp::Noop,
+                      const_cast<NDArray&>(outputs[layernorm::kStd]).CreateMKLDNNData(mean_var_md));
+
+  auto output_mem      = CreateMKLDNNMem(outputs[layernorm::kOut], fwd_pd->dst_desc(), req);
+  auto scale_shift_mem = GetScaleShiftMem(inputs[layernorm::kGamma], inputs[layernorm::kBeta]);
+
+  mkldnn_args_map_t args = {{MKLDNN_ARG_SRC, *inputs[layernorm::kData].GetMKLDNNData()},
+                            {MKLDNN_ARG_DST, *output_mem.second},
+                            {MKLDNN_ARG_MEAN, *mean_mem.second},
+                            {MKLDNN_ARG_VARIANCE, *variance_mem.second},
+                            {MKLDNN_ARG_SCALE_SHIFT, scale_shift_mem}};
+
+  MKLDNNStream::Get()->RegisterPrimArgs(*fwd, args);
+  CommitOutput(outputs[layernorm::kOut], output_mem);
+  CommitOutput(outputs[layernorm::kMean], mean_mem);
+  CommitOutput(outputs[layernorm::kStd], variance_mem);
+  MKLDNNStream::Get()->Submit();
+}
+
+MKLDNNLayerNormBwd::MKLDNNLayerNormBwd(const LayerNormParam& param,
+                                       const std::vector<NDArray>& inputs,
+                                       const mkldnn::memory::desc& data_md,
+                                       const mkldnn::memory::desc& diff_md)
+    : fwd_pd(MKLDNNLayerNormFwd::CreatePrimitiveDesc(param, data_md)),
+      bwd_pd(CreatePrimitiveDesc(param, data_md, diff_md, *fwd_pd)) {
+  bwd = std::make_shared<layernorm_bwd_t>(*bwd_pd);
+}
+
+std::shared_ptr<layernorm_bwd_pd_t> MKLDNNLayerNormBwd::CreatePrimitiveDesc(
+    const LayerNormParam& param,
+    const mkldnn::memory::desc& data_md,
+    const mkldnn::memory::desc& diff_md,
+    const layernorm_fwd_pd_t& layernorm_fwd_pd) {
+  layernorm_bwd_t::desc layernorm_bwd_desc(dnnl::prop_kind::backward,
+                                           diff_md,
+                                           data_md,
+                                           param.eps,
+                                           dnnl::normalization_flags::use_scale_shift);
+  mkldnn::engine& engine = CpuEngine::Get()->get_engine();
+  return std::make_shared<layernorm_bwd_pd_t>(layernorm_bwd_desc, engine, layernorm_fwd_pd);
+}
+
+void MKLDNNLayerNormBwd::Execute(const std::vector<NDArray>& inputs,
+                                 const std::vector<NDArray>& outputs,
+                                 const std::vector<OpReqType>& req) const {
+  auto scale_shift_mem =
+      GetScaleShiftMem(inputs[layernorm::kBwdGamma], inputs[layernorm::kBwdBeta]);
+  auto diff_weights_ndarray = NDArray(scale_shift_mem.get_desc());
+  const auto bytes          = inputs[layernorm::kBwdGamma].shape()[0] * sizeof(float);

Review comment:
       maybe?:
   ```suggestion
     const auto bytes          = inputs[layernorm::kBwdGamma].shape()[0] * mshadow_sizeof(inputs[layernorm::kBwdGamma].dtype());
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-745086286


   @kpuatamazon which version of oneDNN have you used for the benchmark? There is a change boosting the perf of layer normalization which is going to be included in oneDNN v2.1 release.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-780381128


   @mxnet-bot run ci [windows-cpu, centos-gpu]


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19562: Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-758585020


   Jenkins CI successfully triggered : [windows-gpu]


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19562: Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-780547129


   Jenkins CI successfully triggered : [windows-cpu, centos-gpu]


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] kpuatamazon edited a comment on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
kpuatamazon edited a comment on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-744344661


   I've been using a c5.12xlarge `Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz`.  Assume these are some sort of seconds?  
   
   We should at least do `-march=native` to see if it's just a matter of CPU support i.e. MXNet doesn't seem to enable AVX512 by default and one could add CPUID dispatch.  
   
   Might as well reshape to two dimensions with the axis preserved and everything else multiplied.  The problem is identical for e.g. 100x28x10x10x10 and 280000x10.  Also, those are some really small channels to layer normalize over.  
   
   Also, I feel like the optimal assembly implementation would benefit from a different ordering of the input tensor to allow for pure vertical adds whereas layer normalization is currently setup for horizontal adds.  


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] kpuatamazon edited a comment on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
kpuatamazon edited a comment on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-744344661


   I've been using a c5.12xlarge `Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz`.  Assume these are some sort of seconds?  
   
   We should at least do `-march=native` to see if it's just a matter of CPU support i.e. MXNet doesn't seem to enable AVX512 by default and one could add CPUID dispatch.  
   
   Might as well reshape to two dimensions with the axis preserved and everything else multiplied.  The problem is identical for e.g. 100x28x10x10x10 and 280000x10.  Also, those are some really small channels to layer normalize over.  
   
   Also, I feel like the optimal assembly implementation would benefit from a different ordering of the input tensor to allow for pure vertical adds whereas layer normalization is currently setup for horizontal adds.  I can certainly see how a JIT will do better at e.g. 1000x3 where a single problem is shared in vectorization.  


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19562: Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-729819592


   Hey @bartekkuncer , Thanks for submitting the PR 
   All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands: 
   - To trigger all jobs: @mxnet-bot run ci [all] 
   - To trigger specific jobs: @mxnet-bot run ci [job1, job2] 
   *** 
   **CI supported jobs**: [windows-gpu, sanity, centos-gpu, miscellaneous, website, unix-gpu, centos-cpu, clang, windows-cpu, unix-cpu, edge]
   *** 
   _Note_: 
    Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin. 
   All CI tests must pass before the PR can be merged. 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-758584969


   @mxnet-bot run ci [windows-gpu]


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19562: Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-781241541


   Jenkins CI successfully triggered : [windows-cpu]


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19562: [operator] Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-903205498


   Jenkins CI successfully triggered : [website]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-732973375


   > @bartekkuncer could you add more description in the PR to avoid confusion for the reviewers?
   
   @pengzhao-intel was planning to add them as soon as I fix tests :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] bartekkuncer commented on a change in pull request #19562: [operator] Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on a change in pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#discussion_r692919251



##########
File path: src/operator/nn/mkldnn/mkldnn_layer_norm.cc
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file mkldnn_layer_norm.cc
+ */
+
+#if MXNET_USE_ONEDNN == 1
+
+#include "./mkldnn_layer_norm-inl.h"
+
+namespace mxnet {
+namespace op {
+
+bool SupportMKLDNNLayerNorm(const LayerNormParam& param, const std::vector<NDArray>& inputs) {
+  const mxnet::TShape& shape = inputs[layernorm::kData].shape();
+
+  // Native implementation (which can be found in function LayerNormCPU) is faster than oneDNN's one
+  // for small tensors. Below is the heuristic based on measurements on clx machine deciding whether
+  // the shape is better for oneDNN or native implementation.
+  auto ShapeBetterForMKLDNN = [](const mxnet::TShape& shape) {
+    constexpr size_t shapeLimit = 1024;
+    return shape.Size() / shape[0] >= shapeLimit && shape[0] >= shapeLimit;
+  };
+
+  return (ShapeBetterForMKLDNN(shape) &&
+          (GetRealAxis(param.axis, shape.ndim()) == shape.ndim() - 1) && (shape.ndim() >= 2) &&
+          (shape.ndim() <= 5) &&
+          (inputs[layernorm::kData].dtype() == mshadow::kFloat32 ||
+           inputs[layernorm::kData].dtype() == mshadow::kBfloat16) &&
+          inputs[layernorm::kGamma].dtype() == mshadow::kFloat32 &&
+          inputs[layernorm::kBeta].dtype() == mshadow::kFloat32);
+}
+
+void MKLDNNLayerNormForward(const nnvm::NodeAttrs& attrs,
+                            const OpContext& ctx,
+                            const std::vector<NDArray>& inputs,
+                            const std::vector<OpReqType>& req,
+                            const std::vector<NDArray>& outputs) {
+  const LayerNormParam& param = nnvm::get<LayerNormParam>(attrs.parsed);
+  const auto& fwd             = MKLDNNLayerNormFwd::GetCached(param, ctx, inputs[layernorm::kData]);
+  fwd.Execute(param, ctx, inputs, req[layernorm::kOut], outputs);
+}
+
+MKLDNNLayerNormFwd& MKLDNNLayerNormFwd::GetCached(const LayerNormParam& param,
+                                                  const OpContext& ctx,
+                                                  const NDArray& data) {
+  using layernorm_fwd_map = std::unordered_map<LayerNormSignature, MKLDNNLayerNormFwd, OpHash>;
+#if DMLC_CXX11_THREAD_LOCAL
+  static thread_local layernorm_fwd_map layer_norm_fwds;
+#else
+  static MX_THREAD_LOCAL layernorm_fwd_map layer_norm_fwds;
+#endif
+
+  LayerNormSignature key(param);
+  key.AddSign(data);
+  key.AddSign(param.eps);
+
+  auto it = layer_norm_fwds.find(key);
+  if (it == layer_norm_fwds.end()) {
+    MKLDNNLayerNormFwd fwd(param, data);
+    it = AddToCache(&layer_norm_fwds, key, fwd);
+  }
+  return it->second;
+}
+
+MKLDNNLayerNormFwd::MKLDNNLayerNormFwd(const LayerNormParam& param, const NDArray& data) {
+  const mkldnn::memory::desc data_md = data.GetMKLDNNData()->get_desc();
+  fwd_pd                             = CreatePrimitiveDesc(param, data_md);
+  fwd = std::make_shared<mkldnn::layer_normalization_forward>(*fwd_pd);

Review comment:
       Thanks, I overlooked this one ;)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] fhieber commented on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
fhieber commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-744569353


   Speaking up as a 'customer' of LayerNorm here: Sockeye (and its Transformer models) cares about smaller matrix sizes for LayerNorm, i.e. typically in ranges around `(y, 512) for y in range(2,100)`. It would be great if we could get the performance benefits of the Marian implementation into MXNet.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: [operator] Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-902229587


   @mxnet-bot run ci [centos-gpu]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] anko-intel commented on a change in pull request #19562: [operator] Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
anko-intel commented on a change in pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#discussion_r692819028



##########
File path: src/operator/nn/mkldnn/mkldnn_layer_norm.cc
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file mkldnn_layer_norm.cc
+ */
+
+#if MXNET_USE_ONEDNN == 1
+
+#include "./mkldnn_layer_norm-inl.h"
+
+namespace mxnet {
+namespace op {
+
+bool SupportMKLDNNLayerNorm(const LayerNormParam& param, const std::vector<NDArray>& inputs) {
+  const mxnet::TShape& shape = inputs[layernorm::kData].shape();
+
+  // Native implementation (which can be found in function LayerNormCPU) is faster than oneDNN's one
+  // for small tensors. Below is the heuristic based on measurements on clx machine deciding whether
+  // the shape is better for oneDNN or native implementation.
+  auto ShapeBetterForMKLDNN = [](const mxnet::TShape& shape) {
+    constexpr size_t shapeLimit = 1024;
+    return shape.Size() / shape[0] >= shapeLimit && shape[0] >= shapeLimit;
+  };
+
+  return (ShapeBetterForMKLDNN(shape) &&
+          (GetRealAxis(param.axis, shape.ndim()) == shape.ndim() - 1) && (shape.ndim() >= 2) &&
+          (shape.ndim() <= 5) &&
+          (inputs[layernorm::kData].dtype() == mshadow::kFloat32 ||
+           inputs[layernorm::kData].dtype() == mshadow::kBfloat16) &&
+          inputs[layernorm::kGamma].dtype() == mshadow::kFloat32 &&
+          inputs[layernorm::kBeta].dtype() == mshadow::kFloat32);
+}
+
+void MKLDNNLayerNormForward(const nnvm::NodeAttrs& attrs,
+                            const OpContext& ctx,
+                            const std::vector<NDArray>& inputs,
+                            const std::vector<OpReqType>& req,
+                            const std::vector<NDArray>& outputs) {
+  const LayerNormParam& param = nnvm::get<LayerNormParam>(attrs.parsed);
+  const auto& fwd             = MKLDNNLayerNormFwd::GetCached(param, ctx, inputs[layernorm::kData]);
+  fwd.Execute(param, ctx, inputs, req[layernorm::kOut], outputs);
+}
+
+MKLDNNLayerNormFwd& MKLDNNLayerNormFwd::GetCached(const LayerNormParam& param,
+                                                  const OpContext& ctx,
+                                                  const NDArray& data) {
+  using layernorm_fwd_map = std::unordered_map<LayerNormSignature, MKLDNNLayerNormFwd, OpHash>;
+#if DMLC_CXX11_THREAD_LOCAL
+  static thread_local layernorm_fwd_map layer_norm_fwds;
+#else
+  static MX_THREAD_LOCAL layernorm_fwd_map layer_norm_fwds;
+#endif
+
+  LayerNormSignature key(param);
+  key.AddSign(data);
+  key.AddSign(param.eps);
+
+  auto it = layer_norm_fwds.find(key);
+  if (it == layer_norm_fwds.end()) {
+    MKLDNNLayerNormFwd fwd(param, data);
+    it = AddToCache(&layer_norm_fwds, key, fwd);
+  }
+  return it->second;
+}
+
+MKLDNNLayerNormFwd::MKLDNNLayerNormFwd(const LayerNormParam& param, const NDArray& data) {
+  const mkldnn::memory::desc data_md = data.GetMKLDNNData()->get_desc();
+  fwd_pd                             = CreatePrimitiveDesc(param, data_md);
+  fwd = std::make_shared<mkldnn::layer_normalization_forward>(*fwd_pd);
+}
+
+std::shared_ptr<layernorm_fwd_pd_t> MKLDNNLayerNormFwd::CreatePrimitiveDesc(
+    const LayerNormParam& param,
+    const mkldnn::memory::desc& src_md) {
+  layernorm_fwd_t::desc fwd_desc(mkldnn::prop_kind::forward_training,
+                                 src_md,
+                                 param.eps,
+                                 dnnl::normalization_flags::use_scale_shift);
+  mkldnn::engine& engine = CpuEngine::Get()->get_engine();
+  return std::make_shared<layernorm_fwd_pd_t>(fwd_desc, engine);
+}
+
+inline mkldnn::memory::desc GetMeanVarDesc(const mkldnn::memory::data_type& dtype,
+                                           const mxnet::TShape& _shape) {
+  const auto ndim = _shape.ndim();
+
+  mkldnn::memory::dims shape(ndim, 1), strides(ndim, 1);
+  shape[0] = _shape[0];
+  for (int i = ndim - 1; i > 0; --i) {
+    shape[i]       = _shape[i];
+    strides[i - 1] = strides[i] * shape[i];
+  }
+
+  return mkldnn::memory::desc{shape, dtype, strides};
+}
+
+inline mkldnn::memory GetScaleShiftMem(const NDArray& gamma, const NDArray& beta) {
+  // OneDNN takes gamma and beta as one SCALE_SHIFT tensor when both scale and shift are used. In
+  // mxnet scale is called gamma and shift is called beta.
+  constexpr size_t gammaAndBeta = 2;
+  CHECK_EQ(gamma.shape()[0], beta.shape()[0]);
+  const mkldnn::memory::desc scale_shift_md(mkldnn::memory::dims{gammaAndBeta, gamma.shape()[0]},
+                                            get_mkldnn_type(gamma.dtype()),
+                                            mkldnn::memory::format_tag::nc);
+  auto scale_shift_mem = mkldnn::memory(scale_shift_md, CpuEngine::Get()->get_engine());
+  char* ptr            = reinterpret_cast<char*>(scale_shift_mem.get_data_handle());
+  const size_t bytes   = scale_shift_md.get_size() / gammaAndBeta;
+  memcpy(ptr, gamma.data().dptr_, bytes);
+  memcpy(ptr + bytes, beta.data().dptr_, bytes);
+  return scale_shift_mem;
+}
+
+void MKLDNNLayerNormFwd::Execute(const LayerNormParam& param,
+                                 const OpContext& ctx,
+                                 const std::vector<NDArray>& inputs,
+                                 const OpReqType& req,
+                                 const std::vector<NDArray>& outputs) const {
+  auto mean_var_md = GetMeanVarDesc(get_mkldnn_type(outputs[layernorm::kMean].dtype()),
+                                    outputs[layernorm::kMean].shape());
+  auto mean_mem    = mkldnn_output_t(
+      OutDataOp::Noop,
+      const_cast<NDArray&>(outputs[layernorm::kMean]).CreateMKLDNNData(mean_var_md));
+  auto variance_mem =
+      mkldnn_output_t(OutDataOp::Noop,
+                      const_cast<NDArray&>(outputs[layernorm::kStd]).CreateMKLDNNData(mean_var_md));
+
+  auto output_mem      = CreateMKLDNNMem(outputs[layernorm::kOut], fwd_pd->dst_desc(), req);
+  auto scale_shift_mem = GetScaleShiftMem(inputs[layernorm::kGamma], inputs[layernorm::kBeta]);
+
+  mkldnn_args_map_t args = {{MKLDNN_ARG_SRC, *inputs[layernorm::kData].GetMKLDNNData()},
+                            {MKLDNN_ARG_DST, *output_mem.second},
+                            {MKLDNN_ARG_MEAN, *mean_mem.second},
+                            {MKLDNN_ARG_VARIANCE, *variance_mem.second},
+                            {MKLDNN_ARG_SCALE_SHIFT, scale_shift_mem}};
+
+  MKLDNNStream::Get()->RegisterPrimArgs(*fwd, args);
+  CommitOutput(outputs[layernorm::kOut], output_mem);
+  CommitOutput(outputs[layernorm::kMean], mean_mem);
+  CommitOutput(outputs[layernorm::kStd], variance_mem);
+  MKLDNNStream::Get()->Submit();
+}
+
+MKLDNNLayerNormBwd::MKLDNNLayerNormBwd(const LayerNormParam& param,
+                                       const std::vector<NDArray>& inputs,
+                                       const mkldnn::memory::desc& data_md,
+                                       const mkldnn::memory::desc& diff_md)
+    : fwd_pd(MKLDNNLayerNormFwd::CreatePrimitiveDesc(param, data_md)),
+      bwd_pd(CreatePrimitiveDesc(param, data_md, diff_md, *fwd_pd)) {
+  bwd = std::make_shared<layernorm_bwd_t>(*bwd_pd);
+}
+
+std::shared_ptr<layernorm_bwd_pd_t> MKLDNNLayerNormBwd::CreatePrimitiveDesc(
+    const LayerNormParam& param,
+    const mkldnn::memory::desc& data_md,
+    const mkldnn::memory::desc& diff_md,
+    const layernorm_fwd_pd_t& layernorm_fwd_pd) {
+  layernorm_bwd_t::desc layernorm_bwd_desc(dnnl::prop_kind::backward,
+                                           diff_md,
+                                           data_md,
+                                           param.eps,
+                                           dnnl::normalization_flags::use_scale_shift);
+  mkldnn::engine& engine = CpuEngine::Get()->get_engine();
+  return std::make_shared<layernorm_bwd_pd_t>(layernorm_bwd_desc, engine, layernorm_fwd_pd);
+}
+
+void MKLDNNLayerNormBwd::Execute(const std::vector<NDArray>& inputs,
+                                 const std::vector<NDArray>& outputs,
+                                 const std::vector<OpReqType>& req) const {
+  auto scale_shift_mem =
+      GetScaleShiftMem(inputs[layernorm::kBwdGamma], inputs[layernorm::kBwdBeta]);
+  auto diff_weights_ndarray = NDArray(scale_shift_mem.get_desc());
+  const auto bytes          = inputs[layernorm::kBwdGamma].shape()[0] * sizeof(float);

Review comment:
       maybe?:
   ```suggestion
     const auto bytes          = inputs[layernorm::kBwdGamma].shape()[0] * mshadow_sizeof(mshadow::kFloat32);
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] bartekkuncer commented on a change in pull request #19562: [operator] Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on a change in pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#discussion_r692919337



##########
File path: src/operator/nn/mkldnn/mkldnn_layer_norm.cc
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file mkldnn_layer_norm.cc
+ */
+
+#if MXNET_USE_ONEDNN == 1
+
+#include "./mkldnn_layer_norm-inl.h"
+
+namespace mxnet {
+namespace op {
+
+bool SupportMKLDNNLayerNorm(const LayerNormParam& param, const std::vector<NDArray>& inputs) {
+  const mxnet::TShape& shape = inputs[layernorm::kData].shape();
+
+  // Native implementation (which can be found in function LayerNormCPU) is faster than oneDNN's one
+  // for small tensors. Below is the heuristic based on measurements on clx machine deciding whether
+  // the shape is better for oneDNN or native implementation.
+  auto ShapeBetterForMKLDNN = [](const mxnet::TShape& shape) {
+    constexpr size_t shapeLimit = 1024;
+    return shape.Size() / shape[0] >= shapeLimit && shape[0] >= shapeLimit;
+  };
+
+  return (ShapeBetterForMKLDNN(shape) &&
+          (GetRealAxis(param.axis, shape.ndim()) == shape.ndim() - 1) && (shape.ndim() >= 2) &&
+          (shape.ndim() <= 5) &&
+          (inputs[layernorm::kData].dtype() == mshadow::kFloat32 ||
+           inputs[layernorm::kData].dtype() == mshadow::kBfloat16) &&
+          inputs[layernorm::kGamma].dtype() == mshadow::kFloat32 &&
+          inputs[layernorm::kBeta].dtype() == mshadow::kFloat32);
+}
+
+void MKLDNNLayerNormForward(const nnvm::NodeAttrs& attrs,
+                            const OpContext& ctx,
+                            const std::vector<NDArray>& inputs,
+                            const std::vector<OpReqType>& req,
+                            const std::vector<NDArray>& outputs) {
+  const LayerNormParam& param = nnvm::get<LayerNormParam>(attrs.parsed);
+  const auto& fwd             = MKLDNNLayerNormFwd::GetCached(param, ctx, inputs[layernorm::kData]);
+  fwd.Execute(param, ctx, inputs, req[layernorm::kOut], outputs);
+}
+
+MKLDNNLayerNormFwd& MKLDNNLayerNormFwd::GetCached(const LayerNormParam& param,
+                                                  const OpContext& ctx,
+                                                  const NDArray& data) {
+  using layernorm_fwd_map = std::unordered_map<LayerNormSignature, MKLDNNLayerNormFwd, OpHash>;
+#if DMLC_CXX11_THREAD_LOCAL
+  static thread_local layernorm_fwd_map layer_norm_fwds;
+#else
+  static MX_THREAD_LOCAL layernorm_fwd_map layer_norm_fwds;
+#endif
+
+  LayerNormSignature key(param);
+  key.AddSign(data);
+  key.AddSign(param.eps);

Review comment:
       removed




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] bartekkuncer commented on a change in pull request #19562: [operator] Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on a change in pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#discussion_r692882144



##########
File path: src/operator/nn/mkldnn/mkldnn_layer_norm.cc
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file mkldnn_layer_norm.cc
+ */
+
+#if MXNET_USE_ONEDNN == 1
+
+#include "./mkldnn_layer_norm-inl.h"
+
+namespace mxnet {
+namespace op {
+
+bool SupportMKLDNNLayerNorm(const LayerNormParam& param, const std::vector<NDArray>& inputs) {
+  const mxnet::TShape& shape = inputs[layernorm::kData].shape();
+
+  // Native implementation (which can be found in function LayerNormCPU) is faster than oneDNN's one
+  // for small tensors. Below is the heuristic based on measurements on clx machine deciding whether
+  // the shape is better for oneDNN or native implementation.
+  auto ShapeBetterForMKLDNN = [](const mxnet::TShape& shape) {
+    constexpr size_t shapeLimit = 1024;
+    return shape.Size() / shape[0] >= shapeLimit && shape[0] >= shapeLimit;
+  };
+
+  return (ShapeBetterForMKLDNN(shape) &&
+          (GetRealAxis(param.axis, shape.ndim()) == shape.ndim() - 1) && (shape.ndim() >= 2) &&
+          (shape.ndim() <= 5) &&
+          (inputs[layernorm::kData].dtype() == mshadow::kFloat32 ||
+           inputs[layernorm::kData].dtype() == mshadow::kBfloat16) &&
+          inputs[layernorm::kGamma].dtype() == mshadow::kFloat32 &&
+          inputs[layernorm::kBeta].dtype() == mshadow::kFloat32);
+}
+
+void MKLDNNLayerNormForward(const nnvm::NodeAttrs& attrs,
+                            const OpContext& ctx,
+                            const std::vector<NDArray>& inputs,
+                            const std::vector<OpReqType>& req,
+                            const std::vector<NDArray>& outputs) {
+  const LayerNormParam& param = nnvm::get<LayerNormParam>(attrs.parsed);
+  const auto& fwd             = MKLDNNLayerNormFwd::GetCached(param, ctx, inputs[layernorm::kData]);
+  fwd.Execute(param, ctx, inputs, req[layernorm::kOut], outputs);
+}
+
+MKLDNNLayerNormFwd& MKLDNNLayerNormFwd::GetCached(const LayerNormParam& param,
+                                                  const OpContext& ctx,
+                                                  const NDArray& data) {
+  using layernorm_fwd_map = std::unordered_map<LayerNormSignature, MKLDNNLayerNormFwd, OpHash>;
+#if DMLC_CXX11_THREAD_LOCAL
+  static thread_local layernorm_fwd_map layer_norm_fwds;
+#else
+  static MX_THREAD_LOCAL layernorm_fwd_map layer_norm_fwds;
+#endif
+
+  LayerNormSignature key(param);
+  key.AddSign(data);
+  key.AddSign(param.eps);
+
+  auto it = layer_norm_fwds.find(key);
+  if (it == layer_norm_fwds.end()) {
+    MKLDNNLayerNormFwd fwd(param, data);
+    it = AddToCache(&layer_norm_fwds, key, fwd);
+  }
+  return it->second;
+}
+
+MKLDNNLayerNormFwd::MKLDNNLayerNormFwd(const LayerNormParam& param, const NDArray& data) {
+  const mkldnn::memory::desc data_md = data.GetMKLDNNData()->get_desc();
+  fwd_pd                             = CreatePrimitiveDesc(param, data_md);
+  fwd = std::make_shared<mkldnn::layer_normalization_forward>(*fwd_pd);
+}
+
+std::shared_ptr<layernorm_fwd_pd_t> MKLDNNLayerNormFwd::CreatePrimitiveDesc(
+    const LayerNormParam& param,
+    const mkldnn::memory::desc& src_md) {
+  layernorm_fwd_t::desc fwd_desc(mkldnn::prop_kind::forward_training,
+                                 src_md,
+                                 param.eps,
+                                 dnnl::normalization_flags::use_scale_shift);
+  mkldnn::engine& engine = CpuEngine::Get()->get_engine();
+  return std::make_shared<layernorm_fwd_pd_t>(fwd_desc, engine);
+}
+
+inline mkldnn::memory::desc GetMeanVarDesc(const mkldnn::memory::data_type& dtype,
+                                           const mxnet::TShape& _shape) {
+  const auto ndim = _shape.ndim();
+
+  mkldnn::memory::dims shape(ndim, 1), strides(ndim, 1);
+  shape[0] = _shape[0];
+  for (int i = ndim - 1; i > 0; --i) {
+    shape[i]       = _shape[i];
+    strides[i - 1] = strides[i] * shape[i];
+  }
+
+  return mkldnn::memory::desc{shape, dtype, strides};
+}
+
+inline mkldnn::memory GetScaleShiftMem(const NDArray& gamma, const NDArray& beta) {
+  // OneDNN takes gamma and beta as one SCALE_SHIFT tensor when both scale and shift are used. In
+  // mxnet scale is called gamma and shift is called beta.
+  constexpr size_t gammaAndBeta = 2;
+  CHECK_EQ(gamma.shape()[0], beta.shape()[0]);
+  const mkldnn::memory::desc scale_shift_md(mkldnn::memory::dims{gammaAndBeta, gamma.shape()[0]},
+                                            get_mkldnn_type(gamma.dtype()),
+                                            mkldnn::memory::format_tag::nc);
+  auto scale_shift_mem = mkldnn::memory(scale_shift_md, CpuEngine::Get()->get_engine());
+  char* ptr            = reinterpret_cast<char*>(scale_shift_mem.get_data_handle());
+  const size_t bytes   = scale_shift_md.get_size() / gammaAndBeta;
+  memcpy(ptr, gamma.data().dptr_, bytes);
+  memcpy(ptr + bytes, beta.data().dptr_, bytes);
+  return scale_shift_mem;
+}
+
+void MKLDNNLayerNormFwd::Execute(const LayerNormParam& param,
+                                 const OpContext& ctx,
+                                 const std::vector<NDArray>& inputs,
+                                 const OpReqType& req,
+                                 const std::vector<NDArray>& outputs) const {
+  auto mean_var_md = GetMeanVarDesc(get_mkldnn_type(outputs[layernorm::kMean].dtype()),
+                                    outputs[layernorm::kMean].shape());
+  auto mean_mem    = mkldnn_output_t(
+      OutDataOp::Noop,
+      const_cast<NDArray&>(outputs[layernorm::kMean]).CreateMKLDNNData(mean_var_md));
+  auto variance_mem =
+      mkldnn_output_t(OutDataOp::Noop,
+                      const_cast<NDArray&>(outputs[layernorm::kStd]).CreateMKLDNNData(mean_var_md));
+
+  auto output_mem      = CreateMKLDNNMem(outputs[layernorm::kOut], fwd_pd->dst_desc(), req);
+  auto scale_shift_mem = GetScaleShiftMem(inputs[layernorm::kGamma], inputs[layernorm::kBeta]);
+
+  mkldnn_args_map_t args = {{MKLDNN_ARG_SRC, *inputs[layernorm::kData].GetMKLDNNData()},
+                            {MKLDNN_ARG_DST, *output_mem.second},
+                            {MKLDNN_ARG_MEAN, *mean_mem.second},
+                            {MKLDNN_ARG_VARIANCE, *variance_mem.second},
+                            {MKLDNN_ARG_SCALE_SHIFT, scale_shift_mem}};
+
+  MKLDNNStream::Get()->RegisterPrimArgs(*fwd, args);
+  CommitOutput(outputs[layernorm::kOut], output_mem);
+  CommitOutput(outputs[layernorm::kMean], mean_mem);
+  CommitOutput(outputs[layernorm::kStd], variance_mem);
+  MKLDNNStream::Get()->Submit();
+}
+
+MKLDNNLayerNormBwd::MKLDNNLayerNormBwd(const LayerNormParam& param,
+                                       const std::vector<NDArray>& inputs,
+                                       const mkldnn::memory::desc& data_md,
+                                       const mkldnn::memory::desc& diff_md)
+    : fwd_pd(MKLDNNLayerNormFwd::CreatePrimitiveDesc(param, data_md)),
+      bwd_pd(CreatePrimitiveDesc(param, data_md, diff_md, *fwd_pd)) {
+  bwd = std::make_shared<layernorm_bwd_t>(*bwd_pd);
+}
+
+std::shared_ptr<layernorm_bwd_pd_t> MKLDNNLayerNormBwd::CreatePrimitiveDesc(
+    const LayerNormParam& param,
+    const mkldnn::memory::desc& data_md,
+    const mkldnn::memory::desc& diff_md,
+    const layernorm_fwd_pd_t& layernorm_fwd_pd) {
+  layernorm_bwd_t::desc layernorm_bwd_desc(dnnl::prop_kind::backward,
+                                           diff_md,
+                                           data_md,
+                                           param.eps,
+                                           dnnl::normalization_flags::use_scale_shift);
+  mkldnn::engine& engine = CpuEngine::Get()->get_engine();
+  return std::make_shared<layernorm_bwd_pd_t>(layernorm_bwd_desc, engine, layernorm_fwd_pd);
+}
+
+void MKLDNNLayerNormBwd::Execute(const std::vector<NDArray>& inputs,
+                                 const std::vector<NDArray>& outputs,
+                                 const std::vector<OpReqType>& req) const {
+  auto scale_shift_mem =
+      GetScaleShiftMem(inputs[layernorm::kBwdGamma], inputs[layernorm::kBwdBeta]);
+  auto diff_weights_ndarray = NDArray(scale_shift_mem.get_desc());
+  const auto bytes          = inputs[layernorm::kBwdGamma].shape()[0] * sizeof(float);

Review comment:
       Done.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-742603472


   > Hi, can we compare https://github.com/apache/incubator-mxnet/pull/19601 ?
   
   @kpuatamazon Sorry for the late response, was waiting for the layer norm optimization in oneDNN. Below are the results I got using marian and my onednn implementation. It looks like oneDNN is faster in most cases. Can you tell me what CPU are you using?
   
   
   threads amount | 28 threads | 28 threads | 4 threads | 4 threads
   -- | -- | -- | -- | --
   execution | marian | mx w/ onednn | marian | mx w/ onednn
   1000x5 | 0,0213 | 0,032 | 0,0272 | 0,029
   1000x100 | 0,03 | 0,0859 | 0,0664 | 0,0421
   300x512 | 0,0255 | 0,1505 | 0,0667 | 0,073
   500x512 | 0,0305 | 0,2436 | 0,0859 | 0,0449
   1000x2048 | 0,1396 | 0,0616 | 0,4103 | 0,2594
   1000x3 | 0,0156 | 0,0239 | 0,018 | 0,0242
   45x512 | 0,0152 | 0,038 | 0,0252 | 0,0377
   1000x5x100 | 0,0652 | 0,0457 | 0,2194 | 0,117
   1000x8x100 | 0,0963 | 0,0538 | 0,3147 | 0,1594
   300x512x512 | 8,0384 | 7,8368 | 31,491 | 28,0296
   500x512x10 | 0,6231 | 0,3557 | 3,9009 | 1,0595
   1000x5x2048 | 2,1114 | 1,574 | 4,9501 | 3,8672
   1000x2048x3 | 2,6274 | 1,3618 | 12,9816 | 5,8291
   45x512x512 | 2,2648 | 1,7278 | 5,4215 | 4,6426
   1000x5x30x200 | 3,6479 | 3,5311 | 13,3026 | 11,3592
   100x100x10x300 | 3,6913 | 3,9369 | 13,4574 | 10,93
   300x512x45x20 | 24,2647 | 16,6294 | 146,4946 | 76,6008
   50x512x40x30 | 5,3653 | 4,0195 | 27,1334 | 15,8884
   100x2048x10x10 | 6,2846 | 3,4051 | 34,9686 | 13,3237
   1000x3x10x200 | 0,9976 | 0,5507 | 2,1544 | 1,0273
   45x52x300x45 | 4,7193 | 3,9909 | 22,6179 | 14,0623
   100x5x30x20x10 | 0,7225 | 0,3956 | 4,5843 | 1,3655
   100x4x100x10x30 | 2,7491 | 1,9519 | 10,9089 | 6,3355
   300x52x2x45x20 | 5,7044 | 3,9567 | 30,7142 | 15,74
   500x52x3x40x30 | 13,8315 | 10,8504 | 80,604 | 46,8657
   100x28x10x10x10 | 0,6829 | 0,4126 | 4,3192 | 1,2705
   100x3x10x18x200 | 2,1217 | 1,752 | 5,232 | 4,2747
   45x512x30x4x45 | 15,4316 | 12,9341 | 86,8337 | 54,7685
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-780547068


   @mxnet-bot run ci [windows-cpu, centos-gpu]


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] bgawrych commented on a change in pull request #19562: [operator] Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
bgawrych commented on a change in pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#discussion_r692904008



##########
File path: src/operator/nn/mkldnn/mkldnn_layer_norm.cc
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file mkldnn_layer_norm.cc
+ */
+
+#if MXNET_USE_ONEDNN == 1
+
+#include "./mkldnn_layer_norm-inl.h"
+
+namespace mxnet {
+namespace op {
+
+bool SupportMKLDNNLayerNorm(const LayerNormParam& param, const std::vector<NDArray>& inputs) {
+  const mxnet::TShape& shape = inputs[layernorm::kData].shape();
+
+  // Native implementation (which can be found in function LayerNormCPU) is faster than oneDNN's one
+  // for small tensors. Below is the heuristic based on measurements on clx machine deciding whether
+  // the shape is better for oneDNN or native implementation.
+  auto ShapeBetterForMKLDNN = [](const mxnet::TShape& shape) {
+    constexpr size_t shapeLimit = 1024;
+    return shape.Size() / shape[0] >= shapeLimit && shape[0] >= shapeLimit;
+  };
+
+  return (ShapeBetterForMKLDNN(shape) &&
+          (GetRealAxis(param.axis, shape.ndim()) == shape.ndim() - 1) && (shape.ndim() >= 2) &&
+          (shape.ndim() <= 5) &&
+          (inputs[layernorm::kData].dtype() == mshadow::kFloat32 ||
+           inputs[layernorm::kData].dtype() == mshadow::kBfloat16) &&
+          inputs[layernorm::kGamma].dtype() == mshadow::kFloat32 &&
+          inputs[layernorm::kBeta].dtype() == mshadow::kFloat32);
+}
+
+void MKLDNNLayerNormForward(const nnvm::NodeAttrs& attrs,
+                            const OpContext& ctx,
+                            const std::vector<NDArray>& inputs,
+                            const std::vector<OpReqType>& req,
+                            const std::vector<NDArray>& outputs) {
+  const LayerNormParam& param = nnvm::get<LayerNormParam>(attrs.parsed);
+  const auto& fwd             = MKLDNNLayerNormFwd::GetCached(param, ctx, inputs[layernorm::kData]);
+  fwd.Execute(param, ctx, inputs, req[layernorm::kOut], outputs);
+}
+
+MKLDNNLayerNormFwd& MKLDNNLayerNormFwd::GetCached(const LayerNormParam& param,
+                                                  const OpContext& ctx,
+                                                  const NDArray& data) {
+  using layernorm_fwd_map = std::unordered_map<LayerNormSignature, MKLDNNLayerNormFwd, OpHash>;
+#if DMLC_CXX11_THREAD_LOCAL
+  static thread_local layernorm_fwd_map layer_norm_fwds;
+#else
+  static MX_THREAD_LOCAL layernorm_fwd_map layer_norm_fwds;
+#endif
+
+  LayerNormSignature key(param);
+  key.AddSign(data);
+  key.AddSign(param.eps);

Review comment:
       Isn't eps already added by passing param to key?

##########
File path: src/operator/nn/mkldnn/mkldnn_layer_norm.cc
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file mkldnn_layer_norm.cc
+ */
+
+#if MXNET_USE_ONEDNN == 1
+
+#include "./mkldnn_layer_norm-inl.h"
+
+namespace mxnet {
+namespace op {
+
+bool SupportMKLDNNLayerNorm(const LayerNormParam& param, const std::vector<NDArray>& inputs) {
+  const mxnet::TShape& shape = inputs[layernorm::kData].shape();
+
+  // Native implementation (which can be found in function LayerNormCPU) is faster than oneDNN's one
+  // for small tensors. Below is the heuristic based on measurements on clx machine deciding whether
+  // the shape is better for oneDNN or native implementation.
+  auto ShapeBetterForMKLDNN = [](const mxnet::TShape& shape) {
+    constexpr size_t shapeLimit = 1024;
+    return shape.Size() / shape[0] >= shapeLimit && shape[0] >= shapeLimit;
+  };
+
+  return (ShapeBetterForMKLDNN(shape) &&
+          (GetRealAxis(param.axis, shape.ndim()) == shape.ndim() - 1) && (shape.ndim() >= 2) &&
+          (shape.ndim() <= 5) &&
+          (inputs[layernorm::kData].dtype() == mshadow::kFloat32 ||
+           inputs[layernorm::kData].dtype() == mshadow::kBfloat16) &&
+          inputs[layernorm::kGamma].dtype() == mshadow::kFloat32 &&
+          inputs[layernorm::kBeta].dtype() == mshadow::kFloat32);
+}
+
+void MKLDNNLayerNormForward(const nnvm::NodeAttrs& attrs,
+                            const OpContext& ctx,
+                            const std::vector<NDArray>& inputs,
+                            const std::vector<OpReqType>& req,
+                            const std::vector<NDArray>& outputs) {
+  const LayerNormParam& param = nnvm::get<LayerNormParam>(attrs.parsed);
+  const auto& fwd             = MKLDNNLayerNormFwd::GetCached(param, ctx, inputs[layernorm::kData]);
+  fwd.Execute(param, ctx, inputs, req[layernorm::kOut], outputs);
+}
+
+MKLDNNLayerNormFwd& MKLDNNLayerNormFwd::GetCached(const LayerNormParam& param,
+                                                  const OpContext& ctx,
+                                                  const NDArray& data) {
+  using layernorm_fwd_map = std::unordered_map<LayerNormSignature, MKLDNNLayerNormFwd, OpHash>;
+#if DMLC_CXX11_THREAD_LOCAL
+  static thread_local layernorm_fwd_map layer_norm_fwds;
+#else
+  static MX_THREAD_LOCAL layernorm_fwd_map layer_norm_fwds;
+#endif
+
+  LayerNormSignature key(param);
+  key.AddSign(data);
+  key.AddSign(param.eps);
+
+  auto it = layer_norm_fwds.find(key);
+  if (it == layer_norm_fwds.end()) {
+    MKLDNNLayerNormFwd fwd(param, data);
+    it = AddToCache(&layer_norm_fwds, key, fwd);
+  }
+  return it->second;
+}
+
+MKLDNNLayerNormFwd::MKLDNNLayerNormFwd(const LayerNormParam& param, const NDArray& data) {
+  const mkldnn::memory::desc data_md = data.GetMKLDNNData()->get_desc();
+  fwd_pd                             = CreatePrimitiveDesc(param, data_md);
+  fwd = std::make_shared<mkldnn::layer_normalization_forward>(*fwd_pd);

Review comment:
       Sometimes you're using defined types by using, sometimes directly from mkldnn namespace (layernorm_fwd_t) - be consistent ;)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] akarbown merged pull request #19562: [operator] Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
akarbown merged pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-781241478


   @mxnet-bot run ci [windows-cpu]
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19562: Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-780381171


   Jenkins CI successfully triggered : [centos-gpu, windows-cpu]


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-mxnet] kpuatamazon edited a comment on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation

Posted by GitBox <gi...@apache.org>.
kpuatamazon edited a comment on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-744344661


   I've been using a c5.12xlarge `Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz`.  Assume these are some sort of seconds?  
   
   We should at least do `-march=native` to see if it's just a matter of CPU support i.e. MXNet doesn't seem to enable AVX512 by default and one could add CPUID dispatch.  
   
   Might as well reshape to two dimensions with the axis preserved and everything else multiplied.  The problem is identical for e.g. 100x28x10x10x10 and 280000x10.  Also, those are some really small channels to layer normalize over.  
   
   Also, I feel like the optimal assembly implementation would benefit from a different ordering of the input tensor to allow for pure vertical adds whereas layer normalization is currently setup for horizontal adds.  I can certainly see how a JIT will do better at e.g. 1000x3 where multiple problems share the same vector.  But oddly that's where marian is doing better.  


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org