You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/11/18 17:08:45 UTC
[GitHub] [incubator-mxnet] bartekkuncer opened a new pull request #19562: Integrate oneDNN layer normalization implementation
bartekkuncer opened a new pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562
Opperf:
| forward | | | backward | |
-- | -- | -- | -- | -- | -- | --
size | w/o | w/ | time diff in % | w/o | w/ | time diff in %
1000x5 | 0,0203 | 0,0338 | 66,50246 | 0,1402 | 0,137 | -2,28245364
1000x100 | 0,0394 | 0,0375 | -4,82234 | 0,6158 | 0,1037 | -83,1601169
300x512 | 0,0428 | 0,0398 | -7,00935 | 0,6597 | 0,071 | -89,2375322
500x512 | 0,0737 | 0,042 | -43,0122 | 1,0916 | 0,0765 | -92,9919384
1000x2048 | 0,2187 | 0,0696 | -68,1756 | 8,6593 | 0,1659 | -98,0841408
1000x3 | 0,02 | 0,0312 | 56 | 0,1145 | 0,1221 | 6,63755459
45x512 | 0,0267 | 0,033 | 23,59551 | 0,2213 | 0,0632 | -71,4414822
1000x5x100 | 0,0968 | 0,0504 | -47,9339 | 2,2083 | 0,1869 | -91,536476
1000x8x100 | 0,1273 | 0,0693 | -45,5617 | 3,4197 | 0,314 | -90,817908
300x512x512 | 6,947 | 5,559 | -19,9798 | 583,0516 | 12,5954 | -97,8397452
500x512x10 | 0,2469 | 0,471 | 90,76549 | 7,883 | 9,5545 | 21,2038564
1000x5x2048 | 1,1808 | 0,4726 | -59,9763 | 69,3604 | 1,5419 | -97,7769736
1000x2048x3 | 0,8903 | 2,3565 | 164,6861 | 23,2045 | 179,1877 | 672,210994
45x512x512 | 1,267 | 0,4315 | -65,9432 | 79,5693 | 1,4785 | -98,1418713
1000x5x30x200 | 2,9532 | 1,8339 | -37,9013 | 190,4334 | 6,9446 | -96,3532658
100x100x10x300 | 2,9831 | 1,83 | -38,6544 | 189,7882 | 6,9442 | -96,3410792
300x512x45x20 | 11,6564 | 13,2278 | 13,48101 | 441,5131 | 214,6873 | -51,3746478
50x512x40x30 | 3,0377 | 2,4979 | -17,77 | 193,4521 | 36,4343 | -81,1662422
100x2048x10x10 | 2,2314 | 3,1573 | 41,49413 | 64,9861 | 85,1543 | 31,0346366
1000x3x10x200 | 0,793 | 0,1644 | -79,2686 | 28,1187 | 1,0968 | -96,0993929
45x52x300x45 | 3,1359 | 2,2834 | -27,1852 | 221,7148 | 21,7037 | -90,2109828
100x5x30x20x10 | 0,286 | 0,4941 | 72,76224 | 9,1756 | 13,4736 | 46,8416234
100x4x100x10x30 | 1,4714 | 1,0342 | -29,7132 | 67,6208 | 10,4669 | -84,5211828
300x52x2x45x20 | 2,7541 | 2,7491 | -0,18155 | 93,0693 | 53,0185 | -43,0333096
500x52x3x40x30 | 8,1896 | 7,7518 | -5,3458 | 602,2581 | 101,6845 | -83,1161258
100x28x10x10x10 | 0,2706 | 0,4627 | 70,99039 | 8,58 | 11,4651 | 33,6258741
100x3x10x18x200 | 1,2413 | 0,4217 | -66,0276 | 53,8475 | 2,4962 | -95,3643159
45x512x30x4x45 | 10,9092 | 9,5348 | -12,5985 | 947,7691 | 79,3489 | -91,6278237
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] pengzhao-intel commented on pull request #19562: Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
pengzhao-intel commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-732940638
@bartekkuncer could you add more description in the PR to avoid confusion for the reviewers?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19562: [operator] Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-902229641
Jenkins CI successfully triggered : [centos-gpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-758478282
@mxnet-bot run ci [unix-gpu, windows-gpu, windows-cpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] kpuatamazon commented on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
kpuatamazon commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-744892721
I made the sizing more systematic.
AVX512 means `__attribute__((target("avx512f,avx512bw,avx512cd,avx512dq,avx512vnni")))`
Inverse means the 1.f/std is computed in advance rather than dividing by std in the loop.
Overall, the Marian implementation seems to win on smaller problem sizes, including the x512 sizes from @fhieber but lose on larger problem sizes.
Of course there are edge cases when the width is not a multiple of 16 and gcc is testing for those edge cases every time, so I see how that could be optimized.
| Shape | Marian | + AVX512 | + AVX512 + inverse | oneDNN |
|--|--|--|--|--|
| 1x 3 | 0.0000363| 0.0000370| 0.0000364| 0.0000453|
| 5x 3 | 0.0000337| 0.0000355| 0.0000348| 0.0000426|
| 10x 3 | 0.0000346| 0.0000352| 0.0000344| 0.0000434|
| 20x 3 | 0.0000337| 0.0000342| 0.0000349| 0.0000438|
| 30x 3 | 0.0000354| 0.0000376| 0.0000371| 0.0000434|
| 40x 3 | 0.0000373| 0.0000396| 0.0000382| 0.0000431|
| 50x 3 | 0.0000381| 0.0000405| 0.0000393| 0.0000436|
| 60x 3 | 0.0000390| 0.0000408| 0.0000403| 0.0000447|
| 70x 3 | 0.0000391| 0.0000376| 0.0000411| 0.0000440|
| 80x 3 | 0.0000403| 0.0000400| 0.0000414| 0.0000438|
| 90x 3 | 0.0000399| 0.0000399| 0.0000392| 0.0000446|
| 100x 3 | 0.0000378| 0.0000409| 0.0000399| 0.0000451|
| 110x 3 | 0.0000385| 0.0000414| 0.0000398| 0.0000446|
| 120x 3 | 0.0000390| 0.0000413| 0.0000417| 0.0000454|
| 130x 3 | 0.0000389| 0.0000429| 0.0000420| 0.0000457|
| 140x 3 | 0.0000402| 0.0000437| 0.0000436| 0.0000452|
| 150x 3 | 0.0000403| 0.0000442| 0.0000432| 0.0000461|
| 200x 3 | 0.0000432| 0.0000480| 0.0000466| 0.0000476|
| 300x 3 | 0.0000476| 0.0000553| 0.0000533| 0.0000506|
| 400x 3 | 0.0000532| 0.0000617| 0.0000604| 0.0000533|
| 500x 3 | 0.0000575| 0.0000694| 0.0000670| 0.0000570|
| 1000x 3 | 0.0000826| 0.0001037| 0.0001029| 0.0000713|
| 2000x 3 | 0.0001340| 0.0001730| 0.0001706| 0.0000988|
| 3000x 3 | 0.0001818| 0.0002431| 0.0002402| 0.0001275|
| 4000x 3 | 0.0002298| 0.0003116| 0.0003092| 0.0001554|
| 5000x 3 | 0.0002777| 0.0003804| 0.0003814| 0.0001832|
|16384x 3 | 0.0008319| 0.0011868| 0.0012013| 0.0005041|
| 1x 10 | 0.0000339| 0.0000387| 0.0000348| 0.0000412|
| 5x 10 | 0.0000336| 0.0000346| 0.0000345| 0.0000423|
| 10x 10 | 0.0000349| 0.0000347| 0.0000343| 0.0000426|
| 20x 10 | 0.0000339| 0.0000369| 0.0000374| 0.0000423|
| 30x 10 | 0.0000371| 0.0000391| 0.0000378| 0.0000439|
| 40x 10 | 0.0000378| 0.0000397| 0.0000395| 0.0000433|
| 50x 10 | 0.0000393| 0.0000407| 0.0000408| 0.0000440|
| 60x 10 | 0.0000398| 0.0000411| 0.0000423| 0.0000444|
| 70x 10 | 0.0000403| 0.0000399| 0.0000416| 0.0000445|
| 80x 10 | 0.0000411| 0.0000414| 0.0000398| 0.0000449|
| 90x 10 | 0.0000410| 0.0000417| 0.0000406| 0.0000452|
| 100x 10 | 0.0000391| 0.0000424| 0.0000416| 0.0000457|
| 110x 10 | 0.0000403| 0.0000433| 0.0000425| 0.0000460|
| 120x 10 | 0.0000404| 0.0000446| 0.0000433| 0.0000465|
| 130x 10 | 0.0000414| 0.0000450| 0.0000440| 0.0000463|
| 140x 10 | 0.0000413| 0.0000460| 0.0000455| 0.0000467|
| 150x 10 | 0.0000420| 0.0000474| 0.0000455| 0.0000471|
| 200x 10 | 0.0000457| 0.0000511| 0.0000497| 0.0000496|
| 300x 10 | 0.0000517| 0.0000598| 0.0000579| 0.0000522|
| 400x 10 | 0.0000584| 0.0000697| 0.0000660| 0.0000560|
| 500x 10 | 0.0000646| 0.0000772| 0.0000759| 0.0000592|
| 1000x 10 | 0.0000956| 0.0001229| 0.0001169| 0.0000773|
| 2000x 10 | 0.0001586| 0.0002063| 0.0002010| 0.0001119|
| 3000x 10 | 0.0002192| 0.0002946| 0.0002856| 0.0001441|
| 4000x 10 | 0.0002827| 0.0003824| 0.0003686| 0.0001793|
| 5000x 10 | 0.0003435| 0.0004679| 0.0004548| 0.0002134|
|16384x 10 | 0.0010506| 0.0015780| 0.0015417| 0.0006026|
| 1x 100 | 0.0000348| 0.0000379| 0.0000405| 0.0000427|
| 5x 100 | 0.0000348| 0.0000346| 0.0000343| 0.0000433|
| 10x 100 | 0.0000337| 0.0000350| 0.0000367| 0.0000436|
| 20x 100 | 0.0000401| 0.0000388| 0.0000395| 0.0000445|
| 30x 100 | 0.0000411| 0.0000406| 0.0000393| 0.0000452|
| 40x 100 | 0.0000379| 0.0000381| 0.0000375| 0.0000462|
| 50x 100 | 0.0000391| 0.0000399| 0.0000384| 0.0000466|
| 60x 100 | 0.0000394| 0.0000412| 0.0000393| 0.0000474|
| 70x 100 | 0.0000422| 0.0000428| 0.0000408| 0.0000489|
| 80x 100 | 0.0000433| 0.0000439| 0.0000408| 0.0000492|
| 90x 100 | 0.0000436| 0.0000445| 0.0000425| 0.0000500|
| 100x 100 | 0.0000448| 0.0000458| 0.0000435| 0.0000510|
| 110x 100 | 0.0000467| 0.0000476| 0.0000448| 0.0000514|
| 120x 100 | 0.0000466| 0.0000481| 0.0000456| 0.0000527|
| 130x 100 | 0.0000487| 0.0000501| 0.0000469| 0.0000529|
| 140x 100 | 0.0000501| 0.0000515| 0.0000479| 0.0000538|
| 150x 100 | 0.0000503| 0.0000517| 0.0000494| 0.0000550|
| 200x 100 | 0.0000556| 0.0000595| 0.0000539| 0.0000592|
| 300x 100 | 0.0000670| 0.0000708| 0.0000639| 0.0000678|
| 400x 100 | 0.0000782| 0.0000825| 0.0000748| 0.0000727|
| 500x 100 | 0.0000898| 0.0000946| 0.0000857| 0.0000824|
| 1000x 100 | 0.0001492| 0.0001591| 0.0001409| 0.0001263|
| 2000x 100 | 0.0002653| 0.0002819| 0.0002536| 0.0002101|
| 3000x 100 | 0.0003822| 0.0004043| 0.0003598| 0.0002898|
| 4000x 100 | 0.0004926| 0.0005266| 0.0004686| 0.0003655|
| 5000x 100 | 0.0006051| 0.0006524| 0.0005765| 0.0004505|
|16384x 100 | 0.0020228| 0.0021531| 0.0019262| 0.0014567|
| 1x 256 | 0.0000374| 0.0000397| 0.0000358| 0.0000434|
| 5x 256 | 0.0000336| 0.0000409| 0.0000335| 0.0000434|
| 10x 256 | 0.0000399| 0.0000370| 0.0000404| 0.0000436|
| 20x 256 | 0.0000423| 0.0000408| 0.0000400| 0.0000450|
| 30x 256 | 0.0000383| 0.0000371| 0.0000373| 0.0000463|
| 40x 256 | 0.0000411| 0.0000394| 0.0000384| 0.0000469|
| 50x 256 | 0.0000418| 0.0000411| 0.0000386| 0.0000476|
| 60x 256 | 0.0000431| 0.0000424| 0.0000407| 0.0000481|
| 70x 256 | 0.0000455| 0.0000441| 0.0000419| 0.0000495|
| 80x 256 | 0.0000465| 0.0000456| 0.0000433| 0.0000496|
| 90x 256 | 0.0000493| 0.0000476| 0.0000445| 0.0000510|
| 100x 256 | 0.0000502| 0.0000495| 0.0000457| 0.0000522|
| 110x 256 | 0.0000524| 0.0000500| 0.0000467| 0.0000534|
| 120x 256 | 0.0000535| 0.0000517| 0.0000475| 0.0000535|
| 130x 256 | 0.0000554| 0.0000534| 0.0000493| 0.0000549|
| 140x 256 | 0.0000573| 0.0000551| 0.0000512| 0.0000553|
| 150x 256 | 0.0000597| 0.0000570| 0.0000521| 0.0000568|
| 200x 256 | 0.0000679| 0.0000639| 0.0000581| 0.0000631|
| 300x 256 | 0.0000850| 0.0000826| 0.0000713| 0.0000709|
| 400x 256 | 0.0001040| 0.0000962| 0.0000854| 0.0000832|
| 500x 256 | 0.0001231| 0.0001130| 0.0001000| 0.0000967|
| 1000x 256 | 0.0002105| 0.0001881| 0.0001694| 0.0001590|
| 2000x 256 | 0.0003913| 0.0003506| 0.0003000| 0.0002847|
| 3000x 256 | 0.0005685| 0.0005093| 0.0004264| 0.0004101|
| 4000x 256 | 0.0007300| 0.0006509| 0.0005445| 0.0005121|
| 5000x 256 | 0.0009223| 0.0008244| 0.0006968| 0.0006512|
|16384x 256 | 0.0036732| 0.0032559| 0.0029458| 0.0025328|
| 1x 512 | 0.0000366| 0.0000392| 0.0000344| 0.0000438|
| 5x 512 | 0.0000394| 0.0000354| 0.0000346| 0.0000441|
| 10x 512 | 0.0000408| 0.0000401| 0.0000381| 0.0000448|
| 20x 512 | 0.0000444| 0.0000435| 0.0000389| 0.0000453|
| 30x 512 | 0.0000410| 0.0000403| 0.0000386| 0.0000474|
| 40x 512 | 0.0000446| 0.0000429| 0.0000403| 0.0000481|
| 50x 512 | 0.0000476| 0.0000463| 0.0000428| 0.0000496|
| 60x 512 | 0.0000507| 0.0000478| 0.0000440| 0.0000510|
| 70x 512 | 0.0000539| 0.0000502| 0.0000458| 0.0000518|
| 80x 512 | 0.0000577| 0.0000538| 0.0000476| 0.0000537|
| 90x 512 | 0.0000602| 0.0000558| 0.0000504| 0.0000572|
| 100x 512 | 0.0000616| 0.0000582| 0.0000518| 0.0000565|
| 110x 512 | 0.0000667| 0.0000607| 0.0000556| 0.0000600|
| 120x 512 | 0.0000689| 0.0000642| 0.0000576| 0.0000612|
| 130x 512 | 0.0000735| 0.0000654| 0.0000587| 0.0000616|
| 140x 512 | 0.0000744| 0.0000695| 0.0000607| 0.0000661|
| 150x 512 | 0.0000759| 0.0000695| 0.0000636| 0.0000653|
| 200x 512 | 0.0000913| 0.0000831| 0.0000738| 0.0000793|
| 300x 512 | 0.0001299| 0.0001136| 0.0001037| 0.0001073|
| 400x 512 | 0.0001585| 0.0001409| 0.0001313| 0.0001338|
| 500x 512 | 0.0001883| 0.0001598| 0.0001498| 0.0001580|
| 1000x 512 | 0.0003369| 0.0002909| 0.0002665| 0.0002632|
| 2000x 512 | 0.0006396| 0.0005457| 0.0004962| 0.0004790|
| 3000x 512 | 0.0009528| 0.0008094| 0.0007397| 0.0006964|
| 4000x 512 | 0.0013894| 0.0011007| 0.0010205| 0.0009181|
| 5000x 512 | 0.0018264| 0.0015710| 0.0015030| 0.0012737|
|16384x 512 | 0.0074474| 0.0067132| 0.0065738| 0.0060986|
| 1x 1024 | 0.0000347| 0.0000357| 0.0000356| 0.0000445|
| 5x 1024 | 0.0000391| 0.0000385| 0.0000362| 0.0000450|
| 10x 1024 | 0.0000383| 0.0000410| 0.0000393| 0.0000459|
| 20x 1024 | 0.0000439| 0.0000408| 0.0000384| 0.0000493|
| 30x 1024 | 0.0000489| 0.0000458| 0.0000417| 0.0000504|
| 40x 1024 | 0.0000557| 0.0000516| 0.0000448| 0.0000527|
| 50x 1024 | 0.0000587| 0.0000545| 0.0000490| 0.0000546|
| 60x 1024 | 0.0000650| 0.0000586| 0.0000516| 0.0000610|
| 70x 1024 | 0.0000707| 0.0000617| 0.0000563| 0.0000616|
| 80x 1024 | 0.0000768| 0.0000683| 0.0000618| 0.0000659|
| 90x 1024 | 0.0000834| 0.0000735| 0.0000695| 0.0000716|
| 100x 1024 | 0.0000886| 0.0000759| 0.0000708| 0.0000757|
| 110x 1024 | 0.0000949| 0.0000833| 0.0000779| 0.0000826|
| 120x 1024 | 0.0001031| 0.0000882| 0.0000841| 0.0000858|
| 130x 1024 | 0.0001088| 0.0000956| 0.0000887| 0.0000903|
| 140x 1024 | 0.0001156| 0.0001010| 0.0000923| 0.0000969|
| 150x 1024 | 0.0001152| 0.0001057| 0.0000978| 0.0001028|
| 200x 1024 | 0.0001450| 0.0001328| 0.0001257| 0.0001313|
| 300x 1024 | 0.0002082| 0.0001793| 0.0001721| 0.0001762|
| 400x 1024 | 0.0002650| 0.0002286| 0.0002192| 0.0002239|
| 500x 1024 | 0.0003157| 0.0002699| 0.0002596| 0.0002592|
| 1000x 1024 | 0.0005968| 0.0005075| 0.0004867| 0.0004764|
| 2000x 1024 | 0.0012529| 0.0010099| 0.0009831| 0.0009186|
| 3000x 1024 | 0.0021209| 0.0017632| 0.0018832| 0.0017245|
| 4000x 1024 | 0.0029924| 0.0025091| 0.0027886| 0.0024449|
| 5000x 1024 | 0.0040340| 0.0034140| 0.0037645| 0.0033816|
|16384x 1024 | 0.0149200| 0.0130993| 0.0142042| 0.0132403|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on pull request #19562: [operator] Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
szha commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-903205489
@mxnet-bot run ci [website]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] kpuatamazon commented on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
kpuatamazon commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-744344661
I've been using a c5.12xlarge `Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz`. Assume these are some sort of seconds?
We should at least do `-march=native` to see if it's just a matter of CPU support i.e. MXNet doesn't seem to enable AVX512 by default and one could add CPUID dispatch.
Might as well reshape to two dimensions with the axis preserved and everything else multiplied right?
Also, I feel like the optimal assembly implementation would benefit from a different ordering of the input tensor to allow for pure vertical adds whereas layer normalization is currently setup for horizontal adds.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19562: Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-758478322
Jenkins CI successfully triggered : [windows-gpu, windows-cpu, unix-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] bartekkuncer edited a comment on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
bartekkuncer edited a comment on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-742603472
> Hi, can we compare https://github.com/apache/incubator-mxnet/pull/19601 ?
@kpuatamazon Sorry for the late response, was waiting for the layer norm optimization in oneDNN. Below are the results I got using marian and my onednn implementation. It looks like oneDNN is faster in most cases. Can you tell me what CPU are you using?
threads amount | 28 threads | 28 threads | 4 threads | 4 threads
-- | -- | -- | -- | --
execution | marian | mx w/ onednn | marian | mx w/ onednn
1000x5 | 0,0213 | 0,032 | 0,0272 | 0,029
1000x100 | 0,03 | 0,0859 | 0,0664 | 0,0421
300x512 | 0,0255 | 0,1505 | 0,0667 | 0,073
500x512 | 0,0305 | 0,2436 | 0,0859 | 0,0449
1000x2048 | 0,1396 | 0,0616 | 0,4103 | 0,2594
1000x3 | 0,0156 | 0,0239 | 0,018 | 0,0242
45x512 | 0,0152 | 0,038 | 0,0252 | 0,0377
1000x5x100 | 0,0652 | 0,0457 | 0,2194 | 0,117
1000x8x100 | 0,0963 | 0,0538 | 0,3147 | 0,1594
300x512x512 | 8,0384 | 7,8368 | 31,491 | 28,0296
500x512x10 | 0,6231 | 0,3557 | 3,9009 | 1,0595
1000x5x2048 | 2,1114 | 1,574 | 4,9501 | 3,8672
1000x2048x3 | 2,6274 | 1,3618 | 12,9816 | 5,8291
45x512x512 | 2,2648 | 1,7278 | 5,4215 | 4,6426
1000x5x30x200 | 3,6479 | 3,5311 | 13,3026 | 11,3592
100x100x10x300 | 3,6913 | 3,9369 | 13,4574 | 10,93
300x512x45x20 | 24,2647 | 16,6294 | 146,4946 | 76,6008
50x512x40x30 | 5,3653 | 4,0195 | 27,1334 | 15,8884
100x2048x10x10 | 6,2846 | 3,4051 | 34,9686 | 13,3237
1000x3x10x200 | 0,9976 | 0,5507 | 2,1544 | 1,0273
45x52x300x45 | 4,7193 | 3,9909 | 22,6179 | 14,0623
100x5x30x20x10 | 0,7225 | 0,3956 | 4,5843 | 1,3655
100x4x100x10x30 | 2,7491 | 1,9519 | 10,9089 | 6,3355
300x52x2x45x20 | 5,7044 | 3,9567 | 30,7142 | 15,74
500x52x3x40x30 | 13,8315 | 10,8504 | 80,604 | 46,8657
100x28x10x10x10 | 0,6829 | 0,4126 | 4,3192 | 1,2705
100x3x10x18x200 | 2,1217 | 1,752 | 5,232 | 4,2747
45x512x30x4x45 | 15,4316 | 12,9341 | 86,8337 | 54,7685
I built mxnet using:
cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DUSE_BLAS=mkl -DUSE_CUDA=0 -DUSE_LAPACK=0 -DUSE_GPERFTOOLS=0 -DUSE_OPENCV=0 ..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] kpuatamazon commented on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
kpuatamazon commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-746568786
I used whatever was in your pull request: c4b6bcebeed0c5a52383f0317ab0a3b2841b8b6b
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] kpuatamazon commented on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
kpuatamazon commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-735953653
Hi, can we compare #19601 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] kpuatamazon edited a comment on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
kpuatamazon edited a comment on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-744344661
I've been using a c5.12xlarge `Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz`. Assume these are some sort of seconds?
We should at least do `-march=native` to see if it's just a matter of CPU support i.e. MXNet doesn't seem to enable AVX512 by default and one could add CPUID dispatch.
Might as well reshape to two dimensions with the axis preserved and everything else multiplied. The problem is identical for e.g. 100x28x10x10x10 and 280000x10. Also, those are some really small channels to layer normalize over.
Also, I feel like the optimal assembly implementation would benefit from a different ordering of the input tensor to allow for pure vertical adds whereas layer normalization is currently setup for horizontal adds. I can certainly see how a JIT will do better at e.g. 1000x3 where multiple problems share the same vector.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] anko-intel commented on a change in pull request #19562: [operator] Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
anko-intel commented on a change in pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#discussion_r692819028
##########
File path: src/operator/nn/mkldnn/mkldnn_layer_norm.cc
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file mkldnn_layer_norm.cc
+ */
+
+#if MXNET_USE_ONEDNN == 1
+
+#include "./mkldnn_layer_norm-inl.h"
+
+namespace mxnet {
+namespace op {
+
+bool SupportMKLDNNLayerNorm(const LayerNormParam& param, const std::vector<NDArray>& inputs) {
+ const mxnet::TShape& shape = inputs[layernorm::kData].shape();
+
+ // Native implementation (which can be found in function LayerNormCPU) is faster than oneDNN's one
+ // for small tensors. Below is the heuristic based on measurements on clx machine deciding whether
+ // the shape is better for oneDNN or native implementation.
+ auto ShapeBetterForMKLDNN = [](const mxnet::TShape& shape) {
+ constexpr size_t shapeLimit = 1024;
+ return shape.Size() / shape[0] >= shapeLimit && shape[0] >= shapeLimit;
+ };
+
+ return (ShapeBetterForMKLDNN(shape) &&
+ (GetRealAxis(param.axis, shape.ndim()) == shape.ndim() - 1) && (shape.ndim() >= 2) &&
+ (shape.ndim() <= 5) &&
+ (inputs[layernorm::kData].dtype() == mshadow::kFloat32 ||
+ inputs[layernorm::kData].dtype() == mshadow::kBfloat16) &&
+ inputs[layernorm::kGamma].dtype() == mshadow::kFloat32 &&
+ inputs[layernorm::kBeta].dtype() == mshadow::kFloat32);
+}
+
+void MKLDNNLayerNormForward(const nnvm::NodeAttrs& attrs,
+ const OpContext& ctx,
+ const std::vector<NDArray>& inputs,
+ const std::vector<OpReqType>& req,
+ const std::vector<NDArray>& outputs) {
+ const LayerNormParam& param = nnvm::get<LayerNormParam>(attrs.parsed);
+ const auto& fwd = MKLDNNLayerNormFwd::GetCached(param, ctx, inputs[layernorm::kData]);
+ fwd.Execute(param, ctx, inputs, req[layernorm::kOut], outputs);
+}
+
+MKLDNNLayerNormFwd& MKLDNNLayerNormFwd::GetCached(const LayerNormParam& param,
+ const OpContext& ctx,
+ const NDArray& data) {
+ using layernorm_fwd_map = std::unordered_map<LayerNormSignature, MKLDNNLayerNormFwd, OpHash>;
+#if DMLC_CXX11_THREAD_LOCAL
+ static thread_local layernorm_fwd_map layer_norm_fwds;
+#else
+ static MX_THREAD_LOCAL layernorm_fwd_map layer_norm_fwds;
+#endif
+
+ LayerNormSignature key(param);
+ key.AddSign(data);
+ key.AddSign(param.eps);
+
+ auto it = layer_norm_fwds.find(key);
+ if (it == layer_norm_fwds.end()) {
+ MKLDNNLayerNormFwd fwd(param, data);
+ it = AddToCache(&layer_norm_fwds, key, fwd);
+ }
+ return it->second;
+}
+
+MKLDNNLayerNormFwd::MKLDNNLayerNormFwd(const LayerNormParam& param, const NDArray& data) {
+ const mkldnn::memory::desc data_md = data.GetMKLDNNData()->get_desc();
+ fwd_pd = CreatePrimitiveDesc(param, data_md);
+ fwd = std::make_shared<mkldnn::layer_normalization_forward>(*fwd_pd);
+}
+
+std::shared_ptr<layernorm_fwd_pd_t> MKLDNNLayerNormFwd::CreatePrimitiveDesc(
+ const LayerNormParam& param,
+ const mkldnn::memory::desc& src_md) {
+ layernorm_fwd_t::desc fwd_desc(mkldnn::prop_kind::forward_training,
+ src_md,
+ param.eps,
+ dnnl::normalization_flags::use_scale_shift);
+ mkldnn::engine& engine = CpuEngine::Get()->get_engine();
+ return std::make_shared<layernorm_fwd_pd_t>(fwd_desc, engine);
+}
+
+inline mkldnn::memory::desc GetMeanVarDesc(const mkldnn::memory::data_type& dtype,
+ const mxnet::TShape& _shape) {
+ const auto ndim = _shape.ndim();
+
+ mkldnn::memory::dims shape(ndim, 1), strides(ndim, 1);
+ shape[0] = _shape[0];
+ for (int i = ndim - 1; i > 0; --i) {
+ shape[i] = _shape[i];
+ strides[i - 1] = strides[i] * shape[i];
+ }
+
+ return mkldnn::memory::desc{shape, dtype, strides};
+}
+
+inline mkldnn::memory GetScaleShiftMem(const NDArray& gamma, const NDArray& beta) {
+ // OneDNN takes gamma and beta as one SCALE_SHIFT tensor when both scale and shift are used. In
+ // mxnet scale is called gamma and shift is called beta.
+ constexpr size_t gammaAndBeta = 2;
+ CHECK_EQ(gamma.shape()[0], beta.shape()[0]);
+ const mkldnn::memory::desc scale_shift_md(mkldnn::memory::dims{gammaAndBeta, gamma.shape()[0]},
+ get_mkldnn_type(gamma.dtype()),
+ mkldnn::memory::format_tag::nc);
+ auto scale_shift_mem = mkldnn::memory(scale_shift_md, CpuEngine::Get()->get_engine());
+ char* ptr = reinterpret_cast<char*>(scale_shift_mem.get_data_handle());
+ const size_t bytes = scale_shift_md.get_size() / gammaAndBeta;
+ memcpy(ptr, gamma.data().dptr_, bytes);
+ memcpy(ptr + bytes, beta.data().dptr_, bytes);
+ return scale_shift_mem;
+}
+
+void MKLDNNLayerNormFwd::Execute(const LayerNormParam& param,
+ const OpContext& ctx,
+ const std::vector<NDArray>& inputs,
+ const OpReqType& req,
+ const std::vector<NDArray>& outputs) const {
+ auto mean_var_md = GetMeanVarDesc(get_mkldnn_type(outputs[layernorm::kMean].dtype()),
+ outputs[layernorm::kMean].shape());
+ auto mean_mem = mkldnn_output_t(
+ OutDataOp::Noop,
+ const_cast<NDArray&>(outputs[layernorm::kMean]).CreateMKLDNNData(mean_var_md));
+ auto variance_mem =
+ mkldnn_output_t(OutDataOp::Noop,
+ const_cast<NDArray&>(outputs[layernorm::kStd]).CreateMKLDNNData(mean_var_md));
+
+ auto output_mem = CreateMKLDNNMem(outputs[layernorm::kOut], fwd_pd->dst_desc(), req);
+ auto scale_shift_mem = GetScaleShiftMem(inputs[layernorm::kGamma], inputs[layernorm::kBeta]);
+
+ mkldnn_args_map_t args = {{MKLDNN_ARG_SRC, *inputs[layernorm::kData].GetMKLDNNData()},
+ {MKLDNN_ARG_DST, *output_mem.second},
+ {MKLDNN_ARG_MEAN, *mean_mem.second},
+ {MKLDNN_ARG_VARIANCE, *variance_mem.second},
+ {MKLDNN_ARG_SCALE_SHIFT, scale_shift_mem}};
+
+ MKLDNNStream::Get()->RegisterPrimArgs(*fwd, args);
+ CommitOutput(outputs[layernorm::kOut], output_mem);
+ CommitOutput(outputs[layernorm::kMean], mean_mem);
+ CommitOutput(outputs[layernorm::kStd], variance_mem);
+ MKLDNNStream::Get()->Submit();
+}
+
+MKLDNNLayerNormBwd::MKLDNNLayerNormBwd(const LayerNormParam& param,
+ const std::vector<NDArray>& inputs,
+ const mkldnn::memory::desc& data_md,
+ const mkldnn::memory::desc& diff_md)
+ : fwd_pd(MKLDNNLayerNormFwd::CreatePrimitiveDesc(param, data_md)),
+ bwd_pd(CreatePrimitiveDesc(param, data_md, diff_md, *fwd_pd)) {
+ bwd = std::make_shared<layernorm_bwd_t>(*bwd_pd);
+}
+
+std::shared_ptr<layernorm_bwd_pd_t> MKLDNNLayerNormBwd::CreatePrimitiveDesc(
+ const LayerNormParam& param,
+ const mkldnn::memory::desc& data_md,
+ const mkldnn::memory::desc& diff_md,
+ const layernorm_fwd_pd_t& layernorm_fwd_pd) {
+ layernorm_bwd_t::desc layernorm_bwd_desc(dnnl::prop_kind::backward,
+ diff_md,
+ data_md,
+ param.eps,
+ dnnl::normalization_flags::use_scale_shift);
+ mkldnn::engine& engine = CpuEngine::Get()->get_engine();
+ return std::make_shared<layernorm_bwd_pd_t>(layernorm_bwd_desc, engine, layernorm_fwd_pd);
+}
+
+void MKLDNNLayerNormBwd::Execute(const std::vector<NDArray>& inputs,
+ const std::vector<NDArray>& outputs,
+ const std::vector<OpReqType>& req) const {
+ auto scale_shift_mem =
+ GetScaleShiftMem(inputs[layernorm::kBwdGamma], inputs[layernorm::kBwdBeta]);
+ auto diff_weights_ndarray = NDArray(scale_shift_mem.get_desc());
+ const auto bytes = inputs[layernorm::kBwdGamma].shape()[0] * sizeof(float);
Review comment:
maybe?:
```suggestion
const auto bytes = inputs[layernorm::kBwdGamma].shape()[0] * mshadow_sizeof(inputs[layernorm::kBwdGamma].dtype());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-745086286
@kpuatamazon which version of oneDNN have you used for the benchmark? There is a change boosting the perf of layer normalization which is going to be included in oneDNN v2.1 release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-780381128
@mxnet-bot run ci [windows-cpu, centos-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19562: Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-758585020
Jenkins CI successfully triggered : [windows-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19562: Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-780547129
Jenkins CI successfully triggered : [windows-cpu, centos-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] kpuatamazon edited a comment on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
kpuatamazon edited a comment on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-744344661
I've been using a c5.12xlarge `Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz`. Assume these are some sort of seconds?
We should at least do `-march=native` to see if it's just a matter of CPU support i.e. MXNet doesn't seem to enable AVX512 by default and one could add CPUID dispatch.
Might as well reshape to two dimensions with the axis preserved and everything else multiplied. The problem is identical for e.g. 100x28x10x10x10 and 280000x10. Also, those are some really small channels to layer normalize over.
Also, I feel like the optimal assembly implementation would benefit from a different ordering of the input tensor to allow for pure vertical adds whereas layer normalization is currently setup for horizontal adds.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] kpuatamazon edited a comment on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
kpuatamazon edited a comment on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-744344661
I've been using a c5.12xlarge `Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz`. Assume these are some sort of seconds?
We should at least do `-march=native` to see if it's just a matter of CPU support i.e. MXNet doesn't seem to enable AVX512 by default and one could add CPUID dispatch.
Might as well reshape to two dimensions with the axis preserved and everything else multiplied. The problem is identical for e.g. 100x28x10x10x10 and 280000x10. Also, those are some really small channels to layer normalize over.
Also, I feel like the optimal assembly implementation would benefit from a different ordering of the input tensor to allow for pure vertical adds whereas layer normalization is currently setup for horizontal adds. I can certainly see how a JIT will do better at e.g. 1000x3 where a single problem is shared in vectorization.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19562: Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-729819592
Hey @bartekkuncer , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:
- To trigger all jobs: @mxnet-bot run ci [all]
- To trigger specific jobs: @mxnet-bot run ci [job1, job2]
***
**CI supported jobs**: [windows-gpu, sanity, centos-gpu, miscellaneous, website, unix-gpu, centos-cpu, clang, windows-cpu, unix-cpu, edge]
***
_Note_:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-758584969
@mxnet-bot run ci [windows-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19562: Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-781241541
Jenkins CI successfully triggered : [windows-cpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19562: [operator] Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-903205498
Jenkins CI successfully triggered : [website]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-732973375
> @bartekkuncer could you add more description in the PR to avoid confusion for the reviewers?
@pengzhao-intel was planning to add them as soon as I fix tests :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] bartekkuncer commented on a change in pull request #19562: [operator] Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on a change in pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#discussion_r692919251
##########
File path: src/operator/nn/mkldnn/mkldnn_layer_norm.cc
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file mkldnn_layer_norm.cc
+ */
+
+#if MXNET_USE_ONEDNN == 1
+
+#include "./mkldnn_layer_norm-inl.h"
+
+namespace mxnet {
+namespace op {
+
+bool SupportMKLDNNLayerNorm(const LayerNormParam& param, const std::vector<NDArray>& inputs) {
+ const mxnet::TShape& shape = inputs[layernorm::kData].shape();
+
+ // Native implementation (which can be found in function LayerNormCPU) is faster than oneDNN's one
+ // for small tensors. Below is the heuristic based on measurements on clx machine deciding whether
+ // the shape is better for oneDNN or native implementation.
+ auto ShapeBetterForMKLDNN = [](const mxnet::TShape& shape) {
+ constexpr size_t shapeLimit = 1024;
+ return shape.Size() / shape[0] >= shapeLimit && shape[0] >= shapeLimit;
+ };
+
+ return (ShapeBetterForMKLDNN(shape) &&
+ (GetRealAxis(param.axis, shape.ndim()) == shape.ndim() - 1) && (shape.ndim() >= 2) &&
+ (shape.ndim() <= 5) &&
+ (inputs[layernorm::kData].dtype() == mshadow::kFloat32 ||
+ inputs[layernorm::kData].dtype() == mshadow::kBfloat16) &&
+ inputs[layernorm::kGamma].dtype() == mshadow::kFloat32 &&
+ inputs[layernorm::kBeta].dtype() == mshadow::kFloat32);
+}
+
+void MKLDNNLayerNormForward(const nnvm::NodeAttrs& attrs,
+ const OpContext& ctx,
+ const std::vector<NDArray>& inputs,
+ const std::vector<OpReqType>& req,
+ const std::vector<NDArray>& outputs) {
+ const LayerNormParam& param = nnvm::get<LayerNormParam>(attrs.parsed);
+ const auto& fwd = MKLDNNLayerNormFwd::GetCached(param, ctx, inputs[layernorm::kData]);
+ fwd.Execute(param, ctx, inputs, req[layernorm::kOut], outputs);
+}
+
+MKLDNNLayerNormFwd& MKLDNNLayerNormFwd::GetCached(const LayerNormParam& param,
+ const OpContext& ctx,
+ const NDArray& data) {
+ using layernorm_fwd_map = std::unordered_map<LayerNormSignature, MKLDNNLayerNormFwd, OpHash>;
+#if DMLC_CXX11_THREAD_LOCAL
+ static thread_local layernorm_fwd_map layer_norm_fwds;
+#else
+ static MX_THREAD_LOCAL layernorm_fwd_map layer_norm_fwds;
+#endif
+
+ LayerNormSignature key(param);
+ key.AddSign(data);
+ key.AddSign(param.eps);
+
+ auto it = layer_norm_fwds.find(key);
+ if (it == layer_norm_fwds.end()) {
+ MKLDNNLayerNormFwd fwd(param, data);
+ it = AddToCache(&layer_norm_fwds, key, fwd);
+ }
+ return it->second;
+}
+
+MKLDNNLayerNormFwd::MKLDNNLayerNormFwd(const LayerNormParam& param, const NDArray& data) {
+ const mkldnn::memory::desc data_md = data.GetMKLDNNData()->get_desc();
+ fwd_pd = CreatePrimitiveDesc(param, data_md);
+ fwd = std::make_shared<mkldnn::layer_normalization_forward>(*fwd_pd);
Review comment:
Thanks, I overlooked this one ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] fhieber commented on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
fhieber commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-744569353
Speaking up as a 'customer' of LayerNorm here: Sockeye (and its Transformer models) cares about smaller matrix sizes for LayerNorm, i.e. typically in ranges around `(y, 512) for y in range(2,100)`. It would be great if we could get the performance benefits of the Marian implementation into MXNet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: [operator] Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-902229587
@mxnet-bot run ci [centos-gpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] anko-intel commented on a change in pull request #19562: [operator] Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
anko-intel commented on a change in pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#discussion_r692819028
##########
File path: src/operator/nn/mkldnn/mkldnn_layer_norm.cc
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file mkldnn_layer_norm.cc
+ */
+
+#if MXNET_USE_ONEDNN == 1
+
+#include "./mkldnn_layer_norm-inl.h"
+
+namespace mxnet {
+namespace op {
+
+bool SupportMKLDNNLayerNorm(const LayerNormParam& param, const std::vector<NDArray>& inputs) {
+ const mxnet::TShape& shape = inputs[layernorm::kData].shape();
+
+ // Native implementation (which can be found in function LayerNormCPU) is faster than oneDNN's one
+ // for small tensors. Below is the heuristic based on measurements on clx machine deciding whether
+ // the shape is better for oneDNN or native implementation.
+ auto ShapeBetterForMKLDNN = [](const mxnet::TShape& shape) {
+ constexpr size_t shapeLimit = 1024;
+ return shape.Size() / shape[0] >= shapeLimit && shape[0] >= shapeLimit;
+ };
+
+ return (ShapeBetterForMKLDNN(shape) &&
+ (GetRealAxis(param.axis, shape.ndim()) == shape.ndim() - 1) && (shape.ndim() >= 2) &&
+ (shape.ndim() <= 5) &&
+ (inputs[layernorm::kData].dtype() == mshadow::kFloat32 ||
+ inputs[layernorm::kData].dtype() == mshadow::kBfloat16) &&
+ inputs[layernorm::kGamma].dtype() == mshadow::kFloat32 &&
+ inputs[layernorm::kBeta].dtype() == mshadow::kFloat32);
+}
+
+void MKLDNNLayerNormForward(const nnvm::NodeAttrs& attrs,
+ const OpContext& ctx,
+ const std::vector<NDArray>& inputs,
+ const std::vector<OpReqType>& req,
+ const std::vector<NDArray>& outputs) {
+ const LayerNormParam& param = nnvm::get<LayerNormParam>(attrs.parsed);
+ const auto& fwd = MKLDNNLayerNormFwd::GetCached(param, ctx, inputs[layernorm::kData]);
+ fwd.Execute(param, ctx, inputs, req[layernorm::kOut], outputs);
+}
+
+MKLDNNLayerNormFwd& MKLDNNLayerNormFwd::GetCached(const LayerNormParam& param,
+ const OpContext& ctx,
+ const NDArray& data) {
+ using layernorm_fwd_map = std::unordered_map<LayerNormSignature, MKLDNNLayerNormFwd, OpHash>;
+#if DMLC_CXX11_THREAD_LOCAL
+ static thread_local layernorm_fwd_map layer_norm_fwds;
+#else
+ static MX_THREAD_LOCAL layernorm_fwd_map layer_norm_fwds;
+#endif
+
+ LayerNormSignature key(param);
+ key.AddSign(data);
+ key.AddSign(param.eps);
+
+ auto it = layer_norm_fwds.find(key);
+ if (it == layer_norm_fwds.end()) {
+ MKLDNNLayerNormFwd fwd(param, data);
+ it = AddToCache(&layer_norm_fwds, key, fwd);
+ }
+ return it->second;
+}
+
+MKLDNNLayerNormFwd::MKLDNNLayerNormFwd(const LayerNormParam& param, const NDArray& data) {
+ const mkldnn::memory::desc data_md = data.GetMKLDNNData()->get_desc();
+ fwd_pd = CreatePrimitiveDesc(param, data_md);
+ fwd = std::make_shared<mkldnn::layer_normalization_forward>(*fwd_pd);
+}
+
+std::shared_ptr<layernorm_fwd_pd_t> MKLDNNLayerNormFwd::CreatePrimitiveDesc(
+ const LayerNormParam& param,
+ const mkldnn::memory::desc& src_md) {
+ layernorm_fwd_t::desc fwd_desc(mkldnn::prop_kind::forward_training,
+ src_md,
+ param.eps,
+ dnnl::normalization_flags::use_scale_shift);
+ mkldnn::engine& engine = CpuEngine::Get()->get_engine();
+ return std::make_shared<layernorm_fwd_pd_t>(fwd_desc, engine);
+}
+
+inline mkldnn::memory::desc GetMeanVarDesc(const mkldnn::memory::data_type& dtype,
+ const mxnet::TShape& _shape) {
+ const auto ndim = _shape.ndim();
+
+ mkldnn::memory::dims shape(ndim, 1), strides(ndim, 1);
+ shape[0] = _shape[0];
+ for (int i = ndim - 1; i > 0; --i) {
+ shape[i] = _shape[i];
+ strides[i - 1] = strides[i] * shape[i];
+ }
+
+ return mkldnn::memory::desc{shape, dtype, strides};
+}
+
+inline mkldnn::memory GetScaleShiftMem(const NDArray& gamma, const NDArray& beta) {
+ // OneDNN takes gamma and beta as one SCALE_SHIFT tensor when both scale and shift are used. In
+ // mxnet scale is called gamma and shift is called beta.
+ constexpr size_t gammaAndBeta = 2;
+ CHECK_EQ(gamma.shape()[0], beta.shape()[0]);
+ const mkldnn::memory::desc scale_shift_md(mkldnn::memory::dims{gammaAndBeta, gamma.shape()[0]},
+ get_mkldnn_type(gamma.dtype()),
+ mkldnn::memory::format_tag::nc);
+ auto scale_shift_mem = mkldnn::memory(scale_shift_md, CpuEngine::Get()->get_engine());
+ char* ptr = reinterpret_cast<char*>(scale_shift_mem.get_data_handle());
+ const size_t bytes = scale_shift_md.get_size() / gammaAndBeta;
+ memcpy(ptr, gamma.data().dptr_, bytes);
+ memcpy(ptr + bytes, beta.data().dptr_, bytes);
+ return scale_shift_mem;
+}
+
+void MKLDNNLayerNormFwd::Execute(const LayerNormParam& param,
+ const OpContext& ctx,
+ const std::vector<NDArray>& inputs,
+ const OpReqType& req,
+ const std::vector<NDArray>& outputs) const {
+ auto mean_var_md = GetMeanVarDesc(get_mkldnn_type(outputs[layernorm::kMean].dtype()),
+ outputs[layernorm::kMean].shape());
+ auto mean_mem = mkldnn_output_t(
+ OutDataOp::Noop,
+ const_cast<NDArray&>(outputs[layernorm::kMean]).CreateMKLDNNData(mean_var_md));
+ auto variance_mem =
+ mkldnn_output_t(OutDataOp::Noop,
+ const_cast<NDArray&>(outputs[layernorm::kStd]).CreateMKLDNNData(mean_var_md));
+
+ auto output_mem = CreateMKLDNNMem(outputs[layernorm::kOut], fwd_pd->dst_desc(), req);
+ auto scale_shift_mem = GetScaleShiftMem(inputs[layernorm::kGamma], inputs[layernorm::kBeta]);
+
+ mkldnn_args_map_t args = {{MKLDNN_ARG_SRC, *inputs[layernorm::kData].GetMKLDNNData()},
+ {MKLDNN_ARG_DST, *output_mem.second},
+ {MKLDNN_ARG_MEAN, *mean_mem.second},
+ {MKLDNN_ARG_VARIANCE, *variance_mem.second},
+ {MKLDNN_ARG_SCALE_SHIFT, scale_shift_mem}};
+
+ MKLDNNStream::Get()->RegisterPrimArgs(*fwd, args);
+ CommitOutput(outputs[layernorm::kOut], output_mem);
+ CommitOutput(outputs[layernorm::kMean], mean_mem);
+ CommitOutput(outputs[layernorm::kStd], variance_mem);
+ MKLDNNStream::Get()->Submit();
+}
+
+MKLDNNLayerNormBwd::MKLDNNLayerNormBwd(const LayerNormParam& param,
+ const std::vector<NDArray>& inputs,
+ const mkldnn::memory::desc& data_md,
+ const mkldnn::memory::desc& diff_md)
+ : fwd_pd(MKLDNNLayerNormFwd::CreatePrimitiveDesc(param, data_md)),
+ bwd_pd(CreatePrimitiveDesc(param, data_md, diff_md, *fwd_pd)) {
+ bwd = std::make_shared<layernorm_bwd_t>(*bwd_pd);
+}
+
+std::shared_ptr<layernorm_bwd_pd_t> MKLDNNLayerNormBwd::CreatePrimitiveDesc(
+ const LayerNormParam& param,
+ const mkldnn::memory::desc& data_md,
+ const mkldnn::memory::desc& diff_md,
+ const layernorm_fwd_pd_t& layernorm_fwd_pd) {
+ layernorm_bwd_t::desc layernorm_bwd_desc(dnnl::prop_kind::backward,
+ diff_md,
+ data_md,
+ param.eps,
+ dnnl::normalization_flags::use_scale_shift);
+ mkldnn::engine& engine = CpuEngine::Get()->get_engine();
+ return std::make_shared<layernorm_bwd_pd_t>(layernorm_bwd_desc, engine, layernorm_fwd_pd);
+}
+
+void MKLDNNLayerNormBwd::Execute(const std::vector<NDArray>& inputs,
+ const std::vector<NDArray>& outputs,
+ const std::vector<OpReqType>& req) const {
+ auto scale_shift_mem =
+ GetScaleShiftMem(inputs[layernorm::kBwdGamma], inputs[layernorm::kBwdBeta]);
+ auto diff_weights_ndarray = NDArray(scale_shift_mem.get_desc());
+ const auto bytes = inputs[layernorm::kBwdGamma].shape()[0] * sizeof(float);
Review comment:
maybe?:
```suggestion
const auto bytes = inputs[layernorm::kBwdGamma].shape()[0] * mshadow_sizeof(mshadow::kFloat32);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] bartekkuncer commented on a change in pull request #19562: [operator] Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on a change in pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#discussion_r692919337
##########
File path: src/operator/nn/mkldnn/mkldnn_layer_norm.cc
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file mkldnn_layer_norm.cc
+ */
+
+#if MXNET_USE_ONEDNN == 1
+
+#include "./mkldnn_layer_norm-inl.h"
+
+namespace mxnet {
+namespace op {
+
+bool SupportMKLDNNLayerNorm(const LayerNormParam& param, const std::vector<NDArray>& inputs) {
+ const mxnet::TShape& shape = inputs[layernorm::kData].shape();
+
+ // Native implementation (which can be found in function LayerNormCPU) is faster than oneDNN's one
+ // for small tensors. Below is the heuristic based on measurements on clx machine deciding whether
+ // the shape is better for oneDNN or native implementation.
+ auto ShapeBetterForMKLDNN = [](const mxnet::TShape& shape) {
+ constexpr size_t shapeLimit = 1024;
+ return shape.Size() / shape[0] >= shapeLimit && shape[0] >= shapeLimit;
+ };
+
+ return (ShapeBetterForMKLDNN(shape) &&
+ (GetRealAxis(param.axis, shape.ndim()) == shape.ndim() - 1) && (shape.ndim() >= 2) &&
+ (shape.ndim() <= 5) &&
+ (inputs[layernorm::kData].dtype() == mshadow::kFloat32 ||
+ inputs[layernorm::kData].dtype() == mshadow::kBfloat16) &&
+ inputs[layernorm::kGamma].dtype() == mshadow::kFloat32 &&
+ inputs[layernorm::kBeta].dtype() == mshadow::kFloat32);
+}
+
+void MKLDNNLayerNormForward(const nnvm::NodeAttrs& attrs,
+ const OpContext& ctx,
+ const std::vector<NDArray>& inputs,
+ const std::vector<OpReqType>& req,
+ const std::vector<NDArray>& outputs) {
+ const LayerNormParam& param = nnvm::get<LayerNormParam>(attrs.parsed);
+ const auto& fwd = MKLDNNLayerNormFwd::GetCached(param, ctx, inputs[layernorm::kData]);
+ fwd.Execute(param, ctx, inputs, req[layernorm::kOut], outputs);
+}
+
+MKLDNNLayerNormFwd& MKLDNNLayerNormFwd::GetCached(const LayerNormParam& param,
+ const OpContext& ctx,
+ const NDArray& data) {
+ using layernorm_fwd_map = std::unordered_map<LayerNormSignature, MKLDNNLayerNormFwd, OpHash>;
+#if DMLC_CXX11_THREAD_LOCAL
+ static thread_local layernorm_fwd_map layer_norm_fwds;
+#else
+ static MX_THREAD_LOCAL layernorm_fwd_map layer_norm_fwds;
+#endif
+
+ LayerNormSignature key(param);
+ key.AddSign(data);
+ key.AddSign(param.eps);
Review comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] bartekkuncer commented on a change in pull request #19562: [operator] Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on a change in pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#discussion_r692882144
##########
File path: src/operator/nn/mkldnn/mkldnn_layer_norm.cc
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file mkldnn_layer_norm.cc
+ */
+
+#if MXNET_USE_ONEDNN == 1
+
+#include "./mkldnn_layer_norm-inl.h"
+
+namespace mxnet {
+namespace op {
+
+bool SupportMKLDNNLayerNorm(const LayerNormParam& param, const std::vector<NDArray>& inputs) {
+ const mxnet::TShape& shape = inputs[layernorm::kData].shape();
+
+ // Native implementation (which can be found in function LayerNormCPU) is faster than oneDNN's one
+ // for small tensors. Below is the heuristic based on measurements on clx machine deciding whether
+ // the shape is better for oneDNN or native implementation.
+ auto ShapeBetterForMKLDNN = [](const mxnet::TShape& shape) {
+ constexpr size_t shapeLimit = 1024;
+ return shape.Size() / shape[0] >= shapeLimit && shape[0] >= shapeLimit;
+ };
+
+ return (ShapeBetterForMKLDNN(shape) &&
+ (GetRealAxis(param.axis, shape.ndim()) == shape.ndim() - 1) && (shape.ndim() >= 2) &&
+ (shape.ndim() <= 5) &&
+ (inputs[layernorm::kData].dtype() == mshadow::kFloat32 ||
+ inputs[layernorm::kData].dtype() == mshadow::kBfloat16) &&
+ inputs[layernorm::kGamma].dtype() == mshadow::kFloat32 &&
+ inputs[layernorm::kBeta].dtype() == mshadow::kFloat32);
+}
+
+void MKLDNNLayerNormForward(const nnvm::NodeAttrs& attrs,
+ const OpContext& ctx,
+ const std::vector<NDArray>& inputs,
+ const std::vector<OpReqType>& req,
+ const std::vector<NDArray>& outputs) {
+ const LayerNormParam& param = nnvm::get<LayerNormParam>(attrs.parsed);
+ const auto& fwd = MKLDNNLayerNormFwd::GetCached(param, ctx, inputs[layernorm::kData]);
+ fwd.Execute(param, ctx, inputs, req[layernorm::kOut], outputs);
+}
+
+MKLDNNLayerNormFwd& MKLDNNLayerNormFwd::GetCached(const LayerNormParam& param,
+ const OpContext& ctx,
+ const NDArray& data) {
+ using layernorm_fwd_map = std::unordered_map<LayerNormSignature, MKLDNNLayerNormFwd, OpHash>;
+#if DMLC_CXX11_THREAD_LOCAL
+ static thread_local layernorm_fwd_map layer_norm_fwds;
+#else
+ static MX_THREAD_LOCAL layernorm_fwd_map layer_norm_fwds;
+#endif
+
+ LayerNormSignature key(param);
+ key.AddSign(data);
+ key.AddSign(param.eps);
+
+ auto it = layer_norm_fwds.find(key);
+ if (it == layer_norm_fwds.end()) {
+ MKLDNNLayerNormFwd fwd(param, data);
+ it = AddToCache(&layer_norm_fwds, key, fwd);
+ }
+ return it->second;
+}
+
+MKLDNNLayerNormFwd::MKLDNNLayerNormFwd(const LayerNormParam& param, const NDArray& data) {
+ const mkldnn::memory::desc data_md = data.GetMKLDNNData()->get_desc();
+ fwd_pd = CreatePrimitiveDesc(param, data_md);
+ fwd = std::make_shared<mkldnn::layer_normalization_forward>(*fwd_pd);
+}
+
+std::shared_ptr<layernorm_fwd_pd_t> MKLDNNLayerNormFwd::CreatePrimitiveDesc(
+ const LayerNormParam& param,
+ const mkldnn::memory::desc& src_md) {
+ layernorm_fwd_t::desc fwd_desc(mkldnn::prop_kind::forward_training,
+ src_md,
+ param.eps,
+ dnnl::normalization_flags::use_scale_shift);
+ mkldnn::engine& engine = CpuEngine::Get()->get_engine();
+ return std::make_shared<layernorm_fwd_pd_t>(fwd_desc, engine);
+}
+
+inline mkldnn::memory::desc GetMeanVarDesc(const mkldnn::memory::data_type& dtype,
+ const mxnet::TShape& _shape) {
+ const auto ndim = _shape.ndim();
+
+ mkldnn::memory::dims shape(ndim, 1), strides(ndim, 1);
+ shape[0] = _shape[0];
+ for (int i = ndim - 1; i > 0; --i) {
+ shape[i] = _shape[i];
+ strides[i - 1] = strides[i] * shape[i];
+ }
+
+ return mkldnn::memory::desc{shape, dtype, strides};
+}
+
+inline mkldnn::memory GetScaleShiftMem(const NDArray& gamma, const NDArray& beta) {
+ // OneDNN takes gamma and beta as one SCALE_SHIFT tensor when both scale and shift are used. In
+ // mxnet scale is called gamma and shift is called beta.
+ constexpr size_t gammaAndBeta = 2;
+ CHECK_EQ(gamma.shape()[0], beta.shape()[0]);
+ const mkldnn::memory::desc scale_shift_md(mkldnn::memory::dims{gammaAndBeta, gamma.shape()[0]},
+ get_mkldnn_type(gamma.dtype()),
+ mkldnn::memory::format_tag::nc);
+ auto scale_shift_mem = mkldnn::memory(scale_shift_md, CpuEngine::Get()->get_engine());
+ char* ptr = reinterpret_cast<char*>(scale_shift_mem.get_data_handle());
+ const size_t bytes = scale_shift_md.get_size() / gammaAndBeta;
+ memcpy(ptr, gamma.data().dptr_, bytes);
+ memcpy(ptr + bytes, beta.data().dptr_, bytes);
+ return scale_shift_mem;
+}
+
+void MKLDNNLayerNormFwd::Execute(const LayerNormParam& param,
+ const OpContext& ctx,
+ const std::vector<NDArray>& inputs,
+ const OpReqType& req,
+ const std::vector<NDArray>& outputs) const {
+ auto mean_var_md = GetMeanVarDesc(get_mkldnn_type(outputs[layernorm::kMean].dtype()),
+ outputs[layernorm::kMean].shape());
+ auto mean_mem = mkldnn_output_t(
+ OutDataOp::Noop,
+ const_cast<NDArray&>(outputs[layernorm::kMean]).CreateMKLDNNData(mean_var_md));
+ auto variance_mem =
+ mkldnn_output_t(OutDataOp::Noop,
+ const_cast<NDArray&>(outputs[layernorm::kStd]).CreateMKLDNNData(mean_var_md));
+
+ auto output_mem = CreateMKLDNNMem(outputs[layernorm::kOut], fwd_pd->dst_desc(), req);
+ auto scale_shift_mem = GetScaleShiftMem(inputs[layernorm::kGamma], inputs[layernorm::kBeta]);
+
+ mkldnn_args_map_t args = {{MKLDNN_ARG_SRC, *inputs[layernorm::kData].GetMKLDNNData()},
+ {MKLDNN_ARG_DST, *output_mem.second},
+ {MKLDNN_ARG_MEAN, *mean_mem.second},
+ {MKLDNN_ARG_VARIANCE, *variance_mem.second},
+ {MKLDNN_ARG_SCALE_SHIFT, scale_shift_mem}};
+
+ MKLDNNStream::Get()->RegisterPrimArgs(*fwd, args);
+ CommitOutput(outputs[layernorm::kOut], output_mem);
+ CommitOutput(outputs[layernorm::kMean], mean_mem);
+ CommitOutput(outputs[layernorm::kStd], variance_mem);
+ MKLDNNStream::Get()->Submit();
+}
+
+MKLDNNLayerNormBwd::MKLDNNLayerNormBwd(const LayerNormParam& param,
+ const std::vector<NDArray>& inputs,
+ const mkldnn::memory::desc& data_md,
+ const mkldnn::memory::desc& diff_md)
+ : fwd_pd(MKLDNNLayerNormFwd::CreatePrimitiveDesc(param, data_md)),
+ bwd_pd(CreatePrimitiveDesc(param, data_md, diff_md, *fwd_pd)) {
+ bwd = std::make_shared<layernorm_bwd_t>(*bwd_pd);
+}
+
+std::shared_ptr<layernorm_bwd_pd_t> MKLDNNLayerNormBwd::CreatePrimitiveDesc(
+ const LayerNormParam& param,
+ const mkldnn::memory::desc& data_md,
+ const mkldnn::memory::desc& diff_md,
+ const layernorm_fwd_pd_t& layernorm_fwd_pd) {
+ layernorm_bwd_t::desc layernorm_bwd_desc(dnnl::prop_kind::backward,
+ diff_md,
+ data_md,
+ param.eps,
+ dnnl::normalization_flags::use_scale_shift);
+ mkldnn::engine& engine = CpuEngine::Get()->get_engine();
+ return std::make_shared<layernorm_bwd_pd_t>(layernorm_bwd_desc, engine, layernorm_fwd_pd);
+}
+
+void MKLDNNLayerNormBwd::Execute(const std::vector<NDArray>& inputs,
+ const std::vector<NDArray>& outputs,
+ const std::vector<OpReqType>& req) const {
+ auto scale_shift_mem =
+ GetScaleShiftMem(inputs[layernorm::kBwdGamma], inputs[layernorm::kBwdBeta]);
+ auto diff_weights_ndarray = NDArray(scale_shift_mem.get_desc());
+ const auto bytes = inputs[layernorm::kBwdGamma].shape()[0] * sizeof(float);
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-742603472
> Hi, can we compare https://github.com/apache/incubator-mxnet/pull/19601 ?
@kpuatamazon Sorry for the late response, was waiting for the layer norm optimization in oneDNN. Below are the results I got using marian and my onednn implementation. It looks like oneDNN is faster in most cases. Can you tell me what CPU are you using?
threads amount | 28 threads | 28 threads | 4 threads | 4 threads
-- | -- | -- | -- | --
execution | marian | mx w/ onednn | marian | mx w/ onednn
1000x5 | 0,0213 | 0,032 | 0,0272 | 0,029
1000x100 | 0,03 | 0,0859 | 0,0664 | 0,0421
300x512 | 0,0255 | 0,1505 | 0,0667 | 0,073
500x512 | 0,0305 | 0,2436 | 0,0859 | 0,0449
1000x2048 | 0,1396 | 0,0616 | 0,4103 | 0,2594
1000x3 | 0,0156 | 0,0239 | 0,018 | 0,0242
45x512 | 0,0152 | 0,038 | 0,0252 | 0,0377
1000x5x100 | 0,0652 | 0,0457 | 0,2194 | 0,117
1000x8x100 | 0,0963 | 0,0538 | 0,3147 | 0,1594
300x512x512 | 8,0384 | 7,8368 | 31,491 | 28,0296
500x512x10 | 0,6231 | 0,3557 | 3,9009 | 1,0595
1000x5x2048 | 2,1114 | 1,574 | 4,9501 | 3,8672
1000x2048x3 | 2,6274 | 1,3618 | 12,9816 | 5,8291
45x512x512 | 2,2648 | 1,7278 | 5,4215 | 4,6426
1000x5x30x200 | 3,6479 | 3,5311 | 13,3026 | 11,3592
100x100x10x300 | 3,6913 | 3,9369 | 13,4574 | 10,93
300x512x45x20 | 24,2647 | 16,6294 | 146,4946 | 76,6008
50x512x40x30 | 5,3653 | 4,0195 | 27,1334 | 15,8884
100x2048x10x10 | 6,2846 | 3,4051 | 34,9686 | 13,3237
1000x3x10x200 | 0,9976 | 0,5507 | 2,1544 | 1,0273
45x52x300x45 | 4,7193 | 3,9909 | 22,6179 | 14,0623
100x5x30x20x10 | 0,7225 | 0,3956 | 4,5843 | 1,3655
100x4x100x10x30 | 2,7491 | 1,9519 | 10,9089 | 6,3355
300x52x2x45x20 | 5,7044 | 3,9567 | 30,7142 | 15,74
500x52x3x40x30 | 13,8315 | 10,8504 | 80,604 | 46,8657
100x28x10x10x10 | 0,6829 | 0,4126 | 4,3192 | 1,2705
100x3x10x18x200 | 2,1217 | 1,752 | 5,232 | 4,2747
45x512x30x4x45 | 15,4316 | 12,9341 | 86,8337 | 54,7685
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-780547068
@mxnet-bot run ci [windows-cpu, centos-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] bgawrych commented on a change in pull request #19562: [operator] Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
bgawrych commented on a change in pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#discussion_r692904008
##########
File path: src/operator/nn/mkldnn/mkldnn_layer_norm.cc
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file mkldnn_layer_norm.cc
+ */
+
+#if MXNET_USE_ONEDNN == 1
+
+#include "./mkldnn_layer_norm-inl.h"
+
+namespace mxnet {
+namespace op {
+
+bool SupportMKLDNNLayerNorm(const LayerNormParam& param, const std::vector<NDArray>& inputs) {
+ const mxnet::TShape& shape = inputs[layernorm::kData].shape();
+
+ // Native implementation (which can be found in function LayerNormCPU) is faster than oneDNN's one
+ // for small tensors. Below is the heuristic based on measurements on clx machine deciding whether
+ // the shape is better for oneDNN or native implementation.
+ auto ShapeBetterForMKLDNN = [](const mxnet::TShape& shape) {
+ constexpr size_t shapeLimit = 1024;
+ return shape.Size() / shape[0] >= shapeLimit && shape[0] >= shapeLimit;
+ };
+
+ return (ShapeBetterForMKLDNN(shape) &&
+ (GetRealAxis(param.axis, shape.ndim()) == shape.ndim() - 1) && (shape.ndim() >= 2) &&
+ (shape.ndim() <= 5) &&
+ (inputs[layernorm::kData].dtype() == mshadow::kFloat32 ||
+ inputs[layernorm::kData].dtype() == mshadow::kBfloat16) &&
+ inputs[layernorm::kGamma].dtype() == mshadow::kFloat32 &&
+ inputs[layernorm::kBeta].dtype() == mshadow::kFloat32);
+}
+
+void MKLDNNLayerNormForward(const nnvm::NodeAttrs& attrs,
+ const OpContext& ctx,
+ const std::vector<NDArray>& inputs,
+ const std::vector<OpReqType>& req,
+ const std::vector<NDArray>& outputs) {
+ const LayerNormParam& param = nnvm::get<LayerNormParam>(attrs.parsed);
+ const auto& fwd = MKLDNNLayerNormFwd::GetCached(param, ctx, inputs[layernorm::kData]);
+ fwd.Execute(param, ctx, inputs, req[layernorm::kOut], outputs);
+}
+
+MKLDNNLayerNormFwd& MKLDNNLayerNormFwd::GetCached(const LayerNormParam& param,
+ const OpContext& ctx,
+ const NDArray& data) {
+ using layernorm_fwd_map = std::unordered_map<LayerNormSignature, MKLDNNLayerNormFwd, OpHash>;
+#if DMLC_CXX11_THREAD_LOCAL
+ static thread_local layernorm_fwd_map layer_norm_fwds;
+#else
+ static MX_THREAD_LOCAL layernorm_fwd_map layer_norm_fwds;
+#endif
+
+ LayerNormSignature key(param);
+ key.AddSign(data);
+ key.AddSign(param.eps);
Review comment:
Isn't eps already added by passing param to key?
##########
File path: src/operator/nn/mkldnn/mkldnn_layer_norm.cc
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file mkldnn_layer_norm.cc
+ */
+
+#if MXNET_USE_ONEDNN == 1
+
+#include "./mkldnn_layer_norm-inl.h"
+
+namespace mxnet {
+namespace op {
+
+bool SupportMKLDNNLayerNorm(const LayerNormParam& param, const std::vector<NDArray>& inputs) {
+ const mxnet::TShape& shape = inputs[layernorm::kData].shape();
+
+ // Native implementation (which can be found in function LayerNormCPU) is faster than oneDNN's one
+ // for small tensors. Below is the heuristic based on measurements on clx machine deciding whether
+ // the shape is better for oneDNN or native implementation.
+ auto ShapeBetterForMKLDNN = [](const mxnet::TShape& shape) {
+ constexpr size_t shapeLimit = 1024;
+ return shape.Size() / shape[0] >= shapeLimit && shape[0] >= shapeLimit;
+ };
+
+ return (ShapeBetterForMKLDNN(shape) &&
+ (GetRealAxis(param.axis, shape.ndim()) == shape.ndim() - 1) && (shape.ndim() >= 2) &&
+ (shape.ndim() <= 5) &&
+ (inputs[layernorm::kData].dtype() == mshadow::kFloat32 ||
+ inputs[layernorm::kData].dtype() == mshadow::kBfloat16) &&
+ inputs[layernorm::kGamma].dtype() == mshadow::kFloat32 &&
+ inputs[layernorm::kBeta].dtype() == mshadow::kFloat32);
+}
+
+void MKLDNNLayerNormForward(const nnvm::NodeAttrs& attrs,
+ const OpContext& ctx,
+ const std::vector<NDArray>& inputs,
+ const std::vector<OpReqType>& req,
+ const std::vector<NDArray>& outputs) {
+ const LayerNormParam& param = nnvm::get<LayerNormParam>(attrs.parsed);
+ const auto& fwd = MKLDNNLayerNormFwd::GetCached(param, ctx, inputs[layernorm::kData]);
+ fwd.Execute(param, ctx, inputs, req[layernorm::kOut], outputs);
+}
+
+MKLDNNLayerNormFwd& MKLDNNLayerNormFwd::GetCached(const LayerNormParam& param,
+ const OpContext& ctx,
+ const NDArray& data) {
+ using layernorm_fwd_map = std::unordered_map<LayerNormSignature, MKLDNNLayerNormFwd, OpHash>;
+#if DMLC_CXX11_THREAD_LOCAL
+ static thread_local layernorm_fwd_map layer_norm_fwds;
+#else
+ static MX_THREAD_LOCAL layernorm_fwd_map layer_norm_fwds;
+#endif
+
+ LayerNormSignature key(param);
+ key.AddSign(data);
+ key.AddSign(param.eps);
+
+ auto it = layer_norm_fwds.find(key);
+ if (it == layer_norm_fwds.end()) {
+ MKLDNNLayerNormFwd fwd(param, data);
+ it = AddToCache(&layer_norm_fwds, key, fwd);
+ }
+ return it->second;
+}
+
+MKLDNNLayerNormFwd::MKLDNNLayerNormFwd(const LayerNormParam& param, const NDArray& data) {
+ const mkldnn::memory::desc data_md = data.GetMKLDNNData()->get_desc();
+ fwd_pd = CreatePrimitiveDesc(param, data_md);
+ fwd = std::make_shared<mkldnn::layer_normalization_forward>(*fwd_pd);
Review comment:
Sometimes you're using defined types by using, sometimes directly from mkldnn namespace (layernorm_fwd_t) - be consistent ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] akarbown merged pull request #19562: [operator] Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
akarbown merged pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] bartekkuncer commented on pull request #19562: Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-781241478
@mxnet-bot run ci [windows-cpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19562: Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-780381171
Jenkins CI successfully triggered : [centos-gpu, windows-cpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] kpuatamazon edited a comment on pull request #19562: [WIP]Integrate oneDNN layer normalization implementation
Posted by GitBox <gi...@apache.org>.
kpuatamazon edited a comment on pull request #19562:
URL: https://github.com/apache/incubator-mxnet/pull/19562#issuecomment-744344661
I've been using a c5.12xlarge `Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz`. Assume these are some sort of seconds?
We should at least do `-march=native` to see if it's just a matter of CPU support i.e. MXNet doesn't seem to enable AVX512 by default and one could add CPUID dispatch.
Might as well reshape to two dimensions with the axis preserved and everything else multiplied. The problem is identical for e.g. 100x28x10x10x10 and 280000x10. Also, those are some really small channels to layer normalize over.
Also, I feel like the optimal assembly implementation would benefit from a different ordering of the input tensor to allow for pure vertical adds whereas layer normalization is currently setup for horizontal adds. I can certainly see how a JIT will do better at e.g. 1000x3 where multiple problems share the same vector. But oddly that's where marian is doing better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org