You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2022/11/04 16:53:51 UTC
[GitHub] [tvm] saurabh-shandilya opened a new issue, #13296: [Bug] TVM generates wrong results for ssd300_vgg16
saurabh-shandilya opened a new issue, #13296:
URL: https://github.com/apache/tvm/issues/13296
### Expected behavior
I am trying to run pytorch's ssd300_vgg16 model via TVM's VirtualMachine. I followed the instructions in https://github.com/apache/tvm/issues/10050 to overcome the initial problems and now the network runs fine. However the results are all wrong. They don't match the pytorch reference output and really way off. The code is based on mask-rcnn example provided here - https://tvm.apache.org/docs/how_to/deploy_models/deploy_object_detection_pytorch.html. I've tried targets llvm and cuda both and results are bad for both.
Pytorch reference output *************************
(tensor([[2.4417e+02, 2.2165e+02, 3.6217e+02, 2.9035e+02],
[2.9954e+02, 2.6913e+02, 4.6809e+02, 3.8972e+02],
[1.9778e+02, 1.9647e+02, 2.5620e+02, 3.2252e+02],
[2.7028e+01, 2.0530e+02, 1.0533e+02, 3.5209e+02],
[3.4073e+02, 2.0001e+02, 4.3230e+02, 3.7326e+02],
[1.4415e+02, 2.0797e+02, 1.6326e+02, 2.6495e+02],
[2.2417e+02, 2.0850e+02, 2.7180e+02, 2.6460e+02],
[1.3226e+02, 2.1215e+02, 1.4580e+02, 2.5770e+02],
[1.2254e+02, 2.1110e+02, 1.3493e+02, 2.5713e+02],
[8.0295e+01, 2.1339e+02, 9.8900e+01, 2.6089e+02],
[1.7209e+02, 1.9866e+02, 2.1437e+02, 3.1835e+02],
[8.8501e+01, 2.1271e+02, 1.0255e+02, 2.5695e+02],
[0.0000e+00, 2.3197e+02, 2.0041e+01, 2.5666e+02],
[3.1612e+02, 1.8432e+01, 3.4849e+02, 4.9051e+01],
[1.1125e+02, 2.1442e+02, 1.2177e+02, 2.5141e+02],
[1.5562e+02, 1.3672e+02, 1.8261e+02, 1.7365e+02],
[1.5833e+02, 2.1220e+02, 1.7274e+02, 2.6225e+02],
[1.6501e+02, 2.0565e+02, 2.0250e+02, 2.7752e+02],
[2.2764e+01, 2.3100e+02, 4.4465e+01, 2.6462e+02],
[7.0415e+01, 2.1830e+02, 8.1617e+01, 2.5416e+02],
[1.2488e+02, 2.0766e+02, 1.5897e+02, 2.6208e+02],
[3.8607e+01, 2.0426e+02, 7.9962e+01, 2.6993e+02],
[8.2036e+00, 2.3470e+02, 3.2965e+01, 2.5809e+02],
[3.5287e+02, 1.8980e+02, 4.0411e+02, 2.5538e+02],
[3.5970e+01, 2.1084e+02, 5.3370e+01, 2.5875e+02],
[1.6251e+02, 2.0354e+02, 2.5836e+02, 2.5754e+02],
[1.7772e+01, 1.9012e+02, 2.2498e+02, 3.6969e+02],
[1.2379e+02, 2.1581e+02, 1.3302e+02, 2.4182e+02],
[1.9756e+02, 1.9395e+02, 2.1564e+02, 2.2745e+02],
[6.1785e+01, 3.5410e+00, 8.5044e+01, 3.0485e+01],
[1.2857e+02, 1.2811e+01, 1.5189e+02, 6.3675e+01],
[1.7101e+02, 2.0569e+02, 1.8972e+02, 2.4509e+02],
[1.7452e+02, 2.0692e+02, 1.8599e+02, 2.2773e+02],
[9.3479e+01, 2.1707e+02, 1.0389e+02, 2.4277e+02],
[1.4877e+02, 2.0663e+02, 1.8458e+02, 2.6856e+02],
[1.3445e+02, 2.1610e+02, 1.4422e+02, 2.4222e+02],
[4.7994e+02, 1.9391e+02, 4.9662e+02, 2.4127e+02],
[1.7817e+02, 2.0492e+02, 1.8938e+02, 2.2127e+02],
[5.0056e+01, 2.0740e+02, 7.0734e+01, 2.4314e+02],
[7.1253e+01, 2.1274e+02, 8.1918e+01, 2.3385e+02],
[1.8651e+02, 2.0381e+02, 2.0017e+02, 2.2575e+02],
[8.5727e+01, 2.1584e+02, 9.7125e+01, 2.3654e+02],
[1.4228e+02, 1.4399e+02, 1.6137e+02, 1.7483e+02],
[1.5047e+02, 2.1326e+02, 1.5857e+02, 2.2819e+02],
[2.1438e+02, 1.9680e+02, 2.2627e+02, 2.1194e+02],
[1.5042e+02, 2.2576e+02, 2.7536e+02, 2.7424e+02],
[1.9586e+02, 2.0598e+02, 2.6463e+02, 2.8514e+02],
[1.8147e+02, 2.0504e+02, 2.0199e+02, 2.4421e+02],
[1.5039e+02, 2.1429e+02, 1.6195e+02, 2.3567e+02],
[1.3733e+02, 2.1751e+02, 1.4331e+02, 2.2806e+02],
[1.6395e+02, 2.1343e+02, 1.7102e+02, 2.2759e+02],
[3.4431e+02, 2.0538e+02, 3.6409e+02, 2.3753e+02],
[7.8160e+01, 2.1411e+02, 8.7881e+01, 2.2854e+02],
[3.5844e+02, 2.0203e+02, 3.7546e+02, 2.2970e+02],
[1.2746e+02, 2.1575e+02, 1.3565e+02, 2.3239e+02],
[2.1206e+02, 1.9874e+02, 2.2869e+02, 2.2567e+02],
[4.3505e+02, 2.8848e+02, 4.9010e+02, 3.7540e+02],
[1.6241e+02, 2.1783e+02, 1.7228e+02, 2.4317e+02],
[1.2555e+02, 2.1764e+02, 1.3199e+02, 2.2843e+02],
[4.1499e+01, 2.1141e+02, 5.5898e+01, 2.3539e+02],
[1.1285e+02, 2.0742e+02, 1.4543e+02, 2.5972e+02],
[8.0152e+01, 1.7333e+02, 9.0892e+01, 1.9259e+02],
[3.0948e+02, 2.0909e+02, 4.2511e+02, 2.5345e+02],
[9.6563e+01, 2.1657e+02, 1.0712e+02, 2.3372e+02],
[5.7967e+01, 2.0704e+02, 1.0758e+02, 3.1279e+02],
[2.2488e+02, 1.9845e+02, 2.7140e+02, 2.9928e+02],
[1.3866e+02, 2.1548e+02, 1.4698e+02, 2.3203e+02],
[1.8180e+02, 1.8349e+02, 2.8806e+02, 3.3766e+02],
[2.0762e+02, 2.0114e+02, 2.3317e+02, 2.5045e+02],
[3.4167e+02, 2.0053e+02, 3.9220e+02, 3.2931e+02],
[1.5281e+02, 2.0886e+02, 1.6065e+02, 2.2091e+02],
[8.4341e+01, 2.1279e+02, 9.2747e+01, 2.2836e+02],
[7.7487e+01, 1.5585e+02, 9.2186e+01, 1.8693e+02],
[9.2294e+01, 2.1602e+02, 1.0089e+02, 2.2844e+02],
[1.6553e+02, 2.1531e+02, 1.7464e+02, 2.3194e+02],
[2.2985e+01, 2.0752e+02, 7.4032e+01, 3.2143e+02],
[3.0615e+02, 1.8120e+01, 3.5680e+02, 6.7932e+01],
[1.6703e+02, 2.0833e+02, 1.7473e+02, 2.2049e+02],
[1.8375e+02, 2.0363e+02, 1.9445e+02, 2.1595e+02],
[1.9073e+02, 2.0051e+02, 2.0614e+02, 2.1566e+02],
[9.6328e+01, 2.1541e+02, 1.0373e+02, 2.2783e+02],
[3.3845e+02, 2.1063e+02, 3.4878e+02, 2.2635e+02],
[2.2792e+02, 2.0006e+02, 2.3859e+02, 2.1143e+02],
[1.8884e+02, 1.9485e+02, 2.3018e+02, 2.5356e+02],
[1.1430e+02, 2.1921e+02, 1.2471e+02, 2.4620e+02],
[1.4178e+02, 2.1174e+02, 1.4770e+02, 2.2115e+02],
[1.4568e+02, 2.1502e+02, 1.5330e+02, 2.2734e+02],
[2.4827e+02, 1.8518e+02, 3.5682e+02, 2.2428e+02],
[2.0223e+02, 1.9769e+02, 2.2250e+02, 2.1517e+02],
[3.3993e+02, 1.7070e+02, 3.8588e+02, 2.6537e+02],
[1.8883e+02, 1.9777e+02, 2.0004e+02, 2.1264e+02],
[1.4006e+01, 1.8438e+02, 2.8332e+01, 2.0462e+02],
[3.3563e+02, 2.1111e+02, 3.5040e+02, 2.3479e+02],
[2.3329e+02, 2.0625e+02, 2.7655e+02, 2.3498e+02],
[5.4572e+01, 2.1008e+02, 7.4398e+01, 2.3224e+02],
[2.1197e+02, 1.9120e+02, 2.2927e+02, 2.0688e+02],
[3.5489e+02, 1.9519e+02, 4.1565e+02, 2.9707e+02],
[1.3652e+02, 2.0802e+02, 1.7046e+02, 3.0209e+02],
[3.4080e+02, 2.0861e+02, 3.5142e+02, 2.2133e+02],
[1.3292e+02, 2.1576e+02, 1.4012e+02, 2.2785e+02],
[2.8274e+02, 2.8520e+02, 3.4893e+02, 3.7641e+02],
[3.6473e+02, 1.8967e+02, 3.9806e+02, 2.2421e+02],
[2.8861e+02, 2.6854e+02, 3.7318e+02, 3.4846e+02],
[2.8996e+02, 3.1042e+02, 3.7356e+02, 3.8874e+02],
[1.6363e+01, 2.1840e+02, 5.2250e+01, 2.6814e+02],
[2.2071e+02, 2.1286e+02, 2.5107e+02, 2.5021e+02],
[1.3405e+02, 1.4743e+02, 1.5329e+02, 1.7750e+02],
[9.2970e+01, 2.0701e+02, 1.3279e+02, 2.5923e+02],
[2.2400e+02, 2.0320e+02, 2.4073e+02, 2.2553e+02],
[4.0754e+02, 2.8402e+02, 4.6464e+02, 3.7728e+02],
[2.2244e+02, 2.0799e+02, 2.9705e+02, 2.9214e+02],
[2.4218e+02, 2.1918e+02, 2.7036e+02, 2.7253e+02],
[2.8761e+02, 2.2680e+02, 4.2974e+02, 2.8141e+02],
[3.8448e+02, 2.0716e+02, 4.8258e+02, 2.5537e+02],
[1.9475e+02, 1.8607e+02, 2.1995e+02, 2.0691e+02],
[1.1393e+01, 1.7661e+02, 2.5979e+01, 2.0076e+02],
[1.5708e+02, 2.1535e+02, 1.6607e+02, 2.2792e+02],
[1.7086e+02, 1.5667e+02, 1.8398e+02, 1.7901e+02],
[2.9289e+01, 2.1676e+02, 4.0368e+01, 2.4376e+02],
[3.0202e+02, 2.2264e+02, 3.4987e+02, 2.4309e+02],
[9.5497e+01, 2.3061e+02, 2.2246e+02, 2.7316e+02],
[1.0788e+02, 2.0508e+02, 1.4094e+02, 3.1011e+02],
[2.0186e-02, 2.2156e+02, 4.2131e+01, 2.6417e+02],
[9.7750e+01, 2.0579e+02, 2.2026e+02, 2.4983e+02],
[2.2922e+02, 1.9557e+02, 2.4127e+02, 2.0628e+02],
[1.2902e+02, 2.1268e+02, 1.3497e+02, 2.2192e+02],
[1.7705e+02, 2.1076e+02, 1.9573e+02, 2.2958e+02],
[1.6944e+02, 2.1443e+02, 1.7913e+02, 2.2722e+02],
[3.0167e+02, 5.8253e+00, 3.6347e+02, 7.4312e+01],
[2.5227e+01, 2.1899e+02, 4.2790e+01, 2.5848e+02],
[1.2902e+02, 8.5037e+01, 1.4418e+02, 1.1917e+02],
[4.0324e+02, 2.3191e+02, 4.9387e+02, 2.8018e+02],
[9.5982e+01, 1.7625e+02, 1.0427e+02, 1.9008e+02],
[1.1440e+02, 2.1671e+02, 1.2393e+02, 2.3311e+02],
[2.7586e+02, 2.0602e+02, 2.9494e+02, 2.2460e+02],
[2.4362e+02, 2.1098e+02, 3.0343e+02, 2.6181e+02],
[2.5932e+02, 2.2132e+02, 2.9532e+02, 2.4362e+02],
[2.6578e+02, 2.0738e+02, 2.8359e+02, 2.2531e+02],
[5.3024e+01, 2.0659e+02, 6.7485e+01, 2.2767e+02],
[2.4036e+02, 1.9736e+02, 2.5319e+02, 2.0998e+02],
[4.9054e+02, 2.0318e+02, 4.9961e+02, 2.3729e+02],
[1.2857e+02, 1.2811e+01, 1.5189e+02, 6.3675e+01],
[1.9579e+02, 2.0361e+02, 4.0979e+02, 2.9071e+02],
[2.7404e+02, 2.1132e+02, 2.9425e+02, 2.3381e+02],
[4.6134e+01, 2.1251e+02, 6.1695e+01, 2.3054e+02],
[2.5538e+02, 2.0541e+02, 2.9752e+02, 2.2906e+02],
[3.3615e+02, 1.8595e+02, 3.8039e+02, 2.3592e+02],
[2.3864e+02, 2.0568e+02, 2.5589e+02, 2.2548e+02],
[1.1218e+02, 2.1543e+02, 1.2007e+02, 2.2793e+02],
[1.0378e+02, 2.1641e+02, 1.1373e+02, 2.2895e+02],
[1.2001e+02, 2.1615e+02, 1.2838e+02, 2.2855e+02],
[3.5987e+02, 1.9753e+02, 3.7335e+02, 2.1409e+02],
[4.6829e+02, 1.9821e+02, 4.8096e+02, 2.3969e+02],
[3.8286e+02, 1.7366e+02, 4.3008e+02, 2.3417e+02],
[3.0943e+01, 1.7724e+02, 4.0672e+01, 1.9769e+02],
[3.1755e+01, 1.8841e+02, 4.2735e+01, 2.0402e+02],
[3.8737e+02, 1.8791e+02, 4.3494e+02, 2.9256e+02],
[2.9364e+02, 2.0822e+02, 3.0508e+02, 2.2270e+02],
[2.9925e+02, 7.2756e+00, 3.4284e+02, 5.7217e+01],
[4.1634e+02, 3.2460e+02, 4.9210e+02, 3.8555e+02],
[7.3738e+01, 1.7783e+02, 8.4503e+01, 1.9035e+02],
[1.3443e+02, 1.3395e+02, 1.8067e+02, 1.7858e+02],
[1.4563e+01, 2.2917e+02, 2.6468e+01, 2.5742e+02],
[5.6669e+01, 2.2636e+02, 1.9480e+02, 4.2692e+02],
[3.6371e+02, 1.9748e+02, 3.9239e+02, 2.4168e+02],
[1.5971e+02, 1.8670e+02, 2.0270e+02, 2.3739e+02],
[7.7828e+01, 2.2510e+02, 8.9454e+01, 2.4263e+02],
[2.2024e+02, 2.0717e+02, 2.5804e+02, 2.3412e+02],
[1.0226e+02, 2.1950e+02, 2.1449e+02, 3.2868e+02],
[1.1308e+02, 1.8250e+02, 1.1954e+02, 1.9664e+02],
[1.7721e+02, 2.0155e+02, 1.8551e+02, 2.1441e+02],
[1.7191e+02, 2.0483e+02, 1.8056e+02, 2.1556e+02],
[3.9647e+02, 1.8331e+02, 4.1821e+02, 2.2845e+02],
[7.7429e+01, 2.1386e+02, 1.1451e+02, 2.6637e+02],
[1.4211e+02, 1.8934e+02, 2.3429e+02, 2.9392e+02],
[3.4784e+02, 1.9834e+02, 3.6106e+02, 2.1497e+02],
[2.5161e+02, 2.0948e+02, 2.6821e+02, 2.2636e+02],
[3.1872e+02, 2.0555e+02, 3.3413e+02, 2.2391e+02],
[2.0045e+01, 2.3110e+02, 1.3883e+02, 2.7545e+02],
[2.8869e+02, 2.1542e+02, 3.5679e+02, 2.5840e+02],
[3.1345e+02, 1.6837e+02, 3.5717e+02, 2.6241e+02],
[4.6216e+01, 1.9745e+02, 9.2922e+01, 2.5125e+02],
[2.7416e+02, 2.2358e+02, 3.0673e+02, 2.4237e+02],
[3.9641e+01, 1.2524e+02, 4.5479e+01, 1.3559e+02],
[1.2633e+02, 2.3179e+02, 1.3796e+02, 2.5708e+02],
[8.6657e+01, 2.0912e+02, 9.4916e+01, 2.2119e+02],
[3.2138e+02, 9.9774e+00, 3.5424e+02, 3.9627e+01],
[4.8211e+01, 2.1936e+02, 7.2619e+01, 2.7222e+02],
[2.1365e+02, 1.9896e+02, 2.3527e+02, 2.1488e+02],
[2.9446e+02, 2.2113e+02, 3.0608e+02, 2.3524e+02],
[8.4641e+01, 2.2299e+02, 1.0146e+02, 2.4509e+02],
[1.8999e+02, 1.9031e+02, 2.0572e+02, 2.0601e+02],
[2.3058e+02, 2.3519e+02, 2.6048e+02, 2.7905e+02],
[1.3305e+02, 1.9908e+02, 1.7273e+02, 2.5639e+02],
[1.7619e+01, 8.3585e+01, 2.4199e+01, 1.0389e+02],
[9.8760e+01, 2.0997e+02, 1.0625e+02, 2.2053e+02],
[1.3335e+01, 2.2519e+02, 2.5920e+01, 2.5378e+02],
[1.4663e+01, 2.1846e+02, 2.5995e+01, 2.4036e+02],
[5.6244e+01, 2.2121e+02, 7.6548e+01, 2.4442e+02],
[3.1505e+02, 2.1335e+02, 3.4563e+02, 2.2909e+02]],
grad_fn=<StackBackward0>),
tensor([0.9727, 0.9540, 0.8788, 0.8229, 0.8031, 0.7723, 0.5995, 0.5492, 0.5456,
0.5138, 0.3374, 0.2810, 0.1931, 0.1798, 0.1736, 0.1671, 0.1607, 0.1527,
0.1512, 0.1445, 0.1230, 0.1188, 0.1182, 0.1161, 0.1160, 0.1159, 0.1152,
0.1151, 0.1129, 0.1125, 0.1120, 0.1094, 0.1079, 0.1076, 0.1070, 0.1059,
0.1059, 0.1057, 0.1047, 0.1025, 0.1004, 0.0993, 0.0989, 0.0981, 0.0975,
0.0968, 0.0963, 0.0961, 0.0941, 0.0940, 0.0936, 0.0934, 0.0930, 0.0925,
0.0918, 0.0908, 0.0902, 0.0902, 0.0901, 0.0899, 0.0893, 0.0889, 0.0874,
0.0872, 0.0866, 0.0861, 0.0860, 0.0858, 0.0858, 0.0851, 0.0849, 0.0845,
0.0839, 0.0836, 0.0833, 0.0833, 0.0832, 0.0832, 0.0829, 0.0821, 0.0812,
0.0810, 0.0808, 0.0807, 0.0805, 0.0802, 0.0799, 0.0799, 0.0799, 0.0798,
0.0796, 0.0789, 0.0787, 0.0785, 0.0785, 0.0785, 0.0781, 0.0780, 0.0779,
0.0778, 0.0775, 0.0773, 0.0770, 0.0768, 0.0767, 0.0764, 0.0764, 0.0760,
0.0756, 0.0756, 0.0755, 0.0755, 0.0750, 0.0749, 0.0749, 0.0748, 0.0746,
0.0746, 0.0745, 0.0744, 0.0742, 0.0740, 0.0739, 0.0738, 0.0734, 0.0731,
0.0730, 0.0727, 0.0724, 0.0723, 0.0721, 0.0721, 0.0717, 0.0713, 0.0713,
0.0711, 0.0707, 0.0707, 0.0700, 0.0695, 0.0691, 0.0689, 0.0688, 0.0687,
0.0686, 0.0684, 0.0680, 0.0676, 0.0675, 0.0672, 0.0672, 0.0671, 0.0671,
0.0669, 0.0666, 0.0662, 0.0662, 0.0656, 0.0655, 0.0654, 0.0649, 0.0649,
0.0648, 0.0645, 0.0645, 0.0644, 0.0640, 0.0640, 0.0640, 0.0638, 0.0637,
0.0636, 0.0636, 0.0635, 0.0633, 0.0633, 0.0630, 0.0626, 0.0624, 0.0624,
0.0624, 0.0622, 0.0621, 0.0620, 0.0614, 0.0614, 0.0613, 0.0612, 0.0611,
0.0609, 0.0609, 0.0608, 0.0608, 0.0607, 0.0607, 0.0606, 0.0605, 0.0605,
0.0603, 0.0601], grad_fn=<IndexBackward0>),
tensor([ 3, 2, 1, 1, 1, 1, 3, 1, 1, 1, 1, 1, 3, 1, 1, 10, 1, 1,
3, 1, 1, 1, 3, 1, 1, 3, 1, 1, 1, 85, 10, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 10, 1, 1, 3, 3, 1, 1, 1, 1, 1, 1, 1,
1, 1, 2, 1, 1, 1, 1, 10, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1,
10, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 3, 1, 1,
1, 10, 1, 3, 1, 1, 1, 1, 1, 1, 2, 1, 2, 2, 3, 3, 10, 1,
1, 2, 3, 3, 3, 3, 1, 10, 1, 10, 1, 3, 3, 1, 3, 3, 1, 1,
1, 1, 10, 1, 10, 3, 10, 1, 1, 3, 3, 1, 1, 1, 1, 1, 3, 1,
1, 3, 1, 1, 1, 1, 1, 1, 1, 1, 10, 10, 1, 1, 1, 2, 10, 10,
3, 1, 1, 1, 1, 3, 1, 10, 1, 1, 1, 1, 1, 1, 1, 1, 3, 3,
1, 1, 3, 10, 1, 1, 1, 1, 1, 1, 1, 1, 3, 1, 10, 1, 1, 1,
1, 3]))
### Actual behavior
boxes = array([[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0. , 0. , 28.269907 , 35.20406 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ],
[ 0.8428591, 0. , 19.342596 , 20.993477 ]], dtype=float32), <tvm.nd.NDArray shape=(120,), cuda(0)>
score:
array([0.00351303, 0.00333744, 0.00265404, 0.0024982 , 0.00247583,
0.00239549, 0.00220695, 0.00211583, 0.00201931, 0.00191237,
0.00188276, 0.00166292, 0.00166005, 0.00162623, 0.00150306,
0.00141596, 0.00134664, 0.00130558, 0.001287 , 0.00128138,
0.00121021, 0.00118284, 0.00116823, 0.00111245, 0.00106239,
0.00103616, 0.00102645, 0.00102028, 0.00101472, 0.00100795,
0.00100033, 0.00097783, 0.00094324, 0.00092061, 0.00091043,
0.0008875 , 0.0008824 , 0.0008745 , 0.00087165, 0.00086536,
0.0008624 , 0.00086121, 0.00085094, 0.00084903, 0.0008322 ,
0.00079518, 0.00077264, 0.0007638 , 0.00074951, 0.00074839,
0.00074206, 0.00073168, 0.00071763, 0.0007144 , 0.00070207,
0.00069418, 0.00065389, 0.00064997, 0.00064484, 0.00063796,
0.00062919, 0.00062576, 0.00062513, 0.00062389, 0.00062284,
0.00061086, 0.00060135, 0.00060078, 0.0005794 , 0.00057931,
0.00057921, 0.00057331, 0.00056992, 0.00056539, 0.00056354,
0.00056155, 0.00055505, 0.00054732, 0.00054591, 0.00054574,
0.00054472, 0.00053777, 0.0005349 , 0.00053452, 0.00053267,
0.00053003, 0.00052801, 0.00052588, 0.00051947, 0.00051378,
0.00050899, 0.00050629, 0.00050585, 0.00048741, 0.00048676,
0.00048307, 0.00048295, 0.00047997, 0.00047423, 0.00047306,
0.00047231, 0.00047014, 0.00046257, 0.00045252, 0.00044672,
0.00044112, 0.00044042, 0.00043576, 0.00043001, 0.00039631,
0.00038084, 0.00038023, 0.00037982, 0.0003795 , 0.00037916,
0.00037916, 0.0003784 , 0.00037825, 0.00037816, 0.00037725],
dtype=float32), <tvm.nd.NDArray shape=(120,), cuda(0)>
classes:
array([ 1, 84, 62, 1, 62, 84, 3, 85, 47, 3, 77, 72, 44, 10, 75, 85, 72,
51, 47, 67, 10, 44, 49, 46, 73, 31, 74, 46, 50, 64, 8, 86, 31, 32,
67, 81, 28, 15, 48, 33, 37, 16, 64, 27, 78, 6, 79, 9, 27, 86, 15,
28, 76, 8, 90, 57, 63, 60, 70, 55, 13, 2, 52, 38, 61, 16, 59, 43,
87, 41, 6, 43, 88, 54, 2, 53, 39, 13, 17, 58, 40, 18, 4, 14, 35,
41, 34, 35, 20, 42, 82, 7, 40, 42, 56, 5, 80, 37, 65, 36, 4, 36,
24, 21, 89, 22, 19, 25, 11, 23, 83, 30, 29, 68, 66, 71, 45, 12, 26,
69], dtype=int64)]
### Environment
tvm 0.9.dev0
Windows
torch 1.10.2
torchvision 0.11.3
Any environment details, such as: Operating System, TVM version, etc
### Steps to reproduce
Take the code from
https://tvm.apache.org/docs/how_to/deploy_models/deploy_object_detection_pytorch.html.
Change the torch_model to torchvision.models.detection.ssd300_vgg16
Preferably a minimal script to cause the issue to occur.
### Triage
Please refer to the list of label tags [here](https://github.com/apache/tvm/wiki/Issue-Triage-Labels) to find the relevant tags and add them below in a bullet format (example below).
* needs-triage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on issue #13296: [Bug] TVM generates wrong results for ssd300_vgg16
Posted by GitBox <gi...@apache.org>.
masahi commented on issue #13296:
URL: https://github.com/apache/tvm/issues/13296#issuecomment-1304319071
The following test, which runs only the backbone and the detection head of `ssd300_vgg16`, shows that the output is correct until that point:
https://gist.github.com/masahi/4df835222c514582fbeb345052733013
So I suspect the issue to be somewhere in the post processing, e.g. NMS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on issue #13296: [Bug] TVM generates wrong results for ssd300_vgg16
Posted by GitBox <gi...@apache.org>.
masahi commented on issue #13296:
URL: https://github.com/apache/tvm/issues/13296#issuecomment-1306469202
Hmm strange, usually when we hit a stack overflow issue, the fix is to replace the recursive visitor with the non-recursive one, which is called `MixedModeVisitor` in TVM. But `RelayToTIRTargetHook` is already using the mixed mode visitor https://github.com/apache/tvm/blob/985680ee1ae77ebe51f373df64063f8372e6cb6e/src/relay/transforms/target_hooks.cc#L55. So I don't expect the stack overflow to be coming from this pass.
Can you dig deeper into this pass to see if endless recursive visit is happening there somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on issue #13296: [Bug] TVM generates wrong results for ssd300_vgg16
Posted by GitBox <gi...@apache.org>.
masahi commented on issue #13296:
URL: https://github.com/apache/tvm/issues/13296#issuecomment-1305431118
Okay the bug was due to indexing by a boolean mask at https://github.com/pytorch/vision/blob/main/torchvision/models/detection/ssd.py#L433-L435, which we don't handle correctly. The PR https://github.com/apache/tvm/pull/13306 fixes this, and I confirmed that now the output is correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] saurabh-shandilya commented on issue #13296: [Bug] TVM generates wrong results for ssd300_vgg16
Posted by GitBox <gi...@apache.org>.
saurabh-shandilya commented on issue #13296:
URL: https://github.com/apache/tvm/issues/13296#issuecomment-1317799286
@masahi I moved to the linux build, so nothing more to be done on this. Thanks for quick resolution on this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] saurabh-shandilya commented on issue #13296: [Bug] TVM generates wrong results for ssd300_vgg16
Posted by GitBox <gi...@apache.org>.
saurabh-shandilya commented on issue #13296:
URL: https://github.com/apache/tvm/issues/13296#issuecomment-1306378686
@masahi Added logs based on your suggestion and it seems to be RelayToTIRTargetHook -
[15:32:52] K:\code\tvm_gpu\src\ir\transform.cc:431: RelayToTIRTargetHook
Traceback (most recent call last):
File "K:/code/tvm_gpu/python/object_detection_copy.py", line 201, in <module>
verify_model_in_relay(script_module,img,relay_target)
File "K:/code/tvm_gpu/python/object_detection_copy.py", line 148, in verify_model_in_relay
vm_exec = relay.vm.compile(mod, target=target, params=params)
File "K:\code\tvm_gpu\python\tvm\relay\backend\vm.py", line 79, in compile
compiler.lower(mod, target)
File "K:\code\tvm_gpu\python\tvm\relay\backend\vm.py", line 155, in lower
self._lower(mod, target, target_host)
File "K:\code\tvm_gpu\python\tvm\_ffi\_ctypes\packed_func.py", line 233, in __call__
ctypes.byref(ret_tcode),
OSError: exception: stack overflow
What's the way past this issue then?
For debugging is there a way to dump a specific instance of an op or for every op in the ir?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on issue #13296: [Bug] TVM generates wrong results for ssd300_vgg16
Posted by GitBox <gi...@apache.org>.
masahi commented on issue #13296:
URL: https://github.com/apache/tvm/issues/13296#issuecomment-1304138817
Yeah I noticed this issue before but didn't work on a fix then. I'll do it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac closed issue #13296: [Bug] TVM generates wrong results for ssd300_vgg16
Posted by GitBox <gi...@apache.org>.
comaniac closed issue #13296: [Bug] TVM generates wrong results for ssd300_vgg16
URL: https://github.com/apache/tvm/issues/13296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] saurabh-shandilya commented on issue #13296: [Bug] TVM generates wrong results for ssd300_vgg16
Posted by GitBox <gi...@apache.org>.
saurabh-shandilya commented on issue #13296:
URL: https://github.com/apache/tvm/issues/13296#issuecomment-1308156024
@masahi
I double checked it's coming from this pass only. At least code I am using has MixedModeVisitor::VisitExpr(const Expr& expr) which calls ExpandDataflow which seems like a recursive function to me. So I am not sure if this is entirely non-recursive visitor. There is another version of ExpandDataflow but I don't see that being called in this case.
I meant how to dump the tensor-output of a op at run-time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on issue #13296: [Bug] TVM generates wrong results for ssd300_vgg16
Posted by GitBox <gi...@apache.org>.
masahi commented on issue #13296:
URL: https://github.com/apache/tvm/issues/13296#issuecomment-1306127542
This is a common problem when using TVM on Windows. Can you identify which pass is causing the stack overflow? I think the stack overflow is coming from https://github.com/apache/tvm/blob/main/src/ir/transform.cc#L453, where different relay and TIR passes are applied. So you can add debug statements before / after this line to identify what pass is running when the stack overflow happens. The name of a pass can be printed by `LOG(INFO) << pass_info->name`.
> Would be good to understand the kind of tools /process to debug such/similar issues
For this one I did a very dumb thing: I knew that outputs were correct up to some point in the model. Starting from there, I repeated modifying `torchvision/models/detection/ssd.py` and gradually add more pytorch operations until the output became different. I noticed that the lines https://github.com/pytorch/vision/blob/main/torchvision/models/detection/ssd.py#L433-L435 were converted incorrectly - indexing by mask results in a dynamic shape while the converted Relay model didn't have any dynamism around there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] saurabh-shandilya commented on issue #13296: [Bug] TVM generates wrong results for ssd300_vgg16
Posted by GitBox <gi...@apache.org>.
saurabh-shandilya commented on issue #13296:
URL: https://github.com/apache/tvm/issues/13296#issuecomment-1306081809
@masahi Thanks for a quick response.
With this I am getting a stackoverflow error -
[10:01:49] K:\code\tvm_gpu\src\relay\transforms/let_list.h:54: Warning: letlist not used
Traceback (most recent call last):
File "K:/code/tvm_gpu/python/object_detection.py", line 201, in <module>
verify_model_in_relay(script_module,img,relay_target)
File "K:/code/tvm_gpu/python/object_detection.py", line 148, in verify_model_in_relay
vm_exec = relay.vm.compile(mod, target=target, params=params)
File "K:\code\tvm_gpu\python\tvm\relay\backend\vm.py", line 79, in compile
compiler.lower(mod, target)
File "K:\code\tvm_gpu\python\tvm\relay\backend\vm.py", line 155, in lower
self._lower(mod, target, target_host)
File "K:\code\tvm_gpu\python\tvm\_ffi\_ctypes\packed_func.py", line 233, in __call__
ctypes.byref(ret_tcode),
OSError: exception: stack overflow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org