You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "AngersZhuuuu (via GitHub)" <gi...@apache.org> on 2023/05/31 03:57:13 UTC
[GitHub] [incubator-celeborn] AngersZhuuuu opened a new pull request, #1526: [CELEBORN-621][BUG] Push merged data task timeout and mapended should also remove push states
AngersZhuuuu opened a new pull request, #1526:
URL: https://github.com/apache/incubator-celeborn/pull/1526
### What changes were proposed in this pull request?
Push merged data task timeout and mapended should also remove push states, or it will cause push task timeout
Worker side log shows Map eneded.
```
23/05/30 03:24:07,034 INFO [replicate-server-9-12] Worker: Receive push data from speculative task(shuffle application_1684207607609_2728653_1-2, map 5744, attempt 3), but this mapper has already been ended.
23/05/30 03:27:37,388 INFO [replicate-server-9-12] Worker: Receive push data from speculative task(shuffle application_1684207607609_2728653_1-2, map 5744, attempt 4), but this mapper has already been ended.
```
Client side timeout
```
23/05/30 03:24:35 WARN ShuffleClientImpl: It should never reach here!
23/05/30 03:33:27 ERROR InFlightRequestTracker: After waiting for 660000 ms, there are still 1 batches in flight for hostAndPushPort [10.169.32.167:9094, 10.169.40.155:9094, 10.169.44.214:9094, 10.169.26.228:9094, 10.169.41.168:9094, 10.169.42.148:9094, 10.169.34.89:9094, 10.169.40.211:9094, 10.169.48.234:9094, 10.169.45.23:9094, 10.169.29.105:9094, 10.169.27.225:9094, 10.169.41.19:9094, 10.169.11.231:9094, 10.169.33.147:9094, 10.169.49.34:9094, 10.169.33.219:9094, 10.169.41.224:9094, 10.169.29.91:9094, 10.169.33.164:9094, 10.169.27.208:9094, 10.169.41.152:9094, 10.169.32.84:9094, 10.169.41.207:9094, 10.169.41.36:9094, 10.169.44.86:9094, 10.169.21.26:9094, 10.169.42.165:9094, 10.169.34.144:9094, 10.169.11.80:9094, 10.169.29.24:9094, 10.169.11.87:9094, 10.169.40.227:9094, 10.169.41.20:9094, 10.169.27.153:9094, 10.169.27.93:9094, 10.169.26.82:9094, 10.169.34.161:9094, 10.169.33.28:9094, 10.169.40.99:9094, 10.169.29.163:9094, 10.169.32.151:9094, 10.169.29.98:9094, 10.169.28.214:9094,
10.169.34.170:9094, 10.169.26.91:9094, 10.169.11.134:9094, 10.169.32.26:9094, 10.169.11.71:9094, 10.169.32.160:9094, 10.169.33.95:9094, 10.169.32.223:9094, 10.169.41.102:9094, 10.169.11.222:9094, 10.169.41.86:9094, 10.169.34.15:9094, 10.169.29.82:9094, 10.169.34.169:9094, 10.169.34.97:9094, 10.169.21.203:9094, 10.169.32.92:9094, 10.169.33.211:9094, 10.169.33.156:9094, 10.169.11.149:9094, 10.169.27.217:9094, 10.169.34.168:9094, 10.169.32.101:9094, 10.169.41.232:9094, 10.169.27.216:9094, 10.169.34.81:9094, 10.169.28.100:9094, 10.169.33.37:9094, 10.169.29.146:9094, 10.169.41.215:9094, 10.169.41.144:9094, 10.169.28.213:9094, 10.169.34.208:9094, 10.169.28.158:9094, 10.169.11.72:9094, 10.169.40.163:9094, 10.169.32.214:9094, 10.169.42.157:9094, 10.169.28.149:9094, 10.169.29.164:9094, 10.169.34.152:9094, 10.169.40.219:9094, 10.169.21.93:9094, 10.169.34.32:9094, 10.169.33.21:9094, 10.169.32.102:9094, 10.169.11.79:9094, 10.169.29.83:9094, 10.169.34.39:9094, 10.169.32.25:9094, 10.169.33.36:90
94, 10.169.40.218:9094, 10.169.29.155:9094, 10.169.34.153:9094, 10.169.33.30:9094, 10.169.11.88:9094, 10.169.11.142:9094, 10.169.26.83:9094, 10.169.32.215:9094, 10.169.32.34:9094, 10.169.34.23:9094, 10.169.34.162:9094, 10.169.29.99:9094, 10.169.42.150:9094, 10.169.34.90:9094, 10.169.11.151:9094, 10.169.27.85:9094, 10.169.40.209:9094, 10.169.28.222:9094, 10.169.41.233:9094, 10.169.21.84:9094, 10.169.26.89:9094, 10.169.40.105:9094, 10.169.33.220:9094, 10.169.32.222:9094, 10.169.34.91:9094, 10.169.27.209:9094, 10.169.41.153:9094, 10.169.29.42:9094, 10.169.40.210:9094, 10.169.41.37:9094, 10.169.34.232:9094, 10.169.33.94:9094, 10.169.11.230:9094, 10.169.27.226:9094, 10.169.33.148:9094, 10.169.28.223:9094, 10.169.41.223:9094, 10.169.33.221:9094, 10.169.32.18:9094, 10.169.44.213:9094, 10.169.41.85:9094, 10.169.40.91:9094, 10.169.33.20:9094, 10.169.11.229:9094, 10.169.27.224:9094, 10.169.29.89:9094, 10.169.21.211:9094, 10.169.42.149:9094, 10.169.33.96:9094, 10.169.32.150:9094, 10.169.29.92:
9094, 10.169.32.33:9094, 10.169.28.157:9094, 10.169.41.216:9094, 10.169.26.81:9094, 10.169.33.155:9094, 10.169.41.35:9094, 10.169.40.93:9094, 10.169.41.143:9094, 10.169.33.105:9094, 10.169.21.25:9094, 10.169.32.207:9094, 10.169.42.164:9094, 10.169.32.232:9094, 10.169.32.229:9094, 10.169.32.99:9094, 10.169.40.220:9094, 10.169.28.167:9094, 10.169.34.154:9094, 10.169.40.217:9094, 10.169.21.204:9094, 10.169.32.144:9094, 10.169.33.218:9094, 10.169.29.156:9094, 10.169.33.38:9094, 10.169.33.165:9094, 10.169.34.224:9094, 10.169.27.234:9094, 10.169.45.204:9094, 10.169.40.147:9094, 10.169.41.231:9094, 10.169.28.22:9094, 10.169.26.221:9094, 10.169.45.197:9094, 10.169.34.83:9094, 10.169.11.19:9094, 10.169.28.215:9094, 10.169.29.34:9094, 10.169.41.93:9094, 10.169.11.70:9094, 10.169.41.27:9094, 10.169.29.32:9094, 10.169.40.83:9094, 10.169.33.213:9094, 10.169.45.15:9094, 10.169.45.202:9094, 10.169.41.158:9094, 10.169.34.98:9094, 10.169.29.201:9094, 10.169.27.232:9094, 10.169.29.97:9094, 10.169.34.
40:9094, 10.169.41.30:9094, 10.169.44.94:9094, 10.169.33.88:9094, 10.169.32.159:9094, 10.169.32.230:9094, 10.169.32.41:9094, 10.169.41.103:9094, 10.169.11.224:9094, 10.169.32.94:9094, 10.169.41.151:9094, 10.169.28.165:9094, 10.169.41.208:9094, 10.169.33.163:9094, 10.169.42.106:9094, 10.169.21.33:9094, 10.169.34.234:9094, 10.169.32.149:9094, 10.169.28.150:9094, 10.169.11.214:9094, 10.169.40.165:9094, 10.169.29.148:9094, 10.169.32.152:9094, 10.169.41.161:9094, 10.169.29.39:9094, 10.169.29.211:9094, 10.169.11.22:9094, 10.169.8.25:9094, 10.169.29.90:9094, 10.169.28.102:9094, 10.169.40.98:9094, 10.169.21.209:9094, 10.169.42.151:9094, 10.169.33.234:9094, 10.169.32.36:9094, 10.169.41.84:9094, 10.169.28.168:9094, 10.169.41.154:9094, 10.169.33.217:9094, 10.169.33.166:9094, 10.169.21.91:9094, 10.169.27.211:9094, 10.169.11.212:9094, 10.169.32.98:9094, 10.169.29.38:9094, 10.169.41.38:9094, 10.169.42.16:9094, 10.169.40.89:9094, 10.169.27.83:9094, 10.169.42.105:9094, 10.169.29.165:9094, 10.169.34
.92:9094, 10.169.29.204:9094, 10.169.40.208:9094, 10.169.40.169:9094, 10.169.33.93:9094, 10.169.34.163:9094, 10.169.45.25:9094, 10.169.42.101:9094, 10.169.33.162:9094, 10.169.40.230:9094, 10.169.29.31:9094, 10.169.41.150:9094, 10.169.11.90:9094, 10.169.29.169:9094, 10.169.21.86:9094, 10.169.42.155:9094, 10.169.28.20:9094, 10.169.26.88:9094, 10.169.44.84:9094, 10.169.11.65:9094, 10.169.40.100:9094, 10.169.21.95:9094, 10.169.29.88:9094, 10.169.11.77:9094, 10.169.45.198:9094, 10.169.45.195:9094, 10.169.44.96:9094, 10.169.28.161:9094, 10.169.28.164:9094, 10.169.11.140:9094, 10.169.21.98:9094, 10.169.28.19:9094, 10.169.34.167:9094, 10.169.42.158:9094, 10.169.11.156:9094, 10.169.21.19:9094, 10.169.44.212:9094, 10.169.27.218:9094, 10.169.27.215:9094, 10.169.45.16:9094, 10.169.41.209:9094, 10.169.33.149:9094, 10.169.41.80:9094, 10.169.29.200:9094, 10.169.41.100:9094, 10.169.32.91:9094, 10.169.32.145:9094, 10.169.32.89:9094, 10.169.34.22:9094, 10.169.26.92:9094, 10.169.44.219:9094, 10.169.11
.68:9094, 10.169.29.18:9094, 10.169.41.101:9094, 10.169.21.202:9094, 10.169.21.36:9094, 10.169.29.35:9094, 10.169.11.24:9094, 10.169.41.163:9094, 10.169.33.230:9094, 10.169.34.95:9094, 10.169.8.27:9094, 10.169.33.208:9094, 10.169.44.97:9094, 10.169.28.144:9094, 10.169.41.229:9094, 10.169.11.221:9094, 10.169.40.221:9094, 10.169.32.233:9094, 10.169.32.16:9094, 10.169.41.87:9094, 10.169.29.84:9094, 10.169.41.34:9094, 10.169.21.99:9094, 10.169.11.208:9094, 10.169.41.81:9094, 10.169.28.219:9094, 10.169.40.85:9094, 10.169.40.149:9094, 10.169.45.194:9094, 10.169.28.212:9094, 10.169.49.33:9094, 10.169.33.210:9094, 10.169.41.88:9094, 10.169.40.153:9094, 10.169.34.218:9094, 10.169.40.162:9094, 10.169.34.231:9094, 10.169.32.15:9094, 10.169.26.234:9094, 10.169.34.29:9094, 10.169.34.222:9094, 10.169.34.209:9094, 10.169.34.42:9094, 10.169.34.96:9094, 10.169.8.28:9094, 10.169.33.40:9094, 10.169.40.228:9094, 10.169.32.82:9094, 10.169.33.27:9094, 10.169.29.145:9094, 10.169.45.22:9094, 10.169.42.17:9
094, 10.169.26.230:9094, 10.169.41.170:9094, 10.169.29.25:9094, 10.169.33.103:9094, 10.169.26.85:9094, 10.169.34.215:9094, 10.169.29.152:9094, 10.169.40.156:9094, 10.169.42.20:9094, 10.169.44.87:9094, 10.169.33.80:9094, 10.169.21.89:9094, 10.169.34.150:9094, 10.169.26.227:9094, 10.169.33.34:9094, 10.169.48.233:9094, 10.169.27.94:9094, 10.169.29.207:9094, 10.169.40.106:9094, 10.169.8.21:9094, 10.169.29.149:9094, 10.169.27.80:9094, 10.169.29.106:9094, 10.169.32.35:9094, 10.169.21.42:9094, 10.169.42.168:9094, 10.169.40.224:9094, 10.169.34.147:9094, 10.169.11.91:9094, 10.169.21.85:9094, 10.169.11.81:9094, 10.169.41.225:9094, 10.169.11.133:9094, 10.169.27.231:9094, 10.169.21.29:9094, 10.169.32.39:9094, 10.169.21.32:9094, 10.169.28.151:9094, 10.169.11.150:9094, 10.169.34.157:9094, 10.169.27.228:9094, 10.169.29.210:9094, 10.169.40.214:9094, 10.169.32.42:9094, 10.169.45.26:9094, 10.169.11.215:9094, 10.169.27.84:9094, 10.169.21.92:9094, 10.169.11.74:9094, 10.169.28.148:9094, 10.169.29.15:909
4, 10.169.42.145:9094, 10.169.32.226:9094, 10.169.21.79:9094, 10.169.45.19:9094, 10.169.44.215:9094, 10.169.27.90:9094, 10.169.29.159:9094, 10.169.33.146:9094, 10.169.44.90:9094, 10.169.11.225:9094, 10.169.40.79:9094, 10.169.11.84:9094, 10.169.32.161:9094, 10.169.21.82:9094, 10.169.33.24:9094, 10.169.41.28:9094, 10.169.29.81:9094, 10.169.41.31:9094, 10.169.34.82:9094, 10.169.44.80:9094, 10.169.32.158:9094, 10.169.32.213:9094, 10.169.34.36:9094, 10.169.11.153:9094, 10.169.40.82:9094, 10.169.45.203:9094, 10.169.40.103:9094, 10.169.41.160:9094, 10.169.33.90:9094, 10.169.29.162:9094, 10.169.8.24:9094, 10.169.34.79:9094, 10.169.33.169:9094, 10.169.11.21:9094, 10.169.33.224:9094, 10.169.41.147:9094, 10.169.40.166:9094, 10.169.33.97:9094, 10.169.34.16:9094, 10.169.33.214:9094, 10.169.11.18:9094, 10.169.41.157:9094, 10.169.33.106:9094, 10.169.32.95:9094, 10.169.21.39:9094, 10.169.21.205:9094, 10.169.40.234:9094, 10.169.41.94:9094, 10.169.41.104:9094, 10.169.34.225:9094, 10.169.34.26:9094, 1
0.169.32.168:9094, 10.169.33.87:9094, 10.169.34.99:9094, 10.169.40.146:9094, 10.169.41.18:9094, 10.169.11.28:9094, 10.169.21.88:9094, 10.169.34.27:9094, 10.169.33.16:9094, 10.169.40.87:9094, 10.169.11.92:9094, 10.169.11.130:9094, 10.169.32.105:9094, 10.169.29.86:9094, 10.169.28.154:9094, 10.169.32.22:9094, 10.169.33.33:9094, 10.169.33.152:9094, 10.169.11.226:9094, 10.169.27.81:9094, 10.169.41.219:9094, 10.169.27.213:9094, 10.169.29.151:9094, 10.169.41.41:9094, 10.169.42.18:9094, 10.169.11.75:9094, 10.169.32.38:9094, 10.169.44.91:9094, 10.169.41.98:9094, 10.169.21.200:9094, 10.169.27.97:9094, 10.169.41.106:9094, 10.169.21.31:9094, 10.169.28.210:9094, 10.169.33.168:9094, 10.169.34.221:9094, 10.169.41.156:9094, 10.169.32.80:9094, 10.169.21.38:9094, 10.169.28.104:9094, 10.169.26.233:9094, 10.169.44.209:9094, 10.169.29.28:9094, 10.169.33.159:9094, 10.169.28.226:9094, 10.169.40.167:9094, 10.169.41.212:9094, 10.169.48.229:9094, 10.169.33.83:9094, 10.169.34.94:9094, 10.169.42.161:9094, 10.1
69.34.148:9094, 10.169.21.22:9094, 10.169.40.223:9094, 10.169.11.209:9094, 10.169.33.102:9094, 10.169.28.217:9094, 10.169.11.137:9094, 10.169.40.151:9094, 10.169.32.163:9094, 10.169.34.85:9094, 10.169.29.206:9094, 10.169.44.218:9094, 10.169.32.96:9094, 10.169.33.215:9094, 10.169.33.41:9094, 10.169.34.19:9094, 10.169.29.20:9094, 10.169.41.15:9094, 10.169.33.160:9094, 10.169.28.162:9094, 10.169.40.96:9094, 10.169.11.218:9094, 10.169.34.106:9094, 10.169.40.80:9094, 10.169.33.161:9094, 10.169.27.221:9094, 10.169.44.83:9094, 10.169.27.150:9094, 10.169.33.99:9094, 10.169.41.33:9094, 10.169.11.83:9094, 10.169.40.232:9094, 10.169.29.21:9094, 10.169.11.210:9094, 10.169.28.163:9094, 10.169.40.158:9094, 10.169.32.164:9094, 10.169.40.152:9094, 10.169.44.98:9094, 10.169.26.225:9094, 10.169.29.36:9094, 10.169.21.40:9094, 10.169.45.206:9094, 10.169.34.212:9094, 10.169.26.224:9094, 10.169.11.138:9094, 10.169.40.159:9094, 10.169.33.167:9094, 10.169.41.82:9094, 10.169.28.218:9094, 10.169.34.86:9094,
10.169.27.230:9094, 10.169.32.146:9094, 10.169.41.24:9094, 10.169.41.91:9094, 10.169.11.129:9094, 10.169.40.168:9094, 10.169.44.82:9094, 10.169.40.231:9094, 10.169.34.228:9094, 10.169.28.209:9094, 10.169.21.30:9094, 10.169.11.211:9094, 10.169.32.155:9094, 10.169.28.105:9094, 10.169.45.27:9094, 10.169.32.88:9094, 10.169.8.22:9094, 10.169.11.217:9094, 10.169.40.143:9094, 10.169.41.164:9094, 10.169.40.95:9094, 10.169.33.207:9094, 10.169.11.25:9094, 10.169.33.151:9094, 10.169.11.227:9094, 10.169.29.166:9094, 10.169.40.207:9094, 10.169.41.40:9094, 10.169.27.96:9094, 10.169.41.105:9094, 10.169.34.229:9094, 10.169.32.219:9094, 10.169.42.152:9094, 10.169.34.164:9094, 10.169.27.156:9094, 10.169.40.88:9094, 10.169.33.17:9094, 10.169.32.154:9094, 10.169.29.87:9094, 10.169.21.208:9094, 10.169.33.233:9094, 10.169.11.131:9094, 10.169.32.104:9094, 10.169.41.221:9094, 10.169.29.168:9094, 10.169.41.218:9094, 10.169.33.15:9094, 10.169.40.86:9094, 10.169.32.81:9094, 10.169.41.25:9094, 10.169.32.217:90
94, 10.169.29.27:9094, 10.169.32.21:9094, 10.169.42.154:9094, 10.169.44.89:9094, 10.169.28.155:9094, 10.169.8.23:9094, 10.169.45.199:9094, 10.169.41.42:9094, 10.169.28.227:9094, 10.169.27.222:9094, 10.169.41.39:9094, 10.169.41.90:9094, 10.169.33.98:9094, 10.169.42.19:9094, 10.169.29.37:9094, 10.169.32.23:9094, 10.169.28.145:9094, 10.169.44.99:9094, 10.169.34.211:9094, 10.169.28.170:9094, 10.169.26.223:9094, 10.169.21.201:9094, 10.169.32.210:9094, 10.169.41.83:9094, 10.169.29.85:9094, 10.169.33.231:9094, 10.169.32.106:9094, 10.169.42.104:9094, 10.169.29.95:9094, 10.169.41.228:9094, 10.169.40.102:9094, 10.169.33.143:9094, 10.169.40.81:9094, 10.169.21.96:9094, 10.169.11.219:9094, 10.169.40.150:9094, 10.169.33.25:9094, 10.169.32.162:9094, 10.169.32.227:9094, 10.169.11.85:9094, 10.169.29.80:9094, 10.169.40.215:9094, 10.169.41.32:9094, 10.169.29.158:9094, 10.169.42.144:9094, 10.169.27.91:9094, 10.169.11.234:9094, 10.169.27.219:9094, 10.169.48.230:9094, 10.169.29.143:9094, 10.169.33.23:909
4, 10.169.41.226:9094, 10.169.32.209:9094, 10.169.48.92:9094, 10.169.29.19:9094, 10.169.11.136:9094, 10.169.34.35:9094, 10.169.11.67:9094, 10.169.42.162:9094, 10.169.21.23:9094, 10.169.33.223:9094, 10.169.34.166:9094, 10.169.41.211:9094, 10.169.21.206:9094, 10.169.33.91:9094, 10.169.28.17:9094, 10.169.44.211:9094, 10.169.27.98:9094, 10.169.33.145:9094, 10.169.27.229:9094, 10.169.42.159:9094, 10.169.27.214:9094, 10.169.44.216:9094, 10.169.34.17:9094, 10.169.28.225:9094, 10.169.32.212:9094, 10.169.34.156:9094, 10.169.11.154:9094, 10.169.33.81:9094, 10.169.27.88:9094, 10.169.26.231:9094, 10.169.42.169:9094, 10.169.34.93:9094, 10.169.34.20:9094, 10.169.41.148:9094, 10.169.26.86:9094, 10.169.29.161:9094, 10.169.32.153:9094, 10.169.26.84:9094, 10.169.28.224:9094, 10.169.34.146:9094, 10.169.40.225:9094, 10.169.29.170:9094, 10.169.29.26:9094, 10.169.28.15:9094, 10.169.40.90:9094, 10.169.32.170:9094, 10.169.34.87:9094, 10.169.27.155:9094, 10.169.11.27:9094, 10.169.41.149:9094, 10.169.41.210:
9094, 10.169.28.106:9094, 10.169.41.22:9094, 10.169.33.150:9094, 10.169.11.89:9094, 10.169.28.152:9094, 10.169.33.104:9094, 10.169.34.158:9094, 10.169.11.139:9094, 10.169.32.24:9094, 10.169.11.216:9094, 10.169.40.213:9094, 10.169.32.86:9094, 10.169.41.17:9094, 10.169.41.26:9094, 10.169.41.79:9094, 10.169.41.234:9094, 10.169.49.32:9094, 10.169.27.99:9094, 10.169.28.220:9094, 10.169.21.41:9094, 10.169.33.222:9094, 10.169.32.169:9094, 10.169.11.20:9094, 10.169.40.145:9094, 10.169.34.226:9094, 10.169.21.28:9094, 10.169.26.226:9094, 10.169.42.167:9094, 10.169.34.151:9094, 10.169.34.34:9094, 10.169.34.37:9094, 10.169.34.214:9094, 10.169.40.170:9094, 10.169.41.29:9094, 10.169.29.153:9094, 10.169.32.103:9094, 10.169.33.35:9094, 10.169.41.214:9094, 10.169.29.17:9094, 10.169.45.28:9094, 10.169.41.145:9094, 10.169.33.226:9094, 10.169.21.21:9094, 10.169.29.144:9094, 10.169.33.225:9094, 10.169.21.83:9094, 10.169.41.146:9094, 10.169.40.148:9094, 10.169.42.160:9094, 10.169.34.223:9094, 10.169.44.8
1:9094, 10.169.40.229:9094, 10.169.40.94:9094, 10.169.27.152:9094, 10.169.33.158:9094, 10.169.41.213:9094, 10.169.33.26:9094, 10.169.45.205:9094, 10.169.40.104:9094, 10.169.33.85:9094, 10.169.21.37:9094, 10.169.21.20:9094, 10.169.27.86:9094, 10.169.11.152:9094, 10.169.34.38:9094, 10.169.11.141:9094, 10.169.32.225:9094, 10.169.42.146:9094, 10.169.11.73:9094, 10.169.34.80:9094, 10.169.45.17:9094, 10.169.34.149:9094, 10.169.11.207:9094, 10.169.21.94:9094, 10.169.40.222:9094, 10.169.29.30:9094, 10.169.32.224:9094, 10.169.44.92:9094, 10.169.28.159:9094, 10.169.11.69:9094, 10.169.44.88:9094, 10.169.28.228:9094, 10.169.11.220:9094, 10.169.42.147:9094, 10.169.40.212:9094, 10.169.34.159:9094, 10.169.11.82:9094, 10.169.32.234:9094, 10.169.11.132:9094, 10.169.27.151:9094, 10.169.34.105:9094, 10.169.27.220:9094, 10.169.45.21:9094, 10.169.11.76:9094, 10.169.11.213:9094, 10.169.32.83:9094, 10.169.21.90:9094, 10.169.28.169:9094, 10.169.21.27:9094, 10.169.42.166:9094, 10.169.32.97:9094, 10.169.41.2
3:9094, 10.169.27.79:9094, 10.169.34.28:9094, 10.169.29.205:9094, 10.169.27.210:9094, 10.169.29.154:9094, 10.169.41.169:9094, 10.169.33.29:9094, 10.169.41.155:9094, 10.169.33.216:9094, 10.169.33.82:9094, 10.169.28.208:9094, 10.169.34.213:9094, 10.169.32.40:9094, 10.169.40.144:9094, 10.169.34.227:9094, 10.169.34.24:9094, 10.169.32.156:9094, 10.169.26.87:9094, 10.169.28.221:9094, 10.169.29.147:9094, 10.169.40.84:9094, 10.169.33.89:9094, 10.169.11.26:9094, 10.169.34.230:9094, 10.169.33.22:9094, 10.169.42.15:9094, 10.169.41.165:9094, 10.169.32.87:9094, 10.169.34.41:9094, 10.169.40.226:9094, 10.169.26.232:9094, 10.169.41.16:9094, 10.169.40.161:9094, 10.169.28.211:9094, 10.169.33.79:9094, 10.169.34.145:9094, 10.169.26.229:9094, 10.169.40.154:9094, 10.169.34.220:9094, 10.169.32.166:9094, 10.169.34.217:9094, 10.169.33.92:9094, 10.169.29.209:9094, 10.169.41.89:9094, 10.169.34.18:9094, 10.169.49.35:9094, 10.169.32.90:9094, 10.169.29.150:9094, 10.169.33.212:9094, 10.169.33.19:9094, 10.169.32.9
3:9094, 10.169.41.159:9094, 10.169.44.95:9094, 10.169.32.228:9094, 10.169.32.30:9094, 10.169.42.143:9094, 10.169.11.23:9094, 10.169.28.146:9094, 10.169.29.33:9094, 10.169.11.135:9094, 10.169.34.21:9094, 10.169.33.39:9094, 10.169.21.97:9094, 10.169.32.231:9094, 10.169.41.162:9094, 10.169.29.160:9094, 10.169.40.164:9094, 10.169.33.209:9094, 10.169.34.207:9094, 10.169.41.227:9094, 10.169.33.144:9094, 10.169.27.233:9094, 10.169.29.212:9094, 10.169.29.157:9094, 10.169.40.97:9094, 10.169.33.42:9094, 10.169.40.101:9094, 10.169.28.103:9094, 10.169.21.80:9094, 10.169.42.100:9094, 10.169.21.34:9094, 10.169.26.222:9094, 10.169.11.86:9094, 10.169.32.27:9094, 10.169.34.155:9094, 10.169.34.210:9094, 10.169.45.196:9094, 10.169.27.89:9094, 10.169.11.155:9094, 10.169.44.217:9094, 10.169.41.217:9094, 10.169.33.229:9094, 10.169.32.211:9094, 10.169.28.166:9094, 10.169.27.223:9094, 10.169.29.202:9094, 10.169.26.90:9094, 10.169.32.208:9094, 10.169.21.24:9094, 10.169.44.220:9094, 10.169.33.232:9094, 10.16
9.48.231:9094, 10.169.42.163:9094, 10.169.41.220:9094, 10.169.27.92:9094, 10.169.11.223:9094, 10.169.28.18:9094, 10.169.28.21:9094, 10.169.11.66:9094, 10.169.28.156:9094, 10.169.44.85:9094, 10.169.32.37:9094, 10.169.26.80:9094, 10.169.44.210:9094, 10.169.34.165:9094, 10.169.32.221:9094, 10.169.29.104:9094, 10.169.32.218:9094, 10.169.42.153:9094, 10.169.41.230:9094, 10.169.45.24:9094, 10.169.27.82:9094, 10.169.11.233:9094, 10.169.29.96:9094], which exceeds the current limit 0.
23/05/30 03:33:27 ERROR Executor: Exception in task 5744.4 in stage 3.0 (TID 10124)
com.aliyun.emr.rss.common.exception.RssIOException: wait timeout for task 2-5744-4
at com.aliyun.emr.rss.client.ShuffleClientImpl.limitZeroInFlight(ShuffleClientImpl.java:399)
at com.aliyun.emr.rss.client.ShuffleClientImpl.mapperEnd(ShuffleClientImpl.java:1026)
at org.apache.spark.shuffle.rss.SortBasedShuffleWriter.close(SortBasedShuffleWriter.java:269)
at org.apache.spark.shuffle.rss.SortBasedShuffleWriter.write(SortBasedShuffleWriter.java:166)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1532)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] AngersZhuuuu commented on pull request #1526: [CELEBORN-621][BUG] Push merged data task timeout and mapended should also remove push states
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #1526:
URL: https://github.com/apache/incubator-celeborn/pull/1526#issuecomment-1571230883
ping @waitinfuture @FMX @RexXiong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] AngersZhuuuu merged pull request #1526: [CELEBORN-621][BUG] Push merged data task timeout and mapended should also remove push states
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu merged PR #1526:
URL: https://github.com/apache/incubator-celeborn/pull/1526
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1526: [CELEBORN-621][BUG] Push merged data task timeout and mapended should also remove push states
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1526:
URL: https://github.com/apache/incubator-celeborn/pull/1526#issuecomment-1569467670
## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1526?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#1526](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1526?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (18784af) into [main](https://app.codecov.io/gh/apache/incubator-celeborn/commit/6d5dd50915ef9d70bceda4b65616f3a3feb90caf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (6d5dd50) will **decrease** coverage by `0.16%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## main #1526 +/- ##
==========================================
- Coverage 46.08% 45.92% -0.16%
==========================================
Files 158 158
Lines 9880 9880
Branches 967 967
==========================================
- Hits 4552 4536 -16
- Misses 5030 5043 +13
- Partials 298 301 +3
```
[see 1 file with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1526/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org