You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "László Bodor (Jira)" <ji...@apache.org> on 2024/03/08 08:33:00 UTC
[jira] [Resolved] (HIVE-28051) LLAP: cleanup local folders on startup and periodically
[ https://issues.apache.org/jira/browse/HIVE-28051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
László Bodor resolved HIVE-28051.
---------------------------------
Resolution: Fixed
> LLAP: cleanup local folders on startup and periodically
> -------------------------------------------------------
>
> Key: HIVE-28051
> URL: https://issues.apache.org/jira/browse/HIVE-28051
> Project: Hive
> Issue Type: Improvement
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> When LLAP daemon crashes, there might be leftovers in the hive llap local dirs, which are good candidates to be cleaned up on startup/regularly. Under normal circumstances, these are cleaned up, however, as the LLAP damons is responsible for that, it cannot help in the event of a crash, so it would make sense to make it able to clean up old files.
> A file listing of a problematic cluster where LLAP local dir is configured to /apps/llap/work:
> {code}
> du -sh /apps/llap/work/usercache/hive/appcache/*
> 261M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0005
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0043
> 28K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0049
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0052
> 2.0M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0060
> 5.9G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0069
> 1.3G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0072
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0083
> 1.2G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0085
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0087
> 356M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0089
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0091
> 429M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0095
> 48K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0096
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0097
> 166M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0099
> 1.8G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0100
> 369M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0101
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0102
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0103
> 2.8G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0104
> 16K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0105
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0106
> 592K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0107
> 2.9G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0108
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0109
> 60K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0110
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0111
> 1.6G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0112
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0113
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0114
> 11G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0115
> 3.4G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0116
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0117
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0118
> 646M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0119
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0120
> 8.0K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0121
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0122
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0123
> 164K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0124
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0125
> 115M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0126
> 406M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0127
> 988K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0128
> 8.0K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0129
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0130
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0131
> 94M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0132
> 1.4G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0133
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0134
> 1.2G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0135
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0136
> 771M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0137
> 2.0G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0138
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0139
> 2.2G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0140
> 1.2G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0141
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0142
> 2.0G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0143
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0144
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0145
> 13G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0146
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0147
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0148
> 1.3G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0149
> 3.5G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0150
> 12G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0151
> 6.6G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0152
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0153
> 2.7M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0154
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0155
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0156
> 27G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0157
> 11G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0158
> 1.8M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0159
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0160
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0161
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0162
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0163
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0164
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0165
> 1.5G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0166
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0167
> 2.1M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0168
> 1.3G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0169
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0170
> 289M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0171
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0172
> 18M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0173
> 5.4G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0174
> 8.2G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0175
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0176
> 13G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0177
> 6.0G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0178
> 7.0G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0179
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0180
> 28K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0181
> 976M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0182
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0183
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0184
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0185
> 65M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0186
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0187
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0188
> 940K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0189
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0190
> 2.3G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0191
> 102M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0192
> 4.3G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0193
> 654M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0194
> 3.8G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0195
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0196
> 830M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0197
> 1.2M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0198
> 332M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0199
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0200
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0201
> 5.0G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0202
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0203
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0204
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0205
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0206
> 8.0K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0207
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0208
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0209
> 5.1G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0210
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0211
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0212
> 3.0G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0213
> 43G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0214
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0215
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0216
> 3.4M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0217
> 5.2G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0218
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0219
> 37M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0220
> 1.8M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0221
> 665M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0222
> 4.6G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0223
> 1.2G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0224
> 1.8G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0225
> 5.1G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0226
> 12K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0227
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0228
> 106M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0229
> 1.5G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0230
> 16G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0231
> 1.3G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0232
> 940K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0233
> 4.8G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0234
> 41M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0235
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0236
> 6.1G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0237
> 9.0G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0238
> 6.8G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0239
> 1.2M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0240
> 2.1G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0241
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0242
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0243
> 20K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0244
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0245
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0246
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0247
> 431M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0248
> 3.0G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0249
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0250
> 1.1G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0251
> 15G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0252
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0253
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0254
> 100K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0255
> 20K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0256
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0257
> 2.7G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0258
> 987M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0259
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0260
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0261
> 672M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0262
> 8.0K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0263
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0264
> 7.4G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0265
> 1.6G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0266
> 5.4M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0267
> 5.6G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0268
> 1000M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0269
> 13G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0270
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0271
> 169M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0272
> 3.0M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0273
> 5.7G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0274
> 628K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0275
> 2.0G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0276
> 2.3G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0277
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0278
> 12K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0279
> 95M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0280
> 5.3G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0281
> 32K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0282
> 2.5G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0283
> 244M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0284
> 4.4G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0285
> 3.5G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0286
> 5.7G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0287
> 1.3G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0288
> 15G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0289
> 848M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0290
> 19M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0291
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0292
> 38G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0293
> 326M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0294
> 2.2G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0295
> 4.1G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0296
> 38G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0297
> 444K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0298
> 28K /apps/llap/work/usercache/hive/appcache/application_1702138790934_0299
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0300
> 241M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0301
> 45G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0302
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0303
> 36M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0304
> 679M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0305
> 27G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0306
> 6.4G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0307
> 23M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0308
> 490M /apps/llap/work/usercache/hive/appcache/application_1702138790934_0309
> 0 /apps/llap/work/usercache/hive/appcache/application_1702138790934_0310
> 1.5G /apps/llap/work/usercache/hive/appcache/application_1702138790934_0311
> {code}
> regarding dates:
> {code}
> ls -lrth /apps/llap/work/usercache/hive/appcache/
> total 4.0K
> drwxr-xr-x 2 hive hive 6 Dec 29 11:17 application_1702138790934_0102
> drwxr-xr-x 3 hive hive 17 Jan 1 01:37 application_1702138790934_0107
> drwxr-xr-x 3 hive hive 17 Jan 1 09:04 application_1702138790934_0100
> drwxr-xr-x 2 hive hive 6 Jan 2 05:13 application_1702138790934_0087
> drwxr-xr-x 3 hive hive 17 Jan 2 05:14 application_1702138790934_0101
> drwxr-xr-x 3 hive hive 16 Jan 2 12:31 application_1702138790934_0109
> drwxr-xr-x 2 hive hive 6 Jan 2 12:50 application_1702138790934_0106
> drwxr-xr-x 3 hive hive 18 Jan 2 19:58 application_1702138790934_0043
> drwxr-xr-x 3 hive hive 17 Jan 2 20:05 application_1702138790934_0089
> drwxr-xr-x 4 hive hive 30 Jan 2 21:45 application_1702138790934_0072
> drwxr-xr-x 3 hive hive 16 Jan 2 21:56 application_1702138790934_0115
> drwxr-xr-x 3 hive hive 17 Jan 3 08:48 application_1702138790934_0096
> drwxr-xr-x 3 hive hive 17 Jan 3 08:56 application_1702138790934_0105
> drwxr-xr-x 3 hive hive 17 Jan 3 09:05 application_1702138790934_0060
> drwxr-xr-x 2 hive hive 6 Jan 3 09:08 application_1702138790934_0114
> drwxr-xr-x 2 hive hive 6 Jan 3 09:08 application_1702138790934_0113
> drwxr-xr-x 2 hive hive 6 Jan 3 09:28 application_1702138790934_0118
> drwxr-xr-x 2 hive hive 6 Jan 3 09:40 application_1702138790934_0117
> drwxr-xr-x 3 hive hive 17 Jan 3 11:51 application_1702138790934_0110
> drwxr-xr-x 2 hive hive 6 Jan 3 12:03 application_1702138790934_0091
> drwxr-xr-x 2 hive hive 6 Jan 3 12:05 application_1702138790934_0122
> drwxr-xr-x 3 hive hive 17 Jan 3 12:42 application_1702138790934_0095
> {code}
> so e.g. application_1702138790934_0069 has ~6G stale data from Jan3 (file listing was made at the end of January)
> these local files are typically the ones below (assuming that llap local dirs point to /apps/llap/work)
> {code}
> /apps/llap/work/usercache/hive/appcache/application_1707917402901_0001/3/output/attempt_1707917402901_0001_3_05_000002_0_10414:
> total 16
> drwxr-xr-x 2 hive hive 44 Feb 14 13:41 .
> drwxr-xr-x 63 hive hive 4096 Feb 14 13:41 ..
> -rw-r--r-- 1 hive hive 425 Feb 14 13:41 file.out
> -rw-r--r-- 1 hive hive 32 Feb 14 13:41 file.out.index
> /apps/llap/work/usercache/hive/appcache/application_1707917402901_0001/3/output/attempt_1707917402901_0001_3_05_000005_0_10416:
> total 16
> drwxr-xr-x 2 hive hive 44 Feb 14 13:41 .
> drwxr-xr-x 63 hive hive 4096 Feb 14 13:41 ..
> -rw-r--r-- 1 hive hive 383 Feb 14 13:41 file.out
> -rw-r--r-- 1 hive hive 32 Feb 14 13:41 file.out.index
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)