You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hive.apache.org by vg...@apache.org on 2017/12/07 03:56:04 UTC
[1/2] hive git commit: HIVE-18173 : Improve plans for correlated
subqueries with non-equi predicate (Vineet Garg, reviewed by Ashutosh Chauhan)
Repository: hive
Updated Branches:
refs/heads/master 4d2dc3513 -> b23c3299f
http://git-wip-us.apache.org/repos/asf/hive/blob/b23c3299/ql/src/test/results/clientpositive/spark/subquery_in.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/spark/subquery_in.q.out b/ql/src/test/results/clientpositive/spark/subquery_in.q.out
index e19240b..aada175 100644
--- a/ql/src/test/results/clientpositive/spark/subquery_in.q.out
+++ b/ql/src/test/results/clientpositive/spark/subquery_in.q.out
@@ -844,7 +844,6 @@ POSTHOOK: Input: default@src
97 val_97
98 val_98
98 val_98
-Warning: Shuffle Join JOIN[15][tables = [$hdt$_1, $hdt$_2]] in Work 'Reducer 4' is a cross product
PREHOOK: query: explain select * from src b where b.key in (select distinct key from src a where a.value > b.value)
PREHOOK: type: QUERY
POSTHOOK: query: explain select * from src b where b.key in (select distinct key from src a where a.value > b.value)
@@ -858,8 +857,7 @@ STAGE PLANS:
Spark
Edges:
Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 4 (PARTITION-LEVEL SORT, 2)
- Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 1), Reducer 6 (PARTITION-LEVEL SORT, 1)
- Reducer 6 <- Map 5 (GROUP, 2)
+ Reducer 4 <- Map 3 (GROUP, 2)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -868,50 +866,35 @@ STAGE PLANS:
alias: b
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: (key is not null and value is not null) (type: boolean)
+ predicate: key is not null (type: boolean)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: string)
Map 3
Map Operator Tree:
TableScan
alias: a
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: key is not null (type: boolean)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string), _col1 (type: string)
- Map 5
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: value is not null (type: boolean)
+ predicate: (key is not null and value is not null) (type: boolean)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: value (type: string)
+ keys: key (type: string), value (type: string)
mode: hash
- outputColumnNames: _col0
+ outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Reducer 2
Reduce Operator Tree:
@@ -919,55 +902,40 @@ STAGE PLANS:
condition map:
Left Semi Join 0 to 1
keys:
- 0 _col0 (type: string), _col1 (type: string)
- 1 _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 45832 Data size: 1019683 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 45832 Data size: 1019683 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ 0 _col0 (type: string)
+ 1 _col0 (type: string)
+ outputColumnNames: _col0, _col1, _col3
+ residual filter predicates: {(_col3 > _col1)}
+ Statistics: Num rows: 183 Data size: 1944 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: string), _col1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 183 Data size: 1944 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 183 Data size: 1944 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Reducer 4
Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0
- 1
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 125000 Data size: 2781000 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (_col1 > _col2) (type: boolean)
- Statistics: Num rows: 41666 Data size: 926985 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 41666 Data size: 926985 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 41666 Data size: 926985 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 41666 Data size: 926985 Basic stats: COMPLETE Column stats: NONE
- Reducer 6
- Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: string)
+ keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
- outputColumnNames: _col0
+ outputColumnNames: _col0, _col1
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string)
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: string)
Stage: Stage-0
Fetch Operator
@@ -975,7 +943,6 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join JOIN[15][tables = [$hdt$_1, $hdt$_2]] in Work 'Reducer 4' is a cross product
PREHOOK: query: select * from src b where b.key in (select distinct key from src a where a.value > b.value)
PREHOOK: type: QUERY
PREHOOK: Input: default@src
@@ -984,503 +951,6 @@ POSTHOOK: query: select * from src b where b.key in (select distinct key from sr
POSTHOOK: type: QUERY
POSTHOOK: Input: default@src
#### A masked pattern was here ####
-10 val_10
-100 val_100
-100 val_100
-103 val_103
-103 val_103
-104 val_104
-104 val_104
-105 val_105
-11 val_11
-111 val_111
-113 val_113
-113 val_113
-114 val_114
-116 val_116
-118 val_118
-118 val_118
-119 val_119
-119 val_119
-119 val_119
-12 val_12
-12 val_12
-120 val_120
-120 val_120
-125 val_125
-125 val_125
-126 val_126
-128 val_128
-128 val_128
-128 val_128
-129 val_129
-129 val_129
-131 val_131
-133 val_133
-134 val_134
-134 val_134
-136 val_136
-137 val_137
-137 val_137
-138 val_138
-138 val_138
-138 val_138
-138 val_138
-143 val_143
-145 val_145
-146 val_146
-146 val_146
-149 val_149
-149 val_149
-15 val_15
-15 val_15
-150 val_150
-152 val_152
-152 val_152
-153 val_153
-155 val_155
-156 val_156
-157 val_157
-158 val_158
-160 val_160
-162 val_162
-163 val_163
-164 val_164
-164 val_164
-165 val_165
-165 val_165
-166 val_166
-167 val_167
-167 val_167
-167 val_167
-168 val_168
-169 val_169
-169 val_169
-169 val_169
-169 val_169
-17 val_17
-170 val_170
-172 val_172
-172 val_172
-174 val_174
-174 val_174
-175 val_175
-175 val_175
-176 val_176
-176 val_176
-177 val_177
-178 val_178
-179 val_179
-179 val_179
-18 val_18
-18 val_18
-180 val_180
-181 val_181
-183 val_183
-186 val_186
-187 val_187
-187 val_187
-187 val_187
-189 val_189
-19 val_19
-190 val_190
-191 val_191
-191 val_191
-192 val_192
-193 val_193
-193 val_193
-193 val_193
-194 val_194
-195 val_195
-195 val_195
-196 val_196
-197 val_197
-197 val_197
-199 val_199
-199 val_199
-199 val_199
-2 val_2
-20 val_20
-200 val_200
-200 val_200
-201 val_201
-202 val_202
-203 val_203
-203 val_203
-205 val_205
-205 val_205
-207 val_207
-207 val_207
-208 val_208
-208 val_208
-208 val_208
-209 val_209
-209 val_209
-213 val_213
-213 val_213
-214 val_214
-216 val_216
-216 val_216
-217 val_217
-217 val_217
-218 val_218
-219 val_219
-219 val_219
-221 val_221
-221 val_221
-222 val_222
-223 val_223
-223 val_223
-224 val_224
-224 val_224
-226 val_226
-228 val_228
-229 val_229
-229 val_229
-230 val_230
-230 val_230
-230 val_230
-230 val_230
-230 val_230
-233 val_233
-233 val_233
-235 val_235
-237 val_237
-237 val_237
-238 val_238
-238 val_238
-239 val_239
-239 val_239
-24 val_24
-24 val_24
-241 val_241
-242 val_242
-242 val_242
-244 val_244
-247 val_247
-248 val_248
-249 val_249
-252 val_252
-255 val_255
-255 val_255
-256 val_256
-256 val_256
-257 val_257
-258 val_258
-26 val_26
-26 val_26
-260 val_260
-262 val_262
-263 val_263
-265 val_265
-265 val_265
-266 val_266
-27 val_27
-272 val_272
-272 val_272
-273 val_273
-273 val_273
-273 val_273
-274 val_274
-275 val_275
-277 val_277
-277 val_277
-277 val_277
-277 val_277
-278 val_278
-278 val_278
-28 val_28
-280 val_280
-280 val_280
-281 val_281
-281 val_281
-282 val_282
-282 val_282
-283 val_283
-284 val_284
-285 val_285
-286 val_286
-287 val_287
-288 val_288
-288 val_288
-289 val_289
-291 val_291
-292 val_292
-296 val_296
-298 val_298
-298 val_298
-298 val_298
-30 val_30
-302 val_302
-305 val_305
-306 val_306
-307 val_307
-307 val_307
-308 val_308
-309 val_309
-309 val_309
-310 val_310
-311 val_311
-311 val_311
-311 val_311
-315 val_315
-316 val_316
-316 val_316
-316 val_316
-317 val_317
-317 val_317
-318 val_318
-318 val_318
-318 val_318
-321 val_321
-321 val_321
-322 val_322
-322 val_322
-323 val_323
-325 val_325
-325 val_325
-327 val_327
-327 val_327
-327 val_327
-33 val_33
-331 val_331
-331 val_331
-332 val_332
-333 val_333
-333 val_333
-335 val_335
-336 val_336
-338 val_338
-339 val_339
-34 val_34
-341 val_341
-342 val_342
-342 val_342
-344 val_344
-344 val_344
-345 val_345
-348 val_348
-348 val_348
-348 val_348
-348 val_348
-348 val_348
-35 val_35
-35 val_35
-35 val_35
-351 val_351
-353 val_353
-353 val_353
-356 val_356
-360 val_360
-362 val_362
-364 val_364
-365 val_365
-366 val_366
-367 val_367
-367 val_367
-368 val_368
-369 val_369
-369 val_369
-369 val_369
-37 val_37
-37 val_37
-373 val_373
-374 val_374
-375 val_375
-377 val_377
-378 val_378
-379 val_379
-382 val_382
-382 val_382
-384 val_384
-384 val_384
-384 val_384
-386 val_386
-389 val_389
-392 val_392
-393 val_393
-394 val_394
-395 val_395
-395 val_395
-396 val_396
-396 val_396
-396 val_396
-397 val_397
-397 val_397
-399 val_399
-399 val_399
-4 val_4
-400 val_400
-401 val_401
-401 val_401
-401 val_401
-401 val_401
-401 val_401
-402 val_402
-403 val_403
-403 val_403
-403 val_403
-404 val_404
-404 val_404
-406 val_406
-406 val_406
-406 val_406
-406 val_406
-407 val_407
-409 val_409
-409 val_409
-409 val_409
-41 val_41
-411 val_411
-413 val_413
-413 val_413
-414 val_414
-414 val_414
-417 val_417
-417 val_417
-417 val_417
-418 val_418
-419 val_419
-42 val_42
-42 val_42
-421 val_421
-424 val_424
-424 val_424
-427 val_427
-429 val_429
-429 val_429
-43 val_43
-430 val_430
-430 val_430
-430 val_430
-431 val_431
-431 val_431
-431 val_431
-432 val_432
-435 val_435
-436 val_436
-437 val_437
-438 val_438
-438 val_438
-438 val_438
-439 val_439
-439 val_439
-44 val_44
-443 val_443
-444 val_444
-446 val_446
-448 val_448
-449 val_449
-452 val_452
-453 val_453
-454 val_454
-454 val_454
-454 val_454
-455 val_455
-457 val_457
-458 val_458
-458 val_458
-459 val_459
-459 val_459
-460 val_460
-462 val_462
-462 val_462
-463 val_463
-463 val_463
-466 val_466
-466 val_466
-466 val_466
-467 val_467
-468 val_468
-468 val_468
-468 val_468
-468 val_468
-469 val_469
-469 val_469
-469 val_469
-469 val_469
-469 val_469
-47 val_47
-470 val_470
-472 val_472
-475 val_475
-477 val_477
-478 val_478
-478 val_478
-479 val_479
-480 val_480
-480 val_480
-480 val_480
-481 val_481
-482 val_482
-483 val_483
-484 val_484
-485 val_485
-487 val_487
-489 val_489
-489 val_489
-489 val_489
-489 val_489
-490 val_490
-491 val_491
-492 val_492
-492 val_492
-493 val_493
-494 val_494
-495 val_495
-496 val_496
-497 val_497
-498 val_498
-498 val_498
-498 val_498
-5 val_5
-5 val_5
-5 val_5
-51 val_51
-51 val_51
-53 val_53
-54 val_54
-57 val_57
-58 val_58
-58 val_58
-64 val_64
-65 val_65
-66 val_66
-67 val_67
-67 val_67
-69 val_69
-70 val_70
-70 val_70
-70 val_70
-72 val_72
-72 val_72
-74 val_74
-76 val_76
-76 val_76
-77 val_77
-78 val_78
-8 val_8
-80 val_80
-82 val_82
-83 val_83
-83 val_83
-84 val_84
-84 val_84
-85 val_85
-86 val_86
-87 val_87
-9 val_9
-90 val_90
-90 val_90
-90 val_90
-92 val_92
-95 val_95
-95 val_95
-96 val_96
-97 val_97
-97 val_97
-98 val_98
-98 val_98
PREHOOK: query: select p_mfgr, p_name, p_size
from part
where part.p_size in
http://git-wip-us.apache.org/repos/asf/hive/blob/b23c3299/ql/src/test/results/clientpositive/spark/subquery_multi.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/spark/subquery_multi.q.out b/ql/src/test/results/clientpositive/spark/subquery_multi.q.out
index a4282df..edc1fb7 100644
--- a/ql/src/test/results/clientpositive/spark/subquery_multi.q.out
+++ b/ql/src/test/results/clientpositive/spark/subquery_multi.q.out
@@ -234,8 +234,8 @@ POSTHOOK: Input: default@part_null
17273 almond antique forest lavender goldenrod Manufacturer#3 Brand#35 PROMO ANODIZED TIN 14 JUMBO CASE 1190.27 along the
45261 almond aquamarine floral ivory bisque Manufacturer#4 Brand#42 SMALL PLATED STEEL 27 WRAP CASE 1206.26 careful
48427 almond antique violet mint lemon Manufacturer#4 Brand#42 PROMO POLISHED STEEL 39 SM CASE 1375.42 hely ironic i
-78486 almond azure blanched chiffon midnight Manufacturer#5 Brand#52 LARGE BRUSHED BRASS 23 MED BAG 1464.48 hely blith
78487 NULL Manufacturer#6 Brand#52 LARGE BRUSHED BRASS 23 MED BAG 1464.48 hely blith
+78486 almond azure blanched chiffon midnight Manufacturer#5 Brand#52 LARGE BRUSHED BRASS 23 MED BAG 1464.48 hely blith
192697 almond antique blue firebrick mint Manufacturer#5 Brand#52 MEDIUM BURNISHED TIN 31 LG DRUM 1789.69 ickly ir
Warning: Shuffle Join JOIN[27][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Work 'Reducer 3' is a cross product
PREHOOK: query: explain select * from part_null where p_name IN (select p_name from part_null) AND p_brand NOT IN (select p_name from part_null)
http://git-wip-us.apache.org/repos/asf/hive/blob/b23c3299/ql/src/test/results/clientpositive/spark/subquery_notin.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/spark/subquery_notin.q.out b/ql/src/test/results/clientpositive/spark/subquery_notin.q.out
index 0d12d0d..83b1b50 100644
--- a/ql/src/test/results/clientpositive/spark/subquery_notin.q.out
+++ b/ql/src/test/results/clientpositive/spark/subquery_notin.q.out
@@ -6950,7 +6950,6 @@ PREHOOK: type: DROPTABLE
POSTHOOK: query: drop table t1
POSTHOOK: type: DROPTABLE
Warning: Shuffle Join JOIN[13][tables = [$hdt$_1, $hdt$_2]] in Work 'Reducer 5' is a cross product
-Warning: Shuffle Join JOIN[31][tables = [$hdt$_2, $hdt$_3]] in Work 'Reducer 10' is a cross product
PREHOOK: query: explain select *
from src b
where b.key not in
@@ -6975,14 +6974,12 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 10 <- Map 9 (PARTITION-LEVEL SORT, 1), Reducer 13 (PARTITION-LEVEL SORT, 1)
- Reducer 11 <- Reducer 10 (GROUP, 2)
- Reducer 13 <- Map 12 (GROUP, 2)
+ Reducer 10 <- Map 9 (GROUP, 2)
Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 6 (PARTITION-LEVEL SORT, 2)
- Reducer 3 <- Reducer 11 (PARTITION-LEVEL SORT, 2), Reducer 2 (PARTITION-LEVEL SORT, 2)
+ Reducer 3 <- Reducer 10 (PARTITION-LEVEL SORT, 2), Reducer 2 (PARTITION-LEVEL SORT, 2)
Reducer 5 <- Map 4 (PARTITION-LEVEL SORT, 1), Reducer 8 (PARTITION-LEVEL SORT, 1)
Reducer 6 <- Reducer 5 (GROUP, 2)
- Reducer 8 <- Map 12 (GROUP, 2)
+ Reducer 8 <- Map 7 (GROUP, 2)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -7000,7 +6997,23 @@ STAGE PLANS:
Map-reduce partition columns: _col1 (type: string)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: string)
- Map 12
+ Map 4
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key > '9') (type: boolean)
+ Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: string), value (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: string), _col1 (type: string)
+ Map 7
Map Operator Tree:
TableScan
alias: b
@@ -7019,96 +7032,44 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Map 4
- Map Operator Tree:
- TableScan
- alias: a
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (key > '9') (type: boolean)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string), _col1 (type: string)
Map 9
Map Operator Tree:
TableScan
alias: a
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: (key > '9') (type: boolean)
+ predicate: ((key > '9') and value is not null) (type: boolean)
Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string), _col1 (type: string)
- Reducer 10
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0
- 1
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 41500 Data size: 923146 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (_col2 > _col1) (type: boolean)
- Statistics: Num rows: 13833 Data size: 307707 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string), _col2 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 13833 Data size: 307707 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: string), _col2 (type: string)
+ keys: key (type: string), value (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 13833 Data size: 307707 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 13833 Data size: 307707 Basic stats: COMPLETE Column stats: NONE
- Reducer 11
+ Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ Reducer 10
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 6916 Data size: 153842 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: _col0 is not null (type: boolean)
- Statistics: Num rows: 6916 Data size: 153842 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string), true (type: boolean)
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 6916 Data size: 153842 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 6916 Data size: 153842 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col2 (type: boolean)
- Reducer 13
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string)
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: string), _col2 (type: boolean)
Reducer 2
Reduce Operator Tree:
Join Operator
@@ -7120,20 +7081,21 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col3, _col4
Statistics: Num rows: 7607 Data size: 169226 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 7607 Data size: 169226 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col3 (type: bigint), _col4 (type: bigint)
+ value expressions: _col1 (type: string), _col3 (type: bigint), _col4 (type: bigint)
Reducer 3
Reduce Operator Tree:
Join Operator
condition map:
Left Outer Join 0 to 1
keys:
- 0 _col0 (type: string), _col1 (type: string)
- 1 _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1, _col3, _col4, _col7
+ 0 _col0 (type: string)
+ 1 _col0 (type: string)
+ outputColumnNames: _col0, _col1, _col3, _col4, _col6, _col7
+ residual filter predicates: {(_col1 > _col6)}
Statistics: Num rows: 8367 Data size: 186148 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (not CASE WHEN ((_col3 = 0)) THEN (false) WHEN (_col3 is null) THEN (false) WHEN (_col7 is not null) THEN (true) WHEN (_col0 is null) THEN (null) WHEN ((_col4 < _col3)) THEN (true) ELSE (false) END) (type: boolean)
@@ -7211,7 +7173,6 @@ STAGE PLANS:
ListSink
Warning: Shuffle Join JOIN[13][tables = [$hdt$_1, $hdt$_2]] in Work 'Reducer 5' is a cross product
-Warning: Shuffle Join JOIN[31][tables = [$hdt$_2, $hdt$_3]] in Work 'Reducer 10' is a cross product
PREHOOK: query: select *
from src b
where b.key not in
@@ -7232,67 +7193,85 @@ where b.key not in
POSTHOOK: type: QUERY
POSTHOOK: Input: default@src
#### A masked pattern was here ####
-0 val_0
-0 val_0
-0 val_0
10 val_10
+100 val_100
+100 val_100
103 val_103
103 val_103
-105 val_105
+104 val_104
+104 val_104
+111 val_111
+114 val_114
116 val_116
120 val_120
120 val_120
-125 val_125
-125 val_125
+126 val_126
+128 val_128
+128 val_128
+128 val_128
129 val_129
129 val_129
131 val_131
+133 val_133
134 val_134
134 val_134
-136 val_136
-143 val_143
145 val_145
+146 val_146
+146 val_146
149 val_149
149 val_149
-150 val_150
+15 val_15
+15 val_15
+152 val_152
+152 val_152
155 val_155
157 val_157
-158 val_158
-160 val_160
-163 val_163
-164 val_164
-164 val_164
-166 val_166
+169 val_169
+169 val_169
+169 val_169
+169 val_169
17 val_17
170 val_170
172 val_172
172 val_172
-180 val_180
-183 val_183
+178 val_178
+179 val_179
+179 val_179
+181 val_181
+187 val_187
+187 val_187
+187 val_187
189 val_189
19 val_19
-190 val_190
191 val_191
191 val_191
-193 val_193
-193 val_193
-193 val_193
+192 val_192
+194 val_194
195 val_195
195 val_195
-196 val_196
+199 val_199
+199 val_199
+199 val_199
+2 val_2
20 val_20
-205 val_205
-205 val_205
+200 val_200
+200 val_200
+201 val_201
+202 val_202
+203 val_203
+203 val_203
207 val_207
207 val_207
-209 val_209
-209 val_209
+208 val_208
+208 val_208
+208 val_208
213 val_213
213 val_213
+214 val_214
216 val_216
216 val_216
-217 val_217
-217 val_217
+218 val_218
+222 val_222
223 val_223
223 val_223
224 val_224
@@ -7303,79 +7282,78 @@ POSTHOOK: Input: default@src
233 val_233
233 val_233
235 val_235
-238 val_238
-238 val_238
+237 val_237
+237 val_237
239 val_239
239 val_239
24 val_24
24 val_24
241 val_241
244 val_244
-247 val_247
-248 val_248
255 val_255
255 val_255
+256 val_256
+256 val_256
+257 val_257
258 val_258
26 val_26
26 val_26
-260 val_260
-263 val_263
+262 val_262
265 val_265
265 val_265
266 val_266
-272 val_272
-272 val_272
+27 val_27
273 val_273
273 val_273
273 val_273
-274 val_274
+275 val_275
+278 val_278
+278 val_278
28 val_28
-281 val_281
-281 val_281
+280 val_280
+280 val_280
+282 val_282
+282 val_282
+284 val_284
+285 val_285
286 val_286
+287 val_287
+288 val_288
+288 val_288
291 val_291
-296 val_296
+292 val_292
298 val_298
298 val_298
298 val_298
30 val_30
302 val_302
305 val_305
-306 val_306
-307 val_307
-307 val_307
-308 val_308
-309 val_309
-309 val_309
-315 val_315
+310 val_310
+311 val_311
+311 val_311
+311 val_311
316 val_316
316 val_316
316 val_316
-317 val_317
-317 val_317
-318 val_318
-318 val_318
-318 val_318
321 val_321
321 val_321
-325 val_325
-325 val_325
+323 val_323
33 val_33
-331 val_331
-331 val_331
332 val_332
+333 val_333
+333 val_333
335 val_335
+336 val_336
+338 val_338
339 val_339
-342 val_342
-342 val_342
-345 val_345
+344 val_344
+344 val_344
353 val_353
353 val_353
356 val_356
360 val_360
-366 val_366
-367 val_367
-367 val_367
+362 val_362
+364 val_364
368 val_368
369 val_369
369 val_369
@@ -7383,124 +7361,101 @@ POSTHOOK: Input: default@src
37 val_37
37 val_37
373 val_373
-375 val_375
377 val_377
378 val_378
-379 val_379
-382 val_382
-382 val_382
384 val_384
384 val_384
384 val_384
386 val_386
-389 val_389
-394 val_394
+392 val_392
395 val_395
395 val_395
396 val_396
396 val_396
396 val_396
-399 val_399
-399 val_399
-400 val_400
-401 val_401
-401 val_401
-401 val_401
-401 val_401
-401 val_401
+397 val_397
+397 val_397
402 val_402
-406 val_406
-406 val_406
-406 val_406
-406 val_406
+404 val_404
+404 val_404
407 val_407
-41 val_41
-413 val_413
-413 val_413
-414 val_414
-414 val_414
+411 val_411
+417 val_417
+417 val_417
+417 val_417
+418 val_418
+419 val_419
42 val_42
42 val_42
+424 val_424
+424 val_424
+43 val_43
430 val_430
430 val_430
430 val_430
-431 val_431
-431 val_431
-431 val_431
+432 val_432
436 val_436
-44 val_44
-446 val_446
+437 val_437
+444 val_444
448 val_448
449 val_449
-452 val_452
+453 val_453
+454 val_454
+454 val_454
+454 val_454
+457 val_457
459 val_459
459 val_459
-462 val_462
-462 val_462
-466 val_466
-466 val_466
-466 val_466
-467 val_467
468 val_468
468 val_468
468 val_468
468 val_468
47 val_47
-472 val_472
-480 val_480
-480 val_480
-480 val_480
+477 val_477
+479 val_479
+481 val_481
482 val_482
-484 val_484
+483 val_483
485 val_485
-487 val_487
+489 val_489
+489 val_489
+489 val_489
+489 val_489
490 val_490
+492 val_492
+492 val_492
493 val_493
494 val_494
495 val_495
496 val_496
497 val_497
-498 val_498
-498 val_498
-498 val_498
-5 val_5
-5 val_5
-5 val_5
-51 val_51
-51 val_51
-54 val_54
57 val_57
-58 val_58
-58 val_58
65 val_65
-66 val_66
-69 val_69
-70 val_70
-70 val_70
-70 val_70
+67 val_67
+67 val_67
+72 val_72
+72 val_72
74 val_74
-77 val_77
-78 val_78
+76 val_76
+76 val_76
8 val_8
80 val_80
-84 val_84
-84 val_84
+82 val_82
85 val_85
+86 val_86
87 val_87
9 val_9
92 val_92
-95 val_95
-95 val_95
96 val_96
-100 val_100
-100 val_100
-104 val_104
-104 val_104
+97 val_97
+97 val_97
+0 val_0
+0 val_0
+0 val_0
+105 val_105
11 val_11
-111 val_111
113 val_113
113 val_113
-114 val_114
118 val_118
118 val_118
119 val_119
@@ -7508,36 +7463,32 @@ POSTHOOK: Input: default@src
119 val_119
12 val_12
12 val_12
-126 val_126
-128 val_128
-128 val_128
-128 val_128
-133 val_133
+125 val_125
+125 val_125
+136 val_136
137 val_137
137 val_137
138 val_138
138 val_138
138 val_138
138 val_138
-146 val_146
-146 val_146
-15 val_15
-15 val_15
-152 val_152
-152 val_152
+143 val_143
+150 val_150
153 val_153
156 val_156
+158 val_158
+160 val_160
162 val_162
+163 val_163
+164 val_164
+164 val_164
165 val_165
165 val_165
+166 val_166
167 val_167
167 val_167
167 val_167
168 val_168
-169 val_169
-169 val_169
-169 val_169
-169 val_169
174 val_174
174 val_174
175 val_175
@@ -7545,94 +7496,82 @@ POSTHOOK: Input: default@src
176 val_176
176 val_176
177 val_177
-178 val_178
-179 val_179
-179 val_179
18 val_18
18 val_18
-181 val_181
+180 val_180
+183 val_183
186 val_186
-187 val_187
-187 val_187
-187 val_187
-192 val_192
-194 val_194
+190 val_190
+193 val_193
+193 val_193
+193 val_193
+196 val_196
197 val_197
197 val_197
-199 val_199
-199 val_199
-199 val_199
-2 val_2
-200 val_200
-200 val_200
-201 val_201
-202 val_202
-203 val_203
-203 val_203
-208 val_208
-208 val_208
-208 val_208
-214 val_214
-218 val_218
+205 val_205
+205 val_205
+209 val_209
+209 val_209
+217 val_217
+217 val_217
219 val_219
219 val_219
221 val_221
221 val_221
-222 val_222
226 val_226
230 val_230
230 val_230
230 val_230
230 val_230
230 val_230
-237 val_237
-237 val_237
+238 val_238
+238 val_238
242 val_242
242 val_242
+247 val_247
+248 val_248
249 val_249
252 val_252
-256 val_256
-256 val_256
-257 val_257
-262 val_262
-27 val_27
-275 val_275
+260 val_260
+263 val_263
+272 val_272
+272 val_272
+274 val_274
277 val_277
277 val_277
277 val_277
277 val_277
-278 val_278
-278 val_278
-280 val_280
-280 val_280
-282 val_282
-282 val_282
+281 val_281
+281 val_281
283 val_283
-284 val_284
-285 val_285
-287 val_287
-288 val_288
-288 val_288
289 val_289
-292 val_292
-310 val_310
-311 val_311
-311 val_311
-311 val_311
+296 val_296
+306 val_306
+307 val_307
+307 val_307
+308 val_308
+309 val_309
+309 val_309
+315 val_315
+317 val_317
+317 val_317
+318 val_318
+318 val_318
+318 val_318
322 val_322
322 val_322
-323 val_323
+325 val_325
+325 val_325
327 val_327
327 val_327
327 val_327
-333 val_333
-333 val_333
-336 val_336
-338 val_338
+331 val_331
+331 val_331
34 val_34
341 val_341
-344 val_344
-344 val_344
+342 val_342
+342 val_342
+345 val_345
348 val_348
348 val_348
348 val_348
@@ -7642,93 +7581,115 @@ POSTHOOK: Input: default@src
35 val_35
35 val_35
351 val_351
-362 val_362
-364 val_364
365 val_365
+366 val_366
+367 val_367
+367 val_367
374 val_374
-392 val_392
+375 val_375
+379 val_379
+382 val_382
+382 val_382
+389 val_389
393 val_393
-397 val_397
-397 val_397
+394 val_394
+399 val_399
+399 val_399
4 val_4
+400 val_400
+401 val_401
+401 val_401
+401 val_401
+401 val_401
+401 val_401
403 val_403
403 val_403
403 val_403
-404 val_404
-404 val_404
+406 val_406
+406 val_406
+406 val_406
+406 val_406
409 val_409
409 val_409
409 val_409
-411 val_411
-417 val_417
-417 val_417
-417 val_417
-418 val_418
-419 val_419
+41 val_41
+413 val_413
+413 val_413
+414 val_414
+414 val_414
421 val_421
-424 val_424
-424 val_424
427 val_427
429 val_429
429 val_429
-43 val_43
-432 val_432
+431 val_431
+431 val_431
+431 val_431
435 val_435
-437 val_437
438 val_438
438 val_438
438 val_438
439 val_439
439 val_439
+44 val_44
443 val_443
-444 val_444
-453 val_453
-454 val_454
-454 val_454
-454 val_454
+446 val_446
+452 val_452
455 val_455
-457 val_457
458 val_458
458 val_458
460 val_460
+462 val_462
+462 val_462
463 val_463
463 val_463
+466 val_466
+466 val_466
+466 val_466
+467 val_467
469 val_469
469 val_469
469 val_469
469 val_469
469 val_469
470 val_470
+472 val_472
475 val_475
-477 val_477
478 val_478
478 val_478
-479 val_479
-481 val_481
-483 val_483
-489 val_489
-489 val_489
-489 val_489
-489 val_489
+480 val_480
+480 val_480
+480 val_480
+484 val_484
+487 val_487
491 val_491
-492 val_492
-492 val_492
+498 val_498
+498 val_498
+498 val_498
+5 val_5
+5 val_5
+5 val_5
+51 val_51
+51 val_51
53 val_53
+54 val_54
+58 val_58
+58 val_58
64 val_64
-67 val_67
-67 val_67
-72 val_72
-72 val_72
-76 val_76
-76 val_76
-82 val_82
+66 val_66
+69 val_69
+70 val_70
+70 val_70
+70 val_70
+77 val_77
+78 val_78
83 val_83
83 val_83
-86 val_86
+84 val_84
+84 val_84
90 val_90
90 val_90
90 val_90
-97 val_97
-97 val_97
+95 val_95
+95 val_95
98 val_98
98 val_98
http://git-wip-us.apache.org/repos/asf/hive/blob/b23c3299/ql/src/test/results/clientpositive/subquery_exists.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/subquery_exists.q.out b/ql/src/test/results/clientpositive/subquery_exists.q.out
index b6b31aa..898674b 100644
--- a/ql/src/test/results/clientpositive/subquery_exists.q.out
+++ b/ql/src/test/results/clientpositive/subquery_exists.q.out
@@ -1269,7 +1269,7 @@ STAGE PLANS:
0 _col0 (type: int)
1 _col0 (type: int)
outputColumnNames: _col0, _col1, _col3
- residual filter predicates: {(_col1 <> _col3)}
+ residual filter predicates: {(_col3 <> _col1)}
Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: int), _col1 (type: int)
http://git-wip-us.apache.org/repos/asf/hive/blob/b23c3299/ql/src/test/results/clientpositive/subquery_notexists.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/subquery_notexists.q.out b/ql/src/test/results/clientpositive/subquery_notexists.q.out
index a6175f8..dfe256f 100644
--- a/ql/src/test/results/clientpositive/subquery_notexists.q.out
+++ b/ql/src/test/results/clientpositive/subquery_notexists.q.out
@@ -510,7 +510,7 @@ POSTHOOK: Input: default@src
199 val_199
199 val_199
2 val_2
-Warning: Shuffle Join JOIN[13][tables = [$hdt$_1, $hdt$_2]] in Stage 'Stage-2:MAPRED' is a cross product
+Warning: Shuffle Join JOIN[11][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-1:MAPRED' is a cross product
PREHOOK: query: explain
select *
from src b
@@ -530,115 +530,40 @@ where not exists
)
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-4 is a root stage
- Stage-2 depends on stages: Stage-4
- Stage-3 depends on stages: Stage-2
- Stage-1 depends on stages: Stage-3
+ Stage-2 is a root stage
+ Stage-1 depends on stages: Stage-2
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-4
+ Stage: Stage-2
Map Reduce
Map Operator Tree:
TableScan
- alias: b
+ alias: a
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: key, value
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: ((value > 'val_2') and key is not null) (type: boolean)
+ Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: key (type: string), value (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-2
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: a
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (value > 'val_2') (type: boolean)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string), _col1 (type: string)
- TableScan
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string), _col1 (type: string)
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0
- 1
- outputColumnNames: _col0, _col1, _col2, _col3
- Statistics: Num rows: 41500 Data size: 923146 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: ((_col0 > _col2) and (_col3 <> _col1)) (type: boolean)
- Statistics: Num rows: 13833 Data size: 307707 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string), _col3 (type: string)
- outputColumnNames: _col2, _col3
- Statistics: Num rows: 13833 Data size: 307707 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col2 (type: string), _col3 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 13833 Data size: 307707 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-3
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 13833 Data size: 307707 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 6916 Data size: 153842 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string), true (type: boolean)
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 6916 Data size: 153842 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
table:
@@ -657,36 +582,34 @@ STAGE PLANS:
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ sort order:
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: string), _col1 (type: string)
TableScan
Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 6916 Data size: 153842 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col2 (type: boolean)
+ sort order:
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: string), _col1 (type: string), _col2 (type: boolean)
Reduce Operator Tree:
Join Operator
condition map:
Left Outer Join 0 to 1
keys:
- 0 _col0 (type: string), _col1 (type: string)
- 1 _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1, _col4
- Statistics: Num rows: 7607 Data size: 169226 Basic stats: COMPLETE Column stats: NONE
+ 0
+ 1
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4
+ residual filter predicates: {(_col2 > _col0)} {(_col1 <> _col3)}
+ Statistics: Num rows: 41500 Data size: 922896 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: _col4 is null (type: boolean)
- Statistics: Num rows: 3803 Data size: 84601 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20750 Data size: 461448 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 3803 Data size: 84601 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20750 Data size: 461448 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 3803 Data size: 84601 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20750 Data size: 461448 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
@@ -698,7 +621,7 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join JOIN[13][tables = [$hdt$_1, $hdt$_2]] in Stage 'Stage-2:MAPRED' is a cross product
+Warning: Shuffle Join JOIN[11][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-1:MAPRED' is a cross product
PREHOOK: query: select *
from src b
where not exists
[2/2] hive git commit: HIVE-18173 : Improve plans for correlated
subqueries with non-equi predicate (Vineet Garg, reviewed by Ashutosh Chauhan)
Posted by vg...@apache.org.
HIVE-18173 : Improve plans for correlated subqueries with non-equi predicate (Vineet Garg, reviewed by Ashutosh Chauhan)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/b23c3299
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/b23c3299
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/b23c3299
Branch: refs/heads/master
Commit: b23c3299fb91954b7f100c003f07ee8c0a3f8353
Parents: 4d2dc35
Author: Vineet Garg <vg...@apache.com>
Authored: Wed Dec 6 19:55:14 2017 -0800
Committer: Vineet Garg <vg...@apache.com>
Committed: Wed Dec 6 19:55:14 2017 -0800
----------------------------------------------------------------------
.../test/resources/testconfiguration.properties | 1 +
.../calcite/rules/HiveRelDecorrelator.java | 96 ++-
.../hadoop/hive/ql/parse/CalcitePlanner.java | 167 ++--
.../test/queries/clientpositive/subquery_corr.q | 12 +
.../clientpositive/llap/subquery_corr.q.out | 797 +++++++++++++++++++
.../clientpositive/llap/subquery_exists.q.out | 2 +-
.../clientpositive/llap/subquery_in.q.out | 615 +-------------
.../llap/subquery_in_having.q.out | 6 +-
.../clientpositive/llap/subquery_notin.q.out | 118 +--
.../clientpositive/spark/subquery_exists.q.out | 2 +-
.../clientpositive/spark/subquery_in.q.out | 610 +-------------
.../clientpositive/spark/subquery_multi.q.out | 2 +-
.../clientpositive/spark/subquery_notin.q.out | 613 +++++++-------
.../clientpositive/subquery_exists.q.out | 2 +-
.../clientpositive/subquery_notexists.q.out | 127 +--
15 files changed, 1391 insertions(+), 1779 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/b23c3299/itests/src/test/resources/testconfiguration.properties
----------------------------------------------------------------------
diff --git a/itests/src/test/resources/testconfiguration.properties b/itests/src/test/resources/testconfiguration.properties
index cca1055..c910712 100644
--- a/itests/src/test/resources/testconfiguration.properties
+++ b/itests/src/test/resources/testconfiguration.properties
@@ -754,6 +754,7 @@ minillaplocal.query.files=\
vector_between_in.q,\
offset_limit_ppd_optimizer.q,\
cluster.q,\
+ subquery_corr.q,\
subquery_in.q,\
subquery_multi.q,\
subquery_scalar.q,\
http://git-wip-us.apache.org/repos/asf/hive/blob/b23c3299/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
----------------------------------------------------------------------
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
index d1fe49c..98d140f 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
@@ -104,7 +104,6 @@ import org.apache.calcite.util.ReflectiveVisitor;
import org.apache.calcite.util.Stacks;
import org.apache.calcite.util.Util;
import org.apache.calcite.util.mapping.Mappings;
-import org.apache.hadoop.hive.ql.metadata.Hive;
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelShuttleImpl;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveAggregate;
import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter;
@@ -1131,7 +1130,8 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// we need to keep predicate kind e.g. EQUAL or NOT EQUAL
// so that later while decorrelating LogicalCorrelate appropriate join predicate
// is generated
- def.setPredicateKind((SqlKind)((Pair)e.getNode()).getValue());
+ def.setPredicateKind((SqlOperator) ((Pair)((Pair)e.getNode()).getValue()).getKey());
+ def.setIsLeft((boolean)((Pair)((Pair) e.getNode()).getValue()).getValue());
map.put(def, (Integer)((Pair) e.getNode()).getKey());
}
}
@@ -1170,30 +1170,39 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
* and if found, throws a {@link Util.FoundOne}. */
private void findCorrelationEquivalent(CorRef correlation, RexNode e)
throws Util.FoundOne {
- switch (e.getKind()) {
- // TODO: for now only EQUAL and NOT EQUAL corr predicates are optimized
- //optimize rest of the predicates
- case NOT_EQUALS:
- if((boolean)valueGen.peek()) {
- // we will need value generator
- break;
- }
- case EQUALS:
- final RexCall call = (RexCall) e;
- final List<RexNode> operands = call.getOperands();
- if (references(operands.get(0), correlation)
- && operands.get(1) instanceof RexInputRef) {
- throw new Util.FoundOne(Pair.of(((RexInputRef) operands.get(1)).getIndex(), e.getKind()));
- }
- if (references(operands.get(1), correlation)
- && operands.get(0) instanceof RexInputRef) {
- throw new Util.FoundOne(Pair.of(((RexInputRef) operands.get(0)).getIndex(), e.getKind()));
- }
- break;
+ if(e instanceof RexCall){
+ switch (e.getKind()) {
case AND:
for (RexNode operand : ((RexCall) e).getOperands()) {
findCorrelationEquivalent(correlation, operand);
}
+ default:
+ final RexCall call = (RexCall) e;
+ final List<RexNode> operands = call.getOperands();
+ if(operands.size() == 2) {
+ if (references(operands.get(0), correlation)
+ && operands.get(1) instanceof RexInputRef) {
+ // if call isn't EQUAL type and it has been determined that value generate might be
+ // required we should rather generate value generator
+ if(e.getKind() != SqlKind.EQUALS && (boolean)valueGen.peek()) {
+ return;
+ }
+ throw new Util.FoundOne(Pair.of(((RexInputRef) operands.get(1)).getIndex(),
+ Pair.of(((RexCall) e).getOperator(), true)));
+ }
+ if (references(operands.get(1), correlation)
+ && operands.get(0) instanceof RexInputRef) {
+ // if call isn't EQUAL type and it has been determined that value generate might be
+ // required we should rather generate value generator
+ if(e.getKind() != SqlKind.EQUALS && (boolean)valueGen.peek()) {
+ return;
+ }
+ throw new Util.FoundOne(Pair.of(((RexInputRef) operands.get(0)).getIndex(),
+ Pair.of(((RexCall) e).getOperator(), false)));
+ }
+ break;
+ }
+ }
}
}
@@ -1426,23 +1435,21 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
}
final int newLeftPos = leftFrame.oldToNewOutputs.get(corDef.field);
final int newRightPos = rightOutput.getValue();
- if(corDef.getPredicateKind() == SqlKind.NOT_EQUALS) {
+ SqlOperator callOp = corDef.getPredicateKind() == null ?
+ SqlStdOperatorTable.EQUALS: corDef.getPredicateKind();
+ if(corDef.isLeft) {
conditions.add(
- rexBuilder.makeCall(SqlStdOperatorTable.NOT_EQUALS,
+ rexBuilder.makeCall(callOp,
RexInputRef.of(newLeftPos, newLeftOutput),
new RexInputRef(newLeftFieldCount + newRightPos,
newRightOutput.get(newRightPos).getType())));
-
}
else {
- assert(corDef.getPredicateKind() == null
- || corDef.getPredicateKind() == SqlKind.EQUALS);
conditions.add(
- rexBuilder.makeCall(SqlStdOperatorTable.EQUALS,
- RexInputRef.of(newLeftPos, newLeftOutput),
+ rexBuilder.makeCall(callOp,
new RexInputRef(newLeftFieldCount + newRightPos,
- newRightOutput.get(newRightPos).getType())));
-
+ newRightOutput.get(newRightPos).getType()),
+ RexInputRef.of(newLeftPos, newLeftOutput)));
}
// remove this cor var from output position mapping
@@ -1921,9 +1928,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// there is support of not equal
@Override public RexNode visitCall(final RexCall call) {
if(!valueGenerator) {
- switch (call.getKind()) {
- case EQUALS:
- case NOT_EQUALS:
+ if(call.getOperands().size() == 2) {
final List<RexNode> operands = new ArrayList<>(call.operands);
RexNode o0 = operands.get(0);
RexNode o1 = operands.get(1);
@@ -3018,12 +3023,18 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
static class CorDef implements Comparable<CorDef> {
public final CorrelationId corr;
public final int field;
- private SqlKind predicateKind;
+
+ private SqlOperator predicateKind;
+ // this indicates if corr var is left operand of rex call or not
+ // this is used in decorrelate(logical correlate) to appropriately
+ // create Rex node expression
+ private boolean isLeft;
CorDef(CorrelationId corr, int field) {
this.corr = corr;
this.field = field;
this.predicateKind = null;
+ this.isLeft=false;
}
@Override public String toString() {
@@ -3048,13 +3059,24 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
}
return Integer.compare(field, o.field);
}
- public SqlKind getPredicateKind() {
+
+ public SqlOperator getPredicateKind() {
return predicateKind;
}
- public void setPredicateKind(SqlKind predKind) {
+
+ public void setPredicateKind(SqlOperator predKind) {
this.predicateKind = predKind;
}
+
+ public boolean getIsLeft() {
+ return this.isLeft;
+ }
+
+ public void setIsLeft(boolean isLeft) {
+ this.isLeft = isLeft;
+ }
+
}
/** A map of the locations of
http://git-wip-us.apache.org/repos/asf/hive/blob/b23c3299/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
----------------------------------------------------------------------
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java b/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
index 76c82e2..ba64f97 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
@@ -1355,7 +1355,7 @@ public class CalcitePlanner extends SemanticAnalyzer {
RelNode calciteGenPlan = null;
RelNode calcitePreCboPlan = null;
RelNode calciteOptimizedPlan = null;
- subqueryId = 0;
+ subqueryId = -1;
/*
* recreate cluster, so that it picks up the additional traitDef
@@ -2582,104 +2582,106 @@ public class CalcitePlanner extends SemanticAnalyzer {
private void subqueryRestrictionCheck(QB qb, ASTNode searchCond, RelNode srcRel,
boolean forHavingClause, Set<ASTNode> corrScalarQueries,
Set<ASTNode> scalarQueriesWithAggNoWinNoGby) throws SemanticException {
- List<ASTNode> subQueriesInOriginalTree = SubQueryUtils.findSubQueries(searchCond);
-
- ASTNode clonedSearchCond = (ASTNode) SubQueryUtils.adaptor.dupTree(searchCond);
- List<ASTNode> subQueries = SubQueryUtils.findSubQueries(clonedSearchCond);
- for(int i=0; i<subQueriesInOriginalTree.size(); i++){
- //we do not care about the transformation or rewriting of AST
- // which following statement does
- // we only care about the restriction checks they perform.
- // We plan to get rid of these restrictions later
- int sqIdx = qb.incrNumSubQueryPredicates();
- ASTNode originalSubQueryAST = subQueriesInOriginalTree.get(i);
-
- ASTNode subQueryAST = subQueries.get(i);
- //SubQueryUtils.rewriteParentQueryWhere(clonedSearchCond, subQueryAST);
- Boolean orInSubquery = new Boolean(false);
- Integer subqueryCount = new Integer(0);
- ObjectPair<Boolean, Integer> subqInfo = new ObjectPair<Boolean, Integer>(false, 0);
-
- ASTNode outerQueryExpr = (ASTNode) subQueryAST.getChild(2);
-
- if (outerQueryExpr != null && outerQueryExpr.getType() == HiveParser.TOK_SUBQUERY_EXPR ) {
-
- throw new CalciteSubquerySemanticException(ErrorMsg.UNSUPPORTED_SUBQUERY_EXPRESSION.getMsg(
- outerQueryExpr, "IN/NOT IN subqueries are not allowed in LHS"));
- }
+ List<ASTNode> subQueriesInOriginalTree = SubQueryUtils.findSubQueries(searchCond);
+
+ ASTNode clonedSearchCond = (ASTNode) SubQueryUtils.adaptor.dupTree(searchCond);
+ List<ASTNode> subQueries = SubQueryUtils.findSubQueries(clonedSearchCond);
+ for(int i=0; i<subQueriesInOriginalTree.size(); i++){
+ //we do not care about the transformation or rewriting of AST
+ // which following statement does
+ // we only care about the restriction checks they perform.
+ // We plan to get rid of these restrictions later
+ int sqIdx = qb.incrNumSubQueryPredicates();
+ ASTNode originalSubQueryAST = subQueriesInOriginalTree.get(i);
+
+ ASTNode subQueryAST = subQueries.get(i);
+ //SubQueryUtils.rewriteParentQueryWhere(clonedSearchCond, subQueryAST);
+ Boolean orInSubquery = new Boolean(false);
+ Integer subqueryCount = new Integer(0);
+ ObjectPair<Boolean, Integer> subqInfo = new ObjectPair<Boolean, Integer>(false, 0);
+
+ ASTNode outerQueryExpr = (ASTNode) subQueryAST.getChild(2);
+
+ if (outerQueryExpr != null && outerQueryExpr.getType() == HiveParser.TOK_SUBQUERY_EXPR) {
+
+ throw new CalciteSubquerySemanticException(
+ ErrorMsg.UNSUPPORTED_SUBQUERY_EXPRESSION.getMsg(
+ outerQueryExpr, "IN/NOT IN subqueries are not allowed in LHS"));
+ }
- QBSubQuery subQuery = SubQueryUtils.buildSubQuery(qb.getId(), sqIdx, subQueryAST,
- originalSubQueryAST, ctx);
+ QBSubQuery subQuery = SubQueryUtils.buildSubQuery(qb.getId(), sqIdx, subQueryAST,
+ originalSubQueryAST, ctx);
- RowResolver inputRR = relToHiveRR.get(srcRel);
+ RowResolver inputRR = relToHiveRR.get(srcRel);
- String havingInputAlias = null;
+ String havingInputAlias = null;
- boolean [] subqueryConfig = {false, false};
- subQuery.subqueryRestrictionsCheck(inputRR, forHavingClause,
- havingInputAlias, subqueryConfig);
- if(subqueryConfig[0]) {
- corrScalarQueries.add(originalSubQueryAST);
- }
- if(subqueryConfig[1]) {
- scalarQueriesWithAggNoWinNoGby.add(originalSubQueryAST);
- }
+ boolean [] subqueryConfig = {false, false};
+ subQuery.subqueryRestrictionsCheck(inputRR, forHavingClause,
+ havingInputAlias, subqueryConfig);
+ if(subqueryConfig[0]) {
+ corrScalarQueries.add(originalSubQueryAST);
+ }
+ if(subqueryConfig[1]) {
+ scalarQueriesWithAggNoWinNoGby.add(originalSubQueryAST);
+ }
}
}
private boolean genSubQueryRelNode(QB qb, ASTNode node, RelNode srcRel, boolean forHavingClause,
Map<ASTNode, RelNode> subQueryToRelNode) throws SemanticException {
- Set<ASTNode> corrScalarQueriesWithAgg = new HashSet<ASTNode>();
- Set<ASTNode> scalarQueriesWithAggNoWinNoGby= new HashSet<ASTNode>();
- //disallow subqueries which HIVE doesn't currently support
- subqueryRestrictionCheck(qb, node, srcRel, forHavingClause, corrScalarQueriesWithAgg,
- scalarQueriesWithAggNoWinNoGby);
- Deque<ASTNode> stack = new ArrayDeque<ASTNode>();
- stack.push(node);
+ Set<ASTNode> corrScalarQueriesWithAgg = new HashSet<ASTNode>();
+ Set<ASTNode> scalarQueriesWithAggNoWinNoGby= new HashSet<ASTNode>();
+ //disallow subqueries which HIVE doesn't currently support
+ subqueryRestrictionCheck(qb, node, srcRel, forHavingClause, corrScalarQueriesWithAgg,
+ scalarQueriesWithAggNoWinNoGby);
+ Deque<ASTNode> stack = new ArrayDeque<ASTNode>();
+ stack.push(node);
- boolean isSubQuery = false;
+ boolean isSubQuery = false;
- while (!stack.isEmpty()) {
- ASTNode next = stack.pop();
+ while (!stack.isEmpty()) {
+ ASTNode next = stack.pop();
- switch(next.getType()) {
- case HiveParser.TOK_SUBQUERY_EXPR:
+ switch(next.getType()) {
+ case HiveParser.TOK_SUBQUERY_EXPR:
/*
* Restriction 2.h Subquery isnot allowed in LHS
*/
- if(next.getChildren().size() == 3
- && next.getChild(2).getType() == HiveParser.TOK_SUBQUERY_EXPR){
- throw new CalciteSemanticException(ErrorMsg.UNSUPPORTED_SUBQUERY_EXPRESSION.getMsg(
- next.getChild(2),
- "SubQuery in LHS expressions are not supported."));
- }
- String sbQueryAlias = "sq_" + qb.incrNumSubQueryPredicates();
- QB qbSQ = new QB(qb.getId(), sbQueryAlias, true);
- Phase1Ctx ctx1 = initPhase1Ctx();
- doPhase1((ASTNode)next.getChild(1), qbSQ, ctx1, null);
- getMetaData(qbSQ);
- RelNode subQueryRelNode = genLogicalPlan(qbSQ, false, relToHiveColNameCalcitePosMap.get(srcRel),
- relToHiveRR.get(srcRel));
- subQueryToRelNode.put(next, subQueryRelNode);
- //keep track of subqueries which are scalar, correlated and contains aggregate
- // subquery expression. This will later be special cased in Subquery remove rule
- // for correlated scalar queries with aggregate we have take care of the case where
- // inner aggregate happens on empty result
- if(corrScalarQueriesWithAgg.contains(next)) {
- corrScalarRexSQWithAgg.add(subQueryRelNode);
- }
- if(scalarQueriesWithAggNoWinNoGby.contains(next)) {
- scalarAggNoGbyNoWin.add(subQueryRelNode);
- }
- isSubQuery = true;
- break;
- default:
- int childCount = next.getChildCount();
- for(int i = childCount - 1; i >= 0; i--) {
- stack.push((ASTNode) next.getChild(i));
- }
+ if(next.getChildren().size() == 3
+ && next.getChild(2).getType() == HiveParser.TOK_SUBQUERY_EXPR){
+ throw new CalciteSemanticException(ErrorMsg.UNSUPPORTED_SUBQUERY_EXPRESSION.getMsg(
+ next.getChild(2),
+ "SubQuery in LHS expressions are not supported."));
+ }
+ String sbQueryAlias = "sq_" + qb.incrNumSubQueryPredicates();
+ QB qbSQ = new QB(qb.getId(), sbQueryAlias, true);
+ Phase1Ctx ctx1 = initPhase1Ctx();
+ doPhase1((ASTNode)next.getChild(1), qbSQ, ctx1, null);
+ getMetaData(qbSQ);
+ this.subqueryId++;
+ RelNode subQueryRelNode = genLogicalPlan(qbSQ, false,
+ relToHiveColNameCalcitePosMap.get(srcRel), relToHiveRR.get(srcRel));
+ subQueryToRelNode.put(next, subQueryRelNode);
+ //keep track of subqueries which are scalar, correlated and contains aggregate
+ // subquery expression. This will later be special cased in Subquery remove rule
+ // for correlated scalar queries with aggregate we have take care of the case where
+ // inner aggregate happens on empty result
+ if(corrScalarQueriesWithAgg.contains(next)) {
+ corrScalarRexSQWithAgg.add(subQueryRelNode);
}
+ if(scalarQueriesWithAggNoWinNoGby.contains(next)) {
+ scalarAggNoGbyNoWin.add(subQueryRelNode);
+ }
+ isSubQuery = true;
+ break;
+ default:
+ int childCount = next.getChildCount();
+ for(int i = childCount - 1; i >= 0; i--) {
+ stack.push((ASTNode) next.getChild(i));
+ }
+ }
}
return isSubQuery;
}
@@ -2706,7 +2708,6 @@ public class CalcitePlanner extends SemanticAnalyzer {
this.relToHiveColNameCalcitePosMap.put(filterRel, this.relToHiveColNameCalcitePosMap
.get(srcRel));
relToHiveRR.put(filterRel, relToHiveRR.get(srcRel));
- this.subqueryId++;
return filterRel;
} else {
return genFilterRelNode(searchCond, srcRel, outerNameToPosMap, outerRR, forHavingClause);
http://git-wip-us.apache.org/repos/asf/hive/blob/b23c3299/ql/src/test/queries/clientpositive/subquery_corr.q
----------------------------------------------------------------------
diff --git a/ql/src/test/queries/clientpositive/subquery_corr.q b/ql/src/test/queries/clientpositive/subquery_corr.q
new file mode 100644
index 0000000..10b4c3a
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/subquery_corr.q
@@ -0,0 +1,12 @@
+set hive.mapred.mode=nonstrict;
+set hive.explain.user=false;
+
+-- inner query has non-equi correlated predicate, this shouldn't have value gen
+explain select * from src b where b.key in (select key from src a where b.value > a.value);
+select * from src b where b.key in (select key from src a where b.value > a.value);
+
+explain select * from src b where b.key in (select key from src a where b.value <= a.value);
+select * from src b where b.key in (select key from src a where b.value <= a.value);
+
+explain select * from src b where b.key in (select key from src a where b.value > a.value and b.key < a.key) ;
+select * from src b where b.key in (select key from src a where b.value > a.value and b.key < a.key) ;
http://git-wip-us.apache.org/repos/asf/hive/blob/b23c3299/ql/src/test/results/clientpositive/llap/subquery_corr.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/subquery_corr.q.out b/ql/src/test/results/clientpositive/llap/subquery_corr.q.out
new file mode 100644
index 0000000..5fa47d2
--- /dev/null
+++ b/ql/src/test/results/clientpositive/llap/subquery_corr.q.out
@@ -0,0 +1,797 @@
+PREHOOK: query: explain select * from src b where b.key in (select key from src a where b.value > a.value)
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select * from src b where b.key in (select key from src a where b.value > a.value)
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+#### A masked pattern was here ####
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: key (type: string), value (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: string)
+ Execution mode: llap
+ LLAP IO: no inputs
+ Map 3
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (key is not null and value is not null) (type: boolean)
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: key (type: string), value (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: string)
+ Execution mode: llap
+ LLAP IO: no inputs
+ Reducer 2
+ Execution mode: llap
+ Reduce Operator Tree:
+ Merge Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ keys:
+ 0 _col0 (type: string)
+ 1 _col0 (type: string)
+ outputColumnNames: _col0, _col1, _col3
+ residual filter predicates: {(_col1 > _col3)}
+ Statistics: Num rows: 134 Data size: 36046 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: _col0 (type: string), _col1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 134 Data size: 23852 Basic stats: COMPLETE Column stats: COMPLETE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 134 Data size: 23852 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select * from src b where b.key in (select key from src a where b.value > a.value)
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src
+#### A masked pattern was here ####
+POSTHOOK: query: select * from src b where b.key in (select key from src a where b.value > a.value)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src
+#### A masked pattern was here ####
+PREHOOK: query: explain select * from src b where b.key in (select key from src a where b.value <= a.value)
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select * from src b where b.key in (select key from src a where b.value <= a.value)
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+#### A masked pattern was here ####
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: key (type: string), value (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: string)
+ Execution mode: llap
+ LLAP IO: no inputs
+ Map 3
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (key is not null and value is not null) (type: boolean)
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: key (type: string), value (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: string)
+ Execution mode: llap
+ LLAP IO: no inputs
+ Reducer 2
+ Execution mode: llap
+ Reduce Operator Tree:
+ Merge Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ keys:
+ 0 _col0 (type: string)
+ 1 _col0 (type: string)
+ outputColumnNames: _col0, _col1, _col3
+ residual filter predicates: {(_col1 <= _col3)}
+ Statistics: Num rows: 134 Data size: 36046 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: _col0 (type: string), _col1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 134 Data size: 23852 Basic stats: COMPLETE Column stats: COMPLETE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 134 Data size: 23852 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select * from src b where b.key in (select key from src a where b.value <= a.value)
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src
+#### A masked pattern was here ####
+POSTHOOK: query: select * from src b where b.key in (select key from src a where b.value <= a.value)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src
+#### A masked pattern was here ####
+0 val_0
+0 val_0
+0 val_0
+10 val_10
+100 val_100
+100 val_100
+103 val_103
+103 val_103
+104 val_104
+104 val_104
+105 val_105
+11 val_11
+111 val_111
+113 val_113
+113 val_113
+114 val_114
+116 val_116
+118 val_118
+118 val_118
+119 val_119
+119 val_119
+119 val_119
+12 val_12
+12 val_12
+120 val_120
+120 val_120
+125 val_125
+125 val_125
+126 val_126
+128 val_128
+128 val_128
+128 val_128
+129 val_129
+129 val_129
+131 val_131
+133 val_133
+134 val_134
+134 val_134
+136 val_136
+137 val_137
+137 val_137
+138 val_138
+138 val_138
+138 val_138
+138 val_138
+143 val_143
+145 val_145
+146 val_146
+146 val_146
+149 val_149
+149 val_149
+15 val_15
+15 val_15
+150 val_150
+152 val_152
+152 val_152
+153 val_153
+155 val_155
+156 val_156
+157 val_157
+158 val_158
+160 val_160
+162 val_162
+163 val_163
+164 val_164
+164 val_164
+165 val_165
+165 val_165
+166 val_166
+167 val_167
+167 val_167
+167 val_167
+168 val_168
+169 val_169
+169 val_169
+169 val_169
+169 val_169
+17 val_17
+170 val_170
+172 val_172
+172 val_172
+174 val_174
+174 val_174
+175 val_175
+175 val_175
+176 val_176
+176 val_176
+177 val_177
+178 val_178
+179 val_179
+179 val_179
+18 val_18
+18 val_18
+180 val_180
+181 val_181
+183 val_183
+186 val_186
+187 val_187
+187 val_187
+187 val_187
+189 val_189
+19 val_19
+190 val_190
+191 val_191
+191 val_191
+192 val_192
+193 val_193
+193 val_193
+193 val_193
+194 val_194
+195 val_195
+195 val_195
+196 val_196
+197 val_197
+197 val_197
+199 val_199
+199 val_199
+199 val_199
+2 val_2
+20 val_20
+200 val_200
+200 val_200
+201 val_201
+202 val_202
+203 val_203
+203 val_203
+205 val_205
+205 val_205
+207 val_207
+207 val_207
+208 val_208
+208 val_208
+208 val_208
+209 val_209
+209 val_209
+213 val_213
+213 val_213
+214 val_214
+216 val_216
+216 val_216
+217 val_217
+217 val_217
+218 val_218
+219 val_219
+219 val_219
+221 val_221
+221 val_221
+222 val_222
+223 val_223
+223 val_223
+224 val_224
+224 val_224
+226 val_226
+228 val_228
+229 val_229
+229 val_229
+230 val_230
+230 val_230
+230 val_230
+230 val_230
+230 val_230
+233 val_233
+233 val_233
+235 val_235
+237 val_237
+237 val_237
+238 val_238
+238 val_238
+239 val_239
+239 val_239
+24 val_24
+24 val_24
+241 val_241
+242 val_242
+242 val_242
+244 val_244
+247 val_247
+248 val_248
+249 val_249
+252 val_252
+255 val_255
+255 val_255
+256 val_256
+256 val_256
+257 val_257
+258 val_258
+26 val_26
+26 val_26
+260 val_260
+262 val_262
+263 val_263
+265 val_265
+265 val_265
+266 val_266
+27 val_27
+272 val_272
+272 val_272
+273 val_273
+273 val_273
+273 val_273
+274 val_274
+275 val_275
+277 val_277
+277 val_277
+277 val_277
+277 val_277
+278 val_278
+278 val_278
+28 val_28
+280 val_280
+280 val_280
+281 val_281
+281 val_281
+282 val_282
+282 val_282
+283 val_283
+284 val_284
+285 val_285
+286 val_286
+287 val_287
+288 val_288
+288 val_288
+289 val_289
+291 val_291
+292 val_292
+296 val_296
+298 val_298
+298 val_298
+298 val_298
+30 val_30
+302 val_302
+305 val_305
+306 val_306
+307 val_307
+307 val_307
+308 val_308
+309 val_309
+309 val_309
+310 val_310
+311 val_311
+311 val_311
+311 val_311
+315 val_315
+316 val_316
+316 val_316
+316 val_316
+317 val_317
+317 val_317
+318 val_318
+318 val_318
+318 val_318
+321 val_321
+321 val_321
+322 val_322
+322 val_322
+323 val_323
+325 val_325
+325 val_325
+327 val_327
+327 val_327
+327 val_327
+33 val_33
+331 val_331
+331 val_331
+332 val_332
+333 val_333
+333 val_333
+335 val_335
+336 val_336
+338 val_338
+339 val_339
+34 val_34
+341 val_341
+342 val_342
+342 val_342
+344 val_344
+344 val_344
+345 val_345
+348 val_348
+348 val_348
+348 val_348
+348 val_348
+348 val_348
+35 val_35
+35 val_35
+35 val_35
+351 val_351
+353 val_353
+353 val_353
+356 val_356
+360 val_360
+362 val_362
+364 val_364
+365 val_365
+366 val_366
+367 val_367
+367 val_367
+368 val_368
+369 val_369
+369 val_369
+369 val_369
+37 val_37
+37 val_37
+373 val_373
+374 val_374
+375 val_375
+377 val_377
+378 val_378
+379 val_379
+382 val_382
+382 val_382
+384 val_384
+384 val_384
+384 val_384
+386 val_386
+389 val_389
+392 val_392
+393 val_393
+394 val_394
+395 val_395
+395 val_395
+396 val_396
+396 val_396
+396 val_396
+397 val_397
+397 val_397
+399 val_399
+399 val_399
+4 val_4
+400 val_400
+401 val_401
+401 val_401
+401 val_401
+401 val_401
+401 val_401
+402 val_402
+403 val_403
+403 val_403
+403 val_403
+404 val_404
+404 val_404
+406 val_406
+406 val_406
+406 val_406
+406 val_406
+407 val_407
+409 val_409
+409 val_409
+409 val_409
+41 val_41
+411 val_411
+413 val_413
+413 val_413
+414 val_414
+414 val_414
+417 val_417
+417 val_417
+417 val_417
+418 val_418
+419 val_419
+42 val_42
+42 val_42
+421 val_421
+424 val_424
+424 val_424
+427 val_427
+429 val_429
+429 val_429
+43 val_43
+430 val_430
+430 val_430
+430 val_430
+431 val_431
+431 val_431
+431 val_431
+432 val_432
+435 val_435
+436 val_436
+437 val_437
+438 val_438
+438 val_438
+438 val_438
+439 val_439
+439 val_439
+44 val_44
+443 val_443
+444 val_444
+446 val_446
+448 val_448
+449 val_449
+452 val_452
+453 val_453
+454 val_454
+454 val_454
+454 val_454
+455 val_455
+457 val_457
+458 val_458
+458 val_458
+459 val_459
+459 val_459
+460 val_460
+462 val_462
+462 val_462
+463 val_463
+463 val_463
+466 val_466
+466 val_466
+466 val_466
+467 val_467
+468 val_468
+468 val_468
+468 val_468
+468 val_468
+469 val_469
+469 val_469
+469 val_469
+469 val_469
+469 val_469
+47 val_47
+470 val_470
+472 val_472
+475 val_475
+477 val_477
+478 val_478
+478 val_478
+479 val_479
+480 val_480
+480 val_480
+480 val_480
+481 val_481
+482 val_482
+483 val_483
+484 val_484
+485 val_485
+487 val_487
+489 val_489
+489 val_489
+489 val_489
+489 val_489
+490 val_490
+491 val_491
+492 val_492
+492 val_492
+493 val_493
+494 val_494
+495 val_495
+496 val_496
+497 val_497
+498 val_498
+498 val_498
+498 val_498
+5 val_5
+5 val_5
+5 val_5
+51 val_51
+51 val_51
+53 val_53
+54 val_54
+57 val_57
+58 val_58
+58 val_58
+64 val_64
+65 val_65
+66 val_66
+67 val_67
+67 val_67
+69 val_69
+70 val_70
+70 val_70
+70 val_70
+72 val_72
+72 val_72
+74 val_74
+76 val_76
+76 val_76
+77 val_77
+78 val_78
+8 val_8
+80 val_80
+82 val_82
+83 val_83
+83 val_83
+84 val_84
+84 val_84
+85 val_85
+86 val_86
+87 val_87
+9 val_9
+90 val_90
+90 val_90
+90 val_90
+92 val_92
+95 val_95
+95 val_95
+96 val_96
+97 val_97
+97 val_97
+98 val_98
+98 val_98
+PREHOOK: query: explain select * from src b where b.key in (select key from src a where b.value > a.value and b.key < a.key)
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select * from src b where b.key in (select key from src a where b.value > a.value and b.key < a.key)
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+#### A masked pattern was here ####
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: key (type: string), value (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: string)
+ Execution mode: llap
+ LLAP IO: no inputs
+ Map 3
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (key is not null and value is not null) (type: boolean)
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: key (type: string), key (type: string), value (type: string)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 500 Data size: 132500 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string), _col2 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 500 Data size: 132500 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 500 Data size: 132500 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: string), _col2 (type: string)
+ Execution mode: llap
+ LLAP IO: no inputs
+ Reducer 2
+ Execution mode: llap
+ Reduce Operator Tree:
+ Merge Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ keys:
+ 0 _col0 (type: string)
+ 1 _col0 (type: string)
+ outputColumnNames: _col0, _col1, _col3, _col4
+ residual filter predicates: {(_col0 < _col3)} {(_col1 > _col4)}
+ Statistics: Num rows: 55 Data size: 19580 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: _col0 (type: string), _col1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 55 Data size: 9790 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select * from src b where b.key in (select key from src a where b.value > a.value and b.key < a.key)
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src
+#### A masked pattern was here ####
+POSTHOOK: query: select * from src b where b.key in (select key from src a where b.value > a.value and b.key < a.key)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src
+#### A masked pattern was here ####
http://git-wip-us.apache.org/repos/asf/hive/blob/b23c3299/ql/src/test/results/clientpositive/llap/subquery_exists.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/subquery_exists.q.out b/ql/src/test/results/clientpositive/llap/subquery_exists.q.out
index dfe4240..de1f7ae 100644
--- a/ql/src/test/results/clientpositive/llap/subquery_exists.q.out
+++ b/ql/src/test/results/clientpositive/llap/subquery_exists.q.out
@@ -1326,7 +1326,7 @@ STAGE PLANS:
0 _col0 (type: int)
1 _col0 (type: int)
outputColumnNames: _col0, _col1, _col3
- residual filter predicates: {(_col1 <> _col3)}
+ residual filter predicates: {(_col3 <> _col1)}
Statistics: Num rows: 3 Data size: 26 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: int), _col1 (type: int)
http://git-wip-us.apache.org/repos/asf/hive/blob/b23c3299/ql/src/test/results/clientpositive/llap/subquery_in.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/subquery_in.q.out b/ql/src/test/results/clientpositive/llap/subquery_in.q.out
index 5dcdfdd..df0dc85 100644
--- a/ql/src/test/results/clientpositive/llap/subquery_in.q.out
+++ b/ql/src/test/results/clientpositive/llap/subquery_in.q.out
@@ -885,7 +885,6 @@ POSTHOOK: Input: default@src
97 val_97
98 val_98
98 val_98
-Warning: Shuffle Join MERGEJOIN[30][tables = [$hdt$_1, $hdt$_2]] in Stage 'Reducer 3' is a cross product
PREHOOK: query: explain select * from src b where b.key in (select distinct key from src a where a.value > b.value)
PREHOOK: type: QUERY
POSTHOOK: query: explain select * from src b where b.key in (select distinct key from src a where a.value > b.value)
@@ -900,8 +899,7 @@ STAGE PLANS:
#### A masked pattern was here ####
Edges:
Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
- Reducer 3 <- Map 1 (XPROD_EDGE), Reducer 5 (XPROD_EDGE)
- Reducer 5 <- Map 4 (SIMPLE_EDGE)
+ Reducer 3 <- Map 1 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -910,18 +908,6 @@ STAGE PLANS:
alias: b
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (key is not null and value is not null) (type: boolean)
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
@@ -929,29 +915,24 @@ STAGE PLANS:
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- sort order:
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string), _col1 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 4
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 45500 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: string)
Filter Operator
- predicate: value is not null (type: boolean)
- Statistics: Num rows: 500 Data size: 45500 Basic stats: COMPLETE Column stats: COMPLETE
+ predicate: (key is not null and value is not null) (type: boolean)
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
- keys: value (type: string)
+ keys: key (type: string), value (type: string)
mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 250 Data size: 22750 Basic stats: COMPLETE Column stats: COMPLETE
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 250 Data size: 22750 Basic stats: COMPLETE Column stats: COMPLETE
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
Reducer 2
@@ -961,55 +942,41 @@ STAGE PLANS:
condition map:
Left Semi Join 0 to 1
keys:
- 0 _col0 (type: string), _col1 (type: string)
- 1 _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- File Output Operator
- compressed: false
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0
- 1
- outputColumnNames: _col0, _col1, _col2
- residual filter predicates: {(_col1 > _col2)}
- Statistics: Num rows: 41666 Data size: 11208154 Basic stats: COMPLETE Column stats: COMPLETE
+ 0 _col0 (type: string)
+ 1 _col0 (type: string)
+ outputColumnNames: _col0, _col1, _col3
+ residual filter predicates: {(_col3 > _col1)}
+ Statistics: Num rows: 67 Data size: 18023 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: string), _col1 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 41666 Data size: 7416548 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 10609 Data size: 1888402 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 10609 Data size: 1888402 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 5
+ Statistics: Num rows: 67 Data size: 11926 Basic stats: COMPLETE Column stats: COMPLETE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 67 Data size: 11926 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Reducer 3
Execution mode: llap
Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: string)
+ keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 250 Data size: 22750 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 250 Data size: 22750 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 125 Data size: 22250 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 125 Data size: 22250 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: string)
Stage: Stage-0
Fetch Operator
@@ -1017,7 +984,6 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join MERGEJOIN[30][tables = [$hdt$_1, $hdt$_2]] in Stage 'Reducer 3' is a cross product
PREHOOK: query: select * from src b where b.key in (select distinct key from src a where a.value > b.value)
PREHOOK: type: QUERY
PREHOOK: Input: default@src
@@ -1026,503 +992,6 @@ POSTHOOK: query: select * from src b where b.key in (select distinct key from sr
POSTHOOK: type: QUERY
POSTHOOK: Input: default@src
#### A masked pattern was here ####
-10 val_10
-100 val_100
-100 val_100
-103 val_103
-103 val_103
-104 val_104
-104 val_104
-105 val_105
-11 val_11
-111 val_111
-113 val_113
-113 val_113
-114 val_114
-116 val_116
-118 val_118
-118 val_118
-119 val_119
-119 val_119
-119 val_119
-12 val_12
-12 val_12
-120 val_120
-120 val_120
-125 val_125
-125 val_125
-126 val_126
-128 val_128
-128 val_128
-128 val_128
-129 val_129
-129 val_129
-131 val_131
-133 val_133
-134 val_134
-134 val_134
-136 val_136
-137 val_137
-137 val_137
-138 val_138
-138 val_138
-138 val_138
-138 val_138
-143 val_143
-145 val_145
-146 val_146
-146 val_146
-149 val_149
-149 val_149
-15 val_15
-15 val_15
-150 val_150
-152 val_152
-152 val_152
-153 val_153
-155 val_155
-156 val_156
-157 val_157
-158 val_158
-160 val_160
-162 val_162
-163 val_163
-164 val_164
-164 val_164
-165 val_165
-165 val_165
-166 val_166
-167 val_167
-167 val_167
-167 val_167
-168 val_168
-169 val_169
-169 val_169
-169 val_169
-169 val_169
-17 val_17
-170 val_170
-172 val_172
-172 val_172
-174 val_174
-174 val_174
-175 val_175
-175 val_175
-176 val_176
-176 val_176
-177 val_177
-178 val_178
-179 val_179
-179 val_179
-18 val_18
-18 val_18
-180 val_180
-181 val_181
-183 val_183
-186 val_186
-187 val_187
-187 val_187
-187 val_187
-189 val_189
-19 val_19
-190 val_190
-191 val_191
-191 val_191
-192 val_192
-193 val_193
-193 val_193
-193 val_193
-194 val_194
-195 val_195
-195 val_195
-196 val_196
-197 val_197
-197 val_197
-199 val_199
-199 val_199
-199 val_199
-2 val_2
-20 val_20
-200 val_200
-200 val_200
-201 val_201
-202 val_202
-203 val_203
-203 val_203
-205 val_205
-205 val_205
-207 val_207
-207 val_207
-208 val_208
-208 val_208
-208 val_208
-209 val_209
-209 val_209
-213 val_213
-213 val_213
-214 val_214
-216 val_216
-216 val_216
-217 val_217
-217 val_217
-218 val_218
-219 val_219
-219 val_219
-221 val_221
-221 val_221
-222 val_222
-223 val_223
-223 val_223
-224 val_224
-224 val_224
-226 val_226
-228 val_228
-229 val_229
-229 val_229
-230 val_230
-230 val_230
-230 val_230
-230 val_230
-230 val_230
-233 val_233
-233 val_233
-235 val_235
-237 val_237
-237 val_237
-238 val_238
-238 val_238
-239 val_239
-239 val_239
-24 val_24
-24 val_24
-241 val_241
-242 val_242
-242 val_242
-244 val_244
-247 val_247
-248 val_248
-249 val_249
-252 val_252
-255 val_255
-255 val_255
-256 val_256
-256 val_256
-257 val_257
-258 val_258
-26 val_26
-26 val_26
-260 val_260
-262 val_262
-263 val_263
-265 val_265
-265 val_265
-266 val_266
-27 val_27
-272 val_272
-272 val_272
-273 val_273
-273 val_273
-273 val_273
-274 val_274
-275 val_275
-277 val_277
-277 val_277
-277 val_277
-277 val_277
-278 val_278
-278 val_278
-28 val_28
-280 val_280
-280 val_280
-281 val_281
-281 val_281
-282 val_282
-282 val_282
-283 val_283
-284 val_284
-285 val_285
-286 val_286
-287 val_287
-288 val_288
-288 val_288
-289 val_289
-291 val_291
-292 val_292
-296 val_296
-298 val_298
-298 val_298
-298 val_298
-30 val_30
-302 val_302
-305 val_305
-306 val_306
-307 val_307
-307 val_307
-308 val_308
-309 val_309
-309 val_309
-310 val_310
-311 val_311
-311 val_311
-311 val_311
-315 val_315
-316 val_316
-316 val_316
-316 val_316
-317 val_317
-317 val_317
-318 val_318
-318 val_318
-318 val_318
-321 val_321
-321 val_321
-322 val_322
-322 val_322
-323 val_323
-325 val_325
-325 val_325
-327 val_327
-327 val_327
-327 val_327
-33 val_33
-331 val_331
-331 val_331
-332 val_332
-333 val_333
-333 val_333
-335 val_335
-336 val_336
-338 val_338
-339 val_339
-34 val_34
-341 val_341
-342 val_342
-342 val_342
-344 val_344
-344 val_344
-345 val_345
-348 val_348
-348 val_348
-348 val_348
-348 val_348
-348 val_348
-35 val_35
-35 val_35
-35 val_35
-351 val_351
-353 val_353
-353 val_353
-356 val_356
-360 val_360
-362 val_362
-364 val_364
-365 val_365
-366 val_366
-367 val_367
-367 val_367
-368 val_368
-369 val_369
-369 val_369
-369 val_369
-37 val_37
-37 val_37
-373 val_373
-374 val_374
-375 val_375
-377 val_377
-378 val_378
-379 val_379
-382 val_382
-382 val_382
-384 val_384
-384 val_384
-384 val_384
-386 val_386
-389 val_389
-392 val_392
-393 val_393
-394 val_394
-395 val_395
-395 val_395
-396 val_396
-396 val_396
-396 val_396
-397 val_397
-397 val_397
-399 val_399
-399 val_399
-4 val_4
-400 val_400
-401 val_401
-401 val_401
-401 val_401
-401 val_401
-401 val_401
-402 val_402
-403 val_403
-403 val_403
-403 val_403
-404 val_404
-404 val_404
-406 val_406
-406 val_406
-406 val_406
-406 val_406
-407 val_407
-409 val_409
-409 val_409
-409 val_409
-41 val_41
-411 val_411
-413 val_413
-413 val_413
-414 val_414
-414 val_414
-417 val_417
-417 val_417
-417 val_417
-418 val_418
-419 val_419
-42 val_42
-42 val_42
-421 val_421
-424 val_424
-424 val_424
-427 val_427
-429 val_429
-429 val_429
-43 val_43
-430 val_430
-430 val_430
-430 val_430
-431 val_431
-431 val_431
-431 val_431
-432 val_432
-435 val_435
-436 val_436
-437 val_437
-438 val_438
-438 val_438
-438 val_438
-439 val_439
-439 val_439
-44 val_44
-443 val_443
-444 val_444
-446 val_446
-448 val_448
-449 val_449
-452 val_452
-453 val_453
-454 val_454
-454 val_454
-454 val_454
-455 val_455
-457 val_457
-458 val_458
-458 val_458
-459 val_459
-459 val_459
-460 val_460
-462 val_462
-462 val_462
-463 val_463
-463 val_463
-466 val_466
-466 val_466
-466 val_466
-467 val_467
-468 val_468
-468 val_468
-468 val_468
-468 val_468
-469 val_469
-469 val_469
-469 val_469
-469 val_469
-469 val_469
-47 val_47
-470 val_470
-472 val_472
-475 val_475
-477 val_477
-478 val_478
-478 val_478
-479 val_479
-480 val_480
-480 val_480
-480 val_480
-481 val_481
-482 val_482
-483 val_483
-484 val_484
-485 val_485
-487 val_487
-489 val_489
-489 val_489
-489 val_489
-489 val_489
-490 val_490
-491 val_491
-492 val_492
-492 val_492
-493 val_493
-494 val_494
-495 val_495
-496 val_496
-497 val_497
-498 val_498
-498 val_498
-498 val_498
-5 val_5
-5 val_5
-5 val_5
-51 val_51
-51 val_51
-53 val_53
-54 val_54
-57 val_57
-58 val_58
-58 val_58
-64 val_64
-65 val_65
-66 val_66
-67 val_67
-67 val_67
-69 val_69
-70 val_70
-70 val_70
-70 val_70
-72 val_72
-72 val_72
-74 val_74
-76 val_76
-76 val_76
-77 val_77
-78 val_78
-8 val_8
-80 val_80
-82 val_82
-83 val_83
-83 val_83
-84 val_84
-84 val_84
-85 val_85
-86 val_86
-87 val_87
-9 val_9
-90 val_90
-90 val_90
-90 val_90
-92 val_92
-95 val_95
-95 val_95
-96 val_96
-97 val_97
-97 val_97
-98 val_98
-98 val_98
PREHOOK: query: select p_mfgr, p_name, p_size
from part
where part.p_size in
http://git-wip-us.apache.org/repos/asf/hive/blob/b23c3299/ql/src/test/results/clientpositive/llap/subquery_in_having.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/subquery_in_having.q.out b/ql/src/test/results/clientpositive/llap/subquery_in_having.q.out
index 0ffbaae..50ac656 100644
--- a/ql/src/test/results/clientpositive/llap/subquery_in_having.q.out
+++ b/ql/src/test/results/clientpositive/llap/subquery_in_having.q.out
@@ -1623,7 +1623,7 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col2, _col3
input vertices:
1 Reducer 11
- residual filter predicates: {(_col1 <> _col2)}
+ residual filter predicates: {(_col2 <> _col1)}
Statistics: Num rows: 1 Data size: 553 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: _col3 is null (type: boolean)
@@ -1668,7 +1668,7 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col2, _col3
input vertices:
1 Reducer 12
- residual filter predicates: {(_col1 <> _col2)}
+ residual filter predicates: {(_col2 <> _col1)}
Statistics: Num rows: 1 Data size: 553 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: _col3 is null (type: boolean)
@@ -1711,7 +1711,7 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col2, _col3
input vertices:
1 Reducer 13
- residual filter predicates: {(_col1 <> _col2)}
+ residual filter predicates: {(_col2 <> _col1)}
Statistics: Num rows: 1 Data size: 553 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: _col3 is null (type: boolean)
http://git-wip-us.apache.org/repos/asf/hive/blob/b23c3299/ql/src/test/results/clientpositive/llap/subquery_notin.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/subquery_notin.q.out b/ql/src/test/results/clientpositive/llap/subquery_notin.q.out
index 5da1258..a7233e8 100644
--- a/ql/src/test/results/clientpositive/llap/subquery_notin.q.out
+++ b/ql/src/test/results/clientpositive/llap/subquery_notin.q.out
@@ -7130,8 +7130,7 @@ PREHOOK: query: drop table t1
PREHOOK: type: DROPTABLE
POSTHOOK: query: drop table t1
POSTHOOK: type: DROPTABLE
-Warning: Shuffle Join MERGEJOIN[55][tables = [$hdt$_1, $hdt$_2]] in Stage 'Reducer 4' is a cross product
-Warning: Shuffle Join MERGEJOIN[57][tables = [$hdt$_2, $hdt$_3]] in Stage 'Reducer 6' is a cross product
+Warning: Shuffle Join MERGEJOIN[43][tables = [$hdt$_1, $hdt$_2]] in Stage 'Reducer 4' is a cross product
PREHOOK: query: explain select *
from src b
where b.key not in
@@ -7157,14 +7156,12 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 10 <- Map 8 (SIMPLE_EDGE)
Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 5 (ONE_TO_ONE_EDGE)
- Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 7 (ONE_TO_ONE_EDGE)
- Reducer 4 <- Map 1 (XPROD_EDGE), Reducer 9 (XPROD_EDGE)
+ Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
+ Reducer 4 <- Map 1 (XPROD_EDGE), Reducer 8 (XPROD_EDGE)
Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
- Reducer 6 <- Map 1 (XPROD_EDGE), Reducer 10 (XPROD_EDGE)
- Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
- Reducer 9 <- Map 8 (SIMPLE_EDGE)
+ Reducer 6 <- Map 1 (SIMPLE_EDGE)
+ Reducer 8 <- Map 7 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -7194,19 +7191,21 @@ STAGE PLANS:
Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: string), _col1 (type: string)
Filter Operator
- predicate: (key > '9') (type: boolean)
+ predicate: ((key > '9') and value is not null) (type: boolean)
Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: key (type: string), value (type: string)
+ Group By Operator
+ keys: key (type: string), value (type: string)
+ mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 83 Data size: 14774 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- sort order:
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string), _col1 (type: string)
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 83 Data size: 14774 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
- Map 8
+ Map 7
Map Operator Tree:
TableScan
alias: b
@@ -7221,25 +7220,8 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 250 Data size: 22750 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 250 Data size: 22750 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
- Reducer 10
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 250 Data size: 22750 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 250 Data size: 22750 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
Reducer 2
Execution mode: llap
Reduce Operator Tree:
@@ -7252,11 +7234,11 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col3, _col4
Statistics: Num rows: 500 Data size: 91672 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 500 Data size: 91672 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col3 (type: bigint), _col4 (type: bigint)
+ value expressions: _col1 (type: string), _col3 (type: bigint), _col4 (type: bigint)
Reducer 3
Execution mode: llap
Reduce Operator Tree:
@@ -7264,20 +7246,21 @@ STAGE PLANS:
condition map:
Left Outer Join 0 to 1
keys:
- 0 _col0 (type: string), _col1 (type: string)
- 1 _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1, _col3, _col4, _col7
- Statistics: Num rows: 5833 Data size: 1149606 Basic stats: COMPLETE Column stats: COMPLETE
+ 0 _col0 (type: string)
+ 1 _col0 (type: string)
+ outputColumnNames: _col0, _col1, _col3, _col4, _col6, _col7
+ residual filter predicates: {(_col1 > _col6)}
+ Statistics: Num rows: 500 Data size: 104497 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: (not CASE WHEN ((_col3 = 0)) THEN (false) WHEN (_col3 is null) THEN (false) WHEN (_col7 is not null) THEN (true) WHEN (_col0 is null) THEN (null) WHEN ((_col4 < _col3)) THEN (true) ELSE (false) END) (type: boolean)
- Statistics: Num rows: 2917 Data size: 574910 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 250 Data size: 52304 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: string), _col1 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 2917 Data size: 519226 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
compressed: false
- Statistics: Num rows: 2917 Data size: 519226 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
@@ -7328,51 +7311,25 @@ STAGE PLANS:
Reducer 6
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0
- 1
- outputColumnNames: _col0, _col1, _col2
- residual filter predicates: {(_col2 > _col1)}
- Statistics: Num rows: 13833 Data size: 3721077 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col0 (type: string), _col2 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 13833 Data size: 3721077 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: string), _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 3605 Data size: 641690 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 3605 Data size: 641690 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 7
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 3605 Data size: 641690 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 83 Data size: 14774 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: _col0 is not null (type: boolean)
- Statistics: Num rows: 3605 Data size: 641690 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 83 Data size: 14774 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: string), _col1 (type: string), true (type: boolean)
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 3605 Data size: 656110 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 83 Data size: 15106 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 3605 Data size: 656110 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col2 (type: boolean)
- Reducer 9
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 83 Data size: 15106 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: string), _col2 (type: boolean)
+ Reducer 8
Execution mode: llap
Reduce Operator Tree:
Group By Operator
@@ -7391,8 +7348,7 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join MERGEJOIN[55][tables = [$hdt$_1, $hdt$_2]] in Stage 'Reducer 4' is a cross product
-Warning: Shuffle Join MERGEJOIN[57][tables = [$hdt$_2, $hdt$_3]] in Stage 'Reducer 6' is a cross product
+Warning: Shuffle Join MERGEJOIN[43][tables = [$hdt$_1, $hdt$_2]] in Stage 'Reducer 4' is a cross product
PREHOOK: query: select *
from src b
where b.key not in
http://git-wip-us.apache.org/repos/asf/hive/blob/b23c3299/ql/src/test/results/clientpositive/spark/subquery_exists.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/spark/subquery_exists.q.out b/ql/src/test/results/clientpositive/spark/subquery_exists.q.out
index fb13fb7..9d9957b 100644
--- a/ql/src/test/results/clientpositive/spark/subquery_exists.q.out
+++ b/ql/src/test/results/clientpositive/spark/subquery_exists.q.out
@@ -1287,7 +1287,7 @@ STAGE PLANS:
0 _col0 (type: int)
1 _col0 (type: int)
outputColumnNames: _col0, _col1, _col3
- residual filter predicates: {(_col1 <> _col3)}
+ residual filter predicates: {(_col3 <> _col1)}
Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: int), _col1 (type: int)