You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/04/29 07:01:04 UTC
[GitHub] [hive] kasakrisz opened a new pull request #2231: HIVE-25071: Number of reducers limited to fixed 1 when updating/deleting
kasakrisz opened a new pull request #2231:
URL: https://github.com/apache/hive/pull/2231
### What changes were proposed in this pull request?
When creating Reducer for bucketing
1. remove limitation for number of reducers when updating/deleting
2. Add ROWID to sort columns
### Why are the changes needed?
Limiting number of reducers may leads to performance degradation. See Jira for details.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
```
mvn test -Dtest=TestTxnCommands#testDeleteIn -pl ql -Drat.skip
```
```
mvn test -DskipSparkTests -Dtest=TestMiniLlapLocalCliDriver -Dqfile=llap_acid.q -pl itests/qtest -Pitests
```
Search for log entries like
```
2021-04-28T23:40:53,335 DEBUG [ec46860c-7238-4cbd-b14d-80b2e12fc54e main] parse.ParseDriver: Parsing command:
update orc_llap_n1 set cbigint = 2 where cint = 1
...
2021-04-28T23:40:53,394 INFO [ec46860c-7238-4cbd-b14d-80b2e12fc54e main] optimizer.SetReducerParallelism: Number of reducers determined to be: 2
```
in `hive.log`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] jcamachor commented on a change in pull request #2231: HIVE-25071: Number of reducers limited to fixed 1 when updating/deleting
Posted by GitBox <gi...@apache.org>.
jcamachor commented on a change in pull request #2231:
URL: https://github.com/apache/hive/pull/2231#discussion_r626149398
##########
File path: ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out
##########
@@ -810,18 +810,18 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 265500 Basic stats: COMPLETE Column stats: PARTIAL
Reduce Output Operator
key expressions: _col0 (type: struct<writeid:bigint,bucketid:int,rowid:bigint>)
- null sort order: z
+ null sort order: a
Review comment:
I don't think it will have any impact (row_id should not be null) but why is null order changing? I do not see any related change in your changes?
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
##########
@@ -6945,6 +6941,16 @@ else if (acidOp == Operation.NOT_ACID || acidOp == Operation.INSERT) {
return input;
}
+ // SORT BY ROW__ID ASC
+ private List<Integer> createSortOrderForUpdateDelete(List<ExprNodeDesc> sortCols) {
+ List<Integer> sortOrders;
Review comment:
Can be merged with next line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kasakrisz commented on a change in pull request #2231: HIVE-25071: Number of reducers limited to fixed 1 when updating/deleting
Posted by GitBox <gi...@apache.org>.
kasakrisz commented on a change in pull request #2231:
URL: https://github.com/apache/hive/pull/2231#discussion_r627950752
##########
File path: ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out
##########
@@ -810,18 +810,18 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 265500 Basic stats: COMPLETE Column stats: PARTIAL
Reduce Output Operator
key expressions: _col0 (type: struct<writeid:bigint,bucketid:int,rowid:bigint>)
- null sort order: z
+ null sort order: a
Review comment:
Null order is not depending the setting `hive.default.nulls.last`:
https://github.com/apache/hive/blob/02be85202eb46f7207ce2925b361425d87638973/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L6936
I tried to tackle this in #2227 but I experienced some regression in SMB joins: result set mismatch because rows with nulls introduced.
Let me check how was this handled prior this patch: I think the null order came from the parent RS (which introduce the sort by ROWID) not the child (enforce bucketing + sort by ROWID).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kasakrisz merged pull request #2231: HIVE-25071: Number of reducers limited to fixed 1 when updating/deleting
Posted by GitBox <gi...@apache.org>.
kasakrisz merged pull request #2231:
URL: https://github.com/apache/hive/pull/2231
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kuczoram commented on pull request #2231: HIVE-25071: Number of reducers limited to fixed 1 when updating/deleting
Posted by GitBox <gi...@apache.org>.
kuczoram commented on pull request #2231:
URL: https://github.com/apache/hive/pull/2231#issuecomment-829104540
Hi Krisztian!
Thanks for this patch, it is very interesting.
I would have one question about using multiple reducers. Do you know how it will be guaranteed that all rows with the same bucket would go into the same reducer and at the end to the same file?
I am asking it because during compaction we had issues that rows with the same bucket number went into different reducers and we ended up with corrupted files (when rows with the same bucket numbers went into different files or files contained rows with different bucket numbers). I saw this issue when created an unbucketed table, but inserted a bigger amount of data, so at the end, the table contained multiple bucket files.
I know that compaction is a different story, I am just curious whether or not something similar could happen with deletes/updates using multiple reducers. If you know how the row distribution between the reducers would work for deletes and updates, I would be really grateful if you could share some details.
Thanks and regards,
Marta
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kasakrisz commented on a change in pull request #2231: HIVE-25071: Number of reducers limited to fixed 1 when updating/deleting
Posted by GitBox <gi...@apache.org>.
kasakrisz commented on a change in pull request #2231:
URL: https://github.com/apache/hive/pull/2231#discussion_r629173274
##########
File path: ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out
##########
@@ -810,18 +810,18 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 265500 Basic stats: COMPLETE Column stats: PARTIAL
Reduce Output Operator
key expressions: _col0 (type: struct<writeid:bigint,bucketid:int,rowid:bigint>)
- null sort order: z
+ null sort order: a
Review comment:
I found that SMB join expects nulls first behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kasakrisz commented on pull request #2231: HIVE-25071: Number of reducers limited to fixed 1 when updating/deleting
Posted by GitBox <gi...@apache.org>.
kasakrisz commented on pull request #2231:
URL: https://github.com/apache/hive/pull/2231#issuecomment-829900602
Hi Marta,
Thanks for reviewing this patch.
This is what I found about distributing rows to reducers while I was debugging:
Let's say we have the following statements:
```
create table acidtbl(a int, b int) clustered by (a) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true');
insert ...
delete from acidtbl where a = 1 or a = 3;
```
This case the the plan of the delete statement after ReduceSinkDeDuplication looks like:
```
TS[0]-FIL[8]-SEL[2]-RS[5]-SEL[6]-FS[7]
```
So with Tez we have a mapper: TS[0]-FIL[8]-SEL[2]-RS[5]
and have two reducers each of them has: SEL[6]-FS[7]
RS[5] has
Partition keys: GenericUDFBridge ==> UDFToInteger (Column[_col0])
Sort keys: Column[_col0]
And maxReducers: 2
where _col0 is the row_id coming from SEL[2].
UDFToInteger(<row_id_type>) extracts the bucket_id field which is going to be used to generate a `reducesink.key` in the RS operator. This is going to be passed to the wrapped `OutputCollector` with the row. This case this is an `org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput`. This class is part of Tez which I'm not familiar with but I found that this is where rows are distributed to reducers by the key coming from RS.
Hive/hadoop also has a setting `hive.exec.reducers.max`/`mapreduce.job.reduces`. This limits the maxReducers in RS operator. If the table has more buckets than the max reducers then FileSink operator also distributes the rows into different files. If I understand correctly this is done by the `multiFileSpray` functionality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org