You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2022/05/14 00:25:46 UTC
[GitHub] [hive] zhengchenyu opened a new pull request, #3292: HIVE-25335: Unreasonable setting reduce number, when join big size ta…
zhengchenyu opened a new pull request, #3292:
URL: https://github.com/apache/hive/pull/3292
I found an application which is slow in our cluster, because the proccess bytes of one reduce is very huge, but only two reduce.
when I debug, I found the reason. Because in this sql, one big size table (about 30G) with few row count(about 3.5M), another small size table (about 100M) have more row count (about 3.6M). So JoinStatsRule.process only use 100M to estimate reducer's number. But we need to process 30G byte in fact.
https://issues.apache.org/jira/browse/HIVE-25335
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] closed pull request #3292: HIVE-25335: Unreasonable setting reduce number, when join big size ta…
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #3292: HIVE-25335: Unreasonable setting reduce number, when join big size ta…
URL: https://github.com/apache/hive/pull/3292
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zabetak commented on pull request #3292: HIVE-25335: Unreasonable setting reduce number, when join big size ta…
Posted by GitBox <gi...@apache.org>.
zabetak commented on PR #3292:
URL: https://github.com/apache/hive/pull/3292#issuecomment-1130032347
@zhengchenyu I am not sure what exactly do you mean by saying the unit tests are working in your environment. If you check the failed tests you will see a lot related to the `TestMiniLlapLocalCliDriver`. If you want to run these tests and update the plans you don't need Jenkins or anything else. You can do it by following the steps below:
```
mvn clean install -DskipTests -Pitests
cd itests/qtest
mvn test -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite
```
If you want to run specific tests then you can use the `-Dqfile` option. For more info have a look here: https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ#HiveDeveloperFAQ-HowdoIruntheclientpositive/clientnegativeunittests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zabetak commented on pull request #3292: HIVE-25335: Unreasonable setting reduce number, when join big size ta…
Posted by GitBox <gi...@apache.org>.
zabetak commented on PR #3292:
URL: https://github.com/apache/hive/pull/3292#issuecomment-1136889593
@zhengchenyu I suspect that the error you see on Jenkins has to do with the fact that there are a lot of errors in the tests.
If you run locally and you use the `-Dtest.ouptut.overwrite` then you will not have any errors cause you are updating automatically the "reference files". If you want to see all the errors locally you must remove this parameter.
Having said that if you commit all the changes in the reference files then tests most likely will pass and the Jenkins pipeline may run fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zhengchenyu commented on pull request #3292: HIVE-25335: Unreasonable setting reduce number, when join big size ta…
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on PR #3292:
URL: https://github.com/apache/hive/pull/3292#issuecomment-1128396054
@zabetak UT in my environment is right. Seems error happen in post stage.
Because I change the logical of maxDataSize, so some explain output may changed.
Maybe many explain output should repair, so I need setup a jenkins pipeline. Is there any introducation about hive jenkins pipeline. Many problem happen when I setup the pipeline in my dev enviromnent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] commented on pull request #3292: HIVE-25335: Unreasonable setting reduce number, when join big size ta…
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #3292:
URL: https://github.com/apache/hive/pull/3292#issuecomment-1193429305
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zhengchenyu commented on pull request #3292: HIVE-25335: Unreasonable setting reduce number, when join big size ta…
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on PR #3292:
URL: https://github.com/apache/hive/pull/3292#issuecomment-1136682080
> @zhengchenyu I am not sure what exactly do you mean by saying the unit tests are working in your environment. If you check the failed tests you will see a lot related to the `TestMiniLlapLocalCliDriver`. If you want to run these tests and update the plans you don't need Jenkins or anything else. You can do it by following the steps below:
>
> ```
> mvn clean install -DskipTests -Pitests
> cd itests/qtest
> mvn test -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite
> ```
>
> If you want to run specific tests then you can use the `-Dqfile` option. For more info have a look here: https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ#HiveDeveloperFAQ-HowdoIruntheclientpositive/clientnegativeunittests?
@zabetak
In our compile server, I just test the uni-test in this way. Then all pass.
Seems jenkins fail on this scripts.
```
# removes all stdout and err for passed tests
xmlstarlet ed -L -d 'testsuite/testcase/system-out[count(../failure)=0]' -d 'testsuite/testcase/system-err[count(../failure)=0]' `find . -name 'TEST*xml' -path '*/surefire-reports/*'`
# remove all output.txt files
find . -name '*output.txt' -path '*/surefire-reports/*' -exec unlink "{}" \\;
```
I also execute this scripts, then pass. I don't know the difference between my compile server and this jenkins pipeline.
Maybe I should setup a whole jenkins pipeline to reproduce this error. But jenkinsfile on my jenkins server run failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org