You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "yigress (via GitHub)" <gi...@apache.org> on 2023/02/03 00:59:28 UTC
[GitHub] [hive] yigress opened a new pull request, #4013: HIVE-27016 Invoke optional output committer in TezProcessor
yigress opened a new pull request, #4013:
URL: https://github.com/apache/hive/pull/4013
### What changes were proposed in this pull request?
This is backport of HIVE-24629.
1. TezProcessor should invoke output committer's commitTask after each processor execution.
2. After successful vertex completion, the DAG should invoke the Tez API output committer's commitOutput.
### Why are the changes needed?
For StorageHandler to be able to use customized OutputCommitter in Tez.
### Does this PR introduce _any_ user-facing change?
New config value, otherwise no.
### How was this patch tested?
unit tests, also local test with storagehandler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sunchao commented on pull request #4013: HIVE-27016: Backport HIVE-24629 Invoke optional output committer in TezProcessor
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #4013:
URL: https://github.com/apache/hive/pull/4013#issuecomment-1422032002
Merged, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] yigress commented on pull request #4013: HIVE-27016: Backport HIVE-24629 Invoke optional output committer in TezProcessor
Posted by "yigress (via GitHub)" <gi...@apache.org>.
yigress commented on PR #4013:
URL: https://github.com/apache/hive/pull/4013#issuecomment-1416595363
@amanraj2520 I see you are fixing tests in branch-3, will really appreciate your help here. I run the unit test in hive-4 master locally and it passes, but in branch-3, it just hangs. the command i use is
MAVEN_OPTS="-Xmx6144m" mvn clean test -pl ql -Dos.arch=x86_64 -Dmaven.javadoc.skip=true -Drat.numUnapprovedLicenses=500 -Dtest="org.apache.hadoop.hive.ql.exec.tez.TestTezOutputCommitter"
it seems locally it times out to start a cluster. I tried some other unit test in the ql/tez, some can run locally, some seem to have same problem of hanging.
it is hard to troubleshoot unit test failing if I can't run it locally. How would you run it? Thank you in advance if you can help out!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] amanraj2520 commented on pull request #4013: HIVE-27016: Backport HIVE-24629 Invoke optional output committer in TezProcessor
Posted by "amanraj2520 (via GitHub)" <gi...@apache.org>.
amanraj2520 commented on PR #4013:
URL: https://github.com/apache/hive/pull/4013#issuecomment-1416718749
Hi @yigress I was able to run this test on my local and it failed. I only cherrypicked HIVE-27016 commit.
The way I run the test is I have WSL (Windows Subsystem for Linux) setup on my local machine and I simply use this command to run the test - `mvn test -Dtest=org.apache.hadoop.hive.ql.exec.tez.TestTezOutputCommitter`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sunchao merged pull request #4013: HIVE-27016: Backport HIVE-24629 Invoke optional output committer in TezProcessor
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao merged PR #4013:
URL: https://github.com/apache/hive/pull/4013
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] yigress commented on pull request #4013: HIVE-27016: Backport HIVE-24629 Invoke optional output committer in TezProcessor
Posted by "yigress (via GitHub)" <gi...@apache.org>.
yigress commented on PR #4013:
URL: https://github.com/apache/hive/pull/4013#issuecomment-1419704812
test failures are unrelated. @marton-bod could you help review this backport?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] yigress commented on pull request #4013: HIVE-27016: Backport HIVE-24629 Invoke optional output committer in TezProcessor
Posted by "yigress (via GitHub)" <gi...@apache.org>.
yigress commented on PR #4013:
URL: https://github.com/apache/hive/pull/4013#issuecomment-1416423172
@marton-bod can you help review this backport?
I am not able to run the test locally (it just hangs) but the jenkins test log seems okay but it failed. Can you kindly share if any settings need to make local testing run? thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] yigress commented on pull request #4013: HIVE-27016: Backport HIVE-24629 Invoke optional output committer in TezProcessor
Posted by "yigress (via GitHub)" <gi...@apache.org>.
yigress commented on PR #4013:
URL: https://github.com/apache/hive/pull/4013#issuecomment-1416810178
Thank you @amanraj2520! I corrected the test case, turns out the Driver has changed between branch-3 and master, now the tests are passing in jenkins. Can you help review this?
I will try use WSL to see if i can get tests run locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org