You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by "zhengchenyu (via GitHub)" <gi...@apache.org> on 2023/07/04 10:50:40 UTC
[GitHub] [incubator-uniffle] zhengchenyu opened a new issue, #993: [Improvement][tez] Optimize the method of obtain the application attempt id.
zhengchenyu opened a new issue, #993:
URL: https://github.com/apache/incubator-uniffle/issues/993
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
For now, we obtain the application attempt id from environment variable CONTAINER_ID. But CONTAINER_ID is not set in tez local mode. Then tez local mode will throw error like this:
```
2023-07-04 17:40:12,034 INFO HistoryEventHandler.criticalEvents: [HISTORY][DAG:dag_1688463608398_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=Tokenizer, taskAttemptId=attempt_1688463608398_0001_1_00_000001_0, creationTime=1688463611329, allocationTime=1688463611918, startTime=1688463612008, finishTime=1688463612034, timeTaken=26, status=FAILED, taskFailureType=NON_FATAL, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Error while running task ( failure ) : attempt_1688463608398_0001_1_00_000001_0:java.lang.NullPointerException
at org.apache.hadoop.yarn.api.records.ContainerId.fromString(ContainerId.java:190)
at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:140)
at org.apache.tez.runtime.library.common.sort.impl.RssSorter.<init>(RssSorter.java:123)
at org.apache.tez.runtime.library.output.RssOrderedPartitionedKVOutput.start(RssOrderedPartitionedKVOutput.java:201)
at org.apache.tez.runtime.library.processor.SimpleProcessor.preOp(SimpleProcessor.java:87)
at org.apache.tez.runtime.library.processor.SimpleProcessor.run(SimpleProcessor.java:60)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
```
In fact, we can obtain application attempt id by below way, this way is effective for tez local mode and yarn mode.
```
ApplicationAttemptId.newInstance(getContext().getApplicationId(), getContext().getDAGAttemptNumber());
```
### How should we improve?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zhengchenyu commented on issue #993: [Improvement][tez] Optimize the method of obtain the application attempt id.
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on issue #993:
URL: https://github.com/apache/incubator-uniffle/issues/993#issuecomment-1620047065
The logical of tez local mode is similar to tez yarn mode, so tez local mode is useful way in development mode.
If wanna tez local mode run with uniffle, need below patch:
```
---
.../org/apache/tez/client/LocalClient.java | 26 ++++++++++++++++---
1 file changed, 22 insertions(+), 4 deletions(-)
diff --git a/tez-dag/src/main/java/org/apache/tez/client/LocalClient.java b/tez-dag/src/main/java/org/apache/tez/client/LocalClient.java
index 6baea48..4612645 100644
--- a/tez-dag/src/main/java/org/apache/tez/client/LocalClient.java
+++ b/tez-dag/src/main/java/org/apache/tez/client/LocalClient.java
@@ -45,10 +45,12 @@ import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
import org.apache.hadoop.yarn.api.records.ApplicationReport;
import org.apache.hadoop.yarn.api.records.YarnApplicationState;
import org.apache.hadoop.yarn.util.SystemClock;
+import org.apache.tez.common.ReflectionUtils;
import org.apache.tez.common.TezCommonUtils;
import org.apache.tez.common.TezUtilsInternal;
import org.apache.tez.dag.api.TezConfiguration;
import org.apache.tez.dag.api.TezException;
+import org.apache.tez.dag.api.TezReflectionException;
import org.apache.tez.dag.api.client.DAGClientHandler;
import org.apache.tez.dag.api.records.DAGProtos;
import org.apache.tez.dag.api.records.DAGProtos.AMPluginDescriptorProto;
@@ -312,7 +314,7 @@ public class LocalClient extends FrameworkClient {
}
// Construct, initialize, and start the DAGAppMaster
- ApplicationAttemptId applicationAttemptId = ApplicationAttemptId.newInstance(appId, 0);
+ ApplicationAttemptId applicationAttemptId = ApplicationAttemptId.newInstance(appId, 1);
ContainerId cId = ContainerId.newInstance(applicationAttemptId, 1);
String currentHost = InetAddress.getLocalHost().getHostName();
int nmPort = YarnConfiguration.DEFAULT_NM_PORT;
@@ -360,9 +362,25 @@ public class LocalClient extends FrameworkClient {
getPluginDescriptorInfo(conf, applicationAttemptId.getApplicationId().toString());
- return new DAGAppMaster(applicationAttemptId, cId, currentHost, nmPort, nmHttpPort,
- new SystemClock(), appSubmitTime, isSession, userDir, localDirs, logDirs,
- versionInfo.getVersion(), credentials, jobUserName, amPluginDescriptorProto);
+ String className = this.conf.get("tez.local.appmaster.class", DAGAppMaster.class.getName());
+ try {
+ return ReflectionUtils.createClazzInstance(className,
+ new Class[] {
+ ApplicationAttemptId.class, ContainerId.class, String.class, int.class, int.class,
+ Clock.class, long.class, boolean.class, String.class, String[].class, String[].class,
+ String.class, Credentials.class, String.class, AMPluginDescriptorProto.class
+ },
+ new Object[] {
+ applicationAttemptId, cId, currentHost, nmPort, nmHttpPort,
+ new SystemClock(), appSubmitTime, isSession, userDir, localDirs, logDirs,
+ versionInfo.getVersion(), credentials, jobUserName, amPluginDescriptorProto
+ }
+ );
+ } catch (TezReflectionException e) {
+ return new DAGAppMaster(applicationAttemptId, cId, currentHost, nmPort, nmHttpPort,
+ new SystemClock(), appSubmitTime, isSession, userDir, localDirs, logDirs,
+ versionInfo.getVersion(), credentials, jobUserName, amPluginDescriptorProto);
+ }
}
private AMPluginDescriptorProto getPluginDescriptorInfo(Configuration conf,
--
```
And the tez local application command is:
```
HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_HOME}/*:${TEZ_HOME}/lib/*:rss-client-tez-XXX-shaded.jar
hadoop jar ${TEZ_HOME}/tez-examples-0.9.2.jar wordcount \
-Dhadoop.security.authentication=simple \
-local \
/XXX/examples/tez-examples/data/wordcount/input \
/XXX/examples/tez-examples/data/wordcount/output \
2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #993: [Improvement][tez] Optimize the method of obtain the application attempt id.
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #993:
URL: https://github.com/apache/incubator-uniffle/issues/993#issuecomment-1620052970
https://github.com/apache/incubator-uniffle/tree/master/patch/tez
Could you submit this patch to this directory and add the document in the readme?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi closed issue #993: [Improvement][tez] Optimize the method of obtain the application attempt id.
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi closed issue #993: [Improvement][tez] Optimize the method of obtain the application attempt id.
URL: https://github.com/apache/incubator-uniffle/issues/993
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org