You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zhengchenyu (via GitHub)" <gi...@apache.org> on 2023/06/25 02:28:50 UTC

[GitHub] [incubator-uniffle] zhengchenyu opened a new pull request, #966: [965] Tez: support remote shuffle for tez framework

zhengchenyu opened a new pull request, #966:
URL: https://github.com/apache/incubator-uniffle/pull/966

   ### What changes were proposed in this pull request?
   
   A method to support remote shuffle for tez framework.
   
   ### How was this patch tested?
   
   test in real cluster and integration test.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zhengchenyu commented on pull request #966: [965] Tez: support remote shuffle for tez framework

Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #966:
URL: https://github.com/apache/incubator-uniffle/pull/966#issuecomment-1606055507

   @jerqi Oh, I catch it! Thank you for your advice! I will do it! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zhengchenyu commented on pull request #966: [965] Tez: support remote shuffle for tez framework

Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #966:
URL: https://github.com/apache/incubator-uniffle/pull/966#issuecomment-1605829731

   If we want to run tez job with unffile, two way must do:
   
   * (1) Choose RssDagMaster
    use this config:
   
   | config |  value |
   | --- | --- |
   | tez.am.log.level | INFO,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.RssDAGAppMaster avoidtrim |
   
   * (2) Rename IO Name
   Rename io name in code, no config should be configured.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #966: [#965] Tez: support remote shuffle for tez framework

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #966:
URL: https://github.com/apache/incubator-uniffle/pull/966#discussion_r1245108967


##########
client-tez/src/main/java/org/apache/tez/dag/app/RssDAGAppMaster.java:
##########
@@ -228,15 +242,21 @@ public static void main(String[] args) {
       String jobUserName = System
               .getenv(ApplicationConstants.Environment.USER.name());
 
-      // Command line options
-      Options opts = new Options();
-      opts.addOption(TezConstants.TEZ_SESSION_MODE_CLI_OPTION,
-              false, "Run Tez Application Master in Session mode");
-
-      CommandLine cliParser = new GnuParser().parse(opts, args);
-      boolean sessionModeCliOption = cliParser.hasOption(TezConstants.TEZ_SESSION_MODE_CLI_OPTION);
+      boolean sessionModeCliOption = false;
+      for (int i = 0; i < args.length; i++) {
+        if (args[i].startsWith("-D")) {
+          String[] property = args[i].split("=");
+          if (property.length < 2) {
+            System.setProperty(property[0], "");
+          } else {
+            System.setProperty(property[0], property[1]);
+          }
+        } else if (args[i].contains("--session") || args[i].contains("-s")) {
+          sessionModeCliOption = true;
+        }
+      }

Review Comment:
   ```
   private static final Logger LOG = LoggerFactory.getLogger(RssTezUtils.class);
   ```
   Logger should be initialized after we set property. Because the initialization of Logger depends on some property.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #966: [#965] Tez: support remote shuffle for tez framework

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #966:
URL: https://github.com/apache/incubator-uniffle/pull/966#discussion_r1245108967


##########
client-tez/src/main/java/org/apache/tez/dag/app/RssDAGAppMaster.java:
##########
@@ -228,15 +242,21 @@ public static void main(String[] args) {
       String jobUserName = System
               .getenv(ApplicationConstants.Environment.USER.name());
 
-      // Command line options
-      Options opts = new Options();
-      opts.addOption(TezConstants.TEZ_SESSION_MODE_CLI_OPTION,
-              false, "Run Tez Application Master in Session mode");
-
-      CommandLine cliParser = new GnuParser().parse(opts, args);
-      boolean sessionModeCliOption = cliParser.hasOption(TezConstants.TEZ_SESSION_MODE_CLI_OPTION);
+      boolean sessionModeCliOption = false;
+      for (int i = 0; i < args.length; i++) {
+        if (args[i].startsWith("-D")) {
+          String[] property = args[i].split("=");
+          if (property.length < 2) {
+            System.setProperty(property[0], "");
+          } else {
+            System.setProperty(property[0], property[1]);
+          }
+        } else if (args[i].contains("--session") || args[i].contains("-s")) {
+          sessionModeCliOption = true;
+        }
+      }

Review Comment:
   ```
   private static final Logger LOG = LoggerFactory.getLogger(RssTezUtils.class);
   ```
   Logger should be initialed after we set property. Because the initialization of Logger depends on some property.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #966: [#965] Tez: support remote shuffle for tez framework

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #966:
URL: https://github.com/apache/incubator-uniffle/pull/966#discussion_r1245108967


##########
client-tez/src/main/java/org/apache/tez/dag/app/RssDAGAppMaster.java:
##########
@@ -228,15 +242,21 @@ public static void main(String[] args) {
       String jobUserName = System
               .getenv(ApplicationConstants.Environment.USER.name());
 
-      // Command line options
-      Options opts = new Options();
-      opts.addOption(TezConstants.TEZ_SESSION_MODE_CLI_OPTION,
-              false, "Run Tez Application Master in Session mode");
-
-      CommandLine cliParser = new GnuParser().parse(opts, args);
-      boolean sessionModeCliOption = cliParser.hasOption(TezConstants.TEZ_SESSION_MODE_CLI_OPTION);
+      boolean sessionModeCliOption = false;
+      for (int i = 0; i < args.length; i++) {
+        if (args[i].startsWith("-D")) {
+          String[] property = args[i].split("=");
+          if (property.length < 2) {
+            System.setProperty(property[0], "");
+          } else {
+            System.setProperty(property[0], property[1]);
+          }
+        } else if (args[i].contains("--session") || args[i].contains("-s")) {
+          sessionModeCliOption = true;
+        }
+      }

Review Comment:
    private static final Logger LOG = LoggerFactory.getLogger(RssTezUtils.class);
   Logger should be initialed after we set property. Because the initialization of Logger depends on some property.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zhengchenyu commented on pull request #966: [965] Tez: support remote shuffle for tez framework

Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #966:
URL: https://github.com/apache/incubator-uniffle/pull/966#issuecomment-1605899058

   @jerqi 
   Yes, we can use setProperty. But the key problem is that we could override other config.
   
   For example: 
   -Dtez.root.logger=INFO,CLA, here INFO is the value of config 'tez.am.log.level'. If we use tez.am.launch.cmd-opts, we must override this config. 
   
   We know that the config will override the config which the values is placed between 'tez.am.log.level' and class name.
   
   For MR, the value of yarn.app.mapreduce.am.command-opts is placed before app master class name, if yarn.app.mapreduce.am.profile is disable.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] jerqi commented on pull request #966: [965] Tez: support remote shuffle for tez framework

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #966:
URL: https://github.com/apache/incubator-uniffle/pull/966#issuecomment-1605861511

   > @jerqi
   > 
   > Below is the launch command of appmaster.
   > 
   > ```
   > exec /bin/bash -c "$JAVA_HOME/bin/java  -Xmx???m -Djava.io.tmpdir=$PWD/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/xxx/logs/application_1683514060868_55595/container_e3956_1683514060868_55595_01_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.RssDAGAppMaster avoidtrim,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster --session 1>/xxx/logs/application_1683514060868_55595/container_e3956_1683514060868_55595_01_000001/stdout 2>/xxx/logs/application_1683514060868_55595/container_e3956_1683514060868_55595_01_000001/stderr "
   > ```
   > 
   > "-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC" is the value of tez.am.launch.cmd-opts. What a pity! tez.am.launch.cmd-opts is too far way from class name.
   > 
   > If we use tez.am.launch.cmd-opts, some config(eg: tez.am.log.level) must be involved into tez.am.launch.cmd-opts.
   > 
   > Though it is tricky, tez.am.log.level is nearest from class name, so I use it.
   
   It seems that we can use to call `setProperty` method in the main method to process later -D options after I ask the JVM colleagues in our team.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] jerqi commented on pull request #966: [965] Tez: support remote shuffle for tez framework

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #966:
URL: https://github.com/apache/incubator-uniffle/pull/966#issuecomment-1605831562

   > If we want to run tez job with unffile, two way must do:
   > 
   > * (1) Choose RssDagMaster
   >   use this config:
   > 
   > config	value
   > tez.am.log.level	INFO,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.RssDAGAppMaster avoidtrim
   
   For this point, could we use `tez.am.launch.cmd-opts` ? Because we use `yarn.app.mapreduce.am.command-opts` in the MapReduce framework.
   
   <img width="788" alt="企业微信截图_d3c6f542-ae74-42ad-9790-1c210b0d61b3" src="https://github.com/apache/incubator-uniffle/assets/8159038/a9469bf9-b36a-48ac-b612-3f83df259061">
   It's better to use consistent style.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zhengchenyu commented on pull request #966: [965] Tez: support remote shuffle for tez framework

Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #966:
URL: https://github.com/apache/incubator-uniffle/pull/966#issuecomment-1605855834

   @jerqi 
   
   Below is the launch command of appmaster. 
   
   ```
   exec /bin/bash -c "$JAVA_HOME/bin/java  -Xmx???m -Djava.io.tmpdir=$PWD/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/xxx/logs/application_1683514060868_55595/container_e3956_1683514060868_55595_01_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.RssDAGAppMaster avoidtrim,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster --session 1>/xxx/logs/application_1683514060868_55595/container_e3956_1683514060868_55595_01_000001/stdout 2>/xxx/logs/application_1683514060868_55595/container_e3956_1683514060868_55595_01_000001/stderr "
   ```
   
   "-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC" is the value of tez.am.launch.cmd-opts. What a pity! tez.am.launch.cmd-opts is too far way from class name. 
   
   If we use tez.am.launch.cmd-opts, some config(eg: tez.am.log.level) must be involved into tez.am.launch.cmd-opts.
   
   Though it is tricky, tez.am.log.level is nearest from class name, so i use it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] jerqi commented on pull request #966: [965] Tez: support remote shuffle for tez framework

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #966:
URL: https://github.com/apache/incubator-uniffle/pull/966#issuecomment-1605902156

   > @jerqi Yes, we can use setProperty. But the key problem is that we could override other config.
   > 
   > For example: -Dtez.root.logger=INFO,CLA, here INFO is the value of config 'tez.am.log.level'. If we use tez.am.launch.cmd-opts, we must override this config.
   > 
   > We know that the config will override the config which the values is placed between 'tez.am.log.level' and class name.
   > 
   > For MR, the value of yarn.app.mapreduce.am.command-opts is just placed before app master class name, if yarn.app.mapreduce.am.profile is disable.
   
   You don't get my point. I mean that
   ```
   exec /bin/bash -c "$JAVA_HOME/bin/java  -Xmx???m -Djava.io.tmpdir=$PWD/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC org.apache.tez.dag.app.RssDAGAppMaster  -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/xxx/logs/application_1683514060868_55595/container_e3956_1683514060868_55595_01_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel=''  --session 1>/xxx/logs/application_1683514060868_55595/container_e3956_1683514060868_55595_01_000001/stdout 2>/xxx/logs/application_1683514060868_55595/container_e3956_1683514060868_55595_01_000001/stderr "
   ```
   In RssDAGAppMaster's main method
   ```
   public static void main(String[] args) {
       for (int i = 0; i < args.length; i++) {
           if (args[i].startWith("-D")) {
              String[] property = args[i].split("=");
              System.setProperty(property[0], property[1]);
           }
       }
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zhengchenyu closed pull request #966: [965] Tez: support remote shuffle for tez framework

Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu closed pull request #966: [965] Tez: support remote shuffle for tez framework
URL: https://github.com/apache/incubator-uniffle/pull/966


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] codecov-commenter commented on pull request #966: [965] Tez: support remote shuffle for tez framework

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #966:
URL: https://github.com/apache/incubator-uniffle/pull/966#issuecomment-1605829884

   ## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/966?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#966](https://app.codecov.io/gh/apache/incubator-uniffle/pull/966?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (83e749e) into [master](https://app.codecov.io/gh/apache/incubator-uniffle/commit/0e2422533b9e88209a62ca67714e78e798d85391?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (0e24225) will **increase** coverage by `0.12%`.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master     #966      +/-   ##
   ============================================
   + Coverage     54.93%   55.06%   +0.12%     
   - Complexity     2055     2452     +397     
   ============================================
     Files           327      348      +21     
     Lines         15203    17509    +2306     
     Branches       1174     1675     +501     
   ============================================
   + Hits           8352     9641    +1289     
   - Misses         6382     7281     +899     
   - Partials        469      587     +118     
   ```
   
   
   | [Impacted Files](https://app.codecov.io/gh/apache/incubator-uniffle/pull/966?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...c/main/java/org/apache/tez/common/RssTezUtils.java](https://app.codecov.io/gh/apache/incubator-uniffle/pull/966?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y2xpZW50LXRlei9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvdGV6L2NvbW1vbi9Sc3NUZXpVdGlscy5qYXZh) | `51.94% <0.00%> (ø)` | |
   | [...n/java/org/apache/tez/dag/app/RssDAGAppMaster.java](https://app.codecov.io/gh/apache/incubator-uniffle/pull/966?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y2xpZW50LXRlei9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvdGV6L2RhZy9hcHAvUnNzREFHQXBwTWFzdGVyLmphdmE=) | `6.91% <0.00%> (ø)` | |
   
   ... and [63 files with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-uniffle/pull/966/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] zhengchenyu commented on a diff in pull request #966: [#965] feat(tez): support remote shuffle for tez framework

Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on code in PR #966:
URL: https://github.com/apache/incubator-uniffle/pull/966#discussion_r1246028915


##########
client-tez/src/main/java/org/apache/tez/dag/app/RssDAGAppMaster.java:
##########
@@ -228,15 +242,21 @@ public static void main(String[] args) {
       String jobUserName = System
               .getenv(ApplicationConstants.Environment.USER.name());
 
-      // Command line options
-      Options opts = new Options();
-      opts.addOption(TezConstants.TEZ_SESSION_MODE_CLI_OPTION,
-              false, "Run Tez Application Master in Session mode");
-
-      CommandLine cliParser = new GnuParser().parse(opts, args);
-      boolean sessionModeCliOption = cliParser.hasOption(TezConstants.TEZ_SESSION_MODE_CLI_OPTION);
+      boolean sessionModeCliOption = false;
+      for (int i = 0; i < args.length; i++) {
+        if (args[i].startsWith("-D")) {
+          String[] property = args[i].split("=");
+          if (property.length < 2) {
+            System.setProperty(property[0], "");
+          } else {
+            System.setProperty(property[0], property[1]);
+          }
+        } else if (args[i].contains("--session") || args[i].contains("-s")) {
+          sessionModeCliOption = true;
+        }
+      }

Review Comment:
   That's a great catch! I just found the syslog of app master was missing. I will try to solve this problem.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] jerqi merged pull request #966: [#965] feat(tez): support remote shuffle for tez framework

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi merged PR #966:
URL: https://github.com/apache/incubator-uniffle/pull/966


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org