You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by GitBox <gi...@apache.org> on 2020/01/08 22:41:24 UTC
[GitHub] [submarine] pingsutw opened a new pull request #145: SUBMARINE-308.
Support PyTorch with TonY runtime
pingsutw opened a new pull request #145: SUBMARINE-308. Support PyTorch with TonY runtime
URL: https://github.com/apache/submarine/pull/145
### What is this PR for?
Support PyTorch with TonY runtime
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-308
### How should this be tested?
https://travis-ci.org/pingsutw/hadoop-submarine/builds/634462654
### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw commented on issue #145: SUBMARINE-308.
Support PyTorch with TonY runtime
Posted by GitBox <gi...@apache.org>.
pingsutw commented on issue #145: SUBMARINE-308. Support PyTorch with TonY runtime
URL: https://github.com/apache/submarine/pull/145#issuecomment-572841828
Thanks @adamantal for the review.
I commit a new patch and add uint test for PyTorch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] adamantal commented on a change in pull request #145:
SUBMARINE-308. Support PyTorch with TonY runtime
Posted by GitBox <gi...@apache.org>.
adamantal commented on a change in pull request #145: SUBMARINE-308. Support PyTorch with TonY runtime
URL: https://github.com/apache/submarine/pull/145#discussion_r364862660
##########
File path: submarine-server/server-submitter/submitter-yarn/src/main/java/org/apache/submarine/server/submitter/yarn/YarnUtils.java
##########
@@ -167,4 +194,14 @@ public static Configuration tonyConfFromClientContext(
private YarnUtils() {
}
+
+ private static Resource getResource(Parameter parametersHolder, String option)
+ throws ParseException, YarnException {
+ String ResourceStr =
+ parametersHolder.getOptionValue(option);
+ if (ResourceStr == null) {
Review comment:
```suggestion
if (resourceStr == null) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] adamantal commented on a change in pull request #145:
SUBMARINE-308. Support PyTorch with TonY runtime
Posted by GitBox <gi...@apache.org>.
adamantal commented on a change in pull request #145: SUBMARINE-308. Support PyTorch with TonY runtime
URL: https://github.com/apache/submarine/pull/145#discussion_r364859857
##########
File path: submarine-server/server-submitter/submitter-yarn/src/main/java/org/apache/submarine/server/submitter/yarn/YarnJobSubmitter.java
##########
@@ -54,22 +52,18 @@ public void setTonyClient(TonyClient client) {
@Override
public ApplicationId submitJob(Parameter parameters)
throws IOException {
- if (parameters.getFramework() == Framework.PYTORCH) {
- // we need to throw an exception, as ParametersHolder's parameters field
- // could not be casted to TensorFlowRunJobParameters.
- throw new UnsupportedOperationException(
- "Support \"–-framework\" option for PyTorch in Tony is coming. " +
- "Please check the documentation about how to submit a " +
- "PyTorch job with TonY runtime.");
- }
-
LOG.info("Starting Tony runtime..");
Review comment:
Could you add a log entry here about the type of the framework?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] jojochuang commented on issue #145: SUBMARINE-308.
Support PyTorch with TonY runtime
Posted by GitBox <gi...@apache.org>.
jojochuang commented on issue #145: SUBMARINE-308. Support PyTorch with TonY runtime
URL: https://github.com/apache/submarine/pull/145#issuecomment-572409449
Change looks good. Test failure appears unrelated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw commented on a change in pull request #145:
SUBMARINE-308. Support PyTorch with TonY runtime
Posted by GitBox <gi...@apache.org>.
pingsutw commented on a change in pull request #145: SUBMARINE-308. Support PyTorch with TonY runtime
URL: https://github.com/apache/submarine/pull/145#discussion_r365045550
##########
File path: submarine-server/server-submitter/submitter-yarn/src/main/java/org/apache/submarine/server/submitter/yarn/YarnUtils.java
##########
@@ -167,4 +194,14 @@ public static Configuration tonyConfFromClientContext(
private YarnUtils() {
}
+
+ private static Resource getResource(Parameter parametersHolder, String option)
+ throws ParseException, YarnException {
+ String ResourceStr =
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw commented on a change in pull request #145:
SUBMARINE-308. Support PyTorch with TonY runtime
Posted by GitBox <gi...@apache.org>.
pingsutw commented on a change in pull request #145: SUBMARINE-308. Support PyTorch with TonY runtime
URL: https://github.com/apache/submarine/pull/145#discussion_r365045465
##########
File path: submarine-server/server-submitter/submitter-yarn/src/main/java/org/apache/submarine/server/submitter/yarn/YarnJobSubmitter.java
##########
@@ -54,22 +52,18 @@ public void setTonyClient(TonyClient client) {
@Override
public ApplicationId submitJob(Parameter parameters)
throws IOException {
- if (parameters.getFramework() == Framework.PYTORCH) {
- // we need to throw an exception, as ParametersHolder's parameters field
- // could not be casted to TensorFlowRunJobParameters.
- throw new UnsupportedOperationException(
- "Support \"–-framework\" option for PyTorch in Tony is coming. " +
- "Please check the documentation about how to submit a " +
- "PyTorch job with TonY runtime.");
- }
-
LOG.info("Starting Tony runtime..");
File tonyFinalConfPath = File.createTempFile("temp",
Constants.TONY_FINAL_XML);
// Write user's overridden conf to an xml to be localized.
- Configuration tonyConf = YarnUtils.tonyConfFromClientContext(
- (TensorFlowRunJobParameters) parameters.getParameters());
+ Configuration tonyConf = null;
+ try {
+ tonyConf = YarnUtils.tonyConfFromClientContext(
+ (ParametersHolder) parameters);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to create tony conf from client context", e);
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] adamantal commented on a change in pull request #145:
SUBMARINE-308. Support PyTorch with TonY runtime
Posted by GitBox <gi...@apache.org>.
adamantal commented on a change in pull request #145: SUBMARINE-308. Support PyTorch with TonY runtime
URL: https://github.com/apache/submarine/pull/145#discussion_r364862225
##########
File path: submarine-server/server-submitter/submitter-yarn/src/main/java/org/apache/submarine/server/submitter/yarn/YarnUtils.java
##########
@@ -42,107 +48,128 @@
private static final Log LOG = LogFactory.getLog(YarnUtils.class);
public static Configuration tonyConfFromClientContext(
- TensorFlowRunJobParameters parameters) {
+ ParametersHolder parameters) throws YarnException, ParseException {
Configuration tonyConf = new Configuration();
// Add tony.xml for configuration.
tonyConf.addResource(Constants.TONY_XML);
- tonyConf.setInt(
+ tonyConf.setStrings(TonyConfigurationKeys.FRAMEWORK_NAME,
+ parameters.getFramework().getValue());
+ tonyConf.setStrings(TonyConfigurationKeys.APPLICATION_NAME,
+ parameters.getParameters().getName());
+ tonyConf.setStrings(
TonyConfigurationKeys.getInstancesKey(Constants.WORKER_JOB_NAME),
- parameters.getNumWorkers());
- tonyConf.setInt(
- TonyConfigurationKeys.getInstancesKey(Constants.PS_JOB_NAME),
- parameters.getNumPS());
+ parameters.getOptionValue(CliConstants.N_WORKERS));
+ if (parameters.getOptionValue(CliConstants.N_PS) != null) {
+ tonyConf.setStrings(
+ TonyConfigurationKeys.getInstancesKey(Constants.PS_JOB_NAME),
+ parameters.getOptionValue(CliConstants.N_PS));
+ }
// Resources for PS & Worker
- if (parameters.getPsResource() != null) {
+ if (parameters.getOptionValue(CliConstants.PS_RES) != null) {
+ Resource resource = getResource(parameters, CliConstants.PS_RES);
Review comment:
```suggestion
Resource psResource = getResource(parameters, CliConstants.PS_RES);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw commented on issue #145: SUBMARINE-308.
Support PyTorch with TonY runtime
Posted by GitBox <gi...@apache.org>.
pingsutw commented on issue #145: SUBMARINE-308. Support PyTorch with TonY runtime
URL: https://github.com/apache/submarine/pull/145#issuecomment-573293690
Will merge if no more comments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] adamantal commented on a change in pull request #145:
SUBMARINE-308. Support PyTorch with TonY runtime
Posted by GitBox <gi...@apache.org>.
adamantal commented on a change in pull request #145: SUBMARINE-308. Support PyTorch with TonY runtime
URL: https://github.com/apache/submarine/pull/145#discussion_r364861537
##########
File path: submarine-server/server-submitter/submitter-yarn/src/main/java/org/apache/submarine/server/submitter/yarn/YarnJobSubmitter.java
##########
@@ -54,22 +52,18 @@ public void setTonyClient(TonyClient client) {
@Override
public ApplicationId submitJob(Parameter parameters)
throws IOException {
- if (parameters.getFramework() == Framework.PYTORCH) {
- // we need to throw an exception, as ParametersHolder's parameters field
- // could not be casted to TensorFlowRunJobParameters.
- throw new UnsupportedOperationException(
- "Support \"–-framework\" option for PyTorch in Tony is coming. " +
- "Please check the documentation about how to submit a " +
- "PyTorch job with TonY runtime.");
- }
-
LOG.info("Starting Tony runtime..");
File tonyFinalConfPath = File.createTempFile("temp",
Constants.TONY_FINAL_XML);
// Write user's overridden conf to an xml to be localized.
- Configuration tonyConf = YarnUtils.tonyConfFromClientContext(
- (TensorFlowRunJobParameters) parameters.getParameters());
+ Configuration tonyConf = null;
+ try {
+ tonyConf = YarnUtils.tonyConfFromClientContext(
+ (ParametersHolder) parameters);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to create tony conf from client context", e);
Review comment:
Maybe throw a `SubmarineException` here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw commented on a change in pull request #145:
SUBMARINE-308. Support PyTorch with TonY runtime
Posted by GitBox <gi...@apache.org>.
pingsutw commented on a change in pull request #145: SUBMARINE-308. Support PyTorch with TonY runtime
URL: https://github.com/apache/submarine/pull/145#discussion_r365045534
##########
File path: submarine-server/server-submitter/submitter-yarn/src/main/java/org/apache/submarine/server/submitter/yarn/YarnUtils.java
##########
@@ -42,107 +48,128 @@
private static final Log LOG = LogFactory.getLog(YarnUtils.class);
public static Configuration tonyConfFromClientContext(
- TensorFlowRunJobParameters parameters) {
+ ParametersHolder parameters) throws YarnException, ParseException {
Configuration tonyConf = new Configuration();
// Add tony.xml for configuration.
tonyConf.addResource(Constants.TONY_XML);
- tonyConf.setInt(
+ tonyConf.setStrings(TonyConfigurationKeys.FRAMEWORK_NAME,
+ parameters.getFramework().getValue());
+ tonyConf.setStrings(TonyConfigurationKeys.APPLICATION_NAME,
+ parameters.getParameters().getName());
+ tonyConf.setStrings(
TonyConfigurationKeys.getInstancesKey(Constants.WORKER_JOB_NAME),
- parameters.getNumWorkers());
- tonyConf.setInt(
- TonyConfigurationKeys.getInstancesKey(Constants.PS_JOB_NAME),
- parameters.getNumPS());
+ parameters.getOptionValue(CliConstants.N_WORKERS));
+ if (parameters.getOptionValue(CliConstants.N_PS) != null) {
+ tonyConf.setStrings(
+ TonyConfigurationKeys.getInstancesKey(Constants.PS_JOB_NAME),
+ parameters.getOptionValue(CliConstants.N_PS));
+ }
// Resources for PS & Worker
- if (parameters.getPsResource() != null) {
+ if (parameters.getOptionValue(CliConstants.PS_RES) != null) {
+ Resource resource = getResource(parameters, CliConstants.PS_RES);
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw commented on a change in pull request #145:
SUBMARINE-308. Support PyTorch with TonY runtime
Posted by GitBox <gi...@apache.org>.
pingsutw commented on a change in pull request #145: SUBMARINE-308. Support PyTorch with TonY runtime
URL: https://github.com/apache/submarine/pull/145#discussion_r365045442
##########
File path: submarine-server/server-submitter/submitter-yarn/src/main/java/org/apache/submarine/server/submitter/yarn/YarnJobSubmitter.java
##########
@@ -54,22 +52,18 @@ public void setTonyClient(TonyClient client) {
@Override
public ApplicationId submitJob(Parameter parameters)
throws IOException {
- if (parameters.getFramework() == Framework.PYTORCH) {
- // we need to throw an exception, as ParametersHolder's parameters field
- // could not be casted to TensorFlowRunJobParameters.
- throw new UnsupportedOperationException(
- "Support \"–-framework\" option for PyTorch in Tony is coming. " +
- "Please check the documentation about how to submit a " +
- "PyTorch job with TonY runtime.");
- }
-
LOG.info("Starting Tony runtime..");
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw commented on a change in pull request #145:
SUBMARINE-308. Support PyTorch with TonY runtime
Posted by GitBox <gi...@apache.org>.
pingsutw commented on a change in pull request #145: SUBMARINE-308. Support PyTorch with TonY runtime
URL: https://github.com/apache/submarine/pull/145#discussion_r365045566
##########
File path: submarine-server/server-submitter/submitter-yarn/src/main/java/org/apache/submarine/server/submitter/yarn/YarnUtils.java
##########
@@ -167,4 +194,14 @@ public static Configuration tonyConfFromClientContext(
private YarnUtils() {
}
+
+ private static Resource getResource(Parameter parametersHolder, String option)
+ throws ParseException, YarnException {
+ String ResourceStr =
+ parametersHolder.getOptionValue(option);
+ if (ResourceStr == null) {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] pingsutw commented on a change in pull request #145:
SUBMARINE-308. Support PyTorch with TonY runtime
Posted by GitBox <gi...@apache.org>.
pingsutw commented on a change in pull request #145: SUBMARINE-308. Support PyTorch with TonY runtime
URL: https://github.com/apache/submarine/pull/145#discussion_r365045542
##########
File path: submarine-server/server-submitter/submitter-yarn/src/main/java/org/apache/submarine/server/submitter/yarn/YarnUtils.java
##########
@@ -42,107 +48,128 @@
private static final Log LOG = LogFactory.getLog(YarnUtils.class);
public static Configuration tonyConfFromClientContext(
- TensorFlowRunJobParameters parameters) {
+ ParametersHolder parameters) throws YarnException, ParseException {
Configuration tonyConf = new Configuration();
// Add tony.xml for configuration.
tonyConf.addResource(Constants.TONY_XML);
- tonyConf.setInt(
+ tonyConf.setStrings(TonyConfigurationKeys.FRAMEWORK_NAME,
+ parameters.getFramework().getValue());
+ tonyConf.setStrings(TonyConfigurationKeys.APPLICATION_NAME,
+ parameters.getParameters().getName());
+ tonyConf.setStrings(
TonyConfigurationKeys.getInstancesKey(Constants.WORKER_JOB_NAME),
- parameters.getNumWorkers());
- tonyConf.setInt(
- TonyConfigurationKeys.getInstancesKey(Constants.PS_JOB_NAME),
- parameters.getNumPS());
+ parameters.getOptionValue(CliConstants.N_WORKERS));
+ if (parameters.getOptionValue(CliConstants.N_PS) != null) {
+ tonyConf.setStrings(
+ TonyConfigurationKeys.getInstancesKey(Constants.PS_JOB_NAME),
+ parameters.getOptionValue(CliConstants.N_PS));
+ }
// Resources for PS & Worker
- if (parameters.getPsResource() != null) {
+ if (parameters.getOptionValue(CliConstants.PS_RES) != null) {
+ Resource resource = getResource(parameters, CliConstants.PS_RES);
+
tonyConf.setInt(
TonyConfigurationKeys.getResourceKey(Constants.PS_JOB_NAME,
Constants.VCORES),
- parameters.getPsResource().getVirtualCores());
+ resource.getVirtualCores());
tonyConf.setLong(
TonyConfigurationKeys.getResourceKey(Constants.PS_JOB_NAME,
Constants.MEMORY),
- ResourceUtils.getMemorySize(parameters.getPsResource()));
+ ResourceUtils.getMemorySize(resource));
}
- if (parameters.getWorkerResource() != null) {
+ if (parameters.getOptionValue(CliConstants.WORKER_RES) != null) {
+ Resource resource = getResource(parameters, CliConstants.WORKER_RES);
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] adamantal commented on a change in pull request #145:
SUBMARINE-308. Support PyTorch with TonY runtime
Posted by GitBox <gi...@apache.org>.
adamantal commented on a change in pull request #145: SUBMARINE-308. Support PyTorch with TonY runtime
URL: https://github.com/apache/submarine/pull/145#discussion_r364862343
##########
File path: submarine-server/server-submitter/submitter-yarn/src/main/java/org/apache/submarine/server/submitter/yarn/YarnUtils.java
##########
@@ -42,107 +48,128 @@
private static final Log LOG = LogFactory.getLog(YarnUtils.class);
public static Configuration tonyConfFromClientContext(
- TensorFlowRunJobParameters parameters) {
+ ParametersHolder parameters) throws YarnException, ParseException {
Configuration tonyConf = new Configuration();
// Add tony.xml for configuration.
tonyConf.addResource(Constants.TONY_XML);
- tonyConf.setInt(
+ tonyConf.setStrings(TonyConfigurationKeys.FRAMEWORK_NAME,
+ parameters.getFramework().getValue());
+ tonyConf.setStrings(TonyConfigurationKeys.APPLICATION_NAME,
+ parameters.getParameters().getName());
+ tonyConf.setStrings(
TonyConfigurationKeys.getInstancesKey(Constants.WORKER_JOB_NAME),
- parameters.getNumWorkers());
- tonyConf.setInt(
- TonyConfigurationKeys.getInstancesKey(Constants.PS_JOB_NAME),
- parameters.getNumPS());
+ parameters.getOptionValue(CliConstants.N_WORKERS));
+ if (parameters.getOptionValue(CliConstants.N_PS) != null) {
+ tonyConf.setStrings(
+ TonyConfigurationKeys.getInstancesKey(Constants.PS_JOB_NAME),
+ parameters.getOptionValue(CliConstants.N_PS));
+ }
// Resources for PS & Worker
- if (parameters.getPsResource() != null) {
+ if (parameters.getOptionValue(CliConstants.PS_RES) != null) {
+ Resource resource = getResource(parameters, CliConstants.PS_RES);
+
tonyConf.setInt(
TonyConfigurationKeys.getResourceKey(Constants.PS_JOB_NAME,
Constants.VCORES),
- parameters.getPsResource().getVirtualCores());
+ resource.getVirtualCores());
tonyConf.setLong(
TonyConfigurationKeys.getResourceKey(Constants.PS_JOB_NAME,
Constants.MEMORY),
- ResourceUtils.getMemorySize(parameters.getPsResource()));
+ ResourceUtils.getMemorySize(resource));
}
- if (parameters.getWorkerResource() != null) {
+ if (parameters.getOptionValue(CliConstants.WORKER_RES) != null) {
+ Resource resource = getResource(parameters, CliConstants.WORKER_RES);
Review comment:
```suggestion
Resource workerResource = getResource(parameters, CliConstants.WORKER_RES);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] adamantal commented on a change in pull request #145:
SUBMARINE-308. Support PyTorch with TonY runtime
Posted by GitBox <gi...@apache.org>.
adamantal commented on a change in pull request #145: SUBMARINE-308. Support PyTorch with TonY runtime
URL: https://github.com/apache/submarine/pull/145#discussion_r364862580
##########
File path: submarine-server/server-submitter/submitter-yarn/src/main/java/org/apache/submarine/server/submitter/yarn/YarnUtils.java
##########
@@ -167,4 +194,14 @@ public static Configuration tonyConfFromClientContext(
private YarnUtils() {
}
+
+ private static Resource getResource(Parameter parametersHolder, String option)
+ throws ParseException, YarnException {
+ String ResourceStr =
Review comment:
```suggestion
String resourceStr =
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] asfgit closed pull request #145: SUBMARINE-308. Support
PyTorch with TonY runtime
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #145: SUBMARINE-308. Support PyTorch with TonY runtime
URL: https://github.com/apache/submarine/pull/145
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org