You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@airavata.apache.org by ajinkya-dhamnaskar <gi...@git.apache.org> on 2016/10/04 18:59:45 UTC
[GitHub] airavata pull request #52: Local provider Fix
GitHub user ajinkya-dhamnaskar opened a pull request:
https://github.com/apache/airavata/pull/52
Local provider Fix
1. Added SECURITY_PROTOCOL column to LOCAL_SUBMISSION table.
2. Made security protocol mandatory for local job submission
3. Made local job submission to work with existing implementation.
4. Relaxed process folder privileges to 744 (earlier 644)
5. Provisioned local and third party file transfer in output staging for the same.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ajinkya-dhamnaskar/airavata local-provider
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/airavata/pull/52.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #52
----
commit 8f91bdee95c0b27825fe9f59d72a48dad1168149
Author: Ajinkya Dhamnaskar <ad...@umail.iu.edu>
Date: 2016-10-04T18:58:44Z
AIRAVATA-2113. Fixing local provider
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] airavata pull request #52: Local provider Fix
Posted by shamrath <gi...@git.apache.org>.
Github user shamrath commented on a diff in the pull request:
https://github.com/apache/airavata/pull/52#discussion_r83044484
--- Diff: modules/orchestrator/orchestrator-core/src/main/java/org/apache/airavata/orchestrator/cpi/impl/SimpleOrchestratorImpl.java ---
@@ -406,11 +406,17 @@ private String getTaskDag(List<String> taskIdList) {
DataType type = processOutput.getType();
switch (type) {
case STDOUT :
- processOutput.setValue(appName + ".stdout");
+ if(null == processOutput.getValue() ||
+ (null != processOutput.getValue() && processOutput.getValue().trim().isEmpty())){
--- End diff --
You can simplify this logic to (processOutput.getValue == null || processOutput.getvalue().trim().isEmpty())
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] airavata pull request #52: Local provider Fix
Posted by lahirug <gi...@git.apache.org>.
Github user lahirug commented on a diff in the pull request:
https://github.com/apache/airavata/pull/52#discussion_r82008630
--- Diff: modules/registry/registry-core/src/main/resources/appcatalog-mysql.sql ---
@@ -442,6 +442,7 @@ CREATE TABLE LOCAL_SUBMISSION
(
RESOURCE_JOB_MANAGER_ID VARCHAR (255) NOT NULL,
JOB_SUBMISSION_INTERFACE_ID VARCHAR (255) NOT NULL,
+ SECURITY_PROTOCOL VARCHAR (255) NOT NULL,
--- End diff --
How this is going to affect the production deployment ? Do we have a data migration mechanism in airavata ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] airavata pull request #52: Local provider Fix
Posted by lahirug <gi...@git.apache.org>.
Github user lahirug commented on a diff in the pull request:
https://github.com/apache/airavata/pull/52#discussion_r82008856
--- Diff: modules/gfac/gfac-impl/src/main/java/org/apache/airavata/gfac/impl/task/LocalJobSubmissionTask.java ---
@@ -56,78 +61,85 @@ public void init(Map<String, String> propertyMap) throws TaskException {
@Override
public TaskStatus execute(TaskContext taskContext) {
- /* try {
+ TaskStatus taskStatus = new TaskStatus(TaskState.CREATED);
+ try {
ProcessContext processContext = taskContext.getParentProcessContext();
- // build command with all inputs
- List<String> cmdList = buildCommand(processContext);
- initProcessBuilder(processContext.getApplicationDeploymentDescription(), cmdList);
+ JobModel jobModel = processContext.getJobModel();
+ jobModel.setTaskId(taskContext.getTaskId());
- // extra environment variables
- builder.environment().put(GFacConstants.INPUT_DATA_DIR_VAR_NAME, processContext.getInputDir());
- builder.environment().put(GFacConstants.OUTPUT_DATA_DIR_VAR_NAME, processContext.getOutputDir());
+ RemoteCluster remoteCluster = processContext.getJobSubmissionRemoteCluster();
+ GroovyMap groovyMap = GFacUtils.creatGroovyMap(processContext,taskContext);
- // set working directory
- builder.directory(new File(processContext.getWorkingDir()));
+ String jobId = AiravataUtils.getId("JOB_ID_");
+ jobModel.setJobName(groovyMap.get(Script.JOB_NAME).toString());
+ jobModel.setJobId(jobId);
- // log info
- log.info("Command = " + InputUtils.buildCommand(cmdList));
- log.info("Working dir = " + builder.directory());
- JobModel jobModel = processContext.getJobModel();
- if (jobModel == null) {
- jobModel = new JobModel();
+ ResourceJobManager resourceJobManager = GFacUtils.getResourceJobManager(processContext);
+ JobManagerConfiguration jConfig = null;
+
+ if (resourceJobManager != null) {
+ jConfig = Factory.getJobManagerConfiguration(resourceJobManager);
}
- String jobId = taskContext.getTaskModel().getTaskId();
- jobModel.setJobId(jobId);
- jobModel.setJobDescription("sample local job");
- processContext.setJobModel(jobModel);
- GFacUtils.saveJobStatus(taskContext, jobModel, JobState.SUBMITTED);
- // running cmd
- Process process = builder.start();
-
- Thread standardOutWriter = new InputStreamToFileWriter(process.getInputStream(), processContext.getStdoutLocation());
- Thread standardErrorWriter = new InputStreamToFileWriter(process.getErrorStream(), processContext.getStderrLocation());
-
- // start output threads
- standardOutWriter.setDaemon(true);
- standardErrorWriter.setDaemon(true);
- standardOutWriter.start();
- standardErrorWriter.start();
-
- int returnValue = process.waitFor();
-
- // make sure other two threads are done
- standardOutWriter.join();
- standardErrorWriter.join();
-
- *//*
- * check return value. usually not very helpful to draw conclusions based on return values so don't bother.
- * just provide warning in the log messages
- *//*
- if (returnValue != 0) {
- log.error("Process finished with non zero return value. Process may have failed");
+
--- End diff --
Do you think this has to split in to few methods ? Looks bit long for a single method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] airavata pull request #52: Local provider Fix
Posted by shamrath <gi...@git.apache.org>.
Github user shamrath commented on a diff in the pull request:
https://github.com/apache/airavata/pull/52#discussion_r83044405
--- Diff: modules/orchestrator/orchestrator-core/src/main/java/org/apache/airavata/orchestrator/cpi/impl/SimpleOrchestratorImpl.java ---
@@ -406,11 +406,17 @@ private String getTaskDag(List<String> taskIdList) {
DataType type = processOutput.getType();
switch (type) {
case STDOUT :
- processOutput.setValue(appName + ".stdout");
+ if(null == processOutput.getValue() ||
+ (null != processOutput.getValue() && processOutput.getValue().trim().isEmpty())){
+ processOutput.setValue(appName + ".stdout");
+ }
createOutputDataSatagingTasks(processModel, gatewayId, dataStagingTaskIds, processOutput);
break;
case STDERR:
- processOutput.setValue(appName + ".stderr");
+ if(null == processOutput.getValue() ||
--- End diff --
Simplify the logic
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] airavata pull request #52: Local provider Fix
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/airavata/pull/52
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---