You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by DImuthu Upeksha <di...@gmail.com> on 2015/08/17 09:00:33 UTC
Airavata fails to scp files from localhost
Hi all,
I'm submitting jobs to gw77 server to test functionalities of the GridChem
client. I have input files for a gaussian experiment in the same server. So
I passed the value for input file as
file://airavata@localhost
:/home/airavata/oodt/archive/hpn40deftk7r1tvgo2545lfoqc/c4b4nhtwbs3.inp
When I launch the experiment, I get an error saying
2015-08-17 02:45:13,162 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.ssh.util.GFACSSHUtils - An exception occurred
while connecting to server.Connecting server - localhost:22 connecting user
name - airavata
org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred
while connecting to server.Connecting server - localhost:22 connecting user
name - airavata
What could be the reason for that? As far as I understood, it seems like
Airavata is unable to scp a file from localhost. Full log is as follows.
2015-08-17 02:45:10,995 [pool-3-thread-5] INFO
org.apache.airavata.gfac.core.utils.GFacUtils - This is a new Job, so
creating all the experiment docs from the scratch
2015-08-17 02:45:11,017 [pool-3-thread-5] INFO
org.apache.airavata.gfac.core.utils.GFacUtils - Created the node:
/gfac-experiments/gfac-node0/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262/state
successfully !
2015-08-17 02:45:11,028 [pool-3-thread-5] INFO
org.apache.airavata.gfac.server.GfacServerHandler -
-----------------------------------------------------90-----------------------------------------------------
2015-08-17 02:45:11,028 [pool-3-thread-5] INFO
org.apache.airavata.gfac.server.GfacServerHandler -
Id:dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 : GFac
Received submit job request for the Experiment:
dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 TaskId:
tempNode_73058234-5ac6-4278-83f2-c27e23474182
2015-08-17 02:45:11,735 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
at this point
2015-08-17 02:45:11,739 [pool-8-thread-54] INFO
org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
internal state changed to: ACCEPTED
2015-08-17 02:45:11,882 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
at this point
2015-08-17 02:45:11,882 [pool-8-thread-54] INFO
org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
internal state changed to: INHANDLERSINVOKING
2015-08-17 02:45:11,969 [pool-8-thread-54] INFO
org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler - Invoking
ApplicationDescriptorCheckHandler ...
2015-08-17 02:45:11,985 [pool-8-thread-54] INFO
org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler -
Recoverable data is saving to zk:
/oasis/scratch/comet/scigap/temp_project/airavata-workdirs,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262/Gaussian.stdout,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262/Gaussian.stderr
2015-08-17 02:45:12,206 [pool-8-thread-54] INFO
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster - Listing
directory: comet.sdsc.edu:~/
2015-08-17 02:45:12,412 [pool-8-thread-54] INFO
org.apache.airavata.gfac.ssh.util.GFACSSHUtils - Re-using the same
connection used with the connection string:scigapcomet.sdsc.edu22
2015-08-17 02:45:12,413 [pool-8-thread-54] INFO
org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler - Setup SSH
job directorties
2015-08-17 02:45:12,463 [pool-8-thread-54] INFO
org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler -
Successfully retrieved the Security Context
2015-08-17 02:45:12,463 [pool-8-thread-54] INFO
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster - Creating
directory: comet.sdsc.edu:
/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262
2015-08-17 02:45:12,973 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler - Previously
stored data is wrong so we continue the operations
2015-08-17 02:45:13,162 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.ssh.util.GFACSSHUtils - An exception occurred
while connecting to server.Connecting server - localhost:22 connecting user
name - airavata
org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred
while connecting to server.Connecting server - localhost:22 connecting user
name - airavata
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at
org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:13,164 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.GFacException - Error adding security Context
org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred
while connecting to server.Connecting server - localhost:22 connecting user
name - airavata
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at
org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:13,164 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler - Error
adding security Context
2015-08-17 02:45:13,247 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.GFacException - Error while input File Staging
org.apache.airavata.gfac.GFacException: Error adding security Context
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at
org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - localhost:22
connecting user name - airavata
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:13,247 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.core.handler.GFacHandlerException - Error while
input File Staging
org.apache.airavata.gfac.GFacException: Error adding security Context
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at
org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - localhost:22
connecting user name - airavata
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:13,248 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.GFacException - Error Executing a InFlow Handler
org.apache.airavata.gfac.GFacException: Error adding security Context
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at
org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - localhost:22
connecting user name - airavata
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:13,248 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error
Executing a InFlow Handler
org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error adding security
Context
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at
org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - localhost:22
connecting user name - airavata
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:13,253 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
at this point
2015-08-17 02:45:13,253 [pool-8-thread-54] INFO
org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
internal state changed to: FAILED
2015-08-17 02:45:13,253 [pool-8-thread-54] INFO
org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
Experiment Failed, So removing the ZK entry for the
experimentdimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262
2015-08-17 02:45:13,253 [pool-8-thread-54] INFO
org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
Zookeeper experiment Path:
/gfac-experiments/gfac-node0/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262
2015-08-17 02:45:13,254 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error occured during
updating the statuses of Experiments,tasks or Job statuses to failed,
NullPointerException occurred because at this point there might not have
Job Created
org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow
Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
... 7 more
Caused by: org.apache.airavata.gfac.GFacException: Error adding security
Context
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at
org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - localhost:22
connecting user name - airavata
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:13,254 [pool-8-thread-54] INFO
org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator -
expId - dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262:
Updating workflow node status for
tempNode_41ca8687-37ab-492c-bce0-09058ccc1c02:FAILED
2015-08-17 02:45:13,617 [pool-8-thread-54] INFO
org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
Publishing status to rabbitmq...
2015-08-17 02:45:14,721 [pool-8-thread-54] INFO
org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator -
Updating the experiment status of experiment:
dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 to FAILED
2015-08-17 02:45:14,771 [pool-8-thread-54] INFO
org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
Publishing status to rabbitmq...
2015-08-17 02:45:14,777 [pool-8-thread-54] INFO
org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator -
Clean up task - Acknowledging delivery Tag - 91
2015-08-17 02:45:15,017 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error
Executing a InFlow Handler
org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow
Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
... 7 more
Caused by: org.apache.airavata.gfac.GFacException: Error adding security
Context
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at
org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - localhost:22
connecting user name - airavata
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:15,099 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.GFacException - Error launching the Job
org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:743)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
Handlers:Error Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
... 6 more
Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow
Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
... 7 more
Caused by: org.apache.airavata.gfac.GFacException: Error adding security
Context
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at
org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - localhost:22
connecting user name - airavata
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:15,100 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error inovoking the job
with experiment ID:
dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262:Error
launching the Job
2015-08-17 02:45:15,218 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
at this point
2015-08-17 02:45:15,218 [pool-8-thread-54] INFO
org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
internal state changed to: FAILED
2015-08-17 02:45:15,218 [pool-8-thread-54] INFO
org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
Experiment Failed, So removing the ZK entry for the
experimentdimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262
2015-08-17 02:45:15,218 [pool-8-thread-54] INFO
org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
Zookeeper experiment Path:
/gfac-experiments/gfac-node0/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262
2015-08-17 02:45:15,718 [Thread-9] INFO
org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
Monitor Map is empty, no need to retrieve emails
2015-08-17 02:45:15,769 [pool-8-thread-54] INFO
org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry -
Id:tempNode_73058234-5ac6-4278-83f2-c27e23474182 : Updated task
tempNode_73058234-5ac6-4278-83f2-c27e23474182 status to
TaskStatus(executionState:FAILED, timeOfStateChange:1439793915661).
2015-08-17 02:45:15,787 [pool-8-thread-54] INFO
org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
Publishing status to rabbitmq...
2015-08-17 02:45:15,797 [pool-8-thread-54] INFO
org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator -
expId - dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262:
Updating workflow node status for
tempNode_41ca8687-37ab-492c-bce0-09058ccc1c02:FAILED
2015-08-17 02:45:16,225 [pool-8-thread-54] INFO
org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
Publishing status to rabbitmq...
2015-08-17 02:45:17,316 [pool-8-thread-54] INFO
org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator -
Updating the experiment status of experiment:
dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 to FAILED
2015-08-17 02:45:17,376 [pool-8-thread-54] INFO
org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
Publishing status to rabbitmq...
2015-08-17 02:45:17,385 [pool-8-thread-54] ERROR
org.apache.airavata.common.utils.AiravataZKUtils - Cannot find delivery
Tag in
path:/gfac-experiments/gfac-node0/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262-deliveryTag
for this experiment
2015-08-17 02:45:17,397 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.GFacException - Error launching the Job
org.apache.airavata.gfac.GFacException: Error launching the Job
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:505)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
Handlers:Error Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:743)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
... 5 more
Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
Handlers:Error Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
... 6 more
Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow
Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
... 7 more
Caused by: org.apache.airavata.gfac.GFacException: Error adding security
Context
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at
org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - localhost:22
connecting user name - airavata
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:17,401 [pool-8-thread-54] ERROR
org.apache.airavata.gfac.core.utils.InputHandlerWorker -
org.apache.airavata.gfac.GFacException: Error launching the Job
org.apache.airavata.gfac.GFacException:
org.apache.airavata.gfac.GFacException: Error launching the Job
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:200)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error launching the Job
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:505)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
... 4 more
Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
Handlers:Error Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:743)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
... 5 more
Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
Handlers:Error Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
... 6 more
Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow
Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
... 7 more
Caused by: org.apache.airavata.gfac.GFacException: Error adding security
Context
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at
org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - localhost:22
connecting user name - airavata
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:25,721 [Thread-9] INFO
org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
Monitor Map is empty, no need to retrieve emails
2015-08-17 02:45:35,723 [Thread-9] INFO
org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
Monitor Map is empty, no need to retrieve emails
2015-08-17 02:45:45,724 [Thread-9] INFO
org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
Monitor Map is empty, no need to retrieve emails
Thanks
Dimuthu
--
Regards
W.Dimuthu Upeksha
Undergraduate
Department of Computer Science And Engineering
University of Moratuwa, Sri Lanka
Re: Airavata fails to scp files from localhost
Posted by DImuthu Upeksha <di...@gmail.com>.
Fixed after setting key based authentication for localhost.
On Mon, Aug 17, 2015 at 12:30 PM, DImuthu Upeksha <
dimuthu.upeksha2@gmail.com> wrote:
> Hi all,
>
> I'm submitting jobs to gw77 server to test functionalities of the GridChem
> client. I have input files for a gaussian experiment in the same server. So
> I passed the value for input file as
>
> file://airavata@localhost
> :/home/airavata/oodt/archive/hpn40deftk7r1tvgo2545lfoqc/c4b4nhtwbs3.inp
>
> When I launch the experiment, I get an error saying
>
> 2015-08-17 02:45:13,162 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils - An exception occurred
> while connecting to server.Connecting server - localhost:22 connecting user
> name - airavata
>
> org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred
> while connecting to server.Connecting server - localhost:22 connecting user
> name - airavata
>
> What could be the reason for that? As far as I understood, it seems like
> Airavata is unable to scp a file from localhost. Full log is as follows.
>
> 2015-08-17 02:45:10,995 [pool-3-thread-5] INFO
> org.apache.airavata.gfac.core.utils.GFacUtils - This is a new Job, so
> creating all the experiment docs from the scratch
>
> 2015-08-17 02:45:11,017 [pool-3-thread-5] INFO
> org.apache.airavata.gfac.core.utils.GFacUtils - Created the node:
> /gfac-experiments/gfac-node0/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262/state
> successfully !
>
> 2015-08-17 02:45:11,028 [pool-3-thread-5] INFO
> org.apache.airavata.gfac.server.GfacServerHandler -
> -----------------------------------------------------90-----------------------------------------------------
>
> 2015-08-17 02:45:11,028 [pool-3-thread-5] INFO
> org.apache.airavata.gfac.server.GfacServerHandler -
> Id:dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 : GFac
> Received submit job request for the Experiment:
> dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 TaskId:
> tempNode_73058234-5ac6-4278-83f2-c27e23474182
>
> 2015-08-17 02:45:11,735 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
> at this point
>
> 2015-08-17 02:45:11,739 [pool-8-thread-54] INFO
> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
> internal state changed to: ACCEPTED
>
> 2015-08-17 02:45:11,882 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
> at this point
>
> 2015-08-17 02:45:11,882 [pool-8-thread-54] INFO
> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
> internal state changed to: INHANDLERSINVOKING
>
> 2015-08-17 02:45:11,969 [pool-8-thread-54] INFO
> org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler - Invoking
> ApplicationDescriptorCheckHandler ...
>
> 2015-08-17 02:45:11,985 [pool-8-thread-54] INFO
> org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler -
> Recoverable data is saving to zk:
> /oasis/scratch/comet/scigap/temp_project/airavata-workdirs,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262/Gaussian.stdout,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262/Gaussian.stderr
>
> 2015-08-17 02:45:12,206 [pool-8-thread-54] INFO
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster - Listing
> directory: comet.sdsc.edu:~/
>
> 2015-08-17 02:45:12,412 [pool-8-thread-54] INFO
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils - Re-using the same
> connection used with the connection string:scigapcomet.sdsc.edu22
>
> 2015-08-17 02:45:12,413 [pool-8-thread-54] INFO
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler - Setup SSH
> job directorties
>
> 2015-08-17 02:45:12,463 [pool-8-thread-54] INFO
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler -
> Successfully retrieved the Security Context
>
> 2015-08-17 02:45:12,463 [pool-8-thread-54] INFO
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster - Creating
> directory: comet.sdsc.edu:
> /oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262
>
> 2015-08-17 02:45:12,973 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler - Previously
> stored data is wrong so we continue the operations
>
> 2015-08-17 02:45:13,162 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils - An exception occurred
> while connecting to server.Connecting server - localhost:22 connecting user
> name - airavata
>
> org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred
> while connecting to server.Connecting server - localhost:22 connecting user
> name - airavata
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
>
> at
> org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
>
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: com.jcraft.jsch.JSchException: Auth cancel
>
> at com.jcraft.jsch.Session.connect(Session.java:511)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>
> ... 14 more
>
> 2015-08-17 02:45:13,164 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.GFacException - Error adding security Context
>
> org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred
> while connecting to server.Connecting server - localhost:22 connecting user
> name - airavata
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
>
> at
> org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
>
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: com.jcraft.jsch.JSchException: Auth cancel
>
> at com.jcraft.jsch.Session.connect(Session.java:511)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>
> ... 14 more
>
> 2015-08-17 02:45:13,164 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler - Error
> adding security Context
>
> 2015-08-17 02:45:13,247 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.GFacException - Error while input File Staging
>
> org.apache.airavata.gfac.GFacException: Error adding security Context
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
>
> at
> org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
>
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - localhost:22
> connecting user name - airavata
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
>
> ... 10 more
>
> Caused by: com.jcraft.jsch.JSchException: Auth cancel
>
> at com.jcraft.jsch.Session.connect(Session.java:511)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>
> ... 14 more
>
> 2015-08-17 02:45:13,247 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.core.handler.GFacHandlerException - Error while
> input File Staging
>
> org.apache.airavata.gfac.GFacException: Error adding security Context
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
>
> at
> org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
>
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - localhost:22
> connecting user name - airavata
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
>
> ... 10 more
>
> Caused by: com.jcraft.jsch.JSchException: Auth cancel
>
> at com.jcraft.jsch.Session.connect(Session.java:511)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>
> ... 14 more
>
> 2015-08-17 02:45:13,248 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.GFacException - Error Executing a InFlow Handler
>
> org.apache.airavata.gfac.GFacException: Error adding security Context
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
>
> at
> org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
>
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - localhost:22
> connecting user name - airavata
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
>
> ... 10 more
>
> Caused by: com.jcraft.jsch.JSchException: Auth cancel
>
> at com.jcraft.jsch.Session.connect(Session.java:511)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>
> ... 14 more
>
> 2015-08-17 02:45:13,248 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error
> Executing a InFlow Handler
>
> org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
>
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.airavata.gfac.GFacException: Error adding security
> Context
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
>
> at
> org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
>
> ... 7 more
>
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - localhost:22
> connecting user name - airavata
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
>
> ... 10 more
>
> Caused by: com.jcraft.jsch.JSchException: Auth cancel
>
> at com.jcraft.jsch.Session.connect(Session.java:511)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>
> ... 14 more
>
> 2015-08-17 02:45:13,253 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
> at this point
>
> 2015-08-17 02:45:13,253 [pool-8-thread-54] INFO
> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
> internal state changed to: FAILED
>
> 2015-08-17 02:45:13,253 [pool-8-thread-54] INFO
> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
> Experiment Failed, So removing the ZK entry for the
> experimentdimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262
>
> 2015-08-17 02:45:13,253 [pool-8-thread-54] INFO
> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
> Zookeeper experiment Path:
> /gfac-experiments/gfac-node0/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262
>
> 2015-08-17 02:45:13,254 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error occured during
> updating the statuses of Experiments,tasks or Job statuses to failed,
> NullPointerException occurred because at this point there might not have
> Job Created
>
> org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
> Executing a InFlow Handler
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
>
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
> InFlow Handler
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
>
> ... 7 more
>
> Caused by: org.apache.airavata.gfac.GFacException: Error adding security
> Context
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
>
> at
> org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
>
> ... 7 more
>
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - localhost:22
> connecting user name - airavata
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
>
> ... 10 more
>
> Caused by: com.jcraft.jsch.JSchException: Auth cancel
>
> at com.jcraft.jsch.Session.connect(Session.java:511)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>
> ... 14 more
>
> 2015-08-17 02:45:13,254 [pool-8-thread-54] INFO
> org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator -
> expId - dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262:
> Updating workflow node status for
> tempNode_41ca8687-37ab-492c-bce0-09058ccc1c02:FAILED
>
> 2015-08-17 02:45:13,617 [pool-8-thread-54] INFO
> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
> Publishing status to rabbitmq...
>
> 2015-08-17 02:45:14,721 [pool-8-thread-54] INFO
> org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator -
> Updating the experiment status of experiment:
> dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 to FAILED
>
> 2015-08-17 02:45:14,771 [pool-8-thread-54] INFO
> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
> Publishing status to rabbitmq...
>
> 2015-08-17 02:45:14,777 [pool-8-thread-54] INFO
> org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator -
> Clean up task - Acknowledging delivery Tag - 91
>
> 2015-08-17 02:45:15,017 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error
> Executing a InFlow Handler
>
> org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
> Executing a InFlow Handler
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
>
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
> InFlow Handler
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
>
> ... 7 more
>
> Caused by: org.apache.airavata.gfac.GFacException: Error adding security
> Context
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
>
> at
> org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
>
> ... 7 more
>
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - localhost:22
> connecting user name - airavata
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
>
> ... 10 more
>
> Caused by: com.jcraft.jsch.JSchException: Auth cancel
>
> at com.jcraft.jsch.Session.connect(Session.java:511)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>
> ... 14 more
>
> 2015-08-17 02:45:15,099 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.GFacException - Error launching the Job
>
> org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
> Executing a InFlow Handler
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:743)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
>
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
> Handlers:Error Executing a InFlow Handler
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
>
> ... 6 more
>
> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
> InFlow Handler
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
>
> ... 7 more
>
> Caused by: org.apache.airavata.gfac.GFacException: Error adding security
> Context
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
>
> at
> org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
>
> ... 7 more
>
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - localhost:22
> connecting user name - airavata
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
>
> ... 10 more
>
> Caused by: com.jcraft.jsch.JSchException: Auth cancel
>
> at com.jcraft.jsch.Session.connect(Session.java:511)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>
> ... 14 more
>
> 2015-08-17 02:45:15,100 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error inovoking the job
> with experiment ID:
> dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262:Error
> launching the Job
>
> 2015-08-17 02:45:15,218 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
> at this point
>
> 2015-08-17 02:45:15,218 [pool-8-thread-54] INFO
> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
> internal state changed to: FAILED
>
> 2015-08-17 02:45:15,218 [pool-8-thread-54] INFO
> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
> Experiment Failed, So removing the ZK entry for the
> experimentdimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262
>
> 2015-08-17 02:45:15,218 [pool-8-thread-54] INFO
> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
> Zookeeper experiment Path:
> /gfac-experiments/gfac-node0/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262
>
> 2015-08-17 02:45:15,718 [Thread-9] INFO
> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
>
> 2015-08-17 02:45:15,769 [pool-8-thread-54] INFO
> org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry -
> Id:tempNode_73058234-5ac6-4278-83f2-c27e23474182 : Updated task
> tempNode_73058234-5ac6-4278-83f2-c27e23474182 status to
> TaskStatus(executionState:FAILED, timeOfStateChange:1439793915661).
>
> 2015-08-17 02:45:15,787 [pool-8-thread-54] INFO
> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
> Publishing status to rabbitmq...
>
> 2015-08-17 02:45:15,797 [pool-8-thread-54] INFO
> org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator -
> expId - dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262:
> Updating workflow node status for
> tempNode_41ca8687-37ab-492c-bce0-09058ccc1c02:FAILED
>
> 2015-08-17 02:45:16,225 [pool-8-thread-54] INFO
> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
> Publishing status to rabbitmq...
>
> 2015-08-17 02:45:17,316 [pool-8-thread-54] INFO
> org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator -
> Updating the experiment status of experiment:
> dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 to FAILED
>
> 2015-08-17 02:45:17,376 [pool-8-thread-54] INFO
> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
> Publishing status to rabbitmq...
>
> 2015-08-17 02:45:17,385 [pool-8-thread-54] ERROR
> org.apache.airavata.common.utils.AiravataZKUtils - Cannot find delivery
> Tag in
> path:/gfac-experiments/gfac-node0/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262-deliveryTag
> for this experiment
>
> 2015-08-17 02:45:17,397 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.GFacException - Error launching the Job
>
> org.apache.airavata.gfac.GFacException: Error launching the Job
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:505)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
>
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
> Handlers:Error Executing a InFlow Handler
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:743)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
>
> ... 5 more
>
> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
> Handlers:Error Executing a InFlow Handler
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
>
> ... 6 more
>
> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
> InFlow Handler
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
>
> ... 7 more
>
> Caused by: org.apache.airavata.gfac.GFacException: Error adding security
> Context
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
>
> at
> org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
>
> ... 7 more
>
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - localhost:22
> connecting user name - airavata
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
>
> ... 10 more
>
> Caused by: com.jcraft.jsch.JSchException: Auth cancel
>
> at com.jcraft.jsch.Session.connect(Session.java:511)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>
> ... 14 more
>
> 2015-08-17 02:45:17,401 [pool-8-thread-54] ERROR
> org.apache.airavata.gfac.core.utils.InputHandlerWorker -
> org.apache.airavata.gfac.GFacException: Error launching the Job
>
> org.apache.airavata.gfac.GFacException:
> org.apache.airavata.gfac.GFacException: Error launching the Job
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:200)
>
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.airavata.gfac.GFacException: Error launching the Job
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:505)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
>
> ... 4 more
>
> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
> Handlers:Error Executing a InFlow Handler
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:743)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
>
> ... 5 more
>
> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
> Handlers:Error Executing a InFlow Handler
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
>
> ... 6 more
>
> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
> InFlow Handler
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
>
> ... 7 more
>
> Caused by: org.apache.airavata.gfac.GFacException: Error adding security
> Context
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
>
> at
> org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
>
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
>
> ... 7 more
>
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - localhost:22
> connecting user name - airavata
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
>
> ... 10 more
>
> Caused by: com.jcraft.jsch.JSchException: Auth cancel
>
> at com.jcraft.jsch.Session.connect(Session.java:511)
>
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>
> ... 14 more
>
> 2015-08-17 02:45:25,721 [Thread-9] INFO
> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
>
> 2015-08-17 02:45:35,723 [Thread-9] INFO
> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-17 02:45:45,724 [Thread-9] INFO
> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
>
> Thanks
> Dimuthu
> --
> Regards
>
> W.Dimuthu Upeksha
> Undergraduate
> Department of Computer Science And Engineering
>
> University of Moratuwa, Sri Lanka
>
--
Regards
W.Dimuthu Upeksha
Undergraduate
Department of Computer Science And Engineering
University of Moratuwa, Sri Lanka
Re: Airavata fails to scp files from localhost
Posted by "Pierce, Marlon" <ma...@iu.edu>.
Does SCP on the command line work with your key pair?
From: DImuthu Upeksha <di...@gmail.com>>
Reply-To: "dev@airavata.apache.org<ma...@airavata.apache.org>" <de...@airavata.apache.org>>
Date: Monday, August 17, 2015 at 3:00 AM
To: "dev@airavata.apache.org<ma...@airavata.apache.org>" <de...@airavata.apache.org>>
Subject: Airavata fails to scp files from localhost
Hi all,
I'm submitting jobs to gw77 server to test functionalities of the GridChem client. I have input files for a gaussian experiment in the same server. So I passed the value for input file as
file://airavata@localhost:/home/airavata/oodt/archive/hpn40deftk7r1tvgo2545lfoqc/c4b4nhtwbs3.inp
When I launch the experiment, I get an error saying
2015-08-17 02:45:13,162 [pool-8-thread-54] ERROR org.apache.airavata.gfac.ssh.util.GFACSSHUtils - An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata
org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata
What could be the reason for that? As far as I understood, it seems like Airavata is unable to scp a file from localhost. Full log is as follows.
2015-08-17 02:45:10,995 [pool-3-thread-5] INFO org.apache.airavata.gfac.core.utils.GFacUtils - This is a new Job, so creating all the experiment docs from the scratch
2015-08-17 02:45:11,017 [pool-3-thread-5] INFO org.apache.airavata.gfac.core.utils.GFacUtils - Created the node: /gfac-experiments/gfac-node0/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262/state successfully !
2015-08-17 02:45:11,028 [pool-3-thread-5] INFO org.apache.airavata.gfac.server.GfacServerHandler - -----------------------------------------------------90-----------------------------------------------------
2015-08-17 02:45:11,028 [pool-3-thread-5] INFO org.apache.airavata.gfac.server.GfacServerHandler - Id:dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 : GFac Received submit job request for the Experiment: dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 TaskId: tempNode_73058234-5ac6-4278-83f2-c27e23474182
2015-08-17 02:45:11,735 [pool-8-thread-54] ERROR org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created at this point
2015-08-17 02:45:11,739 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac internal state changed to: ACCEPTED
2015-08-17 02:45:11,882 [pool-8-thread-54] ERROR org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created at this point
2015-08-17 02:45:11,882 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac internal state changed to: INHANDLERSINVOKING
2015-08-17 02:45:11,969 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler - Invoking ApplicationDescriptorCheckHandler ...
2015-08-17 02:45:11,985 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler - Recoverable data is saving to zk: /oasis/scratch/comet/scigap/temp_project/airavata-workdirs,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262/Gaussian.stdout,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262/Gaussian.stderr
2015-08-17 02:45:12,206 [pool-8-thread-54] INFO org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster - Listing directory: comet.sdsc.edu:~/
2015-08-17 02:45:12,412 [pool-8-thread-54] INFO org.apache.airavata.gfac.ssh.util.GFACSSHUtils - Re-using the same connection used with the connection string:scigapcomet.sdsc.edu22
2015-08-17 02:45:12,413 [pool-8-thread-54] INFO org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler - Setup SSH job directorties
2015-08-17 02:45:12,463 [pool-8-thread-54] INFO org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler - Successfully retrieved the Security Context
2015-08-17 02:45:12,463 [pool-8-thread-54] INFO org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster - Creating directory: comet.sdsc.edu:/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262
2015-08-17 02:45:12,973 [pool-8-thread-54] ERROR org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler - Previously stored data is wrong so we continue the operations
2015-08-17 02:45:13,162 [pool-8-thread-54] ERROR org.apache.airavata.gfac.ssh.util.GFACSSHUtils - An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata
org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:13,164 [pool-8-thread-54] ERROR org.apache.airavata.gfac.GFacException - Error adding security Context
org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:13,164 [pool-8-thread-54] ERROR org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler - Error adding security Context
2015-08-17 02:45:13,247 [pool-8-thread-54] ERROR org.apache.airavata.gfac.GFacException - Error while input File Staging
org.apache.airavata.gfac.GFacException: Error adding security Context
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:13,247 [pool-8-thread-54] ERROR org.apache.airavata.gfac.core.handler.GFacHandlerException - Error while input File Staging
org.apache.airavata.gfac.GFacException: Error adding security Context
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:13,248 [pool-8-thread-54] ERROR org.apache.airavata.gfac.GFacException - Error Executing a InFlow Handler
org.apache.airavata.gfac.GFacException: Error adding security Context
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:13,248 [pool-8-thread-54] ERROR org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error Executing a InFlow Handler
org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error adding security Context
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:13,253 [pool-8-thread-54] ERROR org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created at this point
2015-08-17 02:45:13,253 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac internal state changed to: FAILED
2015-08-17 02:45:13,253 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Experiment Failed, So removing the ZK entry for the experimentdimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262
2015-08-17 02:45:13,253 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Zookeeper experiment Path: /gfac-experiments/gfac-node0/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262
2015-08-17 02:45:13,254 [pool-8-thread-54] ERROR org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error occured during updating the statuses of Experiments,tasks or Job statuses to failed, NullPointerException occurred because at this point there might not have Job Created
org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
... 7 more
Caused by: org.apache.airavata.gfac.GFacException: Error adding security Context
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:13,254 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator - expId - dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262: Updating workflow node status for tempNode_41ca8687-37ab-492c-bce0-09058ccc1c02:FAILED
2015-08-17 02:45:13,617 [pool-8-thread-54] INFO org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - Publishing status to rabbitmq...
2015-08-17 02:45:14,721 [pool-8-thread-54] INFO org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator - Updating the experiment status of experiment: dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 to FAILED
2015-08-17 02:45:14,771 [pool-8-thread-54] INFO org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - Publishing status to rabbitmq...
2015-08-17 02:45:14,777 [pool-8-thread-54] INFO org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator - Clean up task - Acknowledging delivery Tag - 91
2015-08-17 02:45:15,017 [pool-8-thread-54] ERROR org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error Executing a InFlow Handler
org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
... 7 more
Caused by: org.apache.airavata.gfac.GFacException: Error adding security Context
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:15,099 [pool-8-thread-54] ERROR org.apache.airavata.gfac.GFacException - Error launching the Job
org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:743)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
... 6 more
Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
... 7 more
Caused by: org.apache.airavata.gfac.GFacException: Error adding security Context
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:15,100 [pool-8-thread-54] ERROR org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error inovoking the job with experiment ID: dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262:Error launching the Job
2015-08-17 02:45:15,218 [pool-8-thread-54] ERROR org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created at this point
2015-08-17 02:45:15,218 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac internal state changed to: FAILED
2015-08-17 02:45:15,218 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Experiment Failed, So removing the ZK entry for the experimentdimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262
2015-08-17 02:45:15,218 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Zookeeper experiment Path: /gfac-experiments/gfac-node0/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262
2015-08-17 02:45:15,718 [Thread-9] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails
2015-08-17 02:45:15,769 [pool-8-thread-54] INFO org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry - Id:tempNode_73058234-5ac6-4278-83f2-c27e23474182 : Updated task tempNode_73058234-5ac6-4278-83f2-c27e23474182 status to TaskStatus(executionState:FAILED, timeOfStateChange:1439793915661).
2015-08-17 02:45:15,787 [pool-8-thread-54] INFO org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - Publishing status to rabbitmq...
2015-08-17 02:45:15,797 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator - expId - dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262: Updating workflow node status for tempNode_41ca8687-37ab-492c-bce0-09058ccc1c02:FAILED
2015-08-17 02:45:16,225 [pool-8-thread-54] INFO org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - Publishing status to rabbitmq...
2015-08-17 02:45:17,316 [pool-8-thread-54] INFO org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator - Updating the experiment status of experiment: dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 to FAILED
2015-08-17 02:45:17,376 [pool-8-thread-54] INFO org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - Publishing status to rabbitmq...
2015-08-17 02:45:17,385 [pool-8-thread-54] ERROR org.apache.airavata.common.utils.AiravataZKUtils - Cannot find delivery Tag in path:/gfac-experiments/gfac-node0/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262-deliveryTag for this experiment
2015-08-17 02:45:17,397 [pool-8-thread-54] ERROR org.apache.airavata.gfac.GFacException - Error launching the Job
org.apache.airavata.gfac.GFacException: Error launching the Job
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:505)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:743)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
... 5 more
Caused by: org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
... 6 more
Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
... 7 more
Caused by: org.apache.airavata.gfac.GFacException: Error adding security Context
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:17,401 [pool-8-thread-54] ERROR org.apache.airavata.gfac.core.utils.InputHandlerWorker - org.apache.airavata.gfac.GFacException: Error launching the Job
org.apache.airavata.gfac.GFacException: org.apache.airavata.gfac.GFacException: Error launching the Job
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:200)
at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error launching the Job
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:505)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183)
... 4 more
Caused by: org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:743)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492)
... 5 more
Caused by: org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686)
... 6 more
Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899)
... 7 more
Caused by: org.apache.airavata.gfac.GFacException: Error adding security Context
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547)
at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148)
at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271)
... 10 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:511)
at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 14 more
2015-08-17 02:45:25,721 [Thread-9] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails
2015-08-17 02:45:35,723 [Thread-9] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails
2015-08-17 02:45:45,724 [Thread-9] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails
Thanks
Dimuthu
--
Regards
W.Dimuthu Upeksha
Undergraduate
Department of Computer Science And Engineering
University of Moratuwa, Sri Lanka