You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by DImuthu Upeksha <di...@gmail.com> on 2015/08/01 11:05:04 UTC
Error while launching Gaussian experiments on gw111 server
Hi,
I'm getting following error when I tried to launch a Gaussian experiment on
gw111.iu.xsede.org <ai...@gw111.iu.xsede.org> server. What could be the
reason for that?
Application interface: Gaussian_cc471eee-35e0-4d2b-8cdc-0023481dda90
Resource host id: comet-ln1.sdsc.edu_0bb9bd78-b5e7-40cf-a5dd-fd6f8bd6b537
2015-08-01 04:54:58,754 [Thread-18] INFO
org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
Monitor Map is empty, no need to retrieve emails
2015-08-01 04:55:08,756 [Thread-18] INFO
org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
Monitor Map is empty, no need to retrieve emails
2015-08-01 04:55:18,756 [Thread-18] INFO
org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
Monitor Map is empty, no need to retrieve emails
2015-08-01 04:55:28,756 [Thread-18] INFO
org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
Monitor Map is empty, no need to retrieve emails
2015-08-01 04:55:38,757 [Thread-18] INFO
org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
Monitor Map is empty, no need to retrieve emails
2015-08-01 04:55:48,758 [Thread-18] INFO
org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
Monitor Map is empty, no need to retrieve emails
2015-08-01 04:55:54,691 [pool-3-thread-23] INFO
org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
Publishing status to rabbitmq...
2015-08-01 04:55:54,700 [pool-3-thread-23] INFO
org.apache.airavata.api.server.handler.AiravataServerHandler -
Id:dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b : Created
new experiment with experiment name dimuthu_experiment_gau4
2015-08-01 04:55:58,758 [Thread-18] INFO
org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
Monitor Map is empty, no need to retrieve emails
2015-08-01 04:56:08,759 [Thread-18] INFO
org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
Monitor Map is empty, no need to retrieve emails
2015-08-01 04:56:18,760 [Thread-18] INFO
org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
Monitor Map is empty, no need to retrieve emails
2015-08-01 04:56:28,760 [Thread-18] INFO
org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
Monitor Map is empty, no need to retrieve emails
2015-08-01 04:56:31,929 [pool-10-thread-17] INFO
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl -
Validation of
org.apache.airavata.orchestrator.core.validator.impl.BatchQueueValidator is
SUCCESSFUL
2015-08-01 04:56:31,931 [pool-10-thread-17] INFO
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl -
Validation of
org.apache.airavata.orchestrator.core.validator.impl.ExperimentStatusValidator
is SUCCESSFUL
2015-08-01 04:56:32,843 [pool-18-thread-14] INFO
org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
Publishing status to rabbitmq...
2015-08-01 04:56:33,025 [pool-18-thread-14] INFO
org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry -
Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task
tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to
TaskStatus(executionState:UNKNOWN, timeOfStateChange:1438419390000).
2015-08-01 04:56:33,592 [pool-18-thread-14] INFO
org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry -
Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task
tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to
TaskStatus(executionState:UNKNOWN, timeOfStateChange:1438419390000).
2015-08-01 04:56:34,060 [pool-18-thread-14] INFO
org.apache.airavata.messaging.core.impl.RabbitMQTaskLaunchPublisher -
Publishing to launch queue ...
2015-08-01 04:56:34,061 [pool-18-thread-14] INFO
org.apache.airavata.messaging.core.impl.RabbitMQTaskLaunchPublisher -
Successfully published to launch queue ...
2015-08-01 04:56:34,130 [pool-13-thread-9] INFO
org.apache.airavata.gfac.core.utils.GFacUtils - This is a new Job, so
creating all the experiment docs from the scratch
2015-08-01 04:56:34,147 [pool-13-thread-9] INFO
org.apache.airavata.gfac.core.utils.GFacUtils - Created the node:
/gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/state
successfully !
2015-08-01 04:56:34,154 [pool-13-thread-9] INFO
org.apache.airavata.gfac.server.GfacServerHandler -
-----------------------------------------------------14-----------------------------------------------------
2015-08-01 04:56:34,154 [pool-13-thread-9] INFO
org.apache.airavata.gfac.server.GfacServerHandler -
Id:dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b : GFac
Received submit job request for the Experiment:
dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b TaskId:
tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6
2015-08-01 04:56:34,749 [pool-19-thread-28] ERROR
org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
at this point
2015-08-01 04:56:34,750 [pool-19-thread-28] INFO
org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
internal state changed to: ACCEPTED
2015-08-01 04:56:34,831 [pool-19-thread-28] ERROR
org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
at this point
2015-08-01 04:56:34,831 [pool-19-thread-28] INFO
org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
internal state changed to: INHANDLERSINVOKING
2015-08-01 04:56:34,880 [pool-19-thread-28] INFO
org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler -
Invoking ApplicationDescriptorCheckHandler ...
2015-08-01 04:56:34,886 [pool-19-thread-28] INFO
org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler -
Recoverable data is saving to zk:
/oasis/scratch/comet/ogce/temp_project/gta-work-dirs,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/Gaussian.stdout,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/Gaussian.stderr
2015-08-01 04:56:35,765 [pool-19-thread-28] ERROR
org.apache.airavata.gfac.GFacException - Error occurred...
org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred
while connecting to server.Connecting server - comet.sdsc.edu:22 connecting
user name - ogce
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
at
org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:512)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 13 more
2015-08-01 04:56:35,766 [pool-19-thread-28] ERROR
org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler - Error
occurred...
2015-08-01 04:56:35,829 [pool-19-thread-28] ERROR
org.apache.airavata.gfac.GFacException - Error while creating
SSHSecurityContext
org.apache.airavata.gfac.GFacException: Error occurred...
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
at
org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - comet.sdsc.edu:22
connecting user name - ogce
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
... 9 more
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:512)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 13 more
2015-08-01 04:56:35,829 [pool-19-thread-28] ERROR
org.apache.airavata.gfac.core.handler.GFacHandlerException - Error while
creating SSHSecurityContext
org.apache.airavata.gfac.GFacException: Error occurred...
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
at
org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - comet.sdsc.edu:22
connecting user name - ogce
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
... 9 more
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:512)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 13 more
2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR
org.apache.airavata.gfac.GFacException - Error Executing a InFlow Handler
org.apache.airavata.gfac.GFacException: Error occurred...
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
at
org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - comet.sdsc.edu:22
connecting user name - ogce
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
... 9 more
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:512)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 13 more
2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR
org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error
Executing a InFlow Handler
org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
at
org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - comet.sdsc.edu:22
connecting user name - ogce
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
... 9 more
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:512)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 13 more
2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR
org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
at this point
2015-08-01 04:56:35,830 [pool-19-thread-28] INFO
org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
internal state changed to: FAILED
2015-08-01 04:56:35,830 [pool-19-thread-28] INFO
org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
Experiment Failed, So removing the ZK entry for the
experimentdimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b
2015-08-01 04:56:35,830 [pool-19-thread-28] INFO
org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
Zookeeper experiment Path:
/gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b
2015-08-01 04:56:35,831 [pool-19-thread-28] ERROR
org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error occured during
updating the statuses of Experiments,tasks or Job statuses to failed,
NullPointerException occurred because at this point there might not have
Job Created
org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow
Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
... 7 more
Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
at
org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - comet.sdsc.edu:22
connecting user name - ogce
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
... 9 more
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:512)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 13 more
2015-08-01 04:56:35,832 [pool-19-thread-28] INFO
org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator -
expId - dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b:
Updating workflow node status for
tempNode_50386c5c-2d49-4f04-b061-024c134be4dc:FAILED
2015-08-01 04:56:36,070 [pool-19-thread-28] INFO
org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
Publishing status to rabbitmq...
2015-08-01 04:56:36,656 [pool-19-thread-28] INFO
org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator -
Updating the experiment status of experiment:
dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b to FAILED
2015-08-01 04:56:36,715 [pool-19-thread-28] INFO
org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
Publishing status to rabbitmq...
2015-08-01 04:56:36,718 [pool-19-thread-28] INFO
org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator -
Clean up task - Acknowledging delivery Tag - 14
2015-08-01 04:56:36,770 [pool-19-thread-28] ERROR
org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error
Executing a InFlow Handler
org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow
Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
... 7 more
Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
at
org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - comet.sdsc.edu:22
connecting user name - ogce
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
... 9 more
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:512)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 13 more
2015-08-01 04:56:36,820 [pool-19-thread-28] ERROR
org.apache.airavata.gfac.GFacException - Error launching the Job
org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
Handlers:Error Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
... 6 more
Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow
Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
... 7 more
Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
at
org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - comet.sdsc.edu:22
connecting user name - ogce
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
... 9 more
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:512)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 13 more
2015-08-01 04:56:36,820 [pool-19-thread-28] ERROR
org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error inovoking the job
with experiment ID:
dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b:Error
launching the Job
2015-08-01 04:56:36,868 [pool-19-thread-28] ERROR
org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
at this point
2015-08-01 04:56:36,868 [pool-19-thread-28] INFO
org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
internal state changed to: FAILED
2015-08-01 04:56:36,868 [pool-19-thread-28] INFO
org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
Experiment Failed, So removing the ZK entry for the
experimentdimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b
2015-08-01 04:56:36,868 [pool-19-thread-28] INFO
org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
Zookeeper experiment Path:
/gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b
2015-08-01 04:56:37,059 [pool-19-thread-28] INFO
org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry -
Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task
tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to
TaskStatus(executionState:FAILED, timeOfStateChange:1438419396978).
2015-08-01 04:56:37,059 [pool-19-thread-28] INFO
org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
Publishing status to rabbitmq...
2015-08-01 04:56:37,059 [pool-19-thread-28] INFO
org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator -
expId - dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b:
Updating workflow node status for
tempNode_50386c5c-2d49-4f04-b061-024c134be4dc:FAILED
2015-08-01 04:56:37,267 [pool-19-thread-28] INFO
org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
Publishing status to rabbitmq...
2015-08-01 04:56:37,889 [pool-19-thread-28] INFO
org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator -
Updating the experiment status of experiment:
dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b to FAILED
2015-08-01 04:56:37,947 [pool-19-thread-28] INFO
org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
Publishing status to rabbitmq...
2015-08-01 04:56:37,949 [pool-19-thread-28] ERROR
org.apache.airavata.common.utils.AiravataZKUtils - Cannot find delivery
Tag in
path:/gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b-deliveryTag
for this experiment
2015-08-01 04:56:37,951 [pool-19-thread-28] ERROR
org.apache.airavata.gfac.GFacException - Error launching the Job
org.apache.airavata.gfac.GFacException: Error launching the Job
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:498)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
Handlers:Error Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
... 5 more
Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
Handlers:Error Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
... 6 more
Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow
Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
... 7 more
Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
at
org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - comet.sdsc.edu:22
connecting user name - ogce
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
... 9 more
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:512)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 13 more
2015-08-01 04:56:37,951 [pool-19-thread-28] ERROR
org.apache.airavata.gfac.core.utils.InputHandlerWorker -
org.apache.airavata.gfac.GFacException: Error launching the Job
org.apache.airavata.gfac.GFacException:
org.apache.airavata.gfac.GFacException: Error launching the Job
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:196)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error launching the Job
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:498)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
... 4 more
Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
Handlers:Error Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
... 5 more
Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
Handlers:Error Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
... 6 more
Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow
Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
... 7 more
Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
at
org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - comet.sdsc.edu:22
connecting user name - ogce
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
... 9 more
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:512)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 13 more
--
Regards
W.Dimuthu Upeksha
Undergraduate
Department of Computer Science And Engineering
University of Moratuwa, Sri Lanka
Re: Error while launching Gaussian experiments on gw111 server
Posted by DImuthu Upeksha <di...@gmail.com>.
Hi Shameera,
Yes it still occurs. I tried to it today also and got following error.
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - comet.sdsc.edu:22
connecting user name - ogce
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
... 9 more
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:512)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 13 more
2015-08-08 05:02:47,487 [pool-19-thread-1] ERROR
org.apache.airavata.gfac.core.utils.InputHandlerWorker -
org.apache.airavata.gfac.GFacException: Error launching the Job
org.apache.airavata.gfac.GFacException:
org.apache.airavata.gfac.GFacException: Error launching the Job
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:196)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.airavata.gfac.GFacException: Error launching the Job
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:498)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
... 4 more
Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
Handlers:Error Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
... 5 more
Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
Handlers:Error Executing a InFlow Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
... 6 more
Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow
Handler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
... 7 more
Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
at
org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
... 7 more
Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
occurred while connecting to server.Connecting server - comet.sdsc.edu:22
connecting user name - ogce
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
at
org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
... 9 more
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:512)
at
org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
... 13 more
On Mon, Aug 3, 2015 at 8:58 PM, Shameera Rathnayaka <sh...@gmail.com>
wrote:
> Hi Dimuthu,
>
> do you still have this issue?I just checked and gw111 has access to comet
> resource.
>
> Thanks,
> Shameera.
>
> On Sat, Aug 1, 2015 at 5:05 AM DImuthu Upeksha <di...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm getting following error when I tried to launch a Gaussian experiment
>> on gw111.iu.xsede.org <ai...@gw111.iu.xsede.org> server. What could
>> be the reason for that?
>>
>> Application interface: Gaussian_cc471eee-35e0-4d2b-8cdc-0023481dda90
>> Resource host id: comet-ln1.sdsc.edu_0bb9bd78-b5e7-40cf-a5dd-fd6f8bd6b537
>>
>> 2015-08-01 04:54:58,754 [Thread-18] INFO
>> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
>> Monitor Map is empty, no need to retrieve emails
>> 2015-08-01 04:55:08,756 [Thread-18] INFO
>> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
>> Monitor Map is empty, no need to retrieve emails
>> 2015-08-01 04:55:18,756 [Thread-18] INFO
>> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
>> Monitor Map is empty, no need to retrieve emails
>> 2015-08-01 04:55:28,756 [Thread-18] INFO
>> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
>> Monitor Map is empty, no need to retrieve emails
>> 2015-08-01 04:55:38,757 [Thread-18] INFO
>> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
>> Monitor Map is empty, no need to retrieve emails
>> 2015-08-01 04:55:48,758 [Thread-18] INFO
>> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
>> Monitor Map is empty, no need to retrieve emails
>> 2015-08-01 04:55:54,691 [pool-3-thread-23] INFO
>> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
>> Publishing status to rabbitmq...
>> 2015-08-01 04:55:54,700 [pool-3-thread-23] INFO
>> org.apache.airavata.api.server.handler.AiravataServerHandler -
>> Id:dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b : Created
>> new experiment with experiment name dimuthu_experiment_gau4
>> 2015-08-01 04:55:58,758 [Thread-18] INFO
>> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
>> Monitor Map is empty, no need to retrieve emails
>> 2015-08-01 04:56:08,759 [Thread-18] INFO
>> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
>> Monitor Map is empty, no need to retrieve emails
>> 2015-08-01 04:56:18,760 [Thread-18] INFO
>> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
>> Monitor Map is empty, no need to retrieve emails
>> 2015-08-01 04:56:28,760 [Thread-18] INFO
>> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
>> Monitor Map is empty, no need to retrieve emails
>> 2015-08-01 04:56:31,929 [pool-10-thread-17] INFO
>> org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl -
>> Validation of
>> org.apache.airavata.orchestrator.core.validator.impl.BatchQueueValidator is
>> SUCCESSFUL
>> 2015-08-01 04:56:31,931 [pool-10-thread-17] INFO
>> org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl -
>> Validation of
>> org.apache.airavata.orchestrator.core.validator.impl.ExperimentStatusValidator
>> is SUCCESSFUL
>> 2015-08-01 04:56:32,843 [pool-18-thread-14] INFO
>> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
>> Publishing status to rabbitmq...
>> 2015-08-01 04:56:33,025 [pool-18-thread-14] INFO
>> org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry -
>> Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task
>> tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to
>> TaskStatus(executionState:UNKNOWN, timeOfStateChange:1438419390000).
>> 2015-08-01 04:56:33,592 [pool-18-thread-14] INFO
>> org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry -
>> Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task
>> tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to
>> TaskStatus(executionState:UNKNOWN, timeOfStateChange:1438419390000).
>> 2015-08-01 04:56:34,060 [pool-18-thread-14] INFO
>> org.apache.airavata.messaging.core.impl.RabbitMQTaskLaunchPublisher -
>> Publishing to launch queue ...
>> 2015-08-01 04:56:34,061 [pool-18-thread-14] INFO
>> org.apache.airavata.messaging.core.impl.RabbitMQTaskLaunchPublisher -
>> Successfully published to launch queue ...
>> 2015-08-01 04:56:34,130 [pool-13-thread-9] INFO
>> org.apache.airavata.gfac.core.utils.GFacUtils - This is a new Job, so
>> creating all the experiment docs from the scratch
>> 2015-08-01 04:56:34,147 [pool-13-thread-9] INFO
>> org.apache.airavata.gfac.core.utils.GFacUtils - Created the node:
>> /gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/state
>> successfully !
>> 2015-08-01 04:56:34,154 [pool-13-thread-9] INFO
>> org.apache.airavata.gfac.server.GfacServerHandler -
>> -----------------------------------------------------14-----------------------------------------------------
>> 2015-08-01 04:56:34,154 [pool-13-thread-9] INFO
>> org.apache.airavata.gfac.server.GfacServerHandler -
>> Id:dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b : GFac
>> Received submit job request for the Experiment:
>> dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b TaskId:
>> tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6
>> 2015-08-01 04:56:34,749 [pool-19-thread-28] ERROR
>> org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
>> at this point
>> 2015-08-01 04:56:34,750 [pool-19-thread-28] INFO
>> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
>> internal state changed to: ACCEPTED
>> 2015-08-01 04:56:34,831 [pool-19-thread-28] ERROR
>> org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
>> at this point
>> 2015-08-01 04:56:34,831 [pool-19-thread-28] INFO
>> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
>> internal state changed to: INHANDLERSINVOKING
>> 2015-08-01 04:56:34,880 [pool-19-thread-28] INFO
>> org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler -
>> Invoking ApplicationDescriptorCheckHandler ...
>> 2015-08-01 04:56:34,886 [pool-19-thread-28] INFO
>> org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler -
>> Recoverable data is saving to zk:
>> /oasis/scratch/comet/ogce/temp_project/gta-work-dirs,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/Gaussian.stdout,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/Gaussian.stderr
>> 2015-08-01 04:56:35,765 [pool-19-thread-28] ERROR
>> org.apache.airavata.gfac.GFacException - Error occurred...
>> org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred
>> while connecting to server.Connecting server - comet.sdsc.edu:22
>> connecting user name - ogce
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
>> at
>> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
>> at
>> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: com.jcraft.jsch.JSchException: Auth fail
>> at com.jcraft.jsch.Session.connect(Session.java:512)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>> ... 13 more
>> 2015-08-01 04:56:35,766 [pool-19-thread-28] ERROR
>> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler - Error
>> occurred...
>> 2015-08-01 04:56:35,829 [pool-19-thread-28] ERROR
>> org.apache.airavata.gfac.GFacException - Error while creating
>> SSHSecurityContext
>> org.apache.airavata.gfac.GFacException: Error occurred...
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
>> at
>> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
>> at
>> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
>> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
>> connecting user name - ogce
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
>> ... 9 more
>> Caused by: com.jcraft.jsch.JSchException: Auth fail
>> at com.jcraft.jsch.Session.connect(Session.java:512)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>> ... 13 more
>> 2015-08-01 04:56:35,829 [pool-19-thread-28] ERROR
>> org.apache.airavata.gfac.core.handler.GFacHandlerException - Error while
>> creating SSHSecurityContext
>> org.apache.airavata.gfac.GFacException: Error occurred...
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
>> at
>> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
>> at
>> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
>> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
>> connecting user name - ogce
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
>> ... 9 more
>> Caused by: com.jcraft.jsch.JSchException: Auth fail
>> at com.jcraft.jsch.Session.connect(Session.java:512)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>> ... 13 more
>> 2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR
>> org.apache.airavata.gfac.GFacException - Error Executing a InFlow Handler
>> org.apache.airavata.gfac.GFacException: Error occurred...
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
>> at
>> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
>> at
>> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
>> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
>> connecting user name - ogce
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
>> ... 9 more
>> Caused by: com.jcraft.jsch.JSchException: Auth fail
>> at com.jcraft.jsch.Session.connect(Session.java:512)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>> ... 13 more
>> 2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR
>> org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error
>> Executing a InFlow Handler
>> org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
>> at
>> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
>> at
>> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
>> ... 7 more
>> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
>> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
>> connecting user name - ogce
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
>> ... 9 more
>> Caused by: com.jcraft.jsch.JSchException: Auth fail
>> at com.jcraft.jsch.Session.connect(Session.java:512)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>> ... 13 more
>> 2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR
>> org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
>> at this point
>> 2015-08-01 04:56:35,830 [pool-19-thread-28] INFO
>> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
>> internal state changed to: FAILED
>> 2015-08-01 04:56:35,830 [pool-19-thread-28] INFO
>> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
>> Experiment Failed, So removing the ZK entry for the
>> experimentdimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b
>> 2015-08-01 04:56:35,830 [pool-19-thread-28] INFO
>> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
>> Zookeeper experiment Path:
>> /gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b
>> 2015-08-01 04:56:35,831 [pool-19-thread-28] ERROR
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error occured during
>> updating the statuses of Experiments,tasks or Job statuses to failed,
>> NullPointerException occurred because at this point there might not have
>> Job Created
>> org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
>> Executing a InFlow Handler
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
>> at
>> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
>> InFlow Handler
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
>> ... 7 more
>> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
>> at
>> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
>> ... 7 more
>> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
>> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
>> connecting user name - ogce
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
>> ... 9 more
>> Caused by: com.jcraft.jsch.JSchException: Auth fail
>> at com.jcraft.jsch.Session.connect(Session.java:512)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>> ... 13 more
>> 2015-08-01 04:56:35,832 [pool-19-thread-28] INFO
>> org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator -
>> expId - dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b:
>> Updating workflow node status for
>> tempNode_50386c5c-2d49-4f04-b061-024c134be4dc:FAILED
>> 2015-08-01 04:56:36,070 [pool-19-thread-28] INFO
>> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
>> Publishing status to rabbitmq...
>> 2015-08-01 04:56:36,656 [pool-19-thread-28] INFO
>> org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator -
>> Updating the experiment status of experiment:
>> dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b to FAILED
>> 2015-08-01 04:56:36,715 [pool-19-thread-28] INFO
>> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
>> Publishing status to rabbitmq...
>> 2015-08-01 04:56:36,718 [pool-19-thread-28] INFO
>> org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator -
>> Clean up task - Acknowledging delivery Tag - 14
>> 2015-08-01 04:56:36,770 [pool-19-thread-28] ERROR
>> org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error
>> Executing a InFlow Handler
>> org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
>> Executing a InFlow Handler
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
>> at
>> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
>> InFlow Handler
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
>> ... 7 more
>> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
>> at
>> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
>> ... 7 more
>> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
>> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
>> connecting user name - ogce
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
>> ... 9 more
>> Caused by: com.jcraft.jsch.JSchException: Auth fail
>> at com.jcraft.jsch.Session.connect(Session.java:512)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>> ... 13 more
>> 2015-08-01 04:56:36,820 [pool-19-thread-28] ERROR
>> org.apache.airavata.gfac.GFacException - Error launching the Job
>> org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
>> Executing a InFlow Handler
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
>> at
>> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
>> Handlers:Error Executing a InFlow Handler
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
>> ... 6 more
>> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
>> InFlow Handler
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
>> ... 7 more
>> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
>> at
>> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
>> ... 7 more
>> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
>> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
>> connecting user name - ogce
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
>> ... 9 more
>> Caused by: com.jcraft.jsch.JSchException: Auth fail
>> at com.jcraft.jsch.Session.connect(Session.java:512)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>> ... 13 more
>> 2015-08-01 04:56:36,820 [pool-19-thread-28] ERROR
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error inovoking the job
>> with experiment ID:
>> dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b:Error
>> launching the Job
>> 2015-08-01 04:56:36,868 [pool-19-thread-28] ERROR
>> org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
>> at this point
>> 2015-08-01 04:56:36,868 [pool-19-thread-28] INFO
>> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
>> internal state changed to: FAILED
>> 2015-08-01 04:56:36,868 [pool-19-thread-28] INFO
>> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
>> Experiment Failed, So removing the ZK entry for the
>> experimentdimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b
>> 2015-08-01 04:56:36,868 [pool-19-thread-28] INFO
>> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
>> Zookeeper experiment Path:
>> /gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b
>> 2015-08-01 04:56:37,059 [pool-19-thread-28] INFO
>> org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry -
>> Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task
>> tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to
>> TaskStatus(executionState:FAILED, timeOfStateChange:1438419396978).
>> 2015-08-01 04:56:37,059 [pool-19-thread-28] INFO
>> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
>> Publishing status to rabbitmq...
>> 2015-08-01 04:56:37,059 [pool-19-thread-28] INFO
>> org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator -
>> expId - dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b:
>> Updating workflow node status for
>> tempNode_50386c5c-2d49-4f04-b061-024c134be4dc:FAILED
>> 2015-08-01 04:56:37,267 [pool-19-thread-28] INFO
>> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
>> Publishing status to rabbitmq...
>> 2015-08-01 04:56:37,889 [pool-19-thread-28] INFO
>> org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator -
>> Updating the experiment status of experiment:
>> dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b to FAILED
>> 2015-08-01 04:56:37,947 [pool-19-thread-28] INFO
>> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
>> Publishing status to rabbitmq...
>> 2015-08-01 04:56:37,949 [pool-19-thread-28] ERROR
>> org.apache.airavata.common.utils.AiravataZKUtils - Cannot find delivery
>> Tag in
>> path:/gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b-deliveryTag
>> for this experiment
>> 2015-08-01 04:56:37,951 [pool-19-thread-28] ERROR
>> org.apache.airavata.gfac.GFacException - Error launching the Job
>> org.apache.airavata.gfac.GFacException: Error launching the Job
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:498)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
>> at
>> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
>> Handlers:Error Executing a InFlow Handler
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
>> ... 5 more
>> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
>> Handlers:Error Executing a InFlow Handler
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
>> ... 6 more
>> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
>> InFlow Handler
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
>> ... 7 more
>> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
>> at
>> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
>> ... 7 more
>> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
>> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
>> connecting user name - ogce
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
>> ... 9 more
>> Caused by: com.jcraft.jsch.JSchException: Auth fail
>> at com.jcraft.jsch.Session.connect(Session.java:512)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>> ... 13 more
>> 2015-08-01 04:56:37,951 [pool-19-thread-28] ERROR
>> org.apache.airavata.gfac.core.utils.InputHandlerWorker -
>> org.apache.airavata.gfac.GFacException: Error launching the Job
>> org.apache.airavata.gfac.GFacException:
>> org.apache.airavata.gfac.GFacException: Error launching the Job
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:196)
>> at
>> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.airavata.gfac.GFacException: Error launching the Job
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:498)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
>> ... 4 more
>> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
>> Handlers:Error Executing a InFlow Handler
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
>> ... 5 more
>> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
>> Handlers:Error Executing a InFlow Handler
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
>> ... 6 more
>> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
>> InFlow Handler
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
>> ... 7 more
>> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
>> at
>> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
>> at
>> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
>> ... 7 more
>> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
>> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
>> connecting user name - ogce
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
>> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
>> at
>> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
>> ... 9 more
>> Caused by: com.jcraft.jsch.JSchException: Auth fail
>> at com.jcraft.jsch.Session.connect(Session.java:512)
>> at
>> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
>> ... 13 more
>>
>> --
>> Regards
>>
>> W.Dimuthu Upeksha
>> Undergraduate
>> Department of Computer Science And Engineering
>>
>> University of Moratuwa, Sri Lanka
>>
> --
> Shameera Rathnayaka
>
--
Regards
W.Dimuthu Upeksha
Undergraduate
Department of Computer Science And Engineering
University of Moratuwa, Sri Lanka
Re: Error while launching Gaussian experiments on gw111 server
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi Dimuthu,
do you still have this issue?I just checked and gw111 has access to comet
resource.
Thanks,
Shameera.
On Sat, Aug 1, 2015 at 5:05 AM DImuthu Upeksha <di...@gmail.com>
wrote:
> Hi,
>
> I'm getting following error when I tried to launch a Gaussian experiment
> on gw111.iu.xsede.org <ai...@gw111.iu.xsede.org> server. What could be
> the reason for that?
>
> Application interface: Gaussian_cc471eee-35e0-4d2b-8cdc-0023481dda90
> Resource host id: comet-ln1.sdsc.edu_0bb9bd78-b5e7-40cf-a5dd-fd6f8bd6b537
>
> 2015-08-01 04:54:58,754 [Thread-18] INFO
> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:55:08,756 [Thread-18] INFO
> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:55:18,756 [Thread-18] INFO
> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:55:28,756 [Thread-18] INFO
> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:55:38,757 [Thread-18] INFO
> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:55:48,758 [Thread-18] INFO
> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:55:54,691 [pool-3-thread-23] INFO
> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
> Publishing status to rabbitmq...
> 2015-08-01 04:55:54,700 [pool-3-thread-23] INFO
> org.apache.airavata.api.server.handler.AiravataServerHandler -
> Id:dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b : Created
> new experiment with experiment name dimuthu_experiment_gau4
> 2015-08-01 04:55:58,758 [Thread-18] INFO
> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:56:08,759 [Thread-18] INFO
> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:56:18,760 [Thread-18] INFO
> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:56:28,760 [Thread-18] INFO
> org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job
> Monitor Map is empty, no need to retrieve emails
> 2015-08-01 04:56:31,929 [pool-10-thread-17] INFO
> org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl -
> Validation of
> org.apache.airavata.orchestrator.core.validator.impl.BatchQueueValidator is
> SUCCESSFUL
> 2015-08-01 04:56:31,931 [pool-10-thread-17] INFO
> org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl -
> Validation of
> org.apache.airavata.orchestrator.core.validator.impl.ExperimentStatusValidator
> is SUCCESSFUL
> 2015-08-01 04:56:32,843 [pool-18-thread-14] INFO
> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
> Publishing status to rabbitmq...
> 2015-08-01 04:56:33,025 [pool-18-thread-14] INFO
> org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry -
> Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task
> tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to
> TaskStatus(executionState:UNKNOWN, timeOfStateChange:1438419390000).
> 2015-08-01 04:56:33,592 [pool-18-thread-14] INFO
> org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry -
> Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task
> tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to
> TaskStatus(executionState:UNKNOWN, timeOfStateChange:1438419390000).
> 2015-08-01 04:56:34,060 [pool-18-thread-14] INFO
> org.apache.airavata.messaging.core.impl.RabbitMQTaskLaunchPublisher -
> Publishing to launch queue ...
> 2015-08-01 04:56:34,061 [pool-18-thread-14] INFO
> org.apache.airavata.messaging.core.impl.RabbitMQTaskLaunchPublisher -
> Successfully published to launch queue ...
> 2015-08-01 04:56:34,130 [pool-13-thread-9] INFO
> org.apache.airavata.gfac.core.utils.GFacUtils - This is a new Job, so
> creating all the experiment docs from the scratch
> 2015-08-01 04:56:34,147 [pool-13-thread-9] INFO
> org.apache.airavata.gfac.core.utils.GFacUtils - Created the node:
> /gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/state
> successfully !
> 2015-08-01 04:56:34,154 [pool-13-thread-9] INFO
> org.apache.airavata.gfac.server.GfacServerHandler -
> -----------------------------------------------------14-----------------------------------------------------
> 2015-08-01 04:56:34,154 [pool-13-thread-9] INFO
> org.apache.airavata.gfac.server.GfacServerHandler -
> Id:dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b : GFac
> Received submit job request for the Experiment:
> dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b TaskId:
> tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6
> 2015-08-01 04:56:34,749 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
> at this point
> 2015-08-01 04:56:34,750 [pool-19-thread-28] INFO
> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
> internal state changed to: ACCEPTED
> 2015-08-01 04:56:34,831 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
> at this point
> 2015-08-01 04:56:34,831 [pool-19-thread-28] INFO
> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
> internal state changed to: INHANDLERSINVOKING
> 2015-08-01 04:56:34,880 [pool-19-thread-28] INFO
> org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler -
> Invoking ApplicationDescriptorCheckHandler ...
> 2015-08-01 04:56:34,886 [pool-19-thread-28] INFO
> org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler -
> Recoverable data is saving to zk:
> /oasis/scratch/comet/ogce/temp_project/gta-work-dirs,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/Gaussian.stdout,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/Gaussian.stderr
> 2015-08-01 04:56:35,765 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.GFacException - Error occurred...
> org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred
> while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:35,766 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler - Error
> occurred...
> 2015-08-01 04:56:35,829 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.GFacException - Error while creating
> SSHSecurityContext
> org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:35,829 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.core.handler.GFacHandlerException - Error while
> creating SSHSecurityContext
> org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.GFacException - Error Executing a InFlow Handler
> org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error
> Executing a InFlow Handler
> org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> ... 7 more
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
> at this point
> 2015-08-01 04:56:35,830 [pool-19-thread-28] INFO
> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
> internal state changed to: FAILED
> 2015-08-01 04:56:35,830 [pool-19-thread-28] INFO
> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
> Experiment Failed, So removing the ZK entry for the
> experimentdimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b
> 2015-08-01 04:56:35,830 [pool-19-thread-28] INFO
> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
> Zookeeper experiment Path:
> /gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b
> 2015-08-01 04:56:35,831 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error occured during
> updating the statuses of Experiments,tasks or Job statuses to failed,
> NullPointerException occurred because at this point there might not have
> Job Created
> org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
> Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
> InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
> ... 7 more
> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> ... 7 more
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:35,832 [pool-19-thread-28] INFO
> org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator -
> expId - dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b:
> Updating workflow node status for
> tempNode_50386c5c-2d49-4f04-b061-024c134be4dc:FAILED
> 2015-08-01 04:56:36,070 [pool-19-thread-28] INFO
> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
> Publishing status to rabbitmq...
> 2015-08-01 04:56:36,656 [pool-19-thread-28] INFO
> org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator -
> Updating the experiment status of experiment:
> dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b to FAILED
> 2015-08-01 04:56:36,715 [pool-19-thread-28] INFO
> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
> Publishing status to rabbitmq...
> 2015-08-01 04:56:36,718 [pool-19-thread-28] INFO
> org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator -
> Clean up task - Acknowledging delivery Tag - 14
> 2015-08-01 04:56:36,770 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error
> Executing a InFlow Handler
> org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
> Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
> InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
> ... 7 more
> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> ... 7 more
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:36,820 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.GFacException - Error launching the Job
> org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error
> Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
> Handlers:Error Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> ... 6 more
> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
> InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
> ... 7 more
> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> ... 7 more
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:36,820 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error inovoking the job
> with experiment ID:
> dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b:Error
> launching the Job
> 2015-08-01 04:56:36,868 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created
> at this point
> 2015-08-01 04:56:36,868 [pool-19-thread-28] INFO
> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac
> internal state changed to: FAILED
> 2015-08-01 04:56:36,868 [pool-19-thread-28] INFO
> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
> Experiment Failed, So removing the ZK entry for the
> experimentdimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b
> 2015-08-01 04:56:36,868 [pool-19-thread-28] INFO
> org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator -
> Zookeeper experiment Path:
> /gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b
> 2015-08-01 04:56:37,059 [pool-19-thread-28] INFO
> org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry -
> Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task
> tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to
> TaskStatus(executionState:FAILED, timeOfStateChange:1438419396978).
> 2015-08-01 04:56:37,059 [pool-19-thread-28] INFO
> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
> Publishing status to rabbitmq...
> 2015-08-01 04:56:37,059 [pool-19-thread-28] INFO
> org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator -
> expId - dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b:
> Updating workflow node status for
> tempNode_50386c5c-2d49-4f04-b061-024c134be4dc:FAILED
> 2015-08-01 04:56:37,267 [pool-19-thread-28] INFO
> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
> Publishing status to rabbitmq...
> 2015-08-01 04:56:37,889 [pool-19-thread-28] INFO
> org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator -
> Updating the experiment status of experiment:
> dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b to FAILED
> 2015-08-01 04:56:37,947 [pool-19-thread-28] INFO
> org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher -
> Publishing status to rabbitmq...
> 2015-08-01 04:56:37,949 [pool-19-thread-28] ERROR
> org.apache.airavata.common.utils.AiravataZKUtils - Cannot find delivery
> Tag in
> path:/gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b-deliveryTag
> for this experiment
> 2015-08-01 04:56:37,951 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.GFacException - Error launching the Job
> org.apache.airavata.gfac.GFacException: Error launching the Job
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:498)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
> Handlers:Error Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> ... 5 more
> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
> Handlers:Error Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> ... 6 more
> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
> InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
> ... 7 more
> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> ... 7 more
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
> 2015-08-01 04:56:37,951 [pool-19-thread-28] ERROR
> org.apache.airavata.gfac.core.utils.InputHandlerWorker -
> org.apache.airavata.gfac.GFacException: Error launching the Job
> org.apache.airavata.gfac.GFacException:
> org.apache.airavata.gfac.GFacException: Error launching the Job
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:196)
> at
> org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.airavata.gfac.GFacException: Error launching the Job
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:498)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
> ... 4 more
> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
> Handlers:Error Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485)
> ... 5 more
> Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
> Handlers:Error Executing a InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679)
> ... 6 more
> Caused by: org.apache.airavata.gfac.GFacException: Error Executing a
> InFlow Handler
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892)
> ... 7 more
> Caused by: org.apache.airavata.gfac.GFacException: Error occurred...
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184)
> at
> org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47)
> at
> org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888)
> ... 7 more
> Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception
> occurred while connecting to server.Connecting server - comet.sdsc.edu:22
> connecting user name - ogce
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94)
> at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44)
> at
> org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172)
> ... 9 more
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:512)
> at
> org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230)
> ... 13 more
>
> --
> Regards
>
> W.Dimuthu Upeksha
> Undergraduate
> Department of Computer Science And Engineering
>
> University of Moratuwa, Sri Lanka
>
--
Shameera Rathnayaka