You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@airavata.apache.org by Suresh Marru <sm...@apache.org> on 2013/04/03 05:34:06 UTC
Re: Job Error Code: 126
On Apr 2, 2013, at 9:42 PM, 안신영 <sy...@etri.re.kr> wrote:
> Thanks Suresh.
> I’m considering Airavata as GUI for our HPC system.
> Our local Resource Manager is slurm, applications will be NGS analysis tools such as bwa, samtools, etc.
This is very good to hear.
> Today, I registered a command(bwa aln) as servies and compose a workflow which contains the service,
> Next I executed, but got some error from Airavata server!
> Would you explain what is the meaning of “Job Error Code: 126”
> How can I check error code? Do I have to search in source code? Or Do you have any document?
I just added a document to the wiki explaining how Airavata integrates with GRAM - [1] .
In short, this is an error from down stream GRAM and not within Airavata itself. Did you test the Globus installation from the command line first? Normally it is not recommended for end users to bleed with lower level grid tools. But in this case, since you are the grid administrator it is a good idea first to make sure the installation and configuration is successfully done end to end. If you need any assistance with GRAM testing you can find better guidance on the globus users list gt-user@globus.org.
Looking through your error message, looks like it is a file system issue. Are your compute nodes and the node where you are running GRAM and GridFTP server same? If not, do not use /tmp for working directory location. GridFTP creates the /tmp on the sever where it is running but your compute node will not be able to see it and job thrashes. Choose a location which is accessible to both your compute nodes and management nodes where you install services and try again.
Cheers,
Suresh
[1] - https://cwiki.apache.org/confluence/display/AIRAVATA/Globus+Grid+Integration
>
> Following is log from server
> ------------------------------------------------------------------------------------------------------------------------
> [INFO] Experiment launched :Workflow2_657cf81a-68f5-48e0-9f22-d43ce34e6c88
> [INFO] -----DATA-----
> [INFO] Start scheduling
> [INFO] -----END DATA-----
> [INFO] Searching registry for some deployed application hosts
> [INFO] Found service on: node08.etri.re.kr
> [INFO] Found service on: node08.etri.re.kr
> [INFO] -----DATA-----
> [INFO] Finish scheduling
> [INFO] -----END DATA-----
> null
> [INFO] Proxy file renewed to /tmp/x509up_uteam2b2feb9f-c9e4-42d8-97de-65ba7081fb22 for the user team with 3600 lifetime.
> [INFO] Creating Directory = node08.etri.re.kr:2811=//tmp
> [INFO] Creating Directory = node08.etri.re.kr:2811=//tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a
> [INFO] Creating Directory = node08.etri.re.kr:2811=//tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a/inputData
> [INFO] Creating Directory = node08.etri.re.kr:2811=//tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a/outputData
> [INFO] -----DATA-----
> [INFO] Start execution
> [INFO] -----END DATA-----
> org.globus.gsi.gssapi.GlobusGSSCredentialImpl@61e34259
> [INFO] -----DATA-----
> [INFO] Finished launching job, Host = node08.etri.re.kr RSL = &( jobtype = "single" )( arguments = "4" "/tmp/genome/bwaindex/chr22.fa" "/tmp/genome/Demo/sample_1.fq.00" "/tmp/genome/DemoOut/sample_1.fq.00.sai" )( environment = ( "inputData" "/tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a/inputData" ) ( "outputData" "/tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a/outputData" ) )( stdout = "/tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a/node08_application.stdout" )( count = "4" )( directory = "/tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a" )( executable = "/home/team/bwa_aln" )( stderr = "/tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a/node08_application.stderr" )( maxwalltime = "30" ) working directory = /tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a temp directory = /tmp Globus GateKeeper Endpoint = node08.etri.re.kr:2119/jobmanager-pbs
> [INFO] -----END DATA-----
> [INFO] JobID = https://node08.etri.re.kr:35051/16290077973982812641/15645714137852419510/
> [INFO] -----DATA-----
> [INFO] JobID=https://node08.etri.re.kr:35051/16290077973982812641/15645714137852419510/
> [INFO] -----END DATA-----
> [INFO] Finished launching job, Host = node08.etri.re.kr RSL = &( jobtype = "single" )( arguments = "4" "/tmp/genome/bwaindex/chr22.fa" "/tmp/genome/Demo/sample_1.fq.00" "/tmp/genome/DemoOut/sample_1.fq.00.sai" )( environment = ( "inputData" "/tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a/inputData" ) ( "outputData" "/tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a/outputData" ) )( stdout = "/tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a/node08_application.stdout" )( count = "4" )( directory = "/tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a" )( executable = "/home/team/bwa_aln" )( stderr = "/tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a/node08_application.stderr" )( maxwalltime = "30" ) working directory = /tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a temp directory = /tmp Globus GateKeeper Endpoint = node08.etri.re.kr:2119/jobmanager-pbs
> [INFO] -----DATA-----
> [INFO] https://node08.etri.re.kr:35051/16290077973982812641/15645714137852419510/
> [INFO] node08.etri.re.kr:2119/jobmanager-pbs
> [INFO] null
> [INFO] null
> [INFO] /O=Grid/OU=GlobusTest/OU=simpleCA-node08.etri.re.kr/OU=local/CN=Team General Member
> [INFO] null
> [INFO] &( jobtype = "single" )( arguments = "4" "/tmp/genome/bwaindex/chr22.fa" "/tmp/genome/Demo/sample_1.fq.00" "/tmp/genome/DemoOut/sample_1.fq.00.sai" )( environment = ( "inputData" "/tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a/inputData" ) ( "outputData" "/tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a/outputData" ) )( stdout = "/tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a/node08_application.stdout" )( count = "4" )( directory = "/tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a" )( executable = "/home/team/bwa_aln" )( stderr = "/tmp/BWA-ALN_Wed_Apr_03_10_14_18_KST_2013_cd417b36-6a9d-4220-bc03-3f52c8641c4a/node08_application.stderr" )( maxwalltime = "30" )
> [INFO] -----END DATA-----
> [INFO] Status is zero
> [INFO] Status of job https://node08.etri.re.kr:35051/16290077973982812641/15645714137852419510/is FAILED
> [INFO] -----DATA-----
> [INFO] Status of job https://node08.etri.re.kr:35051/16290077973982812641/15645714137852419510/is FAILED
> [INFO] -----END DATA-----
> [INFO] Job Error Code: 126
> [ERROR] Context passed was NULL.
> java.lang.RuntimeException: Context passed was NULL.
> at org.apache.airavata.workflow.tracking.impl.ProvenanceNotifierImpl.sendingFault(ProvenanceNotifierImpl.java:496)
> at org.apache.airavata.workflow.tracking.impl.ProvenanceNotifierImpl.sendingFault(ProvenanceNotifierImpl.java:485)
> at org.apache.airavata.core.gfac.notification.impl.WorkflowTrackingNotification.executionFail(WorkflowTrackingNotification.java:108)
> at org.apache.airavata.core.gfac.notification.impl.DefaultNotifier.executionFail(DefaultNotifier.java:135)
> at org.apache.airavata.core.gfac.provider.impl.GramProvider.executeApplication(GramProvider.java:225)
> at org.apache.airavata.core.gfac.provider.AbstractProvider.execute(AbstractProvider.java:69)
> at org.apache.airavata.core.gfac.services.impl.AbstractSimpleService.execute(AbstractSimpleService.java:118)
> at org.apache.airavata.core.gfac.GfacAPI.gridJobSubmit(GfacAPI.java:140)
> at org.apache.airavata.xbaya.invoker.EmbeddedGFacInvoker.invoke(EmbeddedGFacInvoker.java:256)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpreter.handleWSComponent(WorkflowInterpreter.java:749)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpreter.executeDynamically(WorkflowInterpreter.java:533)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpreter.scheduleDynamically(WorkflowInterpreter.java:218)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpretorSkeleton.executeWorkflow(WorkflowInterpretorSkeleton.java:389)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpretorSkeleton.access$400(WorkflowInterpretorSkeleton.java:87)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpretorSkeleton$2.run(WorkflowInterpretorSkeleton.java:382)
> at java.lang.Thread.run(Thread.java:636)
> [INFO] -----DATA-----
> [INFO] Job Protocol : https
> Host name : node08.etri.re.kr
> Port number : 35051
> Url path : 16290077973982812641/15645714137852419510/
> User : null
> Pwd : null
> on host node08.etri.re.kr Job Exit Code = 126
> [INFO] -----END DATA-----
> [ERROR] Job Protocol : https
> Host name : node08.etri.re.kr
> Port number : 35051
> Url path : 16290077973982812641/15645714137852419510/
> User : null
> Pwd : null
> on host node08.etri.re.kr Job Exit Code = 126
> org.apache.airavata.core.gfac.exception.JobSubmissionFault: Job Protocol : https
> Host name : node08.etri.re.kr
> Port number : 35051
> Url path : 16290077973982812641/15645714137852419510/
> User : null
> Pwd : null
> on host node08.etri.re.kr Job Exit Code = 126
> at org.apache.airavata.core.gfac.provider.impl.GramProvider.executeApplication(GramProvider.java:222)
> at org.apache.airavata.core.gfac.provider.AbstractProvider.execute(AbstractProvider.java:69)
> at org.apache.airavata.core.gfac.services.impl.AbstractSimpleService.execute(AbstractSimpleService.java:118)
> at org.apache.airavata.core.gfac.GfacAPI.gridJobSubmit(GfacAPI.java:140)
> at org.apache.airavata.xbaya.invoker.EmbeddedGFacInvoker.invoke(EmbeddedGFacInvoker.java:256)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpreter.handleWSComponent(WorkflowInterpreter.java:749)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpreter.executeDynamically(WorkflowInterpreter.java:533)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpreter.scheduleDynamically(WorkflowInterpreter.java:218)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpretorSkeleton.executeWorkflow(WorkflowInterpretorSkeleton.java:389)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpretorSkeleton.access$400(WorkflowInterpretorSkeleton.java:87)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpretorSkeleton$2.run(WorkflowInterpretorSkeleton.java:382)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.Exception: Job Protocol : https
> Host name : node08.etri.re.kr
> Port number : 35051
> Url path : 16290077973982812641/15645714137852419510/
> User : nul
> Caused by: java.lang.Exception: Job Protocol : https
> Host name : node08.etri.re.kr
> Port number : 35051
> Url path : 16290077973982812641/15645714137852419510/
> User : null
> Pwd : null
> on host node08.etri.re.kr Job Exit Code = 126
> ... 12 more
>
> Regards
> Shinyoung
> From: Suresh Marru [mailto:smarru@apache.org]
> Sent: Wednesday, April 03, 2013 9:11 AM
> To: users@airavata.apache.org
> Subject: Re: A problem on running a workflow through globus!
>
> Hi Shinyoung,
>
> Is there anything specific we can help you in Airavata with? Please feel free to describe your use case and future needs so we can consider in subsequent releases.
>
> Cheers,
> Suresh
>
> On Apr 2, 2013, at 7:57 PM, 안신영 <sy...@etri.re.kr> wrote:
>
>
> The following is the original airavata-server.properties file which is cofigured to use myprox.teragrid.org.
> I changed these 5 properties to my myproxy server configuration.
>
> ###########################################################################
> # Airavata GFac MyProxy GSI credentials to access Grid Resources.
> ###########################################################################
> myproxy.server=myproxy.teragrid.org
> myproxy.user=username
> myproxy.pass=password
> myproxy.life=3600
> # XSEDE Trusted certificates can be downloaded from https://software.xsede.org/security/xsede-certs.tar.gz
> trusted.cert.location=/path/to/trusted/certificates
>
> Regards
> Shinyoung
> 보낸 사람 : "Vijayendra Grampurohit" <vi...@gmail.com>
> 보낸 날짜 : 2013-04-02 21:28:25 ( +09:00 )
> 받는 사람 : users@airavata.apache.org <us...@airavata.apache.org>
> 참조 :
> 제목 : Re: A problem on running a workflow through globus!
>
> Hi
>
> What changes have you done in the airavata-server.properties files? I am still getting those error's.
> Regards
> Vijayendra
>
>
> On Tue, Apr 2, 2013 at 2:14 PM, 안신영 <sy...@etri.re.kr> wrote:
> Hello~
> I solved the previous problems that I had asked.
>
> The reason was missing of configuring myproxy information at airavata-server.properties files.
>
> Thanks.
> Shinyoung
>
> From: 안신영 [mailto:syahn@etri.re.kr]
> Sent: Tuesday, April 02, 2013 3:44 PM
> To: users@airavata.apache.org
> Subject: A problem on running a workflow through globus!
>
> Hello~ guys
> I’m testing that Airavata can really work with globus because I am using slurm now on my HPC.
>
> I installed globus at two machine(one for CA and PBS(actually pbs wrapper for slurm) local Resource management server, the other for airavata-server and xbaya)
> I succeeded globus-job-run in command line between two machine.
>
> Next, I registered a new service(/bin/echo) and composed and ran a simple workflow in XBaya.
> I got a problem of invokingServiceFailed.
> Next is error messages from XBaya.
> Do you have any comment about that?
>
> Shinyoung.
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> <ns:invokingServiceFailed infoModelVersion="2.6"
> xmlns:ns="http://airavata.apache.org/schemas/wft/2011/08">
> <ns:notificationSource ns:serviceID="Workflow1_1f8e7176_b86d_4977_99f8_01d928f5f937" ns:experimentID="Workflow1_1f8e7176-b86d-4977-99f8-01d928f5f937" />
> <ns:timestamp>2013-04-02T15:02:01.166+09:00</ns:timestamp>
> <ns:description>MyProxy get failed. [Caused by: Authentication failed [Caused by: Failure unspecified at GSS-API level [Caused by: Unknown CA]]]: org.apache.airavata.core.gfac.exception.ProviderException: MyProxy get failed. [Caused by: Authentication failed [Caused by: Failure unspecified at GSS-API level [Caused by: Unknown CA]]]</ns:description>
> <ns:annotation>
> <stackTrace>org.apache.airavata.core.gfac.exception.ProviderException: MyProxy get failed. [Caused by: Authentication failed [Caused by: Failure unspecified at GSS-API level [Caused by: Unknown CA]]]
> at org.apache.airavata.core.gfac.provider.impl.GramProvider.makeDirectory(GramProvider.java:128)
> at org.apache.airavata.core.gfac.provider.AbstractProvider.initialize(AbstractProvider.java:48)
> at org.apache.airavata.core.gfac.services.impl.AbstractSimpleService.execute(AbstractSimpleService.java:103)
> at org.apache.airavata.core.gfac.GfacAPI.gridJobSubmit(GfacAPI.java:140)
> at org.apache.airavata.xbaya.invoker.EmbeddedGFacInvoker.invoke(EmbeddedGFacInvoker.java:256)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpreter.handleWSComponent(WorkflowInterpreter.java:749)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpreter.executeDynamically(WorkflowInterpreter.java:533)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpreter.scheduleDynamically(WorkflowInterpreter.java:218)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpretorSkeleton.executeWorkflow(WorkflowInterpretorSkeleton.java:389)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpretorSkeleton.access$400(WorkflowInterpretorSkeleton.java:87)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpretorSkeleton$2.run(WorkflowInterpretorSkeleton.java:382)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: org.apache.airavata.core.gfac.exception.SecurityException: MyProxy get failed. [Caused by: Authentication failed [Caused by: Failure unspecified at GSS-API level [Caused by: Unknown CA]]]
> at org.apache.airavata.core.gfac.context.security.impl.GSISecurityContext.getGssCredentails(GSISecurityContext.java:69)
> at org.apache.airavata.core.gfac.provider.impl.GramProvider.makeDirectory(GramProvider.java:87)
> ... 11 more
> Caused by: org.globus.myproxy.MyProxyException: MyProxy get failed. [Caused by: Authentication failed [Caused by: Failure unspecified at GSS-API level [Caused by: Unknown CA]]]
> at org.globus.myproxy.MyProxy.get(MyProxy.java:1023)
> at org.globus.myproxy.MyProxy.get(MyProxy.java:874)
> at org.globus.myproxy.MyProxy.get(MyProxy.java:841)
> at org.apache.airavata.core.gfac.context.security.impl.utils.MyProxyManager.renewProxy(MyProxyManager.java:80)
> at org.apache.airavata.core.gfac.context.security.impl.GSISecurityContext.getGssCredentails(GSISecurityContext.java:64)
> ... 12 more
> </stackTrace>
> </ns:annotation>
> <ns:receiver ns:serviceID="remote_echo_invoke" ns:workflowID="Workflow1_1f8e7176_b86d_4977_99f8_01d928f5f937" ns:workflowTimestep="0" ns:workflowNodeID="remote_echo_invoke" />
> <ns:failure>
> <ns:trace xsi:type="xs:string" xmlns:xs="http://www.w3.org/2001/XMLSchema"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:ns="http://airavata.apache.org/schemas/wft/2011/08">org.apache.airavata.core.gfac.exception.ProviderException: MyProxy get failed. [Caused by: Authentication failed [Caused by: Failure unspecified at GSS-API level [Caused by: Unknown CA]]]
> at org.apache.airavata.core.gfac.provider.impl.GramProvider.makeDirectory(GramProvider.java:128)
> at org.apache.airavata.core.gfac.provider.AbstractProvider.initialize(AbstractProvider.java:48)
> at org.apache.airavata.core.gfac.services.impl.AbstractSimpleService.execute(AbstractSimpleService.java:103)
> at org.apache.airavata.core.gfac.GfacAPI.gridJobSubmit(GfacAPI.java:140)
> at org.apache.airavata.xbaya.invoker.EmbeddedGFacInvoker.invoke(EmbeddedGFacInvoker.java:256)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpreter.handleWSComponent(WorkflowInterpreter.java:749)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpreter.executeDynamically(WorkflowInterpreter.java:533)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpreter.scheduleDynamically(WorkflowInterpreter.java:218)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpretorSkeleton.executeWorkflow(WorkflowInterpretorSkeleton.java:389)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpretorSkeleton.access$400(WorkflowInterpretorSkeleton.java:87)
> at org.apache.airavata.xbaya.interpretor.WorkflowInterpretorSkeleton$2.run(WorkflowInterpretorSkeleton.java:382)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: org.apache.airavata.core.gfac.exception.SecurityException: MyProxy get failed. [Caused by: Authentication failed [Caused by: Failure unspecified at GSS-API level [Caused by: Unknown CA]]]
> at org.apache.airavata.core.gfac.context.security.impl.GSISecurityContext.getGssCredentails(GSISecurityContext.java:69)
> at org.apache.airavata.core.gfac.provider.impl.GramProvider.makeDirectory(GramProvider.java:87)
> ... 11 more
> Caused by: org.globus.myproxy.MyProxyException: MyProxy get failed. [Caused by: Authentication failed [Caused by: Failure unspecified at GSS-API level [Caused by: Unknown CA]]]
> at org.globus.myproxy.MyProxy.get(MyProxy.java:1023)
> at org.globus.myproxy.MyProxy.get(MyProxy.java:874)
> at org.globus.myproxy.MyProxy.get(MyProxy.java:841)
> at org.apache.airavata.core.gfac.context.security.impl.utils.MyProxyManager.renewProxy(MyProxyManager.java:80)
> at org.apache.airavata.core.gfac.context.security.impl.GSISecurityContext.getGssCredentails(GSISecurityContext.java:64)
> ... 12 more
> </ns:trace>
> </ns:failure>
> </ns:invokingServiceFailed>
>