You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@airavata.apache.org by Aaron Householder <aa...@aucsolutions.com> on 2024/10/13 00:11:59 UTC
Re: Failed but metascheduler did not resubmit job
Hi Lahiru,
Any update on this issue? This is an impediment to getting this rolled out to UltraScan users. My understanding is that if the job fails while verifying and making checks that the metascheduler should try another resource.
Is there anything I can do to help?
Regards,
Aaron
From: Lahiru Jayathilake <la...@apache.org>
Date: Thursday, September 12, 2024 at 1:42 PM
To: users@airavata.apache.org <us...@airavata.apache.org>
Subject: Re: Failed but metascheduler did not resubmit job
Hi Aaron,
Thanks for contacting us. We will look into this issue and get back to you.
Best,
Lahiru
On 2024/09/11 18:45:33 Aaron Householder wrote:
> Hi Airavata,
>
> I’m working on connecting Ultrascan3 to Airavata. As the message below shows, if the job fails the metascheduler might not retry the job. Is there a resource available to take a look at this issue?
>
> Regards,
> Aaron
>
> From: Aaron Householder <aa...@aucsolutions.com>
> Date: Tuesday, September 3, 2024 at 4:42 PM
> To: Airavata Users <us...@airavata.apache.org>
> Subject: Failed but metascheduler did not resubmit job
> Hi Airavata Users,
>
> I had an UltraScan job that seemed to fail without the metascheduler resubmitting the job for completion by another cluster. I received the following in an email:
>
> Your UltraScan job is complete:
>
> Submission Time : 2024-08-26 00:50:05
> Job End Time :
> Mail Time : 2024-08-25 19:54:41
> LIMS Host :
> Analysis ID : US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c
> Request ID : 182 ( uslims3_Demo )
> RunID : demo1_veloc1
> EditID : 21030600161
> Data Type : RA
> Cell/Channel/Wl : 2 / A / 259
> Status : failed
> Cluster : metascheduler
> Job Type : 2DSA-MC
> GFAC Status : FAILED
> GFAC Message : org.apache.airavata.helix.impl.task.TaskOnFailException: Error Code : 23857cb5-5431-43e7-a927-fedd7a929e34, Task TASK_5b1ea99b-750c-49f6-a05b-df7175f141ed failed due to Couldn't find job id in both submitted and verified steps. expId:US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c Couldn't find remote jobId for JobName:A1394806797, both submit and verify steps doesn't return a valid JobId. Hence changing experiment state to Failed
> at org.apache.airavata.helix.impl.task.AiravataTask.onFail(AiravataTask.java:146)
> at org.apache.airavata.helix.impl.task.submission.DefaultJobSubmissionTask.onRun(DefaultJobSubmissionTask.java:192)
> at org.apache.airavata.helix.impl.task.AiravataTask.onRun(AiravataTask.java:437)
> at org.apache.airavata.helix.core.AbstractTask.run(AbstractTask.java:102)
> at org.apache.helix.task.TaskRunner.run(TaskRunner.java:71)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
>
>
> No reservation for this job
> --> Verifying valid submit host (login2)...OK
> --> Verifying valid jobname...OK
> --> Verifying valid ssh keys...OK
> --> Verifying access to desired queue (normal)...OK
> --> Checking available allocation FAILED
> Airavata stderr : ERROR: You have no project in the projectuser.map file (in accounting_check_prod.pl).
>
Re: Failed but metascheduler did not resubmit job
Posted by Lahiru Jayathilake <la...@gmail.com>.
Hi Aaron,
As of now, the mentioned task has not been added to a work plan and is
still open for work. However, given your immediate requirement, you can
resolve it by having an active allocation for your experiment submission,
allowing it to proceed without needing the feature.
Let me know if you need further assistance or clarification.
Thanks,
Lahiru
On Thu, Oct 17, 2024 at 8:16 AM Aaron Householder <aa...@aucsolutions.com>
wrote:
> Hi Lahiru,
>
>
>
> Thank you for getting back to me and writing up the improvement. I’m
> trying to submit an academic paper before December. Given your initial
> review, do you think this feature will be available in time for me to meet
> my deadline?
>
>
>
> Regards,
>
> Aaron
>
>
>
> *From: *Lahiru Jayathilake <la...@gmail.com>
> *Date: *Thursday, October 17, 2024 at 12:41 AM
> *To: *aaron@aucsolutions.com <aa...@aucsolutions.com>
> *Cc: *users@airavata.apache.org <us...@airavata.apache.org>
> *Subject: *Re: Failed but metascheduler did not resubmit job
>
> Hi Aaron,
>
> Thank you for your patience, and I apologize for the delay in getting back
> to you regarding the issue.
>
> After further investigation, I noticed that the current version of the
> Airavata Metascheduler does not support automatically resubmitting jobs to
> different clusters when a job fails after successful submission (e.g., due
> to resource allocation issues). I've now created a task [1] to add this
> feature, which will enable the expected functionality. This enhancement
> will take some time to implement, but we’ll keep you updated on the
> progress.
>
> Please feel free to reach out if you have any further questions or need
> additional information.
>
> [1] - https://issues.apache.org/jira/browse/AIRAVATA-3893
>
>
>
> Thanks,
> Lahiru
>
>
>
> On Wed, Oct 16, 2024 at 7:36 PM Aaron Householder <aa...@aucsolutions.com>
> wrote:
>
> Hi,
>
>
>
> Is there any update?
>
>
>
> Regards,
>
> Aaron
>
>
>
> *From: *Aaron Householder <aa...@aucsolutions.com>
> *Date: *Saturday, October 12, 2024 at 8:12 PM
> *To: *users@airavata.apache.org <us...@airavata.apache.org>
> *Subject: *Re: Failed but metascheduler did not resubmit job
>
> Hi Lahiru,
>
>
>
> Any update on this issue? This is an impediment to getting this rolled out
> to UltraScan users. My understanding is that if the job fails while
> verifying and making checks that the metascheduler should try another
> resource.
>
>
>
> Is there anything I can do to help?
>
>
>
> Regards,
>
> Aaron
>
>
>
> *From: *Lahiru Jayathilake <la...@apache.org>
> *Date: *Thursday, September 12, 2024 at 1:42 PM
> *To: *users@airavata.apache.org <us...@airavata.apache.org>
> *Subject: *Re: Failed but metascheduler did not resubmit job
>
> Hi Aaron,
>
> Thanks for contacting us. We will look into this issue and get back to you.
>
> Best,
> Lahiru
>
> On 2024/09/11 18:45:33 Aaron Householder wrote:
> > Hi Airavata,
> >
> > I’m working on connecting Ultrascan3 to Airavata. As the message below
> shows, if the job fails the metascheduler might not retry the job. Is there
> a resource available to take a look at this issue?
> >
> > Regards,
> > Aaron
> >
> > From: Aaron Householder <aa...@aucsolutions.com>
> > Date: Tuesday, September 3, 2024 at 4:42 PM
> > To: Airavata Users <us...@airavata.apache.org>
> > Subject: Failed but metascheduler did not resubmit job
> > Hi Airavata Users,
> >
> > I had an UltraScan job that seemed to fail without the metascheduler
> resubmitting the job for completion by another cluster. I received the
> following in an email:
> >
> > Your UltraScan job is complete:
> >
> > Submission Time : 2024-08-26 00:50:05
> > Job End Time :
> > Mail Time : 2024-08-25 19:54:41
> > LIMS Host :
> > Analysis ID : US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c
> > Request ID : 182 ( uslims3_Demo )
> > RunID : demo1_veloc1
> > EditID : 21030600161
> > Data Type : RA
> > Cell/Channel/Wl : 2 / A / 259
> > Status : failed
> > Cluster : metascheduler
> > Job Type : 2DSA-MC
> > GFAC Status : FAILED
> > GFAC Message :
> org.apache.airavata.helix.impl.task.TaskOnFailException: Error Code :
> 23857cb5-5431-43e7-a927-fedd7a929e34, Task
> TASK_5b1ea99b-750c-49f6-a05b-df7175f141ed failed due to Couldn't find job
> id in both submitted and verified steps.
> expId:US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c Couldn't find remote
> jobId for JobName:A1394806797, both submit and verify steps doesn't return
> a valid JobId. Hence changing experiment state to Failed
> > at
> org.apache.airavata.helix.impl.task.AiravataTask.onFail(AiravataTask.java:146)
> > at
> org.apache.airavata.helix.impl.task.submission.DefaultJobSubmissionTask.onRun(DefaultJobSubmissionTask.java:192)
> > at
> org.apache.airavata.helix.impl.task.AiravataTask.onRun(AiravataTask.java:437)
> > at
> org.apache.airavata.helix.core.AbstractTask.run(AbstractTask.java:102)
> > at org.apache.helix.task.TaskRunner.run(TaskRunner.java:71)
> > at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> > at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> >
> >
> > No reservation for this job
> > --> Verifying valid submit host (login2)...OK
> > --> Verifying valid jobname...OK
> > --> Verifying valid ssh keys...OK
> > --> Verifying access to desired queue (normal)...OK
> > --> Checking available allocation FAILED
> > Airavata stderr : ERROR: You have no project in the projectuser.map
> file (in accounting_check_prod.pl).
> >
>
>
Re: Failed but metascheduler did not resubmit job
Posted by Aaron Householder <aa...@aucsolutions.com>.
Hi Lahiru,
Thank you for getting back to me and writing up the improvement. I’m trying to submit an academic paper before December. Given your initial review, do you think this feature will be available in time for me to meet my deadline?
Regards,
Aaron
From: Lahiru Jayathilake <la...@gmail.com>
Date: Thursday, October 17, 2024 at 12:41 AM
To: aaron@aucsolutions.com <aa...@aucsolutions.com>
Cc: users@airavata.apache.org <us...@airavata.apache.org>
Subject: Re: Failed but metascheduler did not resubmit job
Hi Aaron,
Thank you for your patience, and I apologize for the delay in getting back to you regarding the issue.
After further investigation, I noticed that the current version of the Airavata Metascheduler does not support automatically resubmitting jobs to different clusters when a job fails after successful submission (e.g., due to resource allocation issues). I've now created a task [1] to add this feature, which will enable the expected functionality. This enhancement will take some time to implement, but we’ll keep you updated on the progress.
Please feel free to reach out if you have any further questions or need additional information.
[1] - https://issues.apache.org/jira/browse/AIRAVATA-3893
Thanks,
Lahiru
On Wed, Oct 16, 2024 at 7:36 PM Aaron Householder <aa...@aucsolutions.com>> wrote:
Hi,
Is there any update?
Regards,
Aaron
From: Aaron Householder <aa...@aucsolutions.com>>
Date: Saturday, October 12, 2024 at 8:12 PM
To: users@airavata.apache.org<ma...@airavata.apache.org> <us...@airavata.apache.org>>
Subject: Re: Failed but metascheduler did not resubmit job
Hi Lahiru,
Any update on this issue? This is an impediment to getting this rolled out to UltraScan users. My understanding is that if the job fails while verifying and making checks that the metascheduler should try another resource.
Is there anything I can do to help?
Regards,
Aaron
From: Lahiru Jayathilake <la...@apache.org>>
Date: Thursday, September 12, 2024 at 1:42 PM
To: users@airavata.apache.org<ma...@airavata.apache.org> <us...@airavata.apache.org>>
Subject: Re: Failed but metascheduler did not resubmit job
Hi Aaron,
Thanks for contacting us. We will look into this issue and get back to you.
Best,
Lahiru
On 2024/09/11 18:45:33 Aaron Householder wrote:
> Hi Airavata,
>
> I’m working on connecting Ultrascan3 to Airavata. As the message below shows, if the job fails the metascheduler might not retry the job. Is there a resource available to take a look at this issue?
>
> Regards,
> Aaron
>
> From: Aaron Householder <aa...@aucsolutions.com>>
> Date: Tuesday, September 3, 2024 at 4:42 PM
> To: Airavata Users <us...@airavata.apache.org>>
> Subject: Failed but metascheduler did not resubmit job
> Hi Airavata Users,
>
> I had an UltraScan job that seemed to fail without the metascheduler resubmitting the job for completion by another cluster. I received the following in an email:
>
> Your UltraScan job is complete:
>
> Submission Time : 2024-08-26 00:50:05
> Job End Time :
> Mail Time : 2024-08-25 19:54:41
> LIMS Host :
> Analysis ID : US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c
> Request ID : 182 ( uslims3_Demo )
> RunID : demo1_veloc1
> EditID : 21030600161
> Data Type : RA
> Cell/Channel/Wl : 2 / A / 259
> Status : failed
> Cluster : metascheduler
> Job Type : 2DSA-MC
> GFAC Status : FAILED
> GFAC Message : org.apache.airavata.helix.impl.task.TaskOnFailException: Error Code : 23857cb5-5431-43e7-a927-fedd7a929e34, Task TASK_5b1ea99b-750c-49f6-a05b-df7175f141ed failed due to Couldn't find job id in both submitted and verified steps. expId:US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c Couldn't find remote jobId for JobName:A1394806797, both submit and verify steps doesn't return a valid JobId. Hence changing experiment state to Failed
> at org.apache.airavata.helix.impl.task.AiravataTask.onFail(AiravataTask.java:146)
> at org.apache.airavata.helix.impl.task.submission.DefaultJobSubmissionTask.onRun(DefaultJobSubmissionTask.java:192)
> at org.apache.airavata.helix.impl.task.AiravataTask.onRun(AiravataTask.java:437)
> at org.apache.airavata.helix.core.AbstractTask.run(AbstractTask.java:102)
> at org.apache.helix.task.TaskRunner.run(TaskRunner.java:71)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
>
>
> No reservation for this job
> --> Verifying valid submit host (login2)...OK
> --> Verifying valid jobname...OK
> --> Verifying valid ssh keys...OK
> --> Verifying access to desired queue (normal)...OK
> --> Checking available allocation FAILED
> Airavata stderr : ERROR: You have no project in the projectuser.map file (in accounting_check_prod.pl<http://accounting_check_prod.pl>).
>
Duplicate - Re: Failed but metascheduler did not resubmit job
Posted by Emre Brookes <em...@mso.umt.edu>.
(resend since this email addres wasn't subscribed to
users@airavata.apache.org)
Hi Eroma,
We understand we do not have a current allocation on LS6.
We expect the metascheduler to reschedule to another cluster for this
error, will this case not be supported?
Our goal for the metascheduler was for any issues, regardless of cause,
to be as transparent as possible for the user.
Allocations could run out, expire or perhaps problems with the
allocation mechanism on the target resource - all
these (and others, we have seen communications issues particularly with
expanse) - should be recoverable by
the metascheduler. This is the point of it. The simple case of a job
running and returning a failed status is likely
the *least* recoverable, since if the job ran correctly, there is likely
a data issue that will cause it to fail elsewhere
(exceptions could be different or memory limits, or the rare case a node
failed during the computation).
Thanks,
Emre
Eroma Abeysinghe wrote:
> Hi Emre, et al,
>
> I looked at the failed experiment Aaron shared. It was executed in ls6.
> I looked at ls6 experiments for the past year and 9 months ago the
> cluster started asking for a project allocation. Before, Ultrascan had
> submitted successfully without a project allocation.
> So right now, to submit to ls6, you need to contact them, provide your
> details and ask for the project allocation number.
>
> The messages we see from ls6 [1]
> Aaron, do you have issues submitting to other clusters, or the
> research to be done in ls6?
>
> Thanks,
> Eroma
>
> ----
> [1]
> A1394806797 STDOUT
> -----------------------------------------------------------------
> Welcome to the Lonestar6 Supercomputer
> -----------------------------------------------------------------
>
> No reservation for this job
> --> Verifying valid submit host (login2)...OK
> --> Verifying valid jobname...OK
> --> Verifying valid ssh keys...OK
> --> Verifying access to desired queue (normal)...OK
> --> Checking available allocation FAILED
> A1394806797 STDERR
> ERROR: You have no project in the projectuser.map file
> (inaccounting_check_prod.pl
> <http://accounting_check_prod.pl/>).
>
> Please report this problem:
> U. of TX users contact (https://portal.tacc.utexas.edu/consulting
> <https://portal.tacc.utexas.edu/consulting>)
> ------
>
> On Fri, Oct 25, 2024 at 11:05 AM Lahiru Jayathilake
> <lahirujayathilake@gmail.com <ma...@gmail.com>> wrote:
>
>
>
> ---------- Forwarded message ---------
> From: *Emre Brookes* <emre.brookes@mso.umt.edu
> <ma...@mso.umt.edu>>
> Date: Tue, Oct 22, 2024 at 4:10 PM
> Subject: Re: Failed but metascheduler did not resubmit job
> To: <users@airavata.apache.org
> <ma...@airavata.apache.org>>, Lahiru Jayathilake
> <lahirujayathilake@gmail.com <ma...@gmail.com>>
> Cc: <aaron@aucsolutions.com <ma...@aucsolutions.com>>
>
>
> Hi Lahiru,
>
> I really appreciate your work on this.
> It's important that we have a working system that we can include
> in our
> progress report to the NIH & I'd like to see Aaron able to finish his
> paper this year.
> If there's anything we can do from our end to expedite this,
> please let
> me know.
>
> Thanks,
> Emre
>
> Lahiru Jayathilake wrote:
> > *EXTERNAL EMAIL*
> >
> > Hi Aaron,
> >
> > Thank you for your patience, and I apologize for the delay in
> getting
> > back to you regarding the issue.
> >
> > After further investigation, I noticed that the current version
> of the
> > Airavata Metascheduler does not support automatically resubmitting
> > jobs to different clusters when a job fails after successful
> > submission (e.g., due to resource allocation issues). I've now
> created
> > a task [1] to add this feature, which will enable the expected
> > functionality. This enhancement will take some time to
> implement, but
> > we’ll keep you updated on the progress.
> >
> > Please feel free to reach out if you have any further questions or
> > need additional information.
> >
> > [1] - https://issues.apache.org/jira/browse/AIRAVATA-3893
> <https://issues.apache.org/jira/browse/AIRAVATA-3893>
>
> > <https://issues.apache.org/jira/browse/AIRAVATA-3893
> <https://issues.apache.org/jira/browse/AIRAVATA-3893>>
> >
> > Thanks,
> > Lahiru
> >
> > On Wed, Oct 16, 2024 at 7:36 PM Aaron Householder
> > <aaron@aucsolutions.com <ma...@aucsolutions.com>
> <mailto:aaron@aucsolutions.com <ma...@aucsolutions.com>>>
> wrote:
> >
> > Hi,
> >
> > Is there any update?
> >
> > Regards,
> >
> > Aaron
> >
> > *From: *Aaron Householder <aaron@aucsolutions.com
> <ma...@aucsolutions.com>
> > <mailto:aaron@aucsolutions.com <ma...@aucsolutions.com>>>
> > *Date: *Saturday, October 12, 2024 at 8:12 PM
> > *To: *users@airavata.apache.org
> <ma...@airavata.apache.org>
> <mailto:users@airavata.apache.org <ma...@airavata.apache.org>>
> > <users@airavata.apache.org
> <ma...@airavata.apache.org>
> <mailto:users@airavata.apache.org <ma...@airavata.apache.org>>>
> > *Subject: *Re: Failed but metascheduler did not resubmit job
> >
> > Hi Lahiru,
> >
> > Any update on this issue? This is an impediment to getting this
> > rolled out to UltraScan users. My understanding is that if
> the job
> > fails while verifying and making checks that the metascheduler
> > should try another resource.
> >
> > Is there anything I can do to help?
> >
> > Regards,
> >
> > Aaron
> >
> > *From: *Lahiru Jayathilake <lahirujayathilake@apache.org
> <ma...@apache.org>
> > <mailto:lahirujayathilake@apache.org
> <ma...@apache.org>>>
> > *Date: *Thursday, September 12, 2024 at 1:42 PM
> > *To: *users@airavata.apache.org
> <ma...@airavata.apache.org>
> <mailto:users@airavata.apache.org <ma...@airavata.apache.org>>
> > <users@airavata.apache.org
> <ma...@airavata.apache.org>
> <mailto:users@airavata.apache.org <ma...@airavata.apache.org>>>
> > *Subject: *Re: Failed but metascheduler did not resubmit job
> >
> > Hi Aaron,
> >
> > Thanks for contacting us. We will look into this issue and get
> > back to you.
> >
> > Best,
> > Lahiru
> >
> > On 2024/09/11 18:45:33 Aaron Householder wrote:
> > > Hi Airavata,
> > >
> > > I’m working on connecting Ultrascan3 to Airavata. As the
> message
> > below shows, if the job fails the metascheduler might not retry
> > the job. Is there a resource available to take a look at
> this issue?
> > >
> > > Regards,
> > > Aaron
> > >
> > > From: Aaron Householder <aaron@aucsolutions.com
> <ma...@aucsolutions.com>
> > <mailto:aaron@aucsolutions.com <ma...@aucsolutions.com>>>
> > > Date: Tuesday, September 3, 2024 at 4:42 PM
> > > To: Airavata Users <users@airavata.apache.org
> <ma...@airavata.apache.org>
> > <mailto:users@airavata.apache.org
> <ma...@airavata.apache.org>>>
> > > Subject: Failed but metascheduler did not resubmit job
> > > Hi Airavata Users,
> > >
> > > I had an UltraScan job that seemed to fail without the
> > metascheduler resubmitting the job for completion by another
> > cluster. I received the following in an email:
> > >
> > > Your UltraScan job is complete:
> > >
> > > Submission Time : 2024-08-26 00:50:05
> > > Job End Time :
> > > Mail Time : 2024-08-25 19:54:41
> > > LIMS Host :
> > > Analysis ID :
> US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c
> > > Request ID : 182 ( uslims3_Demo )
> > > RunID : demo1_veloc1
> > > EditID : 21030600161
> > > Data Type : RA
> > > Cell/Channel/Wl : 2 / A / 259
> > > Status : failed
> > > Cluster : metascheduler
> > > Job Type : 2DSA-MC
> > > GFAC Status : FAILED
> > > GFAC Message :
> > org.apache.airavata.helix.impl.task.TaskOnFailException: Error
> > Code : 23857cb5-5431-43e7-a927-fedd7a929e34, Task
> > TASK_5b1ea99b-750c-49f6-a05b-df7175f141ed failed due to Couldn't
> > find job id in both submitted and verified steps.
> > expId:US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c Couldn't find
> > remote jobId for JobName:A1394806797, both submit and verify
> steps
> > doesn't return a valid JobId. Hence changing experiment state to
> > Failed
> > > at
> >
> org.apache.airavata.helix.impl.task.AiravataTask.onFail(AiravataTask.java:146)
> > > at
> >
> org.apache.airavata.helix.impl.task.submission.DefaultJobSubmissionTask.onRun(DefaultJobSubmissionTask.java:192)
> > > at
> >
> org.apache.airavata.helix.impl.task.AiravataTask.onRun(AiravataTask.java:437)
> > > at
> >
> org.apache.airavata.helix.core.AbstractTask.run(AbstractTask.java:102)
> > > at
> org.apache.helix.task.TaskRunner.run(TaskRunner.java:71)
> > > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > at
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > at
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> > > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > at java.base/java.lang.Thread.run(Thread.java:829)
> > >
> > >
> > > No reservation for this job
> > > --> Verifying valid submit host (login2)...OK
> > > --> Verifying valid jobname...OK
> > > --> Verifying valid ssh keys...OK
> > > --> Verifying access to desired queue (normal)...OK
> > > --> Checking available allocation FAILED
> > > Airavata stderr : ERROR: You have no project in the
> > projectuser.map file (in accounting_check_prod.pl
> <http://accounting_check_prod.pl/>
> > <http://accounting_check_prod.pl/
> <http://accounting_check_prod.pl/>>).
> > >
> >
> >
> > CAUTION: This message originated outside of UT Health San Antonio.
> > Please exercise caution when clicking on links or opening
> attachments.
>
>
>
> -- Thank You,
> Best Regards,
> Eroma
Re: Failed but metascheduler did not resubmit job
Posted by Emre Brookes <em...@mso.umt.edu>.
Hi Eroma,
OK. Is there any timeline to get these capabilities implemented?
Given the current state, how can we formally test the ability of the
metascheduler to reschedule?
My thought was to use LS6 since its allocation expired, but that doesn't
work.
If we supply a fake queue name for that resource, will that cause it to
first try and then reschedule?
Any other ideas for a UltraScan LIMS using
SciGap/ultrascan-airavata-bridge initiated testing ?
Thanks,
Emre
Eroma Abeysinghe wrote:
> Hi Emre,
>
> Currently, what metascheduler does is find a resource from the pool
> confirm its available, can ssh, verify the queue, check the queue
> limit and submit the job (Lahiru also explained previously in this
> thread). After this point, if the job fails due to allocation not
> being available etc... we don't send it back to a new resource. This
> is yet to be implemented.
>
> Thanks,
> Eroma
>
>
>
>
>
> On Fri, Oct 25, 2024 at 12:32 PM Emre Brookes
> <emre.brookes@mso.umt.edu <ma...@mso.umt.edu>> wrote:
>
> Hi Eroma,
>
> We understand we do not have a current allocation on LS6.
> We expect the metascheduler to reschedule to another cluster for this
> error, will this case not be supported?
> Our goal for the metascheduler was for any issues, regardless of
> cause,
> to be as transparent as possible for the user.
> Allocations could run out, expire or perhaps problems with the
> allocation mechanism on the target resource - all
> these (and others, we have seen communications issues particularly
> with
> expanse) - should be recoverable by
> the metascheduler. This is the point of it. The simple case of a job
> running and returning a failed status is likely
> the *least* recoverable, since if the job ran correctly, there is
> likely
> a data issue that will cause it to fail elsewhere
> (exceptions could be different or memory limits, or the rare case
> a node
> failed during the computation).
>
> Thanks,
> Emre
>
>
> Eroma Abeysinghe wrote:
> > Hi Emre, et al,
> >
> > I looked at the failed experiment Aaron shared. It was executed
> in ls6.
> > I looked at ls6 experiments for the past year and 9 months ago the
> > cluster started asking for a project allocation.
> Before, Ultrascan had
> > submitted successfully without a project allocation.
> > So right now, to submit to ls6, you need to contact them,
> provide your
> > details and ask for the project allocation number.
> >
> > The messages we see from ls6 [1]
> > Aaron, do you have issues submitting to other clusters, or the
> > research to be done in ls6?
> >
> > Thanks,
> > Eroma
> >
> > ----
> > [1]
> > A1394806797 STDOUT
> > -----------------------------------------------------------------
> > Welcome to the Lonestar6 Supercomputer
> > -----------------------------------------------------------------
> >
> > No reservation for this job
> > --> Verifying valid submit host (login2)...OK
> > --> Verifying valid jobname...OK
> > --> Verifying valid ssh keys...OK
> > --> Verifying access to desired queue (normal)...OK
> > --> Checking available allocation FAILED
> > A1394806797 STDERR
> > ERROR: You have no project in the projectuser.map file
> (inaccounting_check_prod.pl
> <http://inaccounting_check_prod.pl/>
> <http://accounting_check_prod.pl/
> <http://accounting_check_prod.pl/>>).
> >
> > Please report this problem:
> > U. of TX users contact
> (https://portal.tacc.utexas.edu/consulting
> <https://portal.tacc.utexas.edu/consulting>
> <https://portal.tacc.utexas.edu/consulting
> <https://portal.tacc.utexas.edu/consulting>>)
> > ------
> >
> > On Fri, Oct 25, 2024 at 11:05 AM Lahiru Jayathilake
> > <lahirujayathilake@gmail.com
> <ma...@gmail.com>
> <mailto:lahirujayathilake@gmail.com
> <ma...@gmail.com>>> wrote:
> >
> >
> >
> > ---------- Forwarded message ---------
> > From: *Emre Brookes* <emre.brookes@mso.umt.edu
> <ma...@mso.umt.edu>
> > <mailto:emre.brookes@mso.umt.edu
> <ma...@mso.umt.edu>>>
> > Date: Tue, Oct 22, 2024 at 4:10 PM
> > Subject: Re: Failed but metascheduler did not resubmit job
> > To: <users@airavata.apache.org
> <ma...@airavata.apache.org>
> > <mailto:users@airavata.apache.org
> <ma...@airavata.apache.org>>>, Lahiru Jayathilake
> > <lahirujayathilake@gmail.com
> <ma...@gmail.com>
> <mailto:lahirujayathilake@gmail.com
> <ma...@gmail.com>>>
> > Cc: <aaron@aucsolutions.com <ma...@aucsolutions.com>
> <mailto:aaron@aucsolutions.com <ma...@aucsolutions.com>>>
> >
> >
> > Hi Lahiru,
> >
> > I really appreciate your work on this.
> > It's important that we have a working system that we can include
> > in our
> > progress report to the NIH & I'd like to see Aaron able to
> finish his
> > paper this year.
> > If there's anything we can do from our end to expedite this,
> > please let
> > me know.
> >
> > Thanks,
> > Emre
> >
> > Lahiru Jayathilake wrote:
> > > *EXTERNAL EMAIL*
> > >
> > > Hi Aaron,
> > >
> > > Thank you for your patience, and I apologize for the delay in
> > getting
> > > back to you regarding the issue.
> > >
> > > After further investigation, I noticed that the current
> version
> > of the
> > > Airavata Metascheduler does not support automatically
> resubmitting
> > > jobs to different clusters when a job fails after successful
> > > submission (e.g., due to resource allocation issues). I've now
> > created
> > > a task [1] to add this feature, which will enable the expected
> > > functionality. This enhancement will take some time to
> > implement, but
> > > we’ll keep you updated on the progress.
> > >
> > > Please feel free to reach out if you have any further
> questions or
> > > need additional information.
> > >
> > > [1] - https://issues.apache.org/jira/browse/AIRAVATA-3893
> <https://issues.apache.org/jira/browse/AIRAVATA-3893>
> > <https://issues.apache.org/jira/browse/AIRAVATA-3893
> <https://issues.apache.org/jira/browse/AIRAVATA-3893>>
> >
> > > <https://issues.apache.org/jira/browse/AIRAVATA-3893
> <https://issues.apache.org/jira/browse/AIRAVATA-3893>
> > <https://issues.apache.org/jira/browse/AIRAVATA-3893
> <https://issues.apache.org/jira/browse/AIRAVATA-3893>>>
> > >
> > > Thanks,
> > > Lahiru
> > >
> > > On Wed, Oct 16, 2024 at 7:36 PM Aaron Householder
> > > <aaron@aucsolutions.com <ma...@aucsolutions.com>
> <mailto:aaron@aucsolutions.com <ma...@aucsolutions.com>>
> > <mailto:aaron@aucsolutions.com
> <ma...@aucsolutions.com> <mailto:aaron@aucsolutions.com
> <ma...@aucsolutions.com>>>>
> > wrote:
> > >
> > > Hi,
> > >
> > > Is there any update?
> > >
> > > Regards,
> > >
> > > Aaron
> > >
> > > *From: *Aaron Householder <aaron@aucsolutions.com
> <ma...@aucsolutions.com>
> > <mailto:aaron@aucsolutions.com <ma...@aucsolutions.com>>
> > > <mailto:aaron@aucsolutions.com
> <ma...@aucsolutions.com> <mailto:aaron@aucsolutions.com
> <ma...@aucsolutions.com>>>>
> > > *Date: *Saturday, October 12, 2024 at 8:12 PM
> > > *To: *users@airavata.apache.org
> <ma...@airavata.apache.org>
> > <mailto:users@airavata.apache.org
> <ma...@airavata.apache.org>>
> > <mailto:users@airavata.apache.org
> <ma...@airavata.apache.org>
> <mailto:users@airavata.apache.org <ma...@airavata.apache.org>>>
> > > <users@airavata.apache.org
> <ma...@airavata.apache.org>
> > <mailto:users@airavata.apache.org
> <ma...@airavata.apache.org>>
> > <mailto:users@airavata.apache.org
> <ma...@airavata.apache.org>
> <mailto:users@airavata.apache.org
> <ma...@airavata.apache.org>>>>
> > > *Subject: *Re: Failed but metascheduler did not
> resubmit job
> > >
> > > Hi Lahiru,
> > >
> > > Any update on this issue? This is an impediment to
> getting this
> > > rolled out to UltraScan users. My understanding is that if
> > the job
> > > fails while verifying and making checks that the
> metascheduler
> > > should try another resource.
> > >
> > > Is there anything I can do to help?
> > >
> > > Regards,
> > >
> > > Aaron
> > >
> > > *From: *Lahiru Jayathilake
> <lahirujayathilake@apache.org <ma...@apache.org>
> > <mailto:lahirujayathilake@apache.org
> <ma...@apache.org>>
> > > <mailto:lahirujayathilake@apache.org
> <ma...@apache.org>
> > <mailto:lahirujayathilake@apache.org
> <ma...@apache.org>>>>
> > > *Date: *Thursday, September 12, 2024 at 1:42 PM
> > > *To: *users@airavata.apache.org
> <ma...@airavata.apache.org>
> > <mailto:users@airavata.apache.org
> <ma...@airavata.apache.org>>
> > <mailto:users@airavata.apache.org
> <ma...@airavata.apache.org>
> <mailto:users@airavata.apache.org <ma...@airavata.apache.org>>>
> > > <users@airavata.apache.org
> <ma...@airavata.apache.org>
> > <mailto:users@airavata.apache.org
> <ma...@airavata.apache.org>>
> > <mailto:users@airavata.apache.org
> <ma...@airavata.apache.org>
> <mailto:users@airavata.apache.org
> <ma...@airavata.apache.org>>>>
> > > *Subject: *Re: Failed but metascheduler did not
> resubmit job
> > >
> > > Hi Aaron,
> > >
> > > Thanks for contacting us. We will look into this issue
> and get
> > > back to you.
> > >
> > > Best,
> > > Lahiru
> > >
> > > On 2024/09/11 18:45:33 Aaron Householder wrote:
> > > > Hi Airavata,
> > > >
> > > > I’m working on connecting Ultrascan3 to Airavata. As the
> > message
> > > below shows, if the job fails the metascheduler might
> not retry
> > > the job. Is there a resource available to take a look at
> > this issue?
> > > >
> > > > Regards,
> > > > Aaron
> > > >
> > > > From: Aaron Householder <aaron@aucsolutions.com
> <ma...@aucsolutions.com>
> > <mailto:aaron@aucsolutions.com <ma...@aucsolutions.com>>
> > > <mailto:aaron@aucsolutions.com
> <ma...@aucsolutions.com> <mailto:aaron@aucsolutions.com
> <ma...@aucsolutions.com>>>>
> > > > Date: Tuesday, September 3, 2024 at 4:42 PM
> > > > To: Airavata Users <users@airavata.apache.org
> <ma...@airavata.apache.org>
> > <mailto:users@airavata.apache.org
> <ma...@airavata.apache.org>>
> > > <mailto:users@airavata.apache.org
> <ma...@airavata.apache.org>
> > <mailto:users@airavata.apache.org
> <ma...@airavata.apache.org>>>>
> > > > Subject: Failed but metascheduler did not resubmit job
> > > > Hi Airavata Users,
> > > >
> > > > I had an UltraScan job that seemed to fail without the
> > > metascheduler resubmitting the job for completion by
> another
> > > cluster. I received the following in an email:
> > > >
> > > > Your UltraScan job is complete:
> > > >
> > > > Submission Time : 2024-08-26 00:50:05
> > > > Job End Time :
> > > > Mail Time : 2024-08-25 19:54:41
> > > > LIMS Host :
> > > > Analysis ID :
> > US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c
> > > > Request ID : 182 ( uslims3_Demo )
> > > > RunID : demo1_veloc1
> > > > EditID : 21030600161
> > > > Data Type : RA
> > > > Cell/Channel/Wl : 2 / A / 259
> > > > Status : failed
> > > > Cluster : metascheduler
> > > > Job Type : 2DSA-MC
> > > > GFAC Status : FAILED
> > > > GFAC Message :
> > > org.apache.airavata.helix.impl.task.TaskOnFailException:
> Error
> > > Code : 23857cb5-5431-43e7-a927-fedd7a929e34, Task
> > > TASK_5b1ea99b-750c-49f6-a05b-df7175f141ed failed due
> to Couldn't
> > > find job id in both submitted and verified steps.
> > > expId:US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c
> Couldn't find
> > > remote jobId for JobName:A1394806797, both submit and
> verify
> > steps
> > > doesn't return a valid JobId. Hence changing
> experiment state to
> > > Failed
> > > > at
> > >
> >
> org.apache.airavata.helix.impl.task.AiravataTask.onFail(AiravataTask.java:146)
> > > > at
> > >
> >
> org.apache.airavata.helix.impl.task.submission.DefaultJobSubmissionTask.onRun(DefaultJobSubmissionTask.java:192)
> > > > at
> > >
> >
> org.apache.airavata.helix.impl.task.AiravataTask.onRun(AiravataTask.java:437)
> > > > at
> > >
> >
> org.apache.airavata.helix.core.AbstractTask.run(AbstractTask.java:102)
> > > > at
> > org.apache.helix.task.TaskRunner.run(TaskRunner.java:71)
> > > > at
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > > at
> > >
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > > at
> > >
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> > > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > > at
> java.base/java.lang.Thread.run(Thread.java:829)
> > > >
> > > >
> > > > No reservation for this job
> > > > --> Verifying valid submit host (login2)...OK
> > > > --> Verifying valid jobname...OK
> > > > --> Verifying valid ssh keys...OK
> > > > --> Verifying access to desired queue (normal)...OK
> > > > --> Checking available allocation FAILED
> > > > Airavata stderr : ERROR: You have no project in the
> > > projectuser.map file (in accounting_check_prod.pl
> <http://accounting_check_prod.pl/>
> > <http://accounting_check_prod.pl/
> <http://accounting_check_prod.pl/>>
> > > <http://accounting_check_prod.pl/
> <http://accounting_check_prod.pl/>
> > <http://accounting_check_prod.pl/
> <http://accounting_check_prod.pl/>>>).
> > > >
> > >
> > >
> > > CAUTION: This message originated outside of UT Health San
> Antonio.
> > > Please exercise caution when clicking on links or opening
> > attachments.
> >
> >
> >
> > --
> > Thank You,
> > Best Regards,
> > Eroma
>
>
>
> --
> Thank You,
> Best Regards,
> Eroma
Re: Airavata jobs failing
Posted by Aaron Householder <aa...@aucsolutions.com>.
Hi Lahiru,
I’ve managed to run all the jobs that were stalled.
Thank you.
Aaron
From: Lahiru Jayathilake <la...@gmail.com>
Date: Sunday, December 1, 2024 at 7:15 PM
To: Emre Brookes <em...@mso.umt.edu>
Cc: emre brookes <br...@uthscsa.edu>, users@airavata.apache.org <us...@airavata.apache.org>, Eroma Abeysinghe <er...@gmail.com>, aaron@aucsolutions.com <aa...@aucsolutions.com>
Subject: Re: Airavata jobs failing
Hi Emre,
This issue has been resolved. Could you please verify if everything is in order? Additionally, there’s no limitation on the number of jobs that can be submitted.
Best,
Lahiru
On Sun, Dec 1, 2024 at 1:09 PM Emre Brookes <em...@mso.umt.edu>> wrote:
Hi Lahiru,
Issues again, we submitted many jobs and it seems the Airavata endpoint
is failing, worse than normal...
Error trying to initialize service account user profile: Required field
'accessToken' is unset! Struct:AuthzToken(accessToken:null,
claimsMap:{userName=service-account-lims,
gatewayID=ultrascan})string(1074) "#0
/srv/www/htdocs/common/class/ultrascan-airavata-bridge/lib/Airavata/API/Airavata.php(5947):
Airavata\API\AiravataClient->recv_searchProjects() #1
/srv/www/htdocs/common/class/ultrascan-airavata-bridge/AiravataUtils.php(64):
Airavata\API\AiravataClient->searchProjects(Object(Airavata\Model\Security\AuthzToken),
'ultrascan', 'service-account...', Array, -1, 0) #2
/srv/www/htdocs/common/class/ultrascan-airavata-bridge/AiravataWrapper.php(135):
fetch_projectid(Object(Airavata\API\AiravataClient),
Object(Airavata\Model\Security\AuthzToken), 'ultrascan',
'Emre_Brookes_c3...') #3
/srv/www/htdocs/common/class/submit_airavata.php(213):
SCIGAP\AiravataWrapper->launch_airavata_experiment('uslims.uleth.ca<http://uslims.uleth.ca>',
'Emre_Brookes_c3...', 'US3-AIRA', 'uslims3_Worksho...',
'expanse.sdsc.ed...', 'shared', 32, 1, 1, 114, 'us3', 'NONE', 'NONE',
'/srv/www/htdocs...', 'ab1fd003-7c82-0...', 64000) #4
/srv/www/htdocs/common/class/submit_airavata.php(20):
submit_airavata->createExperiment() #5
/srv/www/htdocs/uslims3/uslims3_Workshop/2DSA_2.php(222):
submit_airavata->submit() #6 {main}"
We will try to limit the number of simultaneous active jobs for further
testing.
Thanks,
Emre
emre brookes wrote:
> Hi Lahiru,
>
> Thanks for the quick work!
>
> I see many seem to have finished now.
> Will check again in the morning to see what happens with the last few.
>
> Is there a practical limit to the number of jobs that can be submitted
> to the metascheduler?
>
> Best regards,
> Emre
>
>
> Lahiru Jayathilake wrote:
>> *EXTERNAL EMAIL*
>>
>> Hi Emre,
>>
>> I hope you're doing well!
>> I’ve checked the issue, and the experiments are now being submitted.
>>
>> Let me know if you notice any further issues.
>>
>> Best,
>> Lahiru
>>
>> On Sat, Nov 30, 2024 at 6:11 PM Emre Brookes
>> <em...@mso.umt.edu> <ma...@mso.umt.edu>>> wrote:
>>
>> Hi,
>>
>> Hope you had a good Thanksgiving & your grant submission is close
>> to done.
>> We submitted 100 jobs via the metascheduler and it seems to have
>> choked
>> further submission.
>> No jobs, even from other servers seem to be going through.
>>
>> e.g. US3-AIRA_1f6e88ee-736c-490e-9d83-fba8adc7bbd0 from
>> uslims.uleth.ca<http://uslims.uleth.ca>
>> <http://uslims.uleth.ca/>
>> seems stuck
>>
>> & these (of many) from uslimstest.genapp.rocks also seem stuck:
>>
>> --------------------------------------------------------------------------------
>> status for US3-AIRA_b41e37e8-cb8c-4834-969a-41db6aee9a8e:
>> "SCHEDULED"
>> compute resource for US3-AIRA_b41e37e8-cb8c-4834-969a-41db6aee9a8e:
>> "Resource not yet assigned"
>> --------------------------------------------------------------------------------
>> --------------------------------------------------------------------------------
>> status for US3-AIRA_d8359418-98dd-460a-b739-57a6ad7d9e9c:
>> "EXECUTING"
>> compute resource for US3-AIRA_d8359418-98dd-460a-b739-57a6ad7d9e9c:
>> "expanse.sdsc.edu<http://expanse.sdsc.edu>
>> <http://expanse.sdsc.edu/>"
>> --------------------------------------------------------------------------------
>> --------------------------------------------------------------------------------
>>
>> Thanks,
>> Emre
>>
>>
>> CAUTION: This message originated outside of UT Health San Antonio.
>> Please exercise caution when clicking on links or opening attachments.
>
> .
>
Re: Airavata jobs failing
Posted by Lahiru Jayathilake <la...@gmail.com>.
Hi Emre,
This issue has been resolved. Could you please verify if everything is
in order? Additionally, there’s no limitation on the number of jobs that
can be submitted.
Best,
Lahiru
On Sun, Dec 1, 2024 at 1:09 PM Emre Brookes <em...@mso.umt.edu>
wrote:
> Hi Lahiru,
>
> Issues again, we submitted many jobs and it seems the Airavata endpoint
> is failing, worse than normal...
>
> Error trying to initialize service account user profile: Required field
> 'accessToken' is unset! Struct:AuthzToken(accessToken:null,
> claimsMap:{userName=service-account-lims,
> gatewayID=ultrascan})string(1074) "#0
> /srv/www/htdocs/common/class/ultrascan-airavata-bridge/lib/Airavata/API/Airavata.php(5947):
>
> Airavata\API\AiravataClient->recv_searchProjects() #1
> /srv/www/htdocs/common/class/ultrascan-airavata-bridge/AiravataUtils.php(64):
>
> Airavata\API\AiravataClient->searchProjects(Object(Airavata\Model\Security\AuthzToken),
>
> 'ultrascan', 'service-account...', Array, -1, 0) #2
> /srv/www/htdocs/common/class/ultrascan-airavata-bridge/AiravataWrapper.php(135):
>
> fetch_projectid(Object(Airavata\API\AiravataClient),
> Object(Airavata\Model\Security\AuthzToken), 'ultrascan',
> 'Emre_Brookes_c3...') #3
> /srv/www/htdocs/common/class/submit_airavata.php(213):
> SCIGAP\AiravataWrapper->launch_airavata_experiment('uslims.uleth.ca',
> 'Emre_Brookes_c3...', 'US3-AIRA', 'uslims3_Worksho...',
> 'expanse.sdsc.ed...', 'shared', 32, 1, 1, 114, 'us3', 'NONE', 'NONE',
> '/srv/www/htdocs...', 'ab1fd003-7c82-0...', 64000) #4
> /srv/www/htdocs/common/class/submit_airavata.php(20):
> submit_airavata->createExperiment() #5
> /srv/www/htdocs/uslims3/uslims3_Workshop/2DSA_2.php(222):
> submit_airavata->submit() #6 {main}"
>
> We will try to limit the number of simultaneous active jobs for further
> testing.
>
> Thanks,
> Emre
>
> emre brookes wrote:
> > Hi Lahiru,
> >
> > Thanks for the quick work!
> >
> > I see many seem to have finished now.
> > Will check again in the morning to see what happens with the last few.
> >
> > Is there a practical limit to the number of jobs that can be submitted
> > to the metascheduler?
> >
> > Best regards,
> > Emre
> >
> >
> > Lahiru Jayathilake wrote:
> >> *EXTERNAL EMAIL*
> >>
> >> Hi Emre,
> >>
> >> I hope you're doing well!
> >> I’ve checked the issue, and the experiments are now being submitted.
> >>
> >> Let me know if you notice any further issues.
> >>
> >> Best,
> >> Lahiru
> >>
> >> On Sat, Nov 30, 2024 at 6:11 PM Emre Brookes
> >> <emre.brookes@mso.umt.edu <ma...@mso.umt.edu>> wrote:
> >>
> >> Hi,
> >>
> >> Hope you had a good Thanksgiving & your grant submission is close
> >> to done.
> >> We submitted 100 jobs via the metascheduler and it seems to have
> >> choked
> >> further submission.
> >> No jobs, even from other servers seem to be going through.
> >>
> >> e.g. US3-AIRA_1f6e88ee-736c-490e-9d83-fba8adc7bbd0 from
> >> uslims.uleth.ca
> >> <http://uslims.uleth.ca/>
> >> seems stuck
> >>
> >> & these (of many) from uslimstest.genapp.rocks also seem stuck:
> >>
> >>
> --------------------------------------------------------------------------------
> >> status for US3-AIRA_b41e37e8-cb8c-4834-969a-41db6aee9a8e:
> >> "SCHEDULED"
> >> compute resource for US3-AIRA_b41e37e8-cb8c-4834-969a-41db6aee9a8e:
> >> "Resource not yet assigned"
> >>
> --------------------------------------------------------------------------------
> >>
> --------------------------------------------------------------------------------
> >> status for US3-AIRA_d8359418-98dd-460a-b739-57a6ad7d9e9c:
> >> "EXECUTING"
> >> compute resource for US3-AIRA_d8359418-98dd-460a-b739-57a6ad7d9e9c:
> >> "expanse.sdsc.edu
> >> <http://expanse.sdsc.edu/>"
> >>
> --------------------------------------------------------------------------------
> >>
> --------------------------------------------------------------------------------
> >>
> >> Thanks,
> >> Emre
> >>
> >>
> >> CAUTION: This message originated outside of UT Health San Antonio.
> >> Please exercise caution when clicking on links or opening attachments.
> >
> > .
> >
>
>
Re: Airavata jobs failing
Posted by Emre Brookes <em...@mso.umt.edu>.
Hi Lahiru,
Issues again, we submitted many jobs and it seems the Airavata endpoint
is failing, worse than normal...
Error trying to initialize service account user profile: Required field
'accessToken' is unset! Struct:AuthzToken(accessToken:null,
claimsMap:{userName=service-account-lims,
gatewayID=ultrascan})string(1074) "#0
/srv/www/htdocs/common/class/ultrascan-airavata-bridge/lib/Airavata/API/Airavata.php(5947):
Airavata\API\AiravataClient->recv_searchProjects() #1
/srv/www/htdocs/common/class/ultrascan-airavata-bridge/AiravataUtils.php(64):
Airavata\API\AiravataClient->searchProjects(Object(Airavata\Model\Security\AuthzToken),
'ultrascan', 'service-account...', Array, -1, 0) #2
/srv/www/htdocs/common/class/ultrascan-airavata-bridge/AiravataWrapper.php(135):
fetch_projectid(Object(Airavata\API\AiravataClient),
Object(Airavata\Model\Security\AuthzToken), 'ultrascan',
'Emre_Brookes_c3...') #3
/srv/www/htdocs/common/class/submit_airavata.php(213):
SCIGAP\AiravataWrapper->launch_airavata_experiment('uslims.uleth.ca',
'Emre_Brookes_c3...', 'US3-AIRA', 'uslims3_Worksho...',
'expanse.sdsc.ed...', 'shared', 32, 1, 1, 114, 'us3', 'NONE', 'NONE',
'/srv/www/htdocs...', 'ab1fd003-7c82-0...', 64000) #4
/srv/www/htdocs/common/class/submit_airavata.php(20):
submit_airavata->createExperiment() #5
/srv/www/htdocs/uslims3/uslims3_Workshop/2DSA_2.php(222):
submit_airavata->submit() #6 {main}"
We will try to limit the number of simultaneous active jobs for further
testing.
Thanks,
Emre
emre brookes wrote:
> Hi Lahiru,
>
> Thanks for the quick work!
>
> I see many seem to have finished now.
> Will check again in the morning to see what happens with the last few.
>
> Is there a practical limit to the number of jobs that can be submitted
> to the metascheduler?
>
> Best regards,
> Emre
>
>
> Lahiru Jayathilake wrote:
>> *EXTERNAL EMAIL*
>>
>> Hi Emre,
>>
>> I hope you're doing well!
>> I’ve checked the issue, and the experiments are now being submitted.
>>
>> Let me know if you notice any further issues.
>>
>> Best,
>> Lahiru
>>
>> On Sat, Nov 30, 2024 at 6:11 PM Emre Brookes
>> <emre.brookes@mso.umt.edu <ma...@mso.umt.edu>> wrote:
>>
>> Hi,
>>
>> Hope you had a good Thanksgiving & your grant submission is close
>> to done.
>> We submitted 100 jobs via the metascheduler and it seems to have
>> choked
>> further submission.
>> No jobs, even from other servers seem to be going through.
>>
>> e.g. US3-AIRA_1f6e88ee-736c-490e-9d83-fba8adc7bbd0 from
>> uslims.uleth.ca
>> <http://uslims.uleth.ca/>
>> seems stuck
>>
>> & these (of many) from uslimstest.genapp.rocks also seem stuck:
>>
>> --------------------------------------------------------------------------------
>> status for US3-AIRA_b41e37e8-cb8c-4834-969a-41db6aee9a8e:
>> "SCHEDULED"
>> compute resource for US3-AIRA_b41e37e8-cb8c-4834-969a-41db6aee9a8e:
>> "Resource not yet assigned"
>> --------------------------------------------------------------------------------
>> --------------------------------------------------------------------------------
>> status for US3-AIRA_d8359418-98dd-460a-b739-57a6ad7d9e9c:
>> "EXECUTING"
>> compute resource for US3-AIRA_d8359418-98dd-460a-b739-57a6ad7d9e9c:
>> "expanse.sdsc.edu
>> <http://expanse.sdsc.edu/>"
>> --------------------------------------------------------------------------------
>> --------------------------------------------------------------------------------
>>
>> Thanks,
>> Emre
>>
>>
>> CAUTION: This message originated outside of UT Health San Antonio.
>> Please exercise caution when clicking on links or opening attachments.
>
> .
>
Re: Airavata jobs failing
Posted by Lahiru Jayathilake <la...@gmail.com>.
Hi Emre,
I hope you're doing well!
I’ve checked the issue, and the experiments are now being submitted.
Let me know if you notice any further issues.
Best,
Lahiru
On Sat, Nov 30, 2024 at 6:11 PM Emre Brookes <em...@mso.umt.edu>
wrote:
> Hi,
>
> Hope you had a good Thanksgiving & your grant submission is close to done.
> We submitted 100 jobs via the metascheduler and it seems to have choked
> further submission.
> No jobs, even from other servers seem to be going through.
>
> e.g. US3-AIRA_1f6e88ee-736c-490e-9d83-fba8adc7bbd0 from uslims.uleth.ca
> seems stuck
>
> & these (of many) from uslimstest.genapp.rocks also seem stuck:
>
>
> --------------------------------------------------------------------------------
> status for US3-AIRA_b41e37e8-cb8c-4834-969a-41db6aee9a8e:
> "SCHEDULED"
> compute resource for US3-AIRA_b41e37e8-cb8c-4834-969a-41db6aee9a8e:
> "Resource not yet assigned"
>
> --------------------------------------------------------------------------------
>
> --------------------------------------------------------------------------------
> status for US3-AIRA_d8359418-98dd-460a-b739-57a6ad7d9e9c:
> "EXECUTING"
> compute resource for US3-AIRA_d8359418-98dd-460a-b739-57a6ad7d9e9c:
> "expanse.sdsc.edu"
>
> --------------------------------------------------------------------------------
>
> --------------------------------------------------------------------------------
>
> Thanks,
> Emre
>
>
Airavata jobs failing
Posted by Emre Brookes <em...@mso.umt.edu>.
Hi,
Hope you had a good Thanksgiving & your grant submission is close to done.
We submitted 100 jobs via the metascheduler and it seems to have choked
further submission.
No jobs, even from other servers seem to be going through.
e.g. US3-AIRA_1f6e88ee-736c-490e-9d83-fba8adc7bbd0 from uslims.uleth.ca
seems stuck
& these (of many) from uslimstest.genapp.rocks also seem stuck:
--------------------------------------------------------------------------------
status for US3-AIRA_b41e37e8-cb8c-4834-969a-41db6aee9a8e:
"SCHEDULED"
compute resource for US3-AIRA_b41e37e8-cb8c-4834-969a-41db6aee9a8e:
"Resource not yet assigned"
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
status for US3-AIRA_d8359418-98dd-460a-b739-57a6ad7d9e9c:
"EXECUTING"
compute resource for US3-AIRA_d8359418-98dd-460a-b739-57a6ad7d9e9c:
"expanse.sdsc.edu"
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Thanks,
Emre
Re: Failed but metascheduler did not resubmit job
Posted by Eroma Abeysinghe <er...@gmail.com>.
Hi Emre,
Currently, what metascheduler does is find a resource from the pool
confirm its available, can ssh, verify the queue, check the queue limit and
submit the job (Lahiru also explained previously in this thread). After
this point, if the job fails due to allocation not being available etc...
we don't send it back to a new resource. This is yet to be implemented.
Thanks,
Eroma
On Fri, Oct 25, 2024 at 12:32 PM Emre Brookes <em...@mso.umt.edu>
wrote:
> Hi Eroma,
>
> We understand we do not have a current allocation on LS6.
> We expect the metascheduler to reschedule to another cluster for this
> error, will this case not be supported?
> Our goal for the metascheduler was for any issues, regardless of cause,
> to be as transparent as possible for the user.
> Allocations could run out, expire or perhaps problems with the
> allocation mechanism on the target resource - all
> these (and others, we have seen communications issues particularly with
> expanse) - should be recoverable by
> the metascheduler. This is the point of it. The simple case of a job
> running and returning a failed status is likely
> the *least* recoverable, since if the job ran correctly, there is likely
> a data issue that will cause it to fail elsewhere
> (exceptions could be different or memory limits, or the rare case a node
> failed during the computation).
>
> Thanks,
> Emre
>
>
> Eroma Abeysinghe wrote:
> > Hi Emre, et al,
> >
> > I looked at the failed experiment Aaron shared. It was executed in ls6.
> > I looked at ls6 experiments for the past year and 9 months ago the
> > cluster started asking for a project allocation. Before, Ultrascan had
> > submitted successfully without a project allocation.
> > So right now, to submit to ls6, you need to contact them, provide your
> > details and ask for the project allocation number.
> >
> > The messages we see from ls6 [1]
> > Aaron, do you have issues submitting to other clusters, or the
> > research to be done in ls6?
> >
> > Thanks,
> > Eroma
> >
> > ----
> > [1]
> > A1394806797 STDOUT
> > -----------------------------------------------------------------
> > Welcome to the Lonestar6 Supercomputer
> > -----------------------------------------------------------------
> >
> > No reservation for this job
> > --> Verifying valid submit host (login2)...OK
> > --> Verifying valid jobname...OK
> > --> Verifying valid ssh keys...OK
> > --> Verifying access to desired queue (normal)...OK
> > --> Checking available allocation FAILED
> > A1394806797 STDERR
> > ERROR: You have no project in the projectuser.map file (
> inaccounting_check_prod.pl <http://accounting_check_prod.pl/>).
> >
> > Please report this problem:
> > U. of TX users contact (https://portal.tacc.utexas.edu/consulting <
> https://portal.tacc.utexas.edu/consulting>)
> > ------
> >
> > On Fri, Oct 25, 2024 at 11:05 AM Lahiru Jayathilake
> > <lahirujayathilake@gmail.com <ma...@gmail.com>>
> wrote:
> >
> >
> >
> > ---------- Forwarded message ---------
> > From: *Emre Brookes* <emre.brookes@mso.umt.edu
> > <ma...@mso.umt.edu>>
> > Date: Tue, Oct 22, 2024 at 4:10 PM
> > Subject: Re: Failed but metascheduler did not resubmit job
> > To: <users@airavata.apache.org
> > <ma...@airavata.apache.org>>, Lahiru Jayathilake
> > <lahirujayathilake@gmail.com <ma...@gmail.com>>
> > Cc: <aaron@aucsolutions.com <ma...@aucsolutions.com>>
> >
> >
> > Hi Lahiru,
> >
> > I really appreciate your work on this.
> > It's important that we have a working system that we can include
> > in our
> > progress report to the NIH & I'd like to see Aaron able to finish his
> > paper this year.
> > If there's anything we can do from our end to expedite this,
> > please let
> > me know.
> >
> > Thanks,
> > Emre
> >
> > Lahiru Jayathilake wrote:
> > > *EXTERNAL EMAIL*
> > >
> > > Hi Aaron,
> > >
> > > Thank you for your patience, and I apologize for the delay in
> > getting
> > > back to you regarding the issue.
> > >
> > > After further investigation, I noticed that the current version
> > of the
> > > Airavata Metascheduler does not support automatically resubmitting
> > > jobs to different clusters when a job fails after successful
> > > submission (e.g., due to resource allocation issues). I've now
> > created
> > > a task [1] to add this feature, which will enable the expected
> > > functionality. This enhancement will take some time to
> > implement, but
> > > we’ll keep you updated on the progress.
> > >
> > > Please feel free to reach out if you have any further questions or
> > > need additional information.
> > >
> > > [1] - https://issues.apache.org/jira/browse/AIRAVATA-3893
> > <https://issues.apache.org/jira/browse/AIRAVATA-3893>
> >
> > > <https://issues.apache.org/jira/browse/AIRAVATA-3893
> > <https://issues.apache.org/jira/browse/AIRAVATA-3893>>
> > >
> > > Thanks,
> > > Lahiru
> > >
> > > On Wed, Oct 16, 2024 at 7:36 PM Aaron Householder
> > > <aaron@aucsolutions.com <ma...@aucsolutions.com>
> > <mailto:aaron@aucsolutions.com <ma...@aucsolutions.com>>>
> > wrote:
> > >
> > > Hi,
> > >
> > > Is there any update?
> > >
> > > Regards,
> > >
> > > Aaron
> > >
> > > *From: *Aaron Householder <aaron@aucsolutions.com
> > <ma...@aucsolutions.com>
> > > <mailto:aaron@aucsolutions.com <mailto:aaron@aucsolutions.com
> >>>
> > > *Date: *Saturday, October 12, 2024 at 8:12 PM
> > > *To: *users@airavata.apache.org
> > <ma...@airavata.apache.org>
> > <mailto:users@airavata.apache.org <mailto:users@airavata.apache.org
> >>
> > > <users@airavata.apache.org
> > <ma...@airavata.apache.org>
> > <mailto:users@airavata.apache.org <mailto:users@airavata.apache.org
> >>>
> > > *Subject: *Re: Failed but metascheduler did not resubmit job
> > >
> > > Hi Lahiru,
> > >
> > > Any update on this issue? This is an impediment to getting this
> > > rolled out to UltraScan users. My understanding is that if
> > the job
> > > fails while verifying and making checks that the metascheduler
> > > should try another resource.
> > >
> > > Is there anything I can do to help?
> > >
> > > Regards,
> > >
> > > Aaron
> > >
> > > *From: *Lahiru Jayathilake <lahirujayathilake@apache.org
> > <ma...@apache.org>
> > > <mailto:lahirujayathilake@apache.org
> > <ma...@apache.org>>>
> > > *Date: *Thursday, September 12, 2024 at 1:42 PM
> > > *To: *users@airavata.apache.org
> > <ma...@airavata.apache.org>
> > <mailto:users@airavata.apache.org <mailto:users@airavata.apache.org
> >>
> > > <users@airavata.apache.org
> > <ma...@airavata.apache.org>
> > <mailto:users@airavata.apache.org <mailto:users@airavata.apache.org
> >>>
> > > *Subject: *Re: Failed but metascheduler did not resubmit job
> > >
> > > Hi Aaron,
> > >
> > > Thanks for contacting us. We will look into this issue and get
> > > back to you.
> > >
> > > Best,
> > > Lahiru
> > >
> > > On 2024/09/11 18:45:33 Aaron Householder wrote:
> > > > Hi Airavata,
> > > >
> > > > I’m working on connecting Ultrascan3 to Airavata. As the
> > message
> > > below shows, if the job fails the metascheduler might not retry
> > > the job. Is there a resource available to take a look at
> > this issue?
> > > >
> > > > Regards,
> > > > Aaron
> > > >
> > > > From: Aaron Householder <aaron@aucsolutions.com
> > <ma...@aucsolutions.com>
> > > <mailto:aaron@aucsolutions.com <mailto:aaron@aucsolutions.com
> >>>
> > > > Date: Tuesday, September 3, 2024 at 4:42 PM
> > > > To: Airavata Users <users@airavata.apache.org
> > <ma...@airavata.apache.org>
> > > <mailto:users@airavata.apache.org
> > <ma...@airavata.apache.org>>>
> > > > Subject: Failed but metascheduler did not resubmit job
> > > > Hi Airavata Users,
> > > >
> > > > I had an UltraScan job that seemed to fail without the
> > > metascheduler resubmitting the job for completion by another
> > > cluster. I received the following in an email:
> > > >
> > > > Your UltraScan job is complete:
> > > >
> > > > Submission Time : 2024-08-26 00:50:05
> > > > Job End Time :
> > > > Mail Time : 2024-08-25 19:54:41
> > > > LIMS Host :
> > > > Analysis ID :
> > US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c
> > > > Request ID : 182 ( uslims3_Demo )
> > > > RunID : demo1_veloc1
> > > > EditID : 21030600161
> > > > Data Type : RA
> > > > Cell/Channel/Wl : 2 / A / 259
> > > > Status : failed
> > > > Cluster : metascheduler
> > > > Job Type : 2DSA-MC
> > > > GFAC Status : FAILED
> > > > GFAC Message :
> > > org.apache.airavata.helix.impl.task.TaskOnFailException: Error
> > > Code : 23857cb5-5431-43e7-a927-fedd7a929e34, Task
> > > TASK_5b1ea99b-750c-49f6-a05b-df7175f141ed failed due to
> Couldn't
> > > find job id in both submitted and verified steps.
> > > expId:US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c Couldn't find
> > > remote jobId for JobName:A1394806797, both submit and verify
> > steps
> > > doesn't return a valid JobId. Hence changing experiment state
> to
> > > Failed
> > > > at
> > >
> >
> org.apache.airavata.helix.impl.task.AiravataTask.onFail(AiravataTask.java:146)
> > > > at
> > >
> >
> org.apache.airavata.helix.impl.task.submission.DefaultJobSubmissionTask.onRun(DefaultJobSubmissionTask.java:192)
> > > > at
> > >
> >
> org.apache.airavata.helix.impl.task.AiravataTask.onRun(AiravataTask.java:437)
> > > > at
> > >
> >
> org.apache.airavata.helix.core.AbstractTask.run(AbstractTask.java:102)
> > > > at
> > org.apache.helix.task.TaskRunner.run(TaskRunner.java:71)
> > > > at
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > > at
> > > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > > at
> > >
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> > > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > >
> > > >
> > > > No reservation for this job
> > > > --> Verifying valid submit host (login2)...OK
> > > > --> Verifying valid jobname...OK
> > > > --> Verifying valid ssh keys...OK
> > > > --> Verifying access to desired queue (normal)...OK
> > > > --> Checking available allocation FAILED
> > > > Airavata stderr : ERROR: You have no project in the
> > > projectuser.map file (in accounting_check_prod.pl
> > <http://accounting_check_prod.pl/>
> > > <http://accounting_check_prod.pl/
> > <http://accounting_check_prod.pl/>>).
> > > >
> > >
> > >
> > > CAUTION: This message originated outside of UT Health San Antonio.
> > > Please exercise caution when clicking on links or opening
> > attachments.
> >
> >
> >
> > --
> > Thank You,
> > Best Regards,
> > Eroma
>
>
--
Thank You,
Best Regards,
Eroma
Re: Failed but metascheduler did not resubmit job
Posted by Eroma Abeysinghe <er...@gmail.com>.
Hi Emre, et al,
I looked at the failed experiment Aaron shared. It was executed in ls6.
I looked at ls6 experiments for the past year and 9 months ago the cluster
started asking for a project allocation. Before, Ultrascan had submitted
successfully without a project allocation.
So right now, to submit to ls6, you need to contact them, provide your
details and ask for the project allocation number.
The messages we see from ls6 [1]
Aaron, do you have issues submitting to other clusters, or the research to
be done in ls6?
Thanks,
Eroma
----
[1]
A1394806797 STDOUT
-----------------------------------------------------------------
Welcome to the Lonestar6 Supercomputer
-----------------------------------------------------------------
No reservation for this job
--> Verifying valid submit host (login2)...OK
--> Verifying valid jobname...OK
--> Verifying valid ssh keys...OK
--> Verifying access to desired queue (normal)...OK
--> Checking available allocation FAILED
A1394806797 STDERR
ERROR: You have no project in the projectuser.map file (in
accounting_check_prod.pl).
Please report this problem:
U. of TX users contact (https://portal.tacc.utexas.edu/consulting)
------
On Fri, Oct 25, 2024 at 11:05 AM Lahiru Jayathilake <
lahirujayathilake@gmail.com> wrote:
>
>
> ---------- Forwarded message ---------
> From: Emre Brookes <em...@mso.umt.edu>
> Date: Tue, Oct 22, 2024 at 4:10 PM
> Subject: Re: Failed but metascheduler did not resubmit job
> To: <us...@airavata.apache.org>, Lahiru Jayathilake <
> lahirujayathilake@gmail.com>
> Cc: <aa...@aucsolutions.com>
>
>
> Hi Lahiru,
>
> I really appreciate your work on this.
> It's important that we have a working system that we can include in our
> progress report to the NIH & I'd like to see Aaron able to finish his
> paper this year.
> If there's anything we can do from our end to expedite this, please let
> me know.
>
> Thanks,
> Emre
>
> Lahiru Jayathilake wrote:
> > *EXTERNAL EMAIL*
> >
> > Hi Aaron,
> >
> > Thank you for your patience, and I apologize for the delay in getting
> > back to you regarding the issue.
> >
> > After further investigation, I noticed that the current version of the
> > Airavata Metascheduler does not support automatically resubmitting
> > jobs to different clusters when a job fails after successful
> > submission (e.g., due to resource allocation issues). I've now created
> > a task [1] to add this feature, which will enable the expected
> > functionality. This enhancement will take some time to implement, but
> > we’ll keep you updated on the progress.
> >
> > Please feel free to reach out if you have any further questions or
> > need additional information.
> >
> > [1] - https://issues.apache.org/jira/browse/AIRAVATA-3893
> > <https://issues.apache.org/jira/browse/AIRAVATA-3893>
> >
> > Thanks,
> > Lahiru
> >
> > On Wed, Oct 16, 2024 at 7:36 PM Aaron Householder
> > <aaron@aucsolutions.com <ma...@aucsolutions.com>> wrote:
> >
> > Hi,
> >
> > Is there any update?
> >
> > Regards,
> >
> > Aaron
> >
> > *From: *Aaron Householder <aaron@aucsolutions.com
> > <ma...@aucsolutions.com>>
> > *Date: *Saturday, October 12, 2024 at 8:12 PM
> > *To: *users@airavata.apache.org <ma...@airavata.apache.org>
> > <users@airavata.apache.org <ma...@airavata.apache.org>>
> > *Subject: *Re: Failed but metascheduler did not resubmit job
> >
> > Hi Lahiru,
> >
> > Any update on this issue? This is an impediment to getting this
> > rolled out to UltraScan users. My understanding is that if the job
> > fails while verifying and making checks that the metascheduler
> > should try another resource.
> >
> > Is there anything I can do to help?
> >
> > Regards,
> >
> > Aaron
> >
> > *From: *Lahiru Jayathilake <lahirujayathilake@apache.org
> > <ma...@apache.org>>
> > *Date: *Thursday, September 12, 2024 at 1:42 PM
> > *To: *users@airavata.apache.org <ma...@airavata.apache.org>
> > <users@airavata.apache.org <ma...@airavata.apache.org>>
> > *Subject: *Re: Failed but metascheduler did not resubmit job
> >
> > Hi Aaron,
> >
> > Thanks for contacting us. We will look into this issue and get
> > back to you.
> >
> > Best,
> > Lahiru
> >
> > On 2024/09/11 18:45:33 Aaron Householder wrote:
> > > Hi Airavata,
> > >
> > > I’m working on connecting Ultrascan3 to Airavata. As the message
> > below shows, if the job fails the metascheduler might not retry
> > the job. Is there a resource available to take a look at this issue?
> > >
> > > Regards,
> > > Aaron
> > >
> > > From: Aaron Householder <aaron@aucsolutions.com
> > <ma...@aucsolutions.com>>
> > > Date: Tuesday, September 3, 2024 at 4:42 PM
> > > To: Airavata Users <users@airavata.apache.org
> > <ma...@airavata.apache.org>>
> > > Subject: Failed but metascheduler did not resubmit job
> > > Hi Airavata Users,
> > >
> > > I had an UltraScan job that seemed to fail without the
> > metascheduler resubmitting the job for completion by another
> > cluster. I received the following in an email:
> > >
> > > Your UltraScan job is complete:
> > >
> > > Submission Time : 2024-08-26 00:50:05
> > > Job End Time :
> > > Mail Time : 2024-08-25 19:54:41
> > > LIMS Host :
> > > Analysis ID : US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c
> > > Request ID : 182 ( uslims3_Demo )
> > > RunID : demo1_veloc1
> > > EditID : 21030600161
> > > Data Type : RA
> > > Cell/Channel/Wl : 2 / A / 259
> > > Status : failed
> > > Cluster : metascheduler
> > > Job Type : 2DSA-MC
> > > GFAC Status : FAILED
> > > GFAC Message :
> > org.apache.airavata.helix.impl.task.TaskOnFailException: Error
> > Code : 23857cb5-5431-43e7-a927-fedd7a929e34, Task
> > TASK_5b1ea99b-750c-49f6-a05b-df7175f141ed failed due to Couldn't
> > find job id in both submitted and verified steps.
> > expId:US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c Couldn't find
> > remote jobId for JobName:A1394806797, both submit and verify steps
> > doesn't return a valid JobId. Hence changing experiment state to
> > Failed
> > > at
> >
> org.apache.airavata.helix.impl.task.AiravataTask.onFail(AiravataTask.java:146)
> > > at
> >
> org.apache.airavata.helix.impl.task.submission.DefaultJobSubmissionTask.onRun(DefaultJobSubmissionTask.java:192)
> > > at
> >
> org.apache.airavata.helix.impl.task.AiravataTask.onRun(AiravataTask.java:437)
> > > at
> >
> org.apache.airavata.helix.core.AbstractTask.run(AbstractTask.java:102)
> > > at org.apache.helix.task.TaskRunner.run(TaskRunner.java:71)
> > > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > at
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > at
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> > > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > at java.base/java.lang.Thread.run(Thread.java:829)
> > >
> > >
> > > No reservation for this job
> > > --> Verifying valid submit host (login2)...OK
> > > --> Verifying valid jobname...OK
> > > --> Verifying valid ssh keys...OK
> > > --> Verifying access to desired queue (normal)...OK
> > > --> Checking available allocation FAILED
> > > Airavata stderr : ERROR: You have no project in the
> > projectuser.map file (in accounting_check_prod.pl
> > <http://accounting_check_prod.pl/>).
> > >
> >
> >
> > CAUTION: This message originated outside of UT Health San Antonio.
> > Please exercise caution when clicking on links or opening attachments.
>
>
--
Thank You,
Best Regards,
Eroma
Re: Failed but metascheduler did not resubmit job
Posted by Lahiru Jayathilake <la...@gmail.com>.
Hi Aaron,
Thank you for your patience, and I apologize for the delay in getting back
to you regarding the issue.
After further investigation, I noticed that the current version of the
Airavata Metascheduler does not support automatically resubmitting jobs to
different clusters when a job fails after successful submission (e.g., due
to resource allocation issues). I've now created a task [1] to add this
feature, which will enable the expected functionality. This enhancement
will take some time to implement, but we’ll keep you updated on the
progress.
Please feel free to reach out if you have any further questions or need
additional information.
[1] - https://issues.apache.org/jira/browse/AIRAVATA-3893
Thanks,
Lahiru
On Wed, Oct 16, 2024 at 7:36 PM Aaron Householder <aa...@aucsolutions.com>
wrote:
> Hi,
>
>
>
> Is there any update?
>
>
>
> Regards,
>
> Aaron
>
>
>
> *From: *Aaron Householder <aa...@aucsolutions.com>
> *Date: *Saturday, October 12, 2024 at 8:12 PM
> *To: *users@airavata.apache.org <us...@airavata.apache.org>
> *Subject: *Re: Failed but metascheduler did not resubmit job
>
> Hi Lahiru,
>
>
>
> Any update on this issue? This is an impediment to getting this rolled out
> to UltraScan users. My understanding is that if the job fails while
> verifying and making checks that the metascheduler should try another
> resource.
>
>
>
> Is there anything I can do to help?
>
>
>
> Regards,
>
> Aaron
>
>
>
> *From: *Lahiru Jayathilake <la...@apache.org>
> *Date: *Thursday, September 12, 2024 at 1:42 PM
> *To: *users@airavata.apache.org <us...@airavata.apache.org>
> *Subject: *Re: Failed but metascheduler did not resubmit job
>
> Hi Aaron,
>
> Thanks for contacting us. We will look into this issue and get back to you.
>
> Best,
> Lahiru
>
> On 2024/09/11 18:45:33 Aaron Householder wrote:
> > Hi Airavata,
> >
> > I’m working on connecting Ultrascan3 to Airavata. As the message below
> shows, if the job fails the metascheduler might not retry the job. Is there
> a resource available to take a look at this issue?
> >
> > Regards,
> > Aaron
> >
> > From: Aaron Householder <aa...@aucsolutions.com>
> > Date: Tuesday, September 3, 2024 at 4:42 PM
> > To: Airavata Users <us...@airavata.apache.org>
> > Subject: Failed but metascheduler did not resubmit job
> > Hi Airavata Users,
> >
> > I had an UltraScan job that seemed to fail without the metascheduler
> resubmitting the job for completion by another cluster. I received the
> following in an email:
> >
> > Your UltraScan job is complete:
> >
> > Submission Time : 2024-08-26 00:50:05
> > Job End Time :
> > Mail Time : 2024-08-25 19:54:41
> > LIMS Host :
> > Analysis ID : US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c
> > Request ID : 182 ( uslims3_Demo )
> > RunID : demo1_veloc1
> > EditID : 21030600161
> > Data Type : RA
> > Cell/Channel/Wl : 2 / A / 259
> > Status : failed
> > Cluster : metascheduler
> > Job Type : 2DSA-MC
> > GFAC Status : FAILED
> > GFAC Message :
> org.apache.airavata.helix.impl.task.TaskOnFailException: Error Code :
> 23857cb5-5431-43e7-a927-fedd7a929e34, Task
> TASK_5b1ea99b-750c-49f6-a05b-df7175f141ed failed due to Couldn't find job
> id in both submitted and verified steps.
> expId:US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c Couldn't find remote
> jobId for JobName:A1394806797, both submit and verify steps doesn't return
> a valid JobId. Hence changing experiment state to Failed
> > at
> org.apache.airavata.helix.impl.task.AiravataTask.onFail(AiravataTask.java:146)
> > at
> org.apache.airavata.helix.impl.task.submission.DefaultJobSubmissionTask.onRun(DefaultJobSubmissionTask.java:192)
> > at
> org.apache.airavata.helix.impl.task.AiravataTask.onRun(AiravataTask.java:437)
> > at
> org.apache.airavata.helix.core.AbstractTask.run(AbstractTask.java:102)
> > at org.apache.helix.task.TaskRunner.run(TaskRunner.java:71)
> > at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> > at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> >
> >
> > No reservation for this job
> > --> Verifying valid submit host (login2)...OK
> > --> Verifying valid jobname...OK
> > --> Verifying valid ssh keys...OK
> > --> Verifying access to desired queue (normal)...OK
> > --> Checking available allocation FAILED
> > Airavata stderr : ERROR: You have no project in the projectuser.map
> file (in accounting_check_prod.pl).
> >
>
Re: Failed but metascheduler did not resubmit job
Posted by Aaron Householder <aa...@aucsolutions.com>.
Hi,
Is there any update?
Regards,
Aaron
From: Aaron Householder <aa...@aucsolutions.com>
Date: Saturday, October 12, 2024 at 8:12 PM
To: users@airavata.apache.org <us...@airavata.apache.org>
Subject: Re: Failed but metascheduler did not resubmit job
Hi Lahiru,
Any update on this issue? This is an impediment to getting this rolled out to UltraScan users. My understanding is that if the job fails while verifying and making checks that the metascheduler should try another resource.
Is there anything I can do to help?
Regards,
Aaron
From: Lahiru Jayathilake <la...@apache.org>
Date: Thursday, September 12, 2024 at 1:42 PM
To: users@airavata.apache.org <us...@airavata.apache.org>
Subject: Re: Failed but metascheduler did not resubmit job
Hi Aaron,
Thanks for contacting us. We will look into this issue and get back to you.
Best,
Lahiru
On 2024/09/11 18:45:33 Aaron Householder wrote:
> Hi Airavata,
>
> I’m working on connecting Ultrascan3 to Airavata. As the message below shows, if the job fails the metascheduler might not retry the job. Is there a resource available to take a look at this issue?
>
> Regards,
> Aaron
>
> From: Aaron Householder <aa...@aucsolutions.com>
> Date: Tuesday, September 3, 2024 at 4:42 PM
> To: Airavata Users <us...@airavata.apache.org>
> Subject: Failed but metascheduler did not resubmit job
> Hi Airavata Users,
>
> I had an UltraScan job that seemed to fail without the metascheduler resubmitting the job for completion by another cluster. I received the following in an email:
>
> Your UltraScan job is complete:
>
> Submission Time : 2024-08-26 00:50:05
> Job End Time :
> Mail Time : 2024-08-25 19:54:41
> LIMS Host :
> Analysis ID : US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c
> Request ID : 182 ( uslims3_Demo )
> RunID : demo1_veloc1
> EditID : 21030600161
> Data Type : RA
> Cell/Channel/Wl : 2 / A / 259
> Status : failed
> Cluster : metascheduler
> Job Type : 2DSA-MC
> GFAC Status : FAILED
> GFAC Message : org.apache.airavata.helix.impl.task.TaskOnFailException: Error Code : 23857cb5-5431-43e7-a927-fedd7a929e34, Task TASK_5b1ea99b-750c-49f6-a05b-df7175f141ed failed due to Couldn't find job id in both submitted and verified steps. expId:US3-AIRA_ea2b4a32-27a8-4df4-827c-5fd9367c5e1c Couldn't find remote jobId for JobName:A1394806797, both submit and verify steps doesn't return a valid JobId. Hence changing experiment state to Failed
> at org.apache.airavata.helix.impl.task.AiravataTask.onFail(AiravataTask.java:146)
> at org.apache.airavata.helix.impl.task.submission.DefaultJobSubmissionTask.onRun(DefaultJobSubmissionTask.java:192)
> at org.apache.airavata.helix.impl.task.AiravataTask.onRun(AiravataTask.java:437)
> at org.apache.airavata.helix.core.AbstractTask.run(AbstractTask.java:102)
> at org.apache.helix.task.TaskRunner.run(TaskRunner.java:71)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
>
>
> No reservation for this job
> --> Verifying valid submit host (login2)...OK
> --> Verifying valid jobname...OK
> --> Verifying valid ssh keys...OK
> --> Verifying access to desired queue (normal)...OK
> --> Checking available allocation FAILED
> Airavata stderr : ERROR: You have no project in the projectuser.map file (in accounting_check_prod.pl).
>