You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uima.apache.org by Yi-Wen Liu <yi...@usc.edu> on 2016/02/21 04:30:11 UTC
uima ducc wait_for_completion
Hello,
I am using ducc in a web app, and I need to know when does ducc finish its
work, ie, generate the output file.
I use the two properties in my job file:
cancel_on_interrupt
wait_for_completion
However, when I submitted job file, it did show the status on terminal, but
it finished earlier than the main process, I have to wait for few minutes
for the output file to be generated.
Is there any way to get notified when the main process finishes without
looking at the web server page?
Any suggestion is appreciated!
Thanks,
Yi-Wen
Re: uima ducc wait_for_completion
Posted by Yi-Wen Liu <yi...@usc.edu>.
Hi Eddie,
Thanks! That is very helpful!
Thanks,
Yi-Wen
On Mon, Feb 22, 2016 at 2:59 PM, Eddie Epstein <ea...@gmail.com> wrote:
> Hi Yi-Wen,
>
> Here is an example console from a normal DUCC job, with comments
>
> 2016.02.19 10:33:34 Fri Received
> 2016.02.19 10:33:34 Fri WaitingForDriver ... waiting for the Job
> Driver to initialize successfully
> 2016.02.19 10:33:52 Fri WaitingForResources ... waiting for resources to
> start one or more Job Process (pipelines)
> 2016.02.19 10:34:00 Fri Assigned
> 2016.02.19 10:34:02 Fri Initializing ... one or more
> JP
> launched and doing initialization
> 2016.02.19 10:35:23 Fri Running ... initialization
> completed successfully, processing work items from JD
> 2016.02.19 10:36:14 Fri Completing ... job processes are
> terminating
> 2016.02.19 10:36:31 Fri Completed ... all user processes
> have terminated
> 2016.02.19 10:36:31 Fri completion type: EndOfJob
> 2016.02.19 10:36:31 Fri rationale: state manager detected normal completion
> ... all work items processed
>
> Your console indicates that the job driver was terminated before
> initialization completed.
> There should be a lot more information in the job's log directory. Feel
> free to zip up and email that data to Lou or I, as long as it does *not*
> contain any confidential information.
>
> Regards,
> Eddie
>
>
> On Mon, Feb 22, 2016 at 5:18 PM, Yi-Wen Liu <yi...@usc.edu> wrote:
>
> > Hi,
> >
> > The monitor output from ducc.log:
> >
> > 2015.11.30 02:27:16 Mon Received
> > 2015.11.30 02:27:16 Mon WaitingForDriver
> > 2015.11.30 02:27:44 Mon WaitingForResources
> > 2015.11.30 02:27:56 Mon Assigned
> > 2015.11.30 02:28:04 Mon Initializing
> > 2015.11.30 02:29:35 Mon Completing
> > 2015.11.30 02:30:18 Mon Completed
> > 2015.11.30 02:30:18 Mon completion type: CanceledByDriver
> > 2015.11.30 02:30:18 Mon rationale: unspecified
> >
> >
> >
> > Thanks,
> > Yi-Wen
> >
> > On Sun, Feb 21, 2016 at 1:45 PM, Eddie Epstein <ea...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > Can you please paste the full monitor output from the DUCC job submit
> > here?
> > > Thanks,
> > > Eddie
> > >
> > > On Sun, Feb 21, 2016 at 3:26 PM, Yi-Wen Liu <yi...@usc.edu> wrote:
> > >
> > > > Hi,
> > > >
> > > > Because when I used wait_for_completion, I saw the terminal print out
> > the
> > > > step by step state, and after it finished, the output file was not
> > > > generated yet. The job status on web server page might still stay in
> > > > "waiting for resources" instead of completion.
> > > >
> > > > And sorry for the confusion, I used "main process" to refer to the
> > > process
> > > > generating output (the second one below).
> > > >
> > > > Time axis---->
> > > > *The terminal* status:waiting for driver -> ....->
> > > status:
> > > > completed (point A, no output)
> > > > *The web server page *initializing -> waiting for
> > > >
> > > >
> > >
> >
> resouces............................................................->completion
> > > > (point B, has output)
> > > >
> > > > And my question is when can I know the point when the output is
> > generated
> > > > (point B)?
> > > > If I use wait_for_completion, it is telling me the point A but there
> is
> > > no
> > > > output yet.
> > > >
> > > > If I misunderstand anything please correct me, thanks for your
> > patience!
> > > >
> > > > Thanks,
> > > > Yi-Wen
> > > >
> > > > On Sun, Feb 21, 2016 at 6:05 AM, Lou DeGenaro <
> lou.degenaro@gmail.com>
> > > > wrote:
> > > >
> > > > > Yi-Wen,
> > > > >
> > > > > When submitting a DUCC Job, wait for completion should prevent the
> > > return
> > > > > until all your work items have been processed.
> > > > >
> > > > > What exactly do you mean by "main process"?
> > > > >
> > > > > Lou.
> > > > >
> > > > > On Sat, Feb 20, 2016 at 10:30 PM, Yi-Wen Liu <yi...@usc.edu>
> > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I am using ducc in a web app, and I need to know when does ducc
> > > finish
> > > > > its
> > > > > > work, ie, generate the output file.
> > > > > > I use the two properties in my job file:
> > > > > >
> > > > > > cancel_on_interrupt
> > > > > > wait_for_completion
> > > > > >
> > > > > > However, when I submitted job file, it did show the status on
> > > terminal,
> > > > > but
> > > > > > it finished earlier than the main process, I have to wait for few
> > > > minutes
> > > > > > for the output file to be generated.
> > > > > >
> > > > > > Is there any way to get notified when the main process finishes
> > > without
> > > > > > looking at the web server page?
> > > > > >
> > > > > > Any suggestion is appreciated!
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Yi-Wen
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: uima ducc wait_for_completion
Posted by Eddie Epstein <ea...@gmail.com>.
Hi Yi-Wen,
Here is an example console from a normal DUCC job, with comments
2016.02.19 10:33:34 Fri Received
2016.02.19 10:33:34 Fri WaitingForDriver ... waiting for the Job
Driver to initialize successfully
2016.02.19 10:33:52 Fri WaitingForResources ... waiting for resources to
start one or more Job Process (pipelines)
2016.02.19 10:34:00 Fri Assigned
2016.02.19 10:34:02 Fri Initializing ... one or more JP
launched and doing initialization
2016.02.19 10:35:23 Fri Running ... initialization
completed successfully, processing work items from JD
2016.02.19 10:36:14 Fri Completing ... job processes are
terminating
2016.02.19 10:36:31 Fri Completed ... all user processes
have terminated
2016.02.19 10:36:31 Fri completion type: EndOfJob
2016.02.19 10:36:31 Fri rationale: state manager detected normal completion
... all work items processed
Your console indicates that the job driver was terminated before
initialization completed.
There should be a lot more information in the job's log directory. Feel
free to zip up and email that data to Lou or I, as long as it does *not*
contain any confidential information.
Regards,
Eddie
On Mon, Feb 22, 2016 at 5:18 PM, Yi-Wen Liu <yi...@usc.edu> wrote:
> Hi,
>
> The monitor output from ducc.log:
>
> 2015.11.30 02:27:16 Mon Received
> 2015.11.30 02:27:16 Mon WaitingForDriver
> 2015.11.30 02:27:44 Mon WaitingForResources
> 2015.11.30 02:27:56 Mon Assigned
> 2015.11.30 02:28:04 Mon Initializing
> 2015.11.30 02:29:35 Mon Completing
> 2015.11.30 02:30:18 Mon Completed
> 2015.11.30 02:30:18 Mon completion type: CanceledByDriver
> 2015.11.30 02:30:18 Mon rationale: unspecified
>
>
>
> Thanks,
> Yi-Wen
>
> On Sun, Feb 21, 2016 at 1:45 PM, Eddie Epstein <ea...@gmail.com>
> wrote:
>
> > Hi,
> > Can you please paste the full monitor output from the DUCC job submit
> here?
> > Thanks,
> > Eddie
> >
> > On Sun, Feb 21, 2016 at 3:26 PM, Yi-Wen Liu <yi...@usc.edu> wrote:
> >
> > > Hi,
> > >
> > > Because when I used wait_for_completion, I saw the terminal print out
> the
> > > step by step state, and after it finished, the output file was not
> > > generated yet. The job status on web server page might still stay in
> > > "waiting for resources" instead of completion.
> > >
> > > And sorry for the confusion, I used "main process" to refer to the
> > process
> > > generating output (the second one below).
> > >
> > > Time axis---->
> > > *The terminal* status:waiting for driver -> ....->
> > status:
> > > completed (point A, no output)
> > > *The web server page *initializing -> waiting for
> > >
> > >
> >
> resouces............................................................->completion
> > > (point B, has output)
> > >
> > > And my question is when can I know the point when the output is
> generated
> > > (point B)?
> > > If I use wait_for_completion, it is telling me the point A but there is
> > no
> > > output yet.
> > >
> > > If I misunderstand anything please correct me, thanks for your
> patience!
> > >
> > > Thanks,
> > > Yi-Wen
> > >
> > > On Sun, Feb 21, 2016 at 6:05 AM, Lou DeGenaro <lo...@gmail.com>
> > > wrote:
> > >
> > > > Yi-Wen,
> > > >
> > > > When submitting a DUCC Job, wait for completion should prevent the
> > return
> > > > until all your work items have been processed.
> > > >
> > > > What exactly do you mean by "main process"?
> > > >
> > > > Lou.
> > > >
> > > > On Sat, Feb 20, 2016 at 10:30 PM, Yi-Wen Liu <yi...@usc.edu>
> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > I am using ducc in a web app, and I need to know when does ducc
> > finish
> > > > its
> > > > > work, ie, generate the output file.
> > > > > I use the two properties in my job file:
> > > > >
> > > > > cancel_on_interrupt
> > > > > wait_for_completion
> > > > >
> > > > > However, when I submitted job file, it did show the status on
> > terminal,
> > > > but
> > > > > it finished earlier than the main process, I have to wait for few
> > > minutes
> > > > > for the output file to be generated.
> > > > >
> > > > > Is there any way to get notified when the main process finishes
> > without
> > > > > looking at the web server page?
> > > > >
> > > > > Any suggestion is appreciated!
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Yi-Wen
> > > > >
> > > >
> > >
> >
>
Re: uima ducc wait_for_completion
Posted by Yi-Wen Liu <yi...@usc.edu>.
Hi,
The monitor output from ducc.log:
2015.11.30 02:27:16 Mon Received
2015.11.30 02:27:16 Mon WaitingForDriver
2015.11.30 02:27:44 Mon WaitingForResources
2015.11.30 02:27:56 Mon Assigned
2015.11.30 02:28:04 Mon Initializing
2015.11.30 02:29:35 Mon Completing
2015.11.30 02:30:18 Mon Completed
2015.11.30 02:30:18 Mon completion type: CanceledByDriver
2015.11.30 02:30:18 Mon rationale: unspecified
Thanks,
Yi-Wen
On Sun, Feb 21, 2016 at 1:45 PM, Eddie Epstein <ea...@gmail.com> wrote:
> Hi,
> Can you please paste the full monitor output from the DUCC job submit here?
> Thanks,
> Eddie
>
> On Sun, Feb 21, 2016 at 3:26 PM, Yi-Wen Liu <yi...@usc.edu> wrote:
>
> > Hi,
> >
> > Because when I used wait_for_completion, I saw the terminal print out the
> > step by step state, and after it finished, the output file was not
> > generated yet. The job status on web server page might still stay in
> > "waiting for resources" instead of completion.
> >
> > And sorry for the confusion, I used "main process" to refer to the
> process
> > generating output (the second one below).
> >
> > Time axis---->
> > *The terminal* status:waiting for driver -> ....->
> status:
> > completed (point A, no output)
> > *The web server page *initializing -> waiting for
> >
> >
> resouces............................................................->completion
> > (point B, has output)
> >
> > And my question is when can I know the point when the output is generated
> > (point B)?
> > If I use wait_for_completion, it is telling me the point A but there is
> no
> > output yet.
> >
> > If I misunderstand anything please correct me, thanks for your patience!
> >
> > Thanks,
> > Yi-Wen
> >
> > On Sun, Feb 21, 2016 at 6:05 AM, Lou DeGenaro <lo...@gmail.com>
> > wrote:
> >
> > > Yi-Wen,
> > >
> > > When submitting a DUCC Job, wait for completion should prevent the
> return
> > > until all your work items have been processed.
> > >
> > > What exactly do you mean by "main process"?
> > >
> > > Lou.
> > >
> > > On Sat, Feb 20, 2016 at 10:30 PM, Yi-Wen Liu <yi...@usc.edu> wrote:
> > >
> > > > Hello,
> > > >
> > > > I am using ducc in a web app, and I need to know when does ducc
> finish
> > > its
> > > > work, ie, generate the output file.
> > > > I use the two properties in my job file:
> > > >
> > > > cancel_on_interrupt
> > > > wait_for_completion
> > > >
> > > > However, when I submitted job file, it did show the status on
> terminal,
> > > but
> > > > it finished earlier than the main process, I have to wait for few
> > minutes
> > > > for the output file to be generated.
> > > >
> > > > Is there any way to get notified when the main process finishes
> without
> > > > looking at the web server page?
> > > >
> > > > Any suggestion is appreciated!
> > > >
> > > >
> > > > Thanks,
> > > > Yi-Wen
> > > >
> > >
> >
>
Re: uima ducc wait_for_completion
Posted by Eddie Epstein <ea...@gmail.com>.
Hi,
Can you please paste the full monitor output from the DUCC job submit here?
Thanks,
Eddie
On Sun, Feb 21, 2016 at 3:26 PM, Yi-Wen Liu <yi...@usc.edu> wrote:
> Hi,
>
> Because when I used wait_for_completion, I saw the terminal print out the
> step by step state, and after it finished, the output file was not
> generated yet. The job status on web server page might still stay in
> "waiting for resources" instead of completion.
>
> And sorry for the confusion, I used "main process" to refer to the process
> generating output (the second one below).
>
> Time axis---->
> *The terminal* status:waiting for driver -> ....-> status:
> completed (point A, no output)
> *The web server page *initializing -> waiting for
>
> resouces............................................................->completion
> (point B, has output)
>
> And my question is when can I know the point when the output is generated
> (point B)?
> If I use wait_for_completion, it is telling me the point A but there is no
> output yet.
>
> If I misunderstand anything please correct me, thanks for your patience!
>
> Thanks,
> Yi-Wen
>
> On Sun, Feb 21, 2016 at 6:05 AM, Lou DeGenaro <lo...@gmail.com>
> wrote:
>
> > Yi-Wen,
> >
> > When submitting a DUCC Job, wait for completion should prevent the return
> > until all your work items have been processed.
> >
> > What exactly do you mean by "main process"?
> >
> > Lou.
> >
> > On Sat, Feb 20, 2016 at 10:30 PM, Yi-Wen Liu <yi...@usc.edu> wrote:
> >
> > > Hello,
> > >
> > > I am using ducc in a web app, and I need to know when does ducc finish
> > its
> > > work, ie, generate the output file.
> > > I use the two properties in my job file:
> > >
> > > cancel_on_interrupt
> > > wait_for_completion
> > >
> > > However, when I submitted job file, it did show the status on terminal,
> > but
> > > it finished earlier than the main process, I have to wait for few
> minutes
> > > for the output file to be generated.
> > >
> > > Is there any way to get notified when the main process finishes without
> > > looking at the web server page?
> > >
> > > Any suggestion is appreciated!
> > >
> > >
> > > Thanks,
> > > Yi-Wen
> > >
> >
>
Re: uima ducc wait_for_completion
Posted by Yi-Wen Liu <yi...@usc.edu>.
Hi,
Because when I used wait_for_completion, I saw the terminal print out the
step by step state, and after it finished, the output file was not
generated yet. The job status on web server page might still stay in
"waiting for resources" instead of completion.
And sorry for the confusion, I used "main process" to refer to the process
generating output (the second one below).
Time axis---->
*The terminal* status:waiting for driver -> ....-> status:
completed (point A, no output)
*The web server page *initializing -> waiting for
resouces............................................................->completion
(point B, has output)
And my question is when can I know the point when the output is generated
(point B)?
If I use wait_for_completion, it is telling me the point A but there is no
output yet.
If I misunderstand anything please correct me, thanks for your patience!
Thanks,
Yi-Wen
On Sun, Feb 21, 2016 at 6:05 AM, Lou DeGenaro <lo...@gmail.com>
wrote:
> Yi-Wen,
>
> When submitting a DUCC Job, wait for completion should prevent the return
> until all your work items have been processed.
>
> What exactly do you mean by "main process"?
>
> Lou.
>
> On Sat, Feb 20, 2016 at 10:30 PM, Yi-Wen Liu <yi...@usc.edu> wrote:
>
> > Hello,
> >
> > I am using ducc in a web app, and I need to know when does ducc finish
> its
> > work, ie, generate the output file.
> > I use the two properties in my job file:
> >
> > cancel_on_interrupt
> > wait_for_completion
> >
> > However, when I submitted job file, it did show the status on terminal,
> but
> > it finished earlier than the main process, I have to wait for few minutes
> > for the output file to be generated.
> >
> > Is there any way to get notified when the main process finishes without
> > looking at the web server page?
> >
> > Any suggestion is appreciated!
> >
> >
> > Thanks,
> > Yi-Wen
> >
>
Re: uima ducc wait_for_completion
Posted by Lou DeGenaro <lo...@gmail.com>.
Yi-Wen,
When submitting a DUCC Job, wait for completion should prevent the return
until all your work items have been processed.
What exactly do you mean by "main process"?
Lou.
On Sat, Feb 20, 2016 at 10:30 PM, Yi-Wen Liu <yi...@usc.edu> wrote:
> Hello,
>
> I am using ducc in a web app, and I need to know when does ducc finish its
> work, ie, generate the output file.
> I use the two properties in my job file:
>
> cancel_on_interrupt
> wait_for_completion
>
> However, when I submitted job file, it did show the status on terminal, but
> it finished earlier than the main process, I have to wait for few minutes
> for the output file to be generated.
>
> Is there any way to get notified when the main process finishes without
> looking at the web server page?
>
> Any suggestion is appreciated!
>
>
> Thanks,
> Yi-Wen
>