API Client questions

[Pedro da Silveira <pe...@gmail.com>]

Hi Dev,

I have a few question regarding to Airavata API Client.
I tested both 2 samples from the directory
"apache-airavata-client-0.7-SNAPSHOT/samples". They all ran the echo
example successfully.
I modified the sample "workflow-run" to submit my pre-defined workflow,
which has 4 inputs file "URI", 2 Application Services and total wall time
20 minutes. My workflow ran successfully using API Client as it did on
Xbaya.

1 - One part of the code that got my attention was a call to the method
"getWorkflowComposeContent()", which reads my workflow saved file with
extension ".xwf". I didn't understand the call for this method. Can someone
help me?

airavataAPI.getWorkflowManager().saveWorkflow(getWorkflowComposeContent());


2 - The API client transfered my input correctly from my desktop to
Lonestar, How can I get one output file back from Lonestar to my desktop
using the API? In general, my outputs are just text file and less than
500MB.

3 - I noticed in the monitoring part of the code that a Java Thread is
created and keep the application hanging until the end of the monitoring
execution. Is that possible to create the thread, check the monitor status
and kill thread to check the status later in about N minutes? Or Do I have
to keep the thread alive during the job execution?

Thank you,

Pedro da Silveira

Re: API Client questions

[Saminda Wijeratne <sa...@gmail.com>]

On Tue, Feb 5, 2013 at 4:46 PM, Pedro da Silveira <pe...@gmail.com>wrote:

> Saminda,
>
>
> On Tue, Feb 5, 2013 at 2:18 PM, Saminda Wijeratne <samindaw@gmail.com
> >wrote:
>
> > Hi Pedro,
> >
> > see my comments in-line,
> >
> > On Tue, Feb 5, 2013 at 1:42 PM, Pedro da Silveira <pedrorcs@gmail.com
> > >wrote:
> >
> > > Hi Dev,
> >> >
> >> > I have a few question regarding to Airavata API Client.
> >> > I tested both 2 samples from the directory
> >> > "apache-airavata-client-0.7-SNAPSHOT/samples". They all ran the echo
> >> > example successfully.
> >> > I modified the sample "workflow-run" to submit my pre-defined
> workflow,
> >> > which has 4 inputs file "URI", 2 Application Services and total wall
> >> time
> >> > 20 minutes. My workflow ran successfully using API Client as it did on
> >> > Xbaya.
> >> >
> >> > 1 - One part of the code that got my attention was a call to the
> method
> >> > "getWorkflowComposeContent()", which reads my workflow saved file with
> >> > extension ".xwf". I didn't understand the call for this method. Can
> >> someone
> >> > help me?
> >> >
> >> >
> >>
> airavataAPI.getWorkflowManager().saveWorkflow(getWorkflowComposeContent());
> >> >
> >>  There are two ways to specify what the workflow that needs to be
> >> executed.
> >>    1. Pass the name of the workflow - then the workflow should already
> be
> >>    saved in the registry
> >>    2. Pass the workflow xml as a string
> >> The above command tries to save the workflow to the registry. API user
> has
> >> to pass the workflow xml as a string to the "saveWorkflow" function. The
> >> API will extract the name of the workflow from the xml string & save the
> >> workflow to the registry under that name. This should be a one time
> thing.
> >> That's all it does and has nothing to do with running the workflow at
> this
> >> point.
> >> Then the users can use the 1st way mentioned above to run the workflow
> >> many
> >> times as he/she wants by just passing only the name of the workflow to
> the
> >> runExperiment(...) method.
> >
> >
> > If I understood it correctly, If my workflow is already saved in my
> > register I don't need to run this command again, right?
> >
>
Yep.


>
>
>
> > >
> >> > 2 - The API client transfered my input correctly from my desktop to
> >> > Lonestar, How can I get one output file back from Lonestar to my
> desktop
> >> > using the API? In general, my outputs are just text file and less than
> >> > 500MB.
> >> >
> >> The API client could not have transfered any file from your desktop. I
> >> assume that the Airavata server is running in your machine & the input
> >> paths you may have passed to the experiment are local. The workflow
> >> execution may have taken care of transfering the file in your desktop to
> >> Lonestar. If you require files downloaded back to your desktop then you
> >> need to update your workflow in order to do that.
> >
> >
> >
> Based on your answer "If you require files downloaded back to your desktop
> then you need to update your workflow in order to do that."
>
> What kind of update do I need to do on my workflow?  Do I have to add
> anything in my script that runs on Lonestar?
> My basic understating is that I can setup in my Application Service add
> another Application parameter Output and then provide the type "URI".  What
> is next? Where is the location my output will be saved on my desktop?
>
Well its a little tricky. As far as I know we have grid-ftp capability to
transfer files so as to stage them as input for the next application.
Airavata has no notion of returning files at the end of workflow execution
(to be exact Airavata would not know what files are to transfer anyway
unless the application returns a URI pointing to the file). Theoretically
speaking, if the final result is a file after running a workflow, the user
should manually retrieve the file (using scp/ftp etc.).
Having said that what you could do is to have a new application service
defined in Airavata which points to an application that is in your machine
which accepts a URI as an input (& may or may not have an output). And have
your Lonestar application service output a URI to the file which you want
to be downloaded to your machine & connect that output to the input of your
new application service. (if you are in linux your new application service
could just point to "/bin/echo"). In this way we can trick Airavata to
bring a file back to your machine. Hope I was clear enough... if not please
don't hesitate to ask questions to clarify further.

>
>
>
> > >
> >> > 3 - I noticed in the monitoring part of the code that a Java Thread is
> >> > created and keep the application hanging until the end of the
> monitoring
> >> > execution. Is that possible to create the thread, check the monitor
> >> status
> >> > and kill thread to check the status later in about N minutes? Or Do I
> >> have
> >> > to keep the thread alive during the job execution?
> >> >
> >> Monitoring is used to track the progress of the workflow experiment in
> >> real
> >> time... you can always *not monitor* and check the status at your
> leisure.
> >> Thus the lines,
> >> MonitorWorkflow.monitor(result);
> >> airavataAPI.getExecutionManager().waitForExperimentTermination(result);
> >> are not needed. Important thing to note here is that the command,
> >> airavataAPI.getExecutionManager().runExperiment(workflowName,
> >> workflowInputs)
> >> launches the experiment and returns without waiting for the experiment
> to
> >> finish. Thus the API users should not expect the results to be available
> >> when it tries to read provenance data. Infact it'll take little bit of
> >> time
> >> for the experiment to be registered to the system & initial status of
> the
> >> workflow executions be updated correctly. API users should anticipate
> such
> >> scenarios if they are planning not to use default monitoring or
> >> "waitForExperimentTermination(...)" functionalities.
> >
> >
> >
> My plan is to connect to Airavata-Server to check the status of the job
> every 5 minutes and not keep the socket connection alive until the job
> finishes. Is there any problem doing that?
>
nop.. no problems at all...


> I want to submit more tasks to Lonestar, but now using my Xsede account,
> instead of "ogce".
>
> Thank you for your help Saminda.
>
>
> Please feel free to further ask questions if anything is unclear or if you
> > have any other issues.
> >
> > Regards,
> > Saminda
> >
> >
> > > Thank you,
> > >
> > > Pedro da Silveira
> > >
> >
>

Re: API Client questions

[Pedro da Silveira <pe...@gmail.com>]

Saminda,


On Tue, Feb 5, 2013 at 2:18 PM, Saminda Wijeratne <sa...@gmail.com>wrote:

> Hi Pedro,
>
> see my comments in-line,
>
> On Tue, Feb 5, 2013 at 1:42 PM, Pedro da Silveira <pedrorcs@gmail.com
> >wrote:
>
> > Hi Dev,
>> >
>> > I have a few question regarding to Airavata API Client.
>> > I tested both 2 samples from the directory
>> > "apache-airavata-client-0.7-SNAPSHOT/samples". They all ran the echo
>> > example successfully.
>> > I modified the sample "workflow-run" to submit my pre-defined workflow,
>> > which has 4 inputs file "URI", 2 Application Services and total wall
>> time
>> > 20 minutes. My workflow ran successfully using API Client as it did on
>> > Xbaya.
>> >
>> > 1 - One part of the code that got my attention was a call to the method
>> > "getWorkflowComposeContent()", which reads my workflow saved file with
>> > extension ".xwf". I didn't understand the call for this method. Can
>> someone
>> > help me?
>> >
>> >
>> airavataAPI.getWorkflowManager().saveWorkflow(getWorkflowComposeContent());
>> >
>>  There are two ways to specify what the workflow that needs to be
>> executed.
>>    1. Pass the name of the workflow - then the workflow should already be
>>    saved in the registry
>>    2. Pass the workflow xml as a string
>> The above command tries to save the workflow to the registry. API user has
>> to pass the workflow xml as a string to the "saveWorkflow" function. The
>> API will extract the name of the workflow from the xml string & save the
>> workflow to the registry under that name. This should be a one time thing.
>> That's all it does and has nothing to do with running the workflow at this
>> point.
>> Then the users can use the 1st way mentioned above to run the workflow
>> many
>> times as he/she wants by just passing only the name of the workflow to the
>> runExperiment(...) method.
>
>
> If I understood it correctly, If my workflow is already saved in my
> register I don't need to run this command again, right?
>




> >
>> > 2 - The API client transfered my input correctly from my desktop to
>> > Lonestar, How can I get one output file back from Lonestar to my desktop
>> > using the API? In general, my outputs are just text file and less than
>> > 500MB.
>> >
>> The API client could not have transfered any file from your desktop. I
>> assume that the Airavata server is running in your machine & the input
>> paths you may have passed to the experiment are local. The workflow
>> execution may have taken care of transfering the file in your desktop to
>> Lonestar. If you require files downloaded back to your desktop then you
>> need to update your workflow in order to do that.
>
>
>
Based on your answer "If you require files downloaded back to your desktop
then you need to update your workflow in order to do that."

What kind of update do I need to do on my workflow?  Do I have to add
anything in my script that runs on Lonestar?
My basic understating is that I can setup in my Application Service add
another Application parameter Output and then provide the type "URI".  What
is next? Where is the location my output will be saved on my desktop?



> >
>> > 3 - I noticed in the monitoring part of the code that a Java Thread is
>> > created and keep the application hanging until the end of the monitoring
>> > execution. Is that possible to create the thread, check the monitor
>> status
>> > and kill thread to check the status later in about N minutes? Or Do I
>> have
>> > to keep the thread alive during the job execution?
>> >
>> Monitoring is used to track the progress of the workflow experiment in
>> real
>> time... you can always *not monitor* and check the status at your leisure.
>> Thus the lines,
>> MonitorWorkflow.monitor(result);
>> airavataAPI.getExecutionManager().waitForExperimentTermination(result);
>> are not needed. Important thing to note here is that the command,
>> airavataAPI.getExecutionManager().runExperiment(workflowName,
>> workflowInputs)
>> launches the experiment and returns without waiting for the experiment to
>> finish. Thus the API users should not expect the results to be available
>> when it tries to read provenance data. Infact it'll take little bit of
>> time
>> for the experiment to be registered to the system & initial status of the
>> workflow executions be updated correctly. API users should anticipate such
>> scenarios if they are planning not to use default monitoring or
>> "waitForExperimentTermination(...)" functionalities.
>
>
>
My plan is to connect to Airavata-Server to check the status of the job
every 5 minutes and not keep the socket connection alive until the job
finishes. Is there any problem doing that?
I want to submit more tasks to Lonestar, but now using my Xsede account,
instead of "ogce".

Thank you for your help Saminda.


Please feel free to further ask questions if anything is unclear or if you
> have any other issues.
>
> Regards,
> Saminda
>
>
> > Thank you,
> >
> > Pedro da Silveira
> >
>

Re: API Client questions

[Saminda Wijeratne <sa...@gmail.com>]

Hi Pedro,

see my comments in-line,

On Tue, Feb 5, 2013 at 1:42 PM, Pedro da Silveira <pe...@gmail.com>wrote:

> Hi Dev,
>
> I have a few question regarding to Airavata API Client.
> I tested both 2 samples from the directory
> "apache-airavata-client-0.7-SNAPSHOT/samples". They all ran the echo
> example successfully.
> I modified the sample "workflow-run" to submit my pre-defined workflow,
> which has 4 inputs file "URI", 2 Application Services and total wall time
> 20 minutes. My workflow ran successfully using API Client as it did on
> Xbaya.
>
> 1 - One part of the code that got my attention was a call to the method
> "getWorkflowComposeContent()", which reads my workflow saved file with
> extension ".xwf". I didn't understand the call for this method. Can someone
> help me?
>
> airavataAPI.getWorkflowManager().saveWorkflow(getWorkflowComposeContent());
>
 There are two ways to specify what the workflow that needs to be executed.

   1. Pass the name of the workflow - then the workflow should already be
   saved in the registry
   2. Pass the workflow xml as a string

The above command tries to save the workflow to the registry. API user has
to pass the workflow xml as a string to the "saveWorkflow" function. The
API will extract the name of the workflow from the xml string & save the
workflow to the registry under that name. This should be a one time thing.
That's all it does and has nothing to do with running the workflow at this
point.
Then the users can use the 1st way mentioned above to run the workflow many
times as he/she wants by just passing only the name of the workflow to the
runExperiment(...) method.


>
> 2 - The API client transfered my input correctly from my desktop to
> Lonestar, How can I get one output file back from Lonestar to my desktop
> using the API? In general, my outputs are just text file and less than
> 500MB.
>
The API client could not have transfered any file from your desktop. I
assume that the Airavata server is running in your machine & the input
paths you may have passed to the experiment are local. The workflow
execution may have taken care of transfering the file in your desktop to
Lonestar. If you require files downloaded back to your desktop then you
need to update your workflow in order to do that.

>
> 3 - I noticed in the monitoring part of the code that a Java Thread is
> created and keep the application hanging until the end of the monitoring
> execution. Is that possible to create the thread, check the monitor status
> and kill thread to check the status later in about N minutes? Or Do I have
> to keep the thread alive during the job execution?
>
Monitoring is used to track the progress of the workflow experiment in real
time... you can always *not monitor* and check the status at your leisure.
Thus the lines,

MonitorWorkflow.monitor(result);
airavataAPI.getExecutionManager().waitForExperimentTermination(result);

are not needed. Important thing to note here is that the command,

airavataAPI.getExecutionManager().runExperiment(workflowName, workflowInputs)

launches the experiment and returns without waiting for the experiment to
finish. Thus the API users should not expect the results to be available
when it tries to read provenance data. Infact it'll take little bit of time
for the experiment to be registered to the system & initial status of the
workflow executions be updated correctly. API users should anticipate such
scenarios if they are planning not to use default monitoring or
"waitForExperimentTermination(...)" functionalities.

Please feel free to further ask questions if anything is unclear or if you
have any other issues.

Regards,
Saminda


> Thank you,
>
> Pedro da Silveira
>