You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Nipun Udara <y....@gmail.com> on 2014/06/11 17:10:06 UTC
ec2 input types
Hi all
current ec2 implementation use only String type inputs. what other input
types will be used with ec2. i would be grateful if someone can give a idea.
regards
Nipun Udara
Re: ec2 input types
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Logically this should be possible to do in a handler, because you can do
the initializing of the environment in one of the handler and store that
data in jobexecution context and consume them in file transfer handlers.
And provider just focus on the submission of the job. In this approach you
have more flexibility of implementing different operaitons. If you
implement everything in provider all the tasks are in one place and not
decoupled. Best way is to implement these thing in each handler and order
them properly. Its totally fine if handlers are not function if they
arrange in the proper order, but please try to give proper error if one
handler get invoked in wrong place hwich will help developer to find issues
with the configuration.
Lahiru
On Fri, Jun 13, 2014 at 3:54 AM, Nipun Udara <y....@gmail.com> wrote:
> Hi lahiru
>
> Thank you for the details. I am currently checking AdvancedSCPInputHandler
> and AdvancedSCPOutputHandler. Since i need to start ec2 host and ssh
> connected to transfer data and also need to access command line tool to
> copy s3 files isnt it ok to do data transfer from provider once we have
> started ec2 host. any idea regarding this.
>
> regards
> Nipun Udara
>
>
> On Thu, Jun 12, 2014 at 7:58 PM, Lahiru Gunathilake <gl...@gmail.com>
> wrote:
>
>> Hi Nipun,
>>
>> I think you can implement this as an handler and if the input/output
>> types are URI you can invoke these handler logics. If we want to add new
>> schemas and thrift models we can focus on them little later.
>>
>> First you can write an handlers to transfer inputs and outputs and write
>> test cases to test them. Please refer SSHInputHandler,SSHOutputHandler and
>> AdvancedSCPInputHandler and AdvancedSCPOutputHandler. Initial two handlers
>> transfer files to the remote resource from airavata host and advance
>> handlers transfer files from third party machine to remote machine.
>>
>> Please try to find whether the API allow to do third party transfer, in
>> that case its easy to have one set of handlers to transfer from remote
>> machine to S3 (basically two handlers one for input one for output).
>>
>> Regards
>> Lahiru
>>
>>
>> On Thu, Jun 12, 2014 at 12:12 AM, Raminderjeet Singh <ramifnu@indiana.edu
>> > wrote:
>>
>>> Hi Nipun,
>>>
>>> S3 [1] is the storage service provided by Amazon and it provide
>>> persistent storage for input/output data. There are Amazon API's provided
>>> to develop clients[2]. To add S3 input/output type in Airavata, you need to
>>> change gfac schema and also thrift model. These 2 places will be merged to
>>> one in 0.13 release. They you can expect client to set input as S3 type and
>>> you need to add handlers to upload and download the data from S3.
>>>
>>> Thanks
>>> Raminder
>>>
>>> 1. http://aws.amazon.com/s3/
>>> 2.
>>> http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/index.html?com/amazonaws/services/s3/transfer/TransferManager.html
>>>
>>>
>>> On Wed, Jun 11, 2014 at 11:53 PM, Nipun Udara <y....@gmail.com>
>>> wrote:
>>>
>>>> Hi raminder singh
>>>>
>>>> I cannot find S3type among other input types. How can i add S3type can
>>>> you give a brief idea about doing that.
>>>>
>>>> Regards
>>>>
>>>>
>>>> On Wed, Jun 11, 2014 at 8:59 PM, Raminder Singh <
>>>> raminderjsingh@gmail.com> wrote:
>>>>
>>>>> URI and S3 types can be used for data movement. We should support SCP,
>>>>> HTTP,FTP data transfers under URI type.
>>>>>
>>>>> Thanks
>>>>> Raminder
>>>>>
>>>>> On Jun 11, 2014, at 11:10 AM, Nipun Udara <y....@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hi all
>>>>> >
>>>>> > current ec2 implementation use only String type inputs. what other
>>>>> input types will be used with ec2. i would be grateful if someone can give
>>>>> a idea.
>>>>> >
>>>>> > regards
>>>>> > Nipun Udara
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Nipun Udara
>>>>
>>>> *Undergraduate*
>>>> *Department of Computer Science & Engineering*
>>>> *University of Moratuwa*
>>>> *Sri Lanka*
>>>>
>>>>
>>>
>>
>>
>> --
>> System Analyst Programmer
>> PTI Lab
>> Indiana University
>>
>
>
>
> --
> Nipun Udara
>
> *Undergraduate*
> *Department of Computer Science & Engineering*
> *University of Moratuwa*
> *Sri Lanka*
>
>
--
System Analyst Programmer
PTI Lab
Indiana University
Re: ec2 input types
Posted by Nipun Udara <y....@gmail.com>.
Hi lahiru
Thank you for the details. I am currently checking AdvancedSCPInputHandler
and AdvancedSCPOutputHandler. Since i need to start ec2 host and ssh
connected to transfer data and also need to access command line tool to
copy s3 files isnt it ok to do data transfer from provider once we have
started ec2 host. any idea regarding this.
regards
Nipun Udara
On Thu, Jun 12, 2014 at 7:58 PM, Lahiru Gunathilake <gl...@gmail.com>
wrote:
> Hi Nipun,
>
> I think you can implement this as an handler and if the input/output types
> are URI you can invoke these handler logics. If we want to add new schemas
> and thrift models we can focus on them little later.
>
> First you can write an handlers to transfer inputs and outputs and write
> test cases to test them. Please refer SSHInputHandler,SSHOutputHandler and
> AdvancedSCPInputHandler and AdvancedSCPOutputHandler. Initial two handlers
> transfer files to the remote resource from airavata host and advance
> handlers transfer files from third party machine to remote machine.
>
> Please try to find whether the API allow to do third party transfer, in
> that case its easy to have one set of handlers to transfer from remote
> machine to S3 (basically two handlers one for input one for output).
>
> Regards
> Lahiru
>
>
> On Thu, Jun 12, 2014 at 12:12 AM, Raminderjeet Singh <ra...@indiana.edu>
> wrote:
>
>> Hi Nipun,
>>
>> S3 [1] is the storage service provided by Amazon and it provide
>> persistent storage for input/output data. There are Amazon API's provided
>> to develop clients[2]. To add S3 input/output type in Airavata, you need to
>> change gfac schema and also thrift model. These 2 places will be merged to
>> one in 0.13 release. They you can expect client to set input as S3 type and
>> you need to add handlers to upload and download the data from S3.
>>
>> Thanks
>> Raminder
>>
>> 1. http://aws.amazon.com/s3/
>> 2.
>> http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/index.html?com/amazonaws/services/s3/transfer/TransferManager.html
>>
>>
>> On Wed, Jun 11, 2014 at 11:53 PM, Nipun Udara <y....@gmail.com>
>> wrote:
>>
>>> Hi raminder singh
>>>
>>> I cannot find S3type among other input types. How can i add S3type can
>>> you give a brief idea about doing that.
>>>
>>> Regards
>>>
>>>
>>> On Wed, Jun 11, 2014 at 8:59 PM, Raminder Singh <
>>> raminderjsingh@gmail.com> wrote:
>>>
>>>> URI and S3 types can be used for data movement. We should support SCP,
>>>> HTTP,FTP data transfers under URI type.
>>>>
>>>> Thanks
>>>> Raminder
>>>>
>>>> On Jun 11, 2014, at 11:10 AM, Nipun Udara <y....@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi all
>>>> >
>>>> > current ec2 implementation use only String type inputs. what other
>>>> input types will be used with ec2. i would be grateful if someone can give
>>>> a idea.
>>>> >
>>>> > regards
>>>> > Nipun Udara
>>>> >
>>>> >
>>>>
>>>>
>>>
>>>
>>> --
>>> Nipun Udara
>>>
>>> *Undergraduate*
>>> *Department of Computer Science & Engineering*
>>> *University of Moratuwa*
>>> *Sri Lanka*
>>>
>>>
>>
>
>
> --
> System Analyst Programmer
> PTI Lab
> Indiana University
>
--
Nipun Udara
*Undergraduate*
*Department of Computer Science & Engineering*
*University of Moratuwa*
*Sri Lanka*
Re: ec2 input types
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Nipun,
I think you can implement this as an handler and if the input/output types
are URI you can invoke these handler logics. If we want to add new schemas
and thrift models we can focus on them little later.
First you can write an handlers to transfer inputs and outputs and write
test cases to test them. Please refer SSHInputHandler,SSHOutputHandler and
AdvancedSCPInputHandler and AdvancedSCPOutputHandler. Initial two handlers
transfer files to the remote resource from airavata host and advance
handlers transfer files from third party machine to remote machine.
Please try to find whether the API allow to do third party transfer, in
that case its easy to have one set of handlers to transfer from remote
machine to S3 (basically two handlers one for input one for output).
Regards
Lahiru
On Thu, Jun 12, 2014 at 12:12 AM, Raminderjeet Singh <ra...@indiana.edu>
wrote:
> Hi Nipun,
>
> S3 [1] is the storage service provided by Amazon and it provide persistent
> storage for input/output data. There are Amazon API's provided to develop
> clients[2]. To add S3 input/output type in Airavata, you need to change
> gfac schema and also thrift model. These 2 places will be merged to one in
> 0.13 release. They you can expect client to set input as S3 type and you
> need to add handlers to upload and download the data from S3.
>
> Thanks
> Raminder
>
> 1. http://aws.amazon.com/s3/
> 2.
> http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/index.html?com/amazonaws/services/s3/transfer/TransferManager.html
>
>
> On Wed, Jun 11, 2014 at 11:53 PM, Nipun Udara <y....@gmail.com>
> wrote:
>
>> Hi raminder singh
>>
>> I cannot find S3type among other input types. How can i add S3type can
>> you give a brief idea about doing that.
>>
>> Regards
>>
>>
>> On Wed, Jun 11, 2014 at 8:59 PM, Raminder Singh <raminderjsingh@gmail.com
>> > wrote:
>>
>>> URI and S3 types can be used for data movement. We should support SCP,
>>> HTTP,FTP data transfers under URI type.
>>>
>>> Thanks
>>> Raminder
>>>
>>> On Jun 11, 2014, at 11:10 AM, Nipun Udara <y....@gmail.com> wrote:
>>>
>>> > Hi all
>>> >
>>> > current ec2 implementation use only String type inputs. what other
>>> input types will be used with ec2. i would be grateful if someone can give
>>> a idea.
>>> >
>>> > regards
>>> > Nipun Udara
>>> >
>>> >
>>>
>>>
>>
>>
>> --
>> Nipun Udara
>>
>> *Undergraduate*
>> *Department of Computer Science & Engineering*
>> *University of Moratuwa*
>> *Sri Lanka*
>>
>>
>
--
System Analyst Programmer
PTI Lab
Indiana University
Re: ec2 input types
Posted by Raminderjeet Singh <ra...@indiana.edu>.
Hi Nipun,
S3 [1] is the storage service provided by Amazon and it provide persistent
storage for input/output data. There are Amazon API's provided to develop
clients[2]. To add S3 input/output type in Airavata, you need to change
gfac schema and also thrift model. These 2 places will be merged to one in
0.13 release. They you can expect client to set input as S3 type and you
need to add handlers to upload and download the data from S3.
Thanks
Raminder
1. http://aws.amazon.com/s3/
2.
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/index.html?com/amazonaws/services/s3/transfer/TransferManager.html
On Wed, Jun 11, 2014 at 11:53 PM, Nipun Udara <y....@gmail.com> wrote:
> Hi raminder singh
>
> I cannot find S3type among other input types. How can i add S3type can you
> give a brief idea about doing that.
>
> Regards
>
>
> On Wed, Jun 11, 2014 at 8:59 PM, Raminder Singh <ra...@gmail.com>
> wrote:
>
>> URI and S3 types can be used for data movement. We should support SCP,
>> HTTP,FTP data transfers under URI type.
>>
>> Thanks
>> Raminder
>>
>> On Jun 11, 2014, at 11:10 AM, Nipun Udara <y....@gmail.com> wrote:
>>
>> > Hi all
>> >
>> > current ec2 implementation use only String type inputs. what other
>> input types will be used with ec2. i would be grateful if someone can give
>> a idea.
>> >
>> > regards
>> > Nipun Udara
>> >
>> >
>>
>>
>
>
> --
> Nipun Udara
>
> *Undergraduate*
> *Department of Computer Science & Engineering*
> *University of Moratuwa*
> *Sri Lanka*
>
>
Re: ec2 input types
Posted by Nipun Udara <y....@gmail.com>.
Hi raminder singh
I cannot find S3type among other input types. How can i add S3type can you
give a brief idea about doing that.
Regards
On Wed, Jun 11, 2014 at 8:59 PM, Raminder Singh <ra...@gmail.com>
wrote:
> URI and S3 types can be used for data movement. We should support SCP,
> HTTP,FTP data transfers under URI type.
>
> Thanks
> Raminder
>
> On Jun 11, 2014, at 11:10 AM, Nipun Udara <y....@gmail.com> wrote:
>
> > Hi all
> >
> > current ec2 implementation use only String type inputs. what other input
> types will be used with ec2. i would be grateful if someone can give a idea.
> >
> > regards
> > Nipun Udara
> >
> >
>
>
--
Nipun Udara
*Undergraduate*
*Department of Computer Science & Engineering*
*University of Moratuwa*
*Sri Lanka*
Re: ec2 input types
Posted by Raminder Singh <ra...@gmail.com>.
URI and S3 types can be used for data movement. We should support SCP, HTTP,FTP data transfers under URI type.
Thanks
Raminder
On Jun 11, 2014, at 11:10 AM, Nipun Udara <y....@gmail.com> wrote:
> Hi all
>
> current ec2 implementation use only String type inputs. what other input types will be used with ec2. i would be grateful if someone can give a idea.
>
> regards
> Nipun Udara
>
>