You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by "Patil, Aniruddha Vivek" <an...@iu.edu> on 2020/04/23 01:37:02 UTC
Apache Airavata MFT : File Autotransfer
Hello,
As per discussions with Prof. Suresh, we are planning on contributing the feature of scheduling file transfers to Airavata's MFT (https://www.mail-archive.com/dev@airavata.apache.org/msg12564.html).
We were considering delivering certain API endpoints that could easily be integrated into the dashboard in the future.
Are there any pointers we could follow that would help us with this?
Regards,
Aniruddha Patil
Re: [External] Re: Apache Airavata MFT : File Autotransfer
Posted by DImuthu Upeksha <di...@gmail.com>.
Only prerequisite is understanding message paths between API, Controller
and Agents. For technology pieces like consul and grpc messaging, you can
easily figure out as there are enough code examples in the code itself.
Once you grab the architecture properly, implementation might be easy or
rather straightforward. I account most of the estimated time for
understanding the core of MFT rather than the implementation part. For
implementation I expect following top level milestones
1. Defining a new transfer type for API [2]
2. Parsing the new transfer type in controller [3]
3. Implementing data structures and timers to handle scheduled transfers in
the controller. This includes handling fault tolerance in case of
controller crashes. This is the place I need a proper design form you on
how you are going to do it
[2]
https://github.com/apache/airavata-mft/blob/master/api/stub/src/main/proto/MFTApi.proto#L9
[3]
https://github.com/apache/airavata-mft/blob/master/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java#L95
Thanks
Dimuthu
On Thu, Apr 23, 2020 at 8:12 PM Patil, Aniruddha Vivek <an...@iu.edu>
wrote:
> Since you mentioned that the project involves complexity, we’d like to know
> 1) If you have a plan in mind that would help us estimate if we could be
> able to realistically complete the project in time.
> 2) A list of prerequisite concepts relevant to this project.
> 3) Our biggest concern is the time constraint due to the end-semester time
> period and hence we wanted to be sure before we commit.
>
> Regards,
> Aniruddha
>
> On Apr 23, 2020, at 5:20 PM, DImuthu Upeksha <di...@gmail.com>
> wrote:
>
> I would say this project is an interesting one to work on if you have some
> quality time to put on. And I can provide some significant amount of help
> if the time is the critical factor. But I guess narrowing down the scope
> might not work for both the project and you as it would be hard for a new
> one to continue from the point where you have stopped given the complexity
> of the project. It's up to you to decide.
>
> Dimuthu
>
> On Thu, Apr 23, 2020 at 4:20 PM Bafna, Nikita <ni...@iu.edu> wrote:
>
>> Hi Dimuthu,
>>
>>
>> Given the short deadline, do you think we can pick up and do small part
>> of out proposal.
>>
>> Or is it okay to take up Google Drive implementation as a transport type?
>>
>>
>> Regards,
>>
>> Nikita
>>
>>
>> *From: *Patil, Aniruddha Vivek <an...@iu.edu>
>> *Sent: *Thursday, April 23, 2020 2:59 PM
>> *To: *dev@airavata.apache.org
>> *Subject: *Re: [External] Re: Apache Airavata MFT : File Autotransfer
>>
>> Hello,
>>
>> As part of our course, we have a soft deadline of 30th April, 2020. We
>> can adjust the scope of this proposal to fit the deadline. However, we
>> would like to use a few more days in order to incorporate this feature
>> completely.
>>
>> Regards,
>> Aniruddha
>>
>> On Apr 23, 2020, at 2:40 PM, DImuthu Upeksha <di...@gmail.com>
>> wrote:
>>
>> Hi,
>>
>> This looks interesting. This is a feature going into the controller of
>> MFT. And you have to define a new Transfer request type for scheduled
>> transfers. Basically it will affect to MFT Controller and MFT API modules.
>> You should have a deep understanding on how messages are processed in
>> controller so I suggest you to setup MFT locally and try out
>> different transfers first. Then you can send a design diagram with some
>> text on how you are going to do it and the timeline for that. Do you have
>> an absolute deadline for this project?
>>
>> [1]
>> https://github.com/apache/airavata-mft/blob/master/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java
>>
>> Thanks
>> Dimuthu
>>
>> On Thu, Apr 23, 2020 at 2:24 PM Patil, Aniruddha Vivek <an...@iu.edu>
>> wrote:
>>
>>> Hello Dimuthu,
>>>
>>> Here is a link to our proposal (
>>> https://github.com/airavata-courses/NSA/wiki/Assignment-3-Proposal)
>>> It has more detailed information about this feature.
>>>
>>> Regards,
>>> Aniruddha
>>>
>>> On Apr 22, 2020, at 10:42 PM, DImuthu Upeksha <
>>> dimuthu.upeksha2@gmail.com> wrote:
>>>
>>> This message was sent from a non-IU address. Please exercise caution
>>> when clicking links or opening attachments from external sources.
>>>
>>> Hi Aniruddha,
>>>
>>> Can you note down more fine grained goals of this feature. May be add
>>> some use cases as well. Then I will be able to help in figuring out where
>>> to implement it.
>>>
>>> Thanks
>>> Dimuthu
>>>
>>> On Wed, Apr 22, 2020 at 9:37 PM Patil, Aniruddha Vivek <an...@iu.edu>
>>> wrote:
>>>
>>>> Hello,
>>>> As per discussions with Prof. Suresh, we are planning on contributing
>>>> the feature of scheduling file transfers to Airavata's MFT (
>>>> https://www.mail-archive.com/dev@airavata.apache.org/msg12564.html).
>>>> We were considering delivering certain API endpoints that could easily
>>>> be integrated into the dashboard in the future.
>>>>
>>>> Are there any pointers we could follow that would help us with this?
>>>>
>>>> Regards,
>>>> Aniruddha Patil
>>>
>>>
>>>
>>
>
Re: [External] Re: Apache Airavata MFT : File Autotransfer
Posted by "Patil, Aniruddha Vivek" <an...@iu.edu>.
Since you mentioned that the project involves complexity, we’d like to know
1) If you have a plan in mind that would help us estimate if we could be able to realistically complete the project in time.
2) A list of prerequisite concepts relevant to this project.
3) Our biggest concern is the time constraint due to the end-semester time period and hence we wanted to be sure before we commit.
Regards,
Aniruddha
On Apr 23, 2020, at 5:20 PM, DImuthu Upeksha <di...@gmail.com>> wrote:
I would say this project is an interesting one to work on if you have some quality time to put on. And I can provide some significant amount of help if the time is the critical factor. But I guess narrowing down the scope might not work for both the project and you as it would be hard for a new one to continue from the point where you have stopped given the complexity of the project. It's up to you to decide.
Dimuthu
On Thu, Apr 23, 2020 at 4:20 PM Bafna, Nikita <ni...@iu.edu>> wrote:
Hi Dimuthu,
Given the short deadline, do you think we can pick up and do small part of out proposal.
Or is it okay to take up Google Drive implementation as a transport type?
Regards,
Nikita
From: Patil, Aniruddha Vivek<ma...@iu.edu>
Sent: Thursday, April 23, 2020 2:59 PM
To: dev@airavata.apache.org<ma...@airavata.apache.org>
Subject: Re: [External] Re: Apache Airavata MFT : File Autotransfer
Hello,
As part of our course, we have a soft deadline of 30th April, 2020. We can adjust the scope of this proposal to fit the deadline. However, we would like to use a few more days in order to incorporate this feature completely.
Regards,
Aniruddha
On Apr 23, 2020, at 2:40 PM, DImuthu Upeksha <di...@gmail.com>> wrote:
Hi,
This looks interesting. This is a feature going into the controller of MFT. And you have to define a new Transfer request type for scheduled transfers. Basically it will affect to MFT Controller and MFT API modules. You should have a deep understanding on how messages are processed in controller so I suggest you to setup MFT locally and try out different transfers first. Then you can send a design diagram with some text on how you are going to do it and the timeline for that. Do you have an absolute deadline for this project?
[1] https://github.com/apache/airavata-mft/blob/master/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java
Thanks
Dimuthu
On Thu, Apr 23, 2020 at 2:24 PM Patil, Aniruddha Vivek <an...@iu.edu>> wrote:
Hello Dimuthu,
Here is a link to our proposal (https://github.com/airavata-courses/NSA/wiki/Assignment-3-Proposal)
It has more detailed information about this feature.
Regards,
Aniruddha
On Apr 22, 2020, at 10:42 PM, DImuthu Upeksha <di...@gmail.com>> wrote:
This message was sent from a non-IU address. Please exercise caution when clicking links or opening attachments from external sources.
Hi Aniruddha,
Can you note down more fine grained goals of this feature. May be add some use cases as well. Then I will be able to help in figuring out where to implement it.
Thanks
Dimuthu
On Wed, Apr 22, 2020 at 9:37 PM Patil, Aniruddha Vivek <an...@iu.edu>> wrote:
Hello,
As per discussions with Prof. Suresh, we are planning on contributing the feature of scheduling file transfers to Airavata's MFT (https://www.mail-archive.com/dev@airavata.apache.org/msg12564.html).
We were considering delivering certain API endpoints that could easily be integrated into the dashboard in the future.
Are there any pointers we could follow that would help us with this?
Regards,
Aniruddha Patil
Re: [External] Re: Apache Airavata MFT : File Autotransfer
Posted by DImuthu Upeksha <di...@gmail.com>.
I would say this project is an interesting one to work on if you have some
quality time to put on. And I can provide some significant amount of help
if the time is the critical factor. But I guess narrowing down the scope
might not work for both the project and you as it would be hard for a new
one to continue from the point where you have stopped given the complexity
of the project. It's up to you to decide.
Dimuthu
On Thu, Apr 23, 2020 at 4:20 PM Bafna, Nikita <ni...@iu.edu> wrote:
> Hi Dimuthu,
>
>
>
> Given the short deadline, do you think we can pick up and do small part of
> out proposal.
>
> Or is it okay to take up Google Drive implementation as a transport type?
>
>
>
> Regards,
>
> Nikita
>
>
>
> *From: *Patil, Aniruddha Vivek <an...@iu.edu>
> *Sent: *Thursday, April 23, 2020 2:59 PM
> *To: *dev@airavata.apache.org
> *Subject: *Re: [External] Re: Apache Airavata MFT : File Autotransfer
>
>
> Hello,
>
> As part of our course, we have a soft deadline of 30th April, 2020. We can
> adjust the scope of this proposal to fit the deadline. However, we would
> like to use a few more days in order to incorporate this feature completely.
>
> Regards,
> Aniruddha
>
> On Apr 23, 2020, at 2:40 PM, DImuthu Upeksha <di...@gmail.com>
> wrote:
>
> Hi,
>
> This looks interesting. This is a feature going into the controller of
> MFT. And you have to define a new Transfer request type for scheduled
> transfers. Basically it will affect to MFT Controller and MFT API modules.
> You should have a deep understanding on how messages are processed in
> controller so I suggest you to setup MFT locally and try out
> different transfers first. Then you can send a design diagram with some
> text on how you are going to do it and the timeline for that. Do you have
> an absolute deadline for this project?
>
> [1]
> https://github.com/apache/airavata-mft/blob/master/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java
>
> Thanks
> Dimuthu
>
> On Thu, Apr 23, 2020 at 2:24 PM Patil, Aniruddha Vivek <an...@iu.edu>
> wrote:
>
>> Hello Dimuthu,
>>
>> Here is a link to our proposal (
>> https://github.com/airavata-courses/NSA/wiki/Assignment-3-Proposal)
>> It has more detailed information about this feature.
>>
>> Regards,
>> Aniruddha
>>
>> On Apr 22, 2020, at 10:42 PM, DImuthu Upeksha <di...@gmail.com>
>> wrote:
>>
>> This message was sent from a non-IU address. Please exercise caution when
>> clicking links or opening attachments from external sources.
>>
>> Hi Aniruddha,
>>
>> Can you note down more fine grained goals of this feature. May be add
>> some use cases as well. Then I will be able to help in figuring out where
>> to implement it.
>>
>> Thanks
>> Dimuthu
>>
>> On Wed, Apr 22, 2020 at 9:37 PM Patil, Aniruddha Vivek <an...@iu.edu>
>> wrote:
>>
>>> Hello,
>>> As per discussions with Prof. Suresh, we are planning on contributing
>>> the feature of scheduling file transfers to Airavata's MFT (
>>> https://www.mail-archive.com/dev@airavata.apache.org/msg12564.html).
>>> We were considering delivering certain API endpoints that could easily
>>> be integrated into the dashboard in the future.
>>>
>>> Are there any pointers we could follow that would help us with this?
>>>
>>> Regards,
>>> Aniruddha Patil
>>
>>
>>
>
RE: [External] Re: Apache Airavata MFT : File Autotransfer
Posted by "Bafna, Nikita" <ni...@iu.edu>.
Hi Dimuthu,
Given the short deadline, do you think we can pick up and do small part of out proposal.
Or is it okay to take up Google Drive implementation as a transport type?
Regards,
Nikita
From: Patil, Aniruddha Vivek<ma...@iu.edu>
Sent: Thursday, April 23, 2020 2:59 PM
To: dev@airavata.apache.org<ma...@airavata.apache.org>
Subject: Re: [External] Re: Apache Airavata MFT : File Autotransfer
Hello,
As part of our course, we have a soft deadline of 30th April, 2020. We can adjust the scope of this proposal to fit the deadline. However, we would like to use a few more days in order to incorporate this feature completely.
Regards,
Aniruddha
On Apr 23, 2020, at 2:40 PM, DImuthu Upeksha <di...@gmail.com>> wrote:
Hi,
This looks interesting. This is a feature going into the controller of MFT. And you have to define a new Transfer request type for scheduled transfers. Basically it will affect to MFT Controller and MFT API modules. You should have a deep understanding on how messages are processed in controller so I suggest you to setup MFT locally and try out different transfers first. Then you can send a design diagram with some text on how you are going to do it and the timeline for that. Do you have an absolute deadline for this project?
[1] https://github.com/apache/airavata-mft/blob/master/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java
Thanks
Dimuthu
On Thu, Apr 23, 2020 at 2:24 PM Patil, Aniruddha Vivek <an...@iu.edu>> wrote:
Hello Dimuthu,
Here is a link to our proposal (https://github.com/airavata-courses/NSA/wiki/Assignment-3-Proposal)
It has more detailed information about this feature.
Regards,
Aniruddha
On Apr 22, 2020, at 10:42 PM, DImuthu Upeksha <di...@gmail.com>> wrote:
This message was sent from a non-IU address. Please exercise caution when clicking links or opening attachments from external sources.
Hi Aniruddha,
Can you note down more fine grained goals of this feature. May be add some use cases as well. Then I will be able to help in figuring out where to implement it.
Thanks
Dimuthu
On Wed, Apr 22, 2020 at 9:37 PM Patil, Aniruddha Vivek <an...@iu.edu>> wrote:
Hello,
As per discussions with Prof. Suresh, we are planning on contributing the feature of scheduling file transfers to Airavata's MFT (https://www.mail-archive.com/dev@airavata.apache.org/msg12564.html).
We were considering delivering certain API endpoints that could easily be integrated into the dashboard in the future.
Are there any pointers we could follow that would help us with this?
Regards,
Aniruddha Patil
Re: [External] Re: Apache Airavata MFT : File Autotransfer
Posted by "Patil, Aniruddha Vivek" <an...@iu.edu>.
Hello,
As part of our course, we have a soft deadline of 30th April, 2020. We can adjust the scope of this proposal to fit the deadline. However, we would like to use a few more days in order to incorporate this feature completely.
Regards,
Aniruddha
On Apr 23, 2020, at 2:40 PM, DImuthu Upeksha <di...@gmail.com>> wrote:
Hi,
This looks interesting. This is a feature going into the controller of MFT. And you have to define a new Transfer request type for scheduled transfers. Basically it will affect to MFT Controller and MFT API modules. You should have a deep understanding on how messages are processed in controller so I suggest you to setup MFT locally and try out different transfers first. Then you can send a design diagram with some text on how you are going to do it and the timeline for that. Do you have an absolute deadline for this project?
[1] https://github.com/apache/airavata-mft/blob/master/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java
Thanks
Dimuthu
On Thu, Apr 23, 2020 at 2:24 PM Patil, Aniruddha Vivek <an...@iu.edu>> wrote:
Hello Dimuthu,
Here is a link to our proposal (https://github.com/airavata-courses/NSA/wiki/Assignment-3-Proposal)
It has more detailed information about this feature.
Regards,
Aniruddha
On Apr 22, 2020, at 10:42 PM, DImuthu Upeksha <di...@gmail.com>> wrote:
This message was sent from a non-IU address. Please exercise caution when clicking links or opening attachments from external sources.
Hi Aniruddha,
Can you note down more fine grained goals of this feature. May be add some use cases as well. Then I will be able to help in figuring out where to implement it.
Thanks
Dimuthu
On Wed, Apr 22, 2020 at 9:37 PM Patil, Aniruddha Vivek <an...@iu.edu>> wrote:
Hello,
As per discussions with Prof. Suresh, we are planning on contributing the feature of scheduling file transfers to Airavata's MFT (https://www.mail-archive.com/dev@airavata.apache.org/msg12564.html).
We were considering delivering certain API endpoints that could easily be integrated into the dashboard in the future.
Are there any pointers we could follow that would help us with this?
Regards,
Aniruddha Patil
Re: [External] Re: Apache Airavata MFT : File Autotransfer
Posted by DImuthu Upeksha <di...@gmail.com>.
Hi,
This looks interesting. This is a feature going into the controller of MFT.
And you have to define a new Transfer request type for scheduled transfers.
Basically it will affect to MFT Controller and MFT API modules. You should
have a deep understanding on how messages are processed in controller so I
suggest you to setup MFT locally and try out different transfers first.
Then you can send a design diagram with some text on how you are going to
do it and the timeline for that. Do you have an absolute deadline for this
project?
[1]
https://github.com/apache/airavata-mft/blob/master/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java
Thanks
Dimuthu
On Thu, Apr 23, 2020 at 2:24 PM Patil, Aniruddha Vivek <an...@iu.edu>
wrote:
> Hello Dimuthu,
>
> Here is a link to our proposal (
> https://github.com/airavata-courses/NSA/wiki/Assignment-3-Proposal)
> It has more detailed information about this feature.
>
> Regards,
> Aniruddha
>
> On Apr 22, 2020, at 10:42 PM, DImuthu Upeksha <di...@gmail.com>
> wrote:
>
> This message was sent from a non-IU address. Please exercise caution when
> clicking links or opening attachments from external sources.
>
> Hi Aniruddha,
>
> Can you note down more fine grained goals of this feature. May be add some
> use cases as well. Then I will be able to help in figuring out where to
> implement it.
>
> Thanks
> Dimuthu
>
> On Wed, Apr 22, 2020 at 9:37 PM Patil, Aniruddha Vivek <an...@iu.edu>
> wrote:
>
>> Hello,
>> As per discussions with Prof. Suresh, we are planning on contributing the
>> feature of scheduling file transfers to Airavata's MFT (
>> https://www.mail-archive.com/dev@airavata.apache.org/msg12564.html).
>> We were considering delivering certain API endpoints that could easily be
>> integrated into the dashboard in the future.
>>
>> Are there any pointers we could follow that would help us with this?
>>
>> Regards,
>> Aniruddha Patil
>
>
>
Re: [External] Re: Apache Airavata MFT : File Autotransfer
Posted by "Patil, Aniruddha Vivek" <an...@iu.edu>.
Hello Dimuthu,
Here is a link to our proposal (https://github.com/airavata-courses/NSA/wiki/Assignment-3-Proposal)
It has more detailed information about this feature.
Regards,
Aniruddha
On Apr 22, 2020, at 10:42 PM, DImuthu Upeksha <di...@gmail.com>> wrote:
This message was sent from a non-IU address. Please exercise caution when clicking links or opening attachments from external sources.
Hi Aniruddha,
Can you note down more fine grained goals of this feature. May be add some use cases as well. Then I will be able to help in figuring out where to implement it.
Thanks
Dimuthu
On Wed, Apr 22, 2020 at 9:37 PM Patil, Aniruddha Vivek <an...@iu.edu>> wrote:
Hello,
As per discussions with Prof. Suresh, we are planning on contributing the feature of scheduling file transfers to Airavata's MFT (https://www.mail-archive.com/dev@airavata.apache.org/msg12564.html).
We were considering delivering certain API endpoints that could easily be integrated into the dashboard in the future.
Are there any pointers we could follow that would help us with this?
Regards,
Aniruddha Patil
Re: Apache Airavata MFT : File Autotransfer
Posted by DImuthu Upeksha <di...@gmail.com>.
Hi Aniruddha,
Can you note down more fine grained goals of this feature. May be add some
use cases as well. Then I will be able to help in figuring out where to
implement it.
Thanks
Dimuthu
On Wed, Apr 22, 2020 at 9:37 PM Patil, Aniruddha Vivek <an...@iu.edu>
wrote:
> Hello,
> As per discussions with Prof. Suresh, we are planning on contributing the
> feature of scheduling file transfers to Airavata's MFT (
> https://www.mail-archive.com/dev@airavata.apache.org/msg12564.html).
> We were considering delivering certain API endpoints that could easily be
> integrated into the dashboard in the future.
>
> Are there any pointers we could follow that would help us with this?
>
> Regards,
> Aniruddha Patil