[jira] [Updated] (AIRAVATA-3346) Implement remote FS abstraction of user storage

["Marcus Christie (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRAVATA-3346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marcus Christie updated AIRAVATA-3346:
--------------------------------------
    Description: 
Running the Airavata Django portal locally, a developer experiences the portal just like a user of a deployed portal, *except* when it comes to gateway user storage, which is only available on the deployed portal. So for example, on seagrid.org, I can log in, view my experiments, and download my output files. If I run the Airavata Django portal locally and have configured it to connect to the same Keycloak and API server, I can also login and view my experiments, but I can't download my output files (or upload files, or clone experiments, etc.).

However, there exists a REST API to get and put files into gateway user storage so in theory it should be possible to create a filesystem abstraction, locally, that uses the REST API underneath to make it appear that the user's files are all available.

The benefit of this is that it opens up local development to be able to test a broader range of use cases. I know when I'm developing the Django portal code, there are some things that are just very hard to test because I don't actually have access to the files locally. Also, for extension developers, this will be helpful too. For example, currently creating a custom output view provider requires using a fake output file to test locally because the data files aren't available.

This seems like a good option: PyFilesystem2 https://pypi.org/project/fs/

One thing, we need token based authentication in the REST API, but I've already started work on that in a separate branch.

Another thing, it could be good to put in this work now since it might ease transitioning to Airavata MFT for the data storage backend. MFT would just be another PyFilesystem2 filesystem implementation.

h5. TODO

- [ ] Need support for Bearer token authentication so that the REST API can be used directly
- [ ] Update API view function so that they are APIViews and use the Django REST Framework authentication/permission support
- [ ] Implement remote API support in ADP SDK user_storage functions

  was:
Running the Airavata Django portal locally, a developer experiences the portal just like a user of a deployed portal, *except* when it comes to gateway user storage, which is only available on the deployed portal. So for example, on seagrid.org, I can log in, view my experiments, and download my output files. If I run the Airavata Django portal locally and have configured it to connect to the same Keycloak and API server, I can also login and view my experiments, but I can't download my output files (or upload files, or clone experiments, etc.).

However, there exists a REST API to get and put files into gateway user storage so in theory it should be possible to create a filesystem abstraction, locally, that uses the REST API underneath to make it appear that the user's files are all available.

The benefit of this is that it opens up local development to be able to test a broader range of use cases. I know when I'm developing the Django portal code, there are some things that are just very hard to test because I don't actually have access to the files locally. Also, for extension developers, this will be helpful too. For example, currently creating a custom output view provider requires using a fake output file to test locally because the data files aren't available.

This seems like a good option: PyFilesystem2 https://pypi.org/project/fs/

One thing, we need token based authentication in the REST API, but I've already started work on that in a separate branch.

Another thing, it could be good to put in this work now since it might ease transitioning to Airavata MFT for the data storage backend. MFT would just be another PyFilesystem2 filesystem implementation.


> Implement remote FS abstraction of user storage
> -----------------------------------------------
>
>                 Key: AIRAVATA-3346
>                 URL: https://issues.apache.org/jira/browse/AIRAVATA-3346
>             Project: Airavata
>          Issue Type: Bug
>          Components: Django Portal
>            Reporter: Marcus Christie
>            Assignee: Marcus Christie
>            Priority: Major
>
> Running the Airavata Django portal locally, a developer experiences the portal just like a user of a deployed portal, *except* when it comes to gateway user storage, which is only available on the deployed portal. So for example, on seagrid.org, I can log in, view my experiments, and download my output files. If I run the Airavata Django portal locally and have configured it to connect to the same Keycloak and API server, I can also login and view my experiments, but I can't download my output files (or upload files, or clone experiments, etc.).
> However, there exists a REST API to get and put files into gateway user storage so in theory it should be possible to create a filesystem abstraction, locally, that uses the REST API underneath to make it appear that the user's files are all available.
> The benefit of this is that it opens up local development to be able to test a broader range of use cases. I know when I'm developing the Django portal code, there are some things that are just very hard to test because I don't actually have access to the files locally. Also, for extension developers, this will be helpful too. For example, currently creating a custom output view provider requires using a fake output file to test locally because the data files aren't available.
> This seems like a good option: PyFilesystem2 https://pypi.org/project/fs/
> One thing, we need token based authentication in the REST API, but I've already started work on that in a separate branch.
> Another thing, it could be good to put in this work now since it might ease transitioning to Airavata MFT for the data storage backend. MFT would just be another PyFilesystem2 filesystem implementation.
> h5. TODO
> - [ ] Need support for Bearer token authentication so that the REST API can be used directly
> - [ ] Update API view function so that they are APIViews and use the Django REST Framework authentication/permission support
> - [ ] Implement remote API support in ADP SDK user_storage functions



--
This message was sent by Atlassian Jira
(v8.3.4#803005)