You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airavata.apache.org by ma...@apache.org on 2020/10/05 21:24:20 UTC
[airavata-django-portal] branch master updated (1522d7b -> deb7fda)
This is an automated email from the ASF dual-hosted git repository.
machristie pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git.
from 1522d7b Merge branch 'AIRAVATA-3346-implement-remote-fs-abstraction-of-user-storage'
new b525003 AIRAVATA-3346 Updating tutorial instructions
new 561296d AIRAVATA-3346 Add authz_token to request in auth backend
new e97799d Merge branch 'AIRAVATA-3346-implement-remote-fs-abstraction-of-user-storage'
new deb7fda AIRAVATA-3346 AIRAVATA-3362 Doc how to access exp data dir in output view provider
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
django_airavata/apps/api/authentication.py | 9 ------
django_airavata/apps/auth/backends.py | 27 ++++++++++++----
django_airavata/apps/auth/utils.py | 4 ++-
docs/dev/custom_output_view_provider.md | 40 +++++++++++++++++++----
docs/tutorial/gateways2019_tutorial.md | 51 ++++--------------------------
5 files changed, 63 insertions(+), 68 deletions(-)
[airavata-django-portal] 01/04: AIRAVATA-3346 Updating tutorial
instructions
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
machristie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git
commit b52500324812de96d427aca7ae56c59ac870c8f1
Author: Marcus Christie <ma...@apache.org>
AuthorDate: Mon Oct 5 12:14:27 2020 -0400
AIRAVATA-3346 Updating tutorial instructions
---
docs/tutorial/gateways2019_tutorial.md | 51 ++++------------------------------
1 file changed, 6 insertions(+), 45 deletions(-)
diff --git a/docs/tutorial/gateways2019_tutorial.md b/docs/tutorial/gateways2019_tutorial.md
index e7ae202..1a1f931 100644
--- a/docs/tutorial/gateways2019_tutorial.md
+++ b/docs/tutorial/gateways2019_tutorial.md
@@ -493,22 +493,7 @@ molecular orbital energies. Then `matplotlib` is used to create two plots of
these values. Finally, the plots are exported as a PNG image that is returns as
a buffer of bytes.
-4. To test this locally we need access to a file to test with. While our local
- portal instance can connect to the Airavata API just like the production
- deployed Django portal instance, only the production deployed Django portal
- has access to the output files generated by users' experiments. So for
- testing purposes we'll define a file to be used when there is no Gaussian log
- file available (this test file will only be used when the Django portal is
- running in `DEBUG` mode).
-
-Just after the `name` attribute of `GaussianEigenvaluesViewProvider` we add the
-following:
-
-```python
-test_output_file = os.path.join(BASE_DIR, "data", "gaussian.log")
-```
-
-5. Altogether, the output_views.py file should have the following contents:
+4. Altogether, the output_views.py file should have the following contents:
```python
import io
@@ -524,7 +509,6 @@ BASE_DIR = os.path.dirname(os.path.abspath(__file__))
class GaussianEigenvaluesViewProvider:
display_type = 'image'
name = "Gaussian Eigenvalues"
- test_output_file = os.path.join(BASE_DIR, "data", "gaussian.log")
def generate_data(self, request, experiment_output, experiment, output_file=None):
@@ -573,7 +557,7 @@ class GaussianEigenvaluesViewProvider:
```
-6. Now we need to register our _output view provider_ with the package metadata
+5. Now we need to register our _output view provider_ with the package metadata
so that the Django Portal will be able to discover it. We add the following
lines to the `entry_points` parameter in the
`$HOME/gateways19-tutorial/setup.py` file:
@@ -592,7 +576,7 @@ gaussian-eigenvalues-plot = gateways19_tutorial.output_views:GaussianEigenvalues
`gateways19_tutorial.output_views` is the module in which the
`GaussianEigenvaluesViewProvider` output view provider class is found.
-7. Now we need to install the _gateways19-tutorial_ package into the Django
+6. Now we need to install the _gateways19-tutorial_ package into the Django
portal's virtual environment.
```bash
@@ -1128,15 +1112,6 @@ $("#run-button").click((e) => {
```
Now that we can launch the experiment we can go ahead and give it a try.
-However, the job will ultimately fail because Airavata won't be able to transfer
-the file back to our locally running Django portal (if we had our locally
-running Django portal running a public SSH server we could configure it so that
-Airavata could SCP the file back to our local instance). But this custom Django
-app is also deployed in the hosted tutorial Django instance so you can run it
-there to verify it works.
-
-6. Try this out in the production deployment at
- <https://testdrive.airavata.org/gateways19_tutorial/hello/>.
### Displaying the experiment output
@@ -1200,18 +1175,11 @@ if (exp.experimentStatus === models.ExperimentState.COMPLETED) {
}
```
-3. However, as noted earlier this won't quite work with our local Django
- instance since it doesn't have access to the output file. That's fine though
- since we can fake the STDOUT text so that we can test our code locally.
- Remove the comment starting with `/* Displaying the experiment output` on
- line 88 and ending on line 116. Here's the update to the `loadExperiments`
- function:
+3. To enable this, remove the comment starting with
+ `/* Displaying the experiment output` on line 88 and ending on line 113.
+ Here's the update to the `loadExperiments` function:
```javascript
-const FAKE_STDOUT = `
-bonjour
-`;
-
function loadExperiments() {
return services.ExperimentSearchService.list({
limit: 5,
@@ -1249,9 +1217,6 @@ function loadExperiments() {
return fetch(stdoutDataProduct.downloadURL, {
credentials: "same-origin",
}).then((result) => result.text());
- } else {
- // If we can't download it, fake it
- return FAKE_STDOUT;
}
})
.then((text) => {
@@ -1263,10 +1228,6 @@ function loadExperiments() {
}
```
-4. You can try out this custom Django app in the production deployed instance at
- <https://testdrive.airavata.org/gateways19_tutorial/hello/> where it really
- does download and parse the standard out.
-
## Resources
You can browser the final version of the _gateways19-tutorial_ code at
[airavata-django-portal] 03/04: Merge branch
'AIRAVATA-3346-implement-remote-fs-abstraction-of-user-storage'
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
machristie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git
commit e97799df455a2869a4ec85a04058afa97c792c54
Merge: b525003 561296d
Author: Marcus Christie <ma...@apache.org>
AuthorDate: Mon Oct 5 14:06:54 2020 -0400
Merge branch 'AIRAVATA-3346-implement-remote-fs-abstraction-of-user-storage'
django_airavata/apps/api/authentication.py | 9 ---------
django_airavata/apps/auth/backends.py | 27 ++++++++++++++++++++-------
django_airavata/apps/auth/utils.py | 4 +++-
3 files changed, 23 insertions(+), 17 deletions(-)
[airavata-django-portal] 04/04: AIRAVATA-3346 AIRAVATA-3362 Doc how
to access exp data dir in output view provider
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
machristie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git
commit deb7fdac49df48a5ada84625fed1a5e71b68a0b6
Author: Marcus Christie <ma...@apache.org>
AuthorDate: Mon Oct 5 15:27:02 2020 -0400
AIRAVATA-3346 AIRAVATA-3362 Doc how to access exp data dir in output view provider
---
docs/dev/custom_output_view_provider.md | 40 ++++++++++++++++++++++++++++-----
1 file changed, 34 insertions(+), 6 deletions(-)
diff --git a/docs/dev/custom_output_view_provider.md b/docs/dev/custom_output_view_provider.md
index 43de789..4410ece 100644
--- a/docs/dev/custom_output_view_provider.md
+++ b/docs/dev/custom_output_view_provider.md
@@ -54,6 +54,20 @@ environment using:
python setup.py develop
```
+### Setting up remote data access
+
+To access the files in the remote deployed Django portal instance in your local
+development environment you need to configure a setting so that your local
+Django instance knows at what URL is the remote deployed Django portal REST API.
+The remote API will be used for accessing data, making your local instance
+behave just like the remote instance. Set the GATEWAY_DATA_STORE_REMOTE_API in
+settings_local.py to have the domain of the remote deployed Django portal:
+
+```
+# Change this to match your remote Django portal instance
+GATEWAY_DATA_STORE_REMOTE_API = 'https://testdrive.airavata.org/api'
+```
+
## Reference
### Output View Provider interface
@@ -63,12 +77,6 @@ the following attributes:
- `display_type`: this should be one of _link_, _image_ or _html_.
- `name`: this is the name of the output view provider displayed to the user.
-- `test_output_file`: (optional) the path to a file to use for testing
- purposes. This file will be passed to the `generate_data` function as the
- `output_file` parameter when the output file isn't available and the Django
- server is running in DEBUG mode. This is helpful when developing a custom
- output view provider in a local Django instance that doesn't have access to
- the output files.
The output view provider class should define the following method:
@@ -194,6 +202,26 @@ output-view-providers and place it second. For example:
would make the `gaussian-eigenvalues-plot` the initial output view provider. The
user can access the default output view provider from the drop down menu.
+### Accessing additional experiment output files
+
+The output view provider is associated with a particular output file, but your
+output view provider can access other files in the experiment data directory. To
+access those files use the `list_experiment_dir` of the
+[user_storage module](https://airavata-django-portal-sdk.readthedocs.io/en/latest/#module-user_storage)
+in the Airavata Django Portal SDK.
+
+```python
+from airavata_django_portal_sdk import user_storage
+def generate_data(self, request, experiment_output, experiment, output_file=None, **kwargs):
+
+ dirs, files = user_storage.list_experiment_dir(request, experiment.experimentId)
+ # ...
+```
+
+`list_experiment_dir` returns a tuple of directories and files in the experiment
+data directory. Each entry is a dictionary of metadata about the directory/file.
+See the SDK documentation for more information.
+
### Interactive parameters
You can add some interactivity to your custom output view provider by adding one
[airavata-django-portal] 02/04: AIRAVATA-3346 Add authz_token to
request in auth backend
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
machristie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git
commit 561296d2533d9821b079aabeba11cd8be330a5b3
Author: Marcus Christie <ma...@apache.org>
AuthorDate: Mon Oct 5 14:06:44 2020 -0400
AIRAVATA-3346 Add authz_token to request in auth backend
---
django_airavata/apps/api/authentication.py | 9 ---------
django_airavata/apps/auth/backends.py | 27 ++++++++++++++++++++-------
django_airavata/apps/auth/utils.py | 4 +++-
3 files changed, 23 insertions(+), 17 deletions(-)
diff --git a/django_airavata/apps/api/authentication.py b/django_airavata/apps/api/authentication.py
index fd282ba..9e12b94 100644
--- a/django_airavata/apps/api/authentication.py
+++ b/django_airavata/apps/api/authentication.py
@@ -3,8 +3,6 @@ import logging
from django.contrib.auth import authenticate
from rest_framework import authentication, exceptions
-from django_airavata.apps.auth.utils import get_authz_token
-
logger = logging.getLogger(__name__)
@@ -19,13 +17,6 @@ class OAuthAuthentication(authentication.BaseAuthentication):
"Token failed to authenticate")
_, token = request.META.get('HTTP_AUTHORIZATION').split()
- # authz_token_middleware has already run, so must manually add
- # the `request.authz_token` attribute
-
- # Must pass user directly since `request.user` access will
- # trigger this Authentication being called again, resulting in
- # an infinite loop
- request.authz_token = get_authz_token(request, user=user, access_token=token)
logger.debug(f"OAuthAuthentication authenticated user {user}")
return (user, token)
except Exception as e:
diff --git a/django_airavata/apps/auth/backends.py b/django_airavata/apps/auth/backends.py
index 1c9591d..5bb832a 100644
--- a/django_airavata/apps/auth/backends.py
+++ b/django_airavata/apps/auth/backends.py
@@ -9,6 +9,8 @@ from django.views.decorators.debug import sensitive_variables
from oauthlib.oauth2 import InvalidGrantError, LegacyApplicationClient
from requests_oauthlib import OAuth2Session
+from django_airavata.apps.auth.utils import get_authz_token
+
from . import utils
logger = logging.getLogger(__name__)
@@ -28,22 +30,24 @@ class KeycloakBackend(object):
password=None,
refresh_token=None):
try:
+ user = None
+ access_token = None
if username and password:
token, userinfo = self._get_token_and_userinfo_password_flow(
username, password)
if token is None: # login failed
return None
self._process_token(request, token)
- return self._process_userinfo(request, userinfo)
+ user = self._process_userinfo(request, userinfo)
+ access_token = token['access_token']
elif 'HTTP_AUTHORIZATION' in request.META:
bearer, token = request.META.get('HTTP_AUTHORIZATION').split()
if bearer != "Bearer":
raise Exception("Unexpected Authorization header")
# implicitly validate token by using it to get userinfo
userinfo = self._get_userinfo_from_token(request, token)
- # Token should be added as a request attribute (request.auth)
- # self._process_token(request, token)
- return self._process_userinfo(request, userinfo)
+ user = self._process_userinfo(request, userinfo)
+ access_token = token
# user is already logged in and can use refresh token
elif request.user and not utils.is_refresh_token_expired(request):
logger.debug("Refreshing token...")
@@ -51,19 +55,28 @@ class KeycloakBackend(object):
self._get_token_and_userinfo_from_refresh_token(request)
self._process_token(request, token)
# user is already logged in
- return request.user
+ user = request.user
+ access_token = token['access_token']
elif refresh_token:
logger.debug("Refreshing supplied token...")
token, userinfo = \
self._get_token_and_userinfo_from_refresh_token(
request, refresh_token=refresh_token)
self._process_token(request, token)
- return self._process_userinfo(request, userinfo)
+ user = self._process_userinfo(request, userinfo)
+ access_token = token['access_token']
else:
token, userinfo = self._get_token_and_userinfo_redirect_flow(
request)
self._process_token(request, token)
- return self._process_userinfo(request, userinfo)
+ user = self._process_userinfo(request, userinfo)
+ access_token = token['access_token']
+ # authz_token_middleware has already run, so must manually add
+ # the `request.authz_token` attribute
+ if user is not None:
+ request.authz_token = get_authz_token(
+ request, user=user, access_token=access_token)
+ return user
except Exception as e:
logger.exception("login failed")
return None
diff --git a/django_airavata/apps/auth/utils.py b/django_airavata/apps/auth/utils.py
index d350850..caf0088 100644
--- a/django_airavata/apps/auth/utils.py
+++ b/django_airavata/apps/auth/utils.py
@@ -17,7 +17,9 @@ from . import models
def get_authz_token(request, user=None, access_token=None):
"""Construct AuthzToken instance from session; refresh token if needed."""
- if not is_access_token_expired(request):
+ if access_token is not None:
+ return _create_authz_token(request, user=user, access_token=access_token)
+ elif not is_access_token_expired(request):
return _create_authz_token(request, user=user, access_token=access_token)
elif not is_refresh_token_expired(request):
# Have backend reauthenticate the user with the refresh token