You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airavata.apache.org by ma...@apache.org on 2022/08/29 18:51:12 UTC

[airavata-django-portal] branch develop updated (ba4cb026 -> 373c21f9)

This is an automated email from the ASF dual-hosted git repository.

machristie pushed a change to branch develop
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git


    from ba4cb026 AIRAVATA-3650 Switch from node-sass to dart-sass
     new c41f6549 AIRAVATA-3649 REST API to list and invoke queue settings calculators
     new 2319627d AIRAVATA-3649 DB field and Admin UI for queueSettingsCalculatorId
     new 68e425fe AIRAVATA-3649 Apply queue settings calculator values in experiment editor
     new 2d355f57 AIRAVATA-3649 Only include queueSettingsCalculatorId if it exists
     new c6617c00 AIRAVATA-3649 debounce calls to queue settings calculator
     new 6a11894c AIRAVATA-3649 Only re-calculate queue settings when inputs or compute resource change
     new b21eede5 AIRAVATA-3649 Make queue settings readonly when calculator is configured
     new 61d181f8 AIRAVATA-3649 Ignore invalid experiment models
     new cf64de70 AIRAVATA-3649 Queue settings calculator docs
     new 15c7da95 AIRAVATA-3649 Upgrade SDK to 1.6.0
     new 373c21f9 Merge branch 'AIRAVATA-3649' into develop

The 11 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../applications/ApplicationInterfaceEditor.vue    |  51 ++++++++--
 ...icationsettings_queue_settings_calculator_id.py |  18 ++++
 django_airavata/apps/api/models.py                 |   1 +
 django_airavata/apps/api/serializers.py            |  42 ++++++---
 .../api/static/django_airavata_api/js/index.js     |   3 +
 .../js/models/ApplicationInterfaceDefinition.js    |   5 +
 .../js/models/QueueSettingsCalculator.js           |   9 ++
 .../django_airavata_api/js/service_config.js       |  15 +++
 django_airavata/apps/api/urls.py                   |   2 +
 django_airavata/apps/api/views.py                  |  31 +++++-
 .../js/components/experiment/ExperimentEditor.vue  |  58 +++++++++---
 .../components/experiment/QueueSettingsEditor.vue  |  72 +++++++++++---
 docs/dev/queue_settings_calculator.md              | 104 +++++++++++++++++++++
 mkdocs.yml                                         |   1 +
 requirements.txt                                   |   2 +-
 15 files changed, 370 insertions(+), 44 deletions(-)
 create mode 100644 django_airavata/apps/api/migrations/0009_applicationsettings_queue_settings_calculator_id.py
 create mode 100644 django_airavata/apps/api/static/django_airavata_api/js/models/QueueSettingsCalculator.js
 create mode 100644 docs/dev/queue_settings_calculator.md


[airavata-django-portal] 10/11: AIRAVATA-3649 Upgrade SDK to 1.6.0

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

machristie pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git

commit 15c7da956ef9b983b04407f0cf3e5e65146258be
Author: Marcus Christie <ma...@apache.org>
AuthorDate: Mon Aug 29 14:50:05 2022 -0400

    AIRAVATA-3649 Upgrade SDK to 1.6.0
---
 requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/requirements.txt b/requirements.txt
index e910a594..46a8dc9f 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -21,7 +21,7 @@ google-api-python-client==1.12.8
 grpcio-tools==1.34.1
 grpcio==1.34.1
 
-airavata-django-portal-sdk==1.4.0
+airavata-django-portal-sdk==1.6.0
 airavata-python-sdk==1.0.2
 
 -e "."


[airavata-django-portal] 05/11: AIRAVATA-3649 debounce calls to queue settings calculator

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

machristie pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git

commit c6617c0030a5765cc230cf9bfb4c2f2667f15c4d
Author: Marcus Christie <ma...@apache.org>
AuthorDate: Mon Aug 29 09:45:44 2022 -0400

    AIRAVATA-3649 debounce calls to queue settings calculator
---
 .../js/components/experiment/ExperimentEditor.vue  | 31 +++++++++++++---------
 1 file changed, 18 insertions(+), 13 deletions(-)

diff --git a/django_airavata/apps/workspace/static/django_airavata_workspace/js/components/experiment/ExperimentEditor.vue b/django_airavata/apps/workspace/static/django_airavata_workspace/js/components/experiment/ExperimentEditor.vue
index 57698d3b..42de7da7 100644
--- a/django_airavata/apps/workspace/static/django_airavata_workspace/js/components/experiment/ExperimentEditor.vue
+++ b/django_airavata/apps/workspace/static/django_airavata_workspace/js/components/experiment/ExperimentEditor.vue
@@ -189,6 +189,7 @@ import InputEditorContainer from "./input-editors/InputEditorContainer.vue";
 import { models, services } from "django-airavata-api";
 import { components, utils } from "django-airavata-common-ui";
 import WorkspaceNoticesManagementContainer from "../notices/WorkspaceNoticesManagementContainer";
+import _ from "lodash";
 
 export default {
   name: "edit-experiment",
@@ -348,21 +349,25 @@ export default {
     inputValueChanged: function () {
       this.localExperiment.evaluateInputDependencies();
     },
+    calculateQueueSettings: _.debounce(async function () {
+      const queueSettingsUpdate = await services.QueueSettingsCalculatorService.calculate(
+        {
+          lookup: this.appInterface.queueSettingsCalculatorId,
+          data: this.localExperiment,
+        },
+        { showSpinner: false }
+      );
+      // Override values in computationalResourceScheduling with the values
+      // returned from the queue settings calculator
+      Object.assign(
+        this.localExperiment.userConfigurationData
+          .computationalResourceScheduling,
+        queueSettingsUpdate
+      );
+    }, 500),
     async experimentChanged() {
       if (this.appInterface.queueSettingsCalculatorId) {
-        const queueSettingsUpdate = await services.QueueSettingsCalculatorService.calculate(
-          {
-            lookup: this.appInterface.queueSettingsCalculatorId,
-            data: this.localExperiment,
-          }
-        );
-        // Override values in computationalResourceScheduling with the values
-        // returned from the queue settings calculator
-        Object.assign(
-          this.localExperiment.userConfigurationData
-            .computationalResourceScheduling,
-          queueSettingsUpdate
-        );
+        this.calculateQueueSettings();
       }
     },
   },


[airavata-django-portal] 11/11: Merge branch 'AIRAVATA-3649' into develop

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

machristie pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git

commit 373c21f99c66f79853f189df267e775df77ec833
Merge: ba4cb026 15c7da95
Author: Marcus Christie <ma...@apache.org>
AuthorDate: Mon Aug 29 14:50:48 2022 -0400

    Merge branch 'AIRAVATA-3649' into develop

 .../applications/ApplicationInterfaceEditor.vue    |  51 ++++++++--
 ...icationsettings_queue_settings_calculator_id.py |  18 ++++
 django_airavata/apps/api/models.py                 |   1 +
 django_airavata/apps/api/serializers.py            |  42 ++++++---
 .../api/static/django_airavata_api/js/index.js     |   3 +
 .../js/models/ApplicationInterfaceDefinition.js    |   5 +
 .../js/models/QueueSettingsCalculator.js           |   9 ++
 .../django_airavata_api/js/service_config.js       |  15 +++
 django_airavata/apps/api/urls.py                   |   2 +
 django_airavata/apps/api/views.py                  |  31 +++++-
 .../js/components/experiment/ExperimentEditor.vue  |  58 +++++++++---
 .../components/experiment/QueueSettingsEditor.vue  |  72 +++++++++++---
 docs/dev/queue_settings_calculator.md              | 104 +++++++++++++++++++++
 mkdocs.yml                                         |   1 +
 requirements.txt                                   |   2 +-
 15 files changed, 370 insertions(+), 44 deletions(-)



[airavata-django-portal] 04/11: AIRAVATA-3649 Only include queueSettingsCalculatorId if it exists

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

machristie pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git

commit 2d355f570f87306560f6169de0d2f7dbc4d2eb1d
Author: Marcus Christie <ma...@apache.org>
AuthorDate: Thu Aug 25 14:57:06 2022 -0400

    AIRAVATA-3649 Only include queueSettingsCalculatorId if it exists
    
    This is to gracefully handle situation where queue
    settings calculator is unavailable (for example, if it isn't installed).
---
 django_airavata/apps/api/serializers.py | 10 ++++++++--
 django_airavata/apps/api/views.py       |  8 ++++----
 2 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/django_airavata/apps/api/serializers.py b/django_airavata/apps/api/serializers.py
index 5ed02848..6618f9e4 100644
--- a/django_airavata/apps/api/serializers.py
+++ b/django_airavata/apps/api/serializers.py
@@ -60,7 +60,11 @@ from airavata.model.workspace.ttypes import (
     NotificationPriority,
     Project
 )
-from airavata_django_portal_sdk import experiment_util, user_storage
+from airavata_django_portal_sdk import (
+    experiment_util,
+    queue_settings_calculators,
+    user_storage
+)
 from django.conf import settings
 from django.contrib.auth import get_user_model
 from django.urls import reverse
@@ -344,7 +348,9 @@ class ApplicationInterfaceDescriptionSerializer(
         application_settings, created = models.ApplicationSettings.objects.get_or_create(
             application_module_id=application_module_id)
         representation["showQueueSettings"] = application_settings.show_queue_settings
-        representation["queueSettingsCalculatorId"] = application_settings.queue_settings_calculator_id
+        # check that queue_settings_calculator_id exists
+        if queue_settings_calculators.exists(application_settings.queue_settings_calculator_id):
+            representation["queueSettingsCalculatorId"] = application_settings.queue_settings_calculator_id
         return representation
 
     def create(self, validated_data):
diff --git a/django_airavata/apps/api/views.py b/django_airavata/apps/api/views.py
index f1d8d6e0..051ba669 100644
--- a/django_airavata/apps/api/views.py
+++ b/django_airavata/apps/api/views.py
@@ -25,7 +25,7 @@ from airavata.model.group.ttypes import ResourcePermissionType
 from airavata.model.user.ttypes import Status
 from airavata_django_portal_sdk import (
     experiment_util,
-    queue_settings,
+    queue_settings_calculators,
     user_storage
 )
 from django.conf import settings
@@ -1872,10 +1872,10 @@ class QueueSettingsCalculatorViewSet(mixins.ListModelMixin, mixins.RetrieveModel
     serializer_class = serializers.QueueSettingsCalculatorSerializer
 
     def get_list(self):
-        return queue_settings.get_all()
+        return queue_settings_calculators.get_all()
 
     def get_instance(self, lookup_value):
-        calcs = queue_settings.get_all()
+        calcs = queue_settings_calculators.get_all()
         calc = [calc for calc in calcs if calc.id == lookup_value]
         if len(calc) == 0:
             return None
@@ -1887,5 +1887,5 @@ class QueueSettingsCalculatorViewSet(mixins.ListModelMixin, mixins.RetrieveModel
         serializer = self.get_serializer(data=request.data)
         serializer.is_valid(raise_exception=True)
         experiment_model = serializer.save()
-        result = queue_settings.calculate_queue_settings(pk, request, experiment_model)
+        result = queue_settings_calculators.calculate_queue_settings(pk, request, experiment_model)
         return Response(result)


[airavata-django-portal] 07/11: AIRAVATA-3649 Make queue settings readonly when calculator is configured

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

machristie pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git

commit b21eede520aa2d8fd76d70ba0ea7ce5329611001
Author: Marcus Christie <ma...@apache.org>
AuthorDate: Mon Aug 29 10:05:13 2022 -0400

    AIRAVATA-3649 Make queue settings readonly when calculator is configured
---
 .../components/experiment/QueueSettingsEditor.vue  | 72 ++++++++++++++++++----
 1 file changed, 60 insertions(+), 12 deletions(-)

diff --git a/django_airavata/apps/workspace/static/django_airavata_workspace/js/components/experiment/QueueSettingsEditor.vue b/django_airavata/apps/workspace/static/django_airavata_workspace/js/components/experiment/QueueSettingsEditor.vue
index da86e071..12aa9ef1 100644
--- a/django_airavata/apps/workspace/static/django_airavata_workspace/js/components/experiment/QueueSettingsEditor.vue
+++ b/django_airavata/apps/workspace/static/django_airavata_workspace/js/components/experiment/QueueSettingsEditor.vue
@@ -2,10 +2,14 @@
   <div v-if="showQueueSettings">
     <div class="row">
       <div class="col">
-        <div class="card border-default" :class="{ 'border-danger': !valid }">
+        <div
+          class="card border-default"
+          :class="{ 'border-danger': !valid, 'is-disabled': disabled }"
+        >
           <b-link
             @click="showConfiguration = !showConfiguration"
             class="card-link text-dark"
+            :disabled="disabled"
           >
             <div class="card-body">
               <h5 class="card-title mb-4">
@@ -106,7 +110,8 @@
                   Max Allowed Cores = {{ maxCPUCount
                   }}<template
                     v-if="
-                      selectedQueueDefault && selectedQueueDefault.cpuPerNode > 0
+                      selectedQueueDefault &&
+                      selectedQueueDefault.cpuPerNode > 0
                     "
                     >. There are {{ selectedQueueDefault.cpuPerNode }} cores per
                     node.
@@ -114,16 +119,47 @@
                 </div>
               </b-form-group>
             </div>
-            <div class="d-flex flex-column" v-if="selectedQueueDefault && selectedQueueDefault.cpuPerNode > 0">
-              <div class="flex-fill"
-                   style="border: 1px solid #6c757d;border-top-right-radius: 10px;margin-top: 51px;border-left-width: 0px;border-bottom-width: 0px;margin-right: 15px;"></div>
-              <b-button size="sm" pill variant="outline-secondary"
-                        v-on:click="enableNodeCountToCpuCheck = !enableNodeCountToCpuCheck">
-                <i v-if="enableNodeCountToCpuCheck" class="fa fa-lock" aria-hidden="true"></i>
+            <div
+              class="d-flex flex-column"
+              v-if="selectedQueueDefault && selectedQueueDefault.cpuPerNode > 0"
+            >
+              <div
+                class="flex-fill"
+                style="
+                  border: 1px solid #6c757d;
+                  border-top-right-radius: 10px;
+                  margin-top: 51px;
+                  border-left-width: 0px;
+                  border-bottom-width: 0px;
+                  margin-right: 15px;
+                "
+              ></div>
+              <b-button
+                size="sm"
+                pill
+                variant="outline-secondary"
+                v-on:click="
+                  enableNodeCountToCpuCheck = !enableNodeCountToCpuCheck
+                "
+              >
+                <i
+                  v-if="enableNodeCountToCpuCheck"
+                  class="fa fa-lock"
+                  aria-hidden="true"
+                ></i>
                 <i v-else class="fa fa-unlock" aria-hidden="true"></i>
               </b-button>
-              <div class="flex-fill"
-                   style="border: 1px solid #6c757d;border-bottom-right-radius: 10px;margin-bottom: 57px;border-left-width: 0px;border-top-width: 0px;margin-right: 15px;"></div>
+              <div
+                class="flex-fill"
+                style="
+                  border: 1px solid #6c757d;
+                  border-bottom-right-radius: 10px;
+                  margin-bottom: 57px;
+                  border-left-width: 0px;
+                  border-top-width: 0px;
+                  margin-right: 15px;
+                "
+              ></div>
             </div>
           </div>
           <b-form-group
@@ -337,6 +373,12 @@ export default {
         ? this.applicationInterface.showQueueSettings
         : false;
     },
+    disabled() {
+      return (
+        this.applicationInterface &&
+        !!this.applicationInterface.queueSettingsCalculatorId
+      );
+    },
   },
   methods: {
     queueChanged: function (queueName) {
@@ -449,7 +491,10 @@ export default {
       }
     },
     nodeCountChanged() {
-      if (this.enableNodeCountToCpuCheck && this.selectedQueueDefault.cpuPerNode > 0) {
+      if (
+        this.enableNodeCountToCpuCheck &&
+        this.selectedQueueDefault.cpuPerNode > 0
+      ) {
         const nodeCount = parseInt(this.data.nodeCount);
         this.data.totalCPUCount = Math.min(
           nodeCount * this.selectedQueueDefault.cpuPerNode,
@@ -458,7 +503,10 @@ export default {
       }
     },
     cpuCountChanged() {
-      if (this.enableNodeCountToCpuCheck && this.selectedQueueDefault.cpuPerNode > 0) {
+      if (
+        this.enableNodeCountToCpuCheck &&
+        this.selectedQueueDefault.cpuPerNode > 0
+      ) {
         const cpuCount = parseInt(this.data.totalCPUCount);
         if (cpuCount > 0) {
           this.data.nodeCount = Math.min(


[airavata-django-portal] 08/11: AIRAVATA-3649 Ignore invalid experiment models

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

machristie pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git

commit 61d181f87bb605c9268e026910b507f2a63a5b28
Author: Marcus Christie <ma...@apache.org>
AuthorDate: Mon Aug 29 11:03:09 2022 -0400

    AIRAVATA-3649 Ignore invalid experiment models
---
 django_airavata/apps/api/views.py | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/django_airavata/apps/api/views.py b/django_airavata/apps/api/views.py
index 051ba669..75fe425e 100644
--- a/django_airavata/apps/api/views.py
+++ b/django_airavata/apps/api/views.py
@@ -1885,7 +1885,9 @@ class QueueSettingsCalculatorViewSet(mixins.ListModelMixin, mixins.RetrieveModel
     def calculate(self, request, pk=None):
 
         serializer = self.get_serializer(data=request.data)
-        serializer.is_valid(raise_exception=True)
-        experiment_model = serializer.save()
-        result = queue_settings_calculators.calculate_queue_settings(pk, request, experiment_model)
+        result = {}
+        # Just ignore invalid experiment model since likely caused by late initialization
+        if serializer.is_valid():
+            experiment_model = serializer.save()
+            result = queue_settings_calculators.calculate_queue_settings(pk, request, experiment_model)
         return Response(result)


[airavata-django-portal] 01/11: AIRAVATA-3649 REST API to list and invoke queue settings calculators

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

machristie pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git

commit c41f65494a091929700384fe523e1f81eba40e64
Author: Marcus Christie <ma...@apache.org>
AuthorDate: Tue Aug 23 17:26:17 2022 -0400

    AIRAVATA-3649 REST API to list and invoke queue settings calculators
---
 django_airavata/apps/api/serializers.py |  5 +++++
 django_airavata/apps/api/urls.py        |  2 ++
 django_airavata/apps/api/views.py       | 29 ++++++++++++++++++++++++++++-
 3 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/django_airavata/apps/api/serializers.py b/django_airavata/apps/api/serializers.py
index 84d32b5b..6780b3c3 100644
--- a/django_airavata/apps/api/serializers.py
+++ b/django_airavata/apps/api/serializers.py
@@ -1162,3 +1162,8 @@ class SettingsSerializer(serializers.Serializer):
     fileUploadMaxFileSize = serializers.IntegerField()
     tusEndpoint = serializers.CharField()
     pgaUrl = serializers.CharField()
+
+
+class QueueSettingsCalculatorSerializer(serializers.Serializer):
+    id = serializers.CharField()
+    name = serializers.CharField()
diff --git a/django_airavata/apps/api/urls.py b/django_airavata/apps/api/urls.py
index 61b27177..f655d648 100644
--- a/django_airavata/apps/api/urls.py
+++ b/django_airavata/apps/api/urls.py
@@ -45,6 +45,8 @@ router.register(r'iam-user-profiles', views.IAMUserViewSet,
                 basename='iam-user-profile')
 router.register(r'unverified-email-users', views.UnverifiedEmailUserViewSet,
                 basename='unverified-email-user-profile')
+router.register(r'queue-settings-calculators', views.QueueSettingsCalculatorViewSet,
+                basename='queue-settings-calculator')
 
 app_name = 'django_airavata_api'
 urlpatterns = [
diff --git a/django_airavata/apps/api/views.py b/django_airavata/apps/api/views.py
index 79816b3c..f1d8d6e0 100644
--- a/django_airavata/apps/api/views.py
+++ b/django_airavata/apps/api/views.py
@@ -23,7 +23,11 @@ from airavata.model.data.movement.ttypes import (
 from airavata.model.experiment.ttypes import ExperimentSearchFields
 from airavata.model.group.ttypes import ResourcePermissionType
 from airavata.model.user.ttypes import Status
-from airavata_django_portal_sdk import experiment_util, user_storage
+from airavata_django_portal_sdk import (
+    experiment_util,
+    queue_settings,
+    user_storage
+)
 from django.conf import settings
 from django.contrib.auth import get_user_model
 from django.core.exceptions import ObjectDoesNotExist, PermissionDenied
@@ -1862,3 +1866,26 @@ def _generate_output_view_data(request):
                                       experiment_id,
                                       test_mode=test_mode,
                                       **params.dict())
+
+
+class QueueSettingsCalculatorViewSet(mixins.ListModelMixin, mixins.RetrieveModelMixin, GenericAPIBackedViewSet):
+    serializer_class = serializers.QueueSettingsCalculatorSerializer
+
+    def get_list(self):
+        return queue_settings.get_all()
+
+    def get_instance(self, lookup_value):
+        calcs = queue_settings.get_all()
+        calc = [calc for calc in calcs if calc.id == lookup_value]
+        if len(calc) == 0:
+            return None
+        return calc[0]
+
+    @action(methods=['post'], detail=True, serializer_class=serializers.ExperimentSerializer)
+    def calculate(self, request, pk=None):
+
+        serializer = self.get_serializer(data=request.data)
+        serializer.is_valid(raise_exception=True)
+        experiment_model = serializer.save()
+        result = queue_settings.calculate_queue_settings(pk, request, experiment_model)
+        return Response(result)


[airavata-django-portal] 06/11: AIRAVATA-3649 Only re-calculate queue settings when inputs or compute resource change

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

machristie pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git

commit 6a11894c7656e851dae8b7b2bd761860051bcb7d
Author: Marcus Christie <ma...@apache.org>
AuthorDate: Mon Aug 29 09:56:48 2022 -0400

    AIRAVATA-3649 Only re-calculate queue settings when inputs or compute resource change
---
 .../js/components/experiment/ExperimentEditor.vue       | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/django_airavata/apps/workspace/static/django_airavata_workspace/js/components/experiment/ExperimentEditor.vue b/django_airavata/apps/workspace/static/django_airavata_workspace/js/components/experiment/ExperimentEditor.vue
index 42de7da7..4485733e 100644
--- a/django_airavata/apps/workspace/static/django_airavata_workspace/js/components/experiment/ExperimentEditor.vue
+++ b/django_airavata/apps/workspace/static/django_airavata_workspace/js/components/experiment/ExperimentEditor.vue
@@ -365,7 +365,12 @@ export default {
         queueSettingsUpdate
       );
     }, 500),
-    async experimentChanged() {
+    experimentInputsChanged() {
+      if (this.appInterface.queueSettingsCalculatorId) {
+        this.calculateQueueSettings();
+      }
+    },
+    resourceHostIdChanged() {
       if (this.appInterface.queueSettingsCalculatorId) {
         this.calculateQueueSettings();
       }
@@ -377,11 +382,19 @@ export default {
     },
     localExperiment: {
       handler() {
-        this.experimentChanged();
         this.edited = true;
       },
       deep: true,
     },
+    "experiment.experimentInputs": {
+      handler() {
+        this.experimentInputsChanged();
+      },
+      deep: true,
+    },
+    "experiment.userConfigurationData.computationalResourceScheduling.resourceHostId": function () {
+      this.resourceHostIdChanged();
+    },
   },
 };
 </script>


[airavata-django-portal] 02/11: AIRAVATA-3649 DB field and Admin UI for queueSettingsCalculatorId

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

machristie pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git

commit 2319627de69a66eaf0f1543e256c155f1336df60
Author: Marcus Christie <ma...@apache.org>
AuthorDate: Thu Aug 25 11:10:56 2022 -0400

    AIRAVATA-3649 DB field and Admin UI for queueSettingsCalculatorId
---
 .../applications/ApplicationInterfaceEditor.vue    | 51 +++++++++++++++++++---
 ...icationsettings_queue_settings_calculator_id.py | 18 ++++++++
 django_airavata/apps/api/models.py                 |  1 +
 django_airavata/apps/api/serializers.py            | 29 +++++++-----
 .../api/static/django_airavata_api/js/index.js     |  3 ++
 .../js/models/ApplicationInterfaceDefinition.js    |  5 +++
 .../js/models/QueueSettingsCalculator.js           |  9 ++++
 .../django_airavata_api/js/service_config.js       | 13 ++++++
 8 files changed, 111 insertions(+), 18 deletions(-)

diff --git a/django_airavata/apps/admin/static/django_airavata_admin/src/components/applications/ApplicationInterfaceEditor.vue b/django_airavata/apps/admin/static/django_airavata_admin/src/components/applications/ApplicationInterfaceEditor.vue
index c33c8c46..3114370f 100644
--- a/django_airavata/apps/admin/static/django_airavata_admin/src/components/applications/ApplicationInterfaceEditor.vue
+++ b/django_airavata/apps/admin/static/django_airavata_admin/src/components/applications/ApplicationInterfaceEditor.vue
@@ -37,6 +37,22 @@
             cores, walltime limit).
           </div>
         </b-form-group>
+        <b-form-group
+          label="Queue Settings Calculator"
+          description="Select function to automatically compute queue settings."
+        >
+          <b-form-select
+            v-model="data.queueSettingsCalculatorId"
+            :options="queueSettingsCalculatorOptions"
+            :disabled="queueSettingsCalculatorOptions.length === 0"
+          >
+            <template slot="first">
+              <option :value="null">
+                If applicable, select a queue settings calculator
+              </option>
+            </template>
+          </b-form-select>
+        </b-form-group>
       </div>
     </div>
     <div class="w-100">
@@ -45,9 +61,13 @@
         label-for="application-description"
       >
         <b-form-textarea
-          id="application-description" :rows="5"
+          id="application-description"
+          :rows="5"
           v-model="data.applicationDescription"
-          :state="!data.applicationDescription || data.applicationDescription.length < 500"
+          :state="
+            !data.applicationDescription ||
+            data.applicationDescription.length < 500
+          "
         >
         </b-form-textarea>
         <b-form-valid-feedback v-if="!!data.applicationDescription">
@@ -120,8 +140,8 @@
 </template>
 
 <script>
-import {models} from "django-airavata-api";
-import {mixins} from "django-airavata-common-ui";
+import { models, services } from "django-airavata-api";
+import { mixins } from "django-airavata-common-ui";
 import ApplicationInputFieldEditor from "./ApplicationInputFieldEditor.vue";
 import ApplicationOutputFieldEditor from "./ApplicationOutputFieldEditor.vue";
 
@@ -144,13 +164,28 @@ export default {
     ApplicationOutputFieldEditor,
     draggable,
   },
+  created() {
+    this.loadQueueSettingsCalculators();
+  },
   computed: {
     trueFalseOptions() {
       return [
-        {text: "True", value: true},
-        {text: "False", value: false},
+        { text: "True", value: true },
+        { text: "False", value: false },
       ];
     },
+    queueSettingsCalculatorOptions() {
+      if (this.queueSettingsCalculators) {
+        return this.queueSettingsCalculators.map((qsc) => {
+          return {
+            text: qsc.name,
+            value: qsc.id,
+          };
+        });
+      } else {
+        return [];
+      }
+    },
   },
   data() {
     return {
@@ -160,6 +195,7 @@ export default {
         handle: ".drag-handle",
       },
       collapseApplicationInputs: false,
+      queueSettingsCalculators: null,
     };
   },
   methods: {
@@ -209,6 +245,9 @@ export default {
     onDragEnd() {
       this.collapseApplicationInputs = false;
     },
+    async loadQueueSettingsCalculators() {
+      this.queueSettingsCalculators = await services.QueueSettingsCalculatorService.list();
+    },
   },
 };
 </script>
diff --git a/django_airavata/apps/api/migrations/0009_applicationsettings_queue_settings_calculator_id.py b/django_airavata/apps/api/migrations/0009_applicationsettings_queue_settings_calculator_id.py
new file mode 100644
index 00000000..bc7bf8fb
--- /dev/null
+++ b/django_airavata/apps/api/migrations/0009_applicationsettings_queue_settings_calculator_id.py
@@ -0,0 +1,18 @@
+# Generated by Django 3.2.15 on 2022-08-25 13:34
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('django_airavata_api', '0008_merge_20220601_1951'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='applicationsettings',
+            name='queue_settings_calculator_id',
+            field=models.CharField(max_length=255, null=True),
+        ),
+    ]
diff --git a/django_airavata/apps/api/models.py b/django_airavata/apps/api/models.py
index 829c88fa..45c88164 100644
--- a/django_airavata/apps/api/models.py
+++ b/django_airavata/apps/api/models.py
@@ -72,3 +72,4 @@ class ApplicationTemplateContextProcessor(models.Model):
 class ApplicationSettings(models.Model):
     application_module_id = models.CharField(max_length=255, unique=True)
     show_queue_settings = models.BooleanField(default=True)
+    queue_settings_calculator_id = models.CharField(max_length=255, null=True)
diff --git a/django_airavata/apps/api/serializers.py b/django_airavata/apps/api/serializers.py
index 6780b3c3..5ed02848 100644
--- a/django_airavata/apps/api/serializers.py
+++ b/django_airavata/apps/api/serializers.py
@@ -336,6 +336,7 @@ class ApplicationInterfaceDescriptionSerializer(
     applicationOutputs = OutputDataObjectTypeSerializer(many=True)
     userHasWriteAccess = serializers.SerializerMethodField()
     showQueueSettings = serializers.BooleanField(required=False)
+    queueSettingsCalculatorId = serializers.CharField(allow_null=True, required=False)
 
     def to_representation(self, instance):
         representation = super().to_representation(instance)
@@ -343,28 +344,32 @@ class ApplicationInterfaceDescriptionSerializer(
         application_settings, created = models.ApplicationSettings.objects.get_or_create(
             application_module_id=application_module_id)
         representation["showQueueSettings"] = application_settings.show_queue_settings
+        representation["queueSettingsCalculatorId"] = application_settings.queue_settings_calculator_id
         return representation
 
     def create(self, validated_data):
-        showQueueSettings = validated_data.pop("showQueueSettings", None)
+        showQueueSettings = validated_data.pop("showQueueSettings", True)
+        queueSettingsCalculatorId = validated_data.pop("queueSettingsCalculatorId", None)
         application_interface = super().create(validated_data)
         application_module_id = application_interface.applicationModules[0]
-        if showQueueSettings is not None:
-            models.ApplicationSettings.objects.update_or_create(
-                application_module_id=application_module_id,
-                defaults={"show_queue_settings": showQueueSettings}
-            )
+        models.ApplicationSettings.objects.update_or_create(
+            application_module_id=application_module_id,
+            defaults={"show_queue_settings": showQueueSettings,
+                      "queue_settings_calculator_id": queueSettingsCalculatorId}
+        )
         return application_interface
 
     def update(self, instance, validated_data):
-        showQueueSettings = validated_data.pop("showQueueSettings", None)
+        defaults = {}
+        if "showQueueSettings" in validated_data:
+            defaults["show_queue_settings"] = validated_data.pop("showQueueSettings")
+        if "queueSettingsCalculatorId" in validated_data:
+            defaults["queue_settings_calculator_id"] = validated_data.pop("queueSettingsCalculatorId")
         application_interface = super().update(instance, validated_data)
         application_module_id = application_interface.applicationModules[0]
-        if showQueueSettings is not None:
-            models.ApplicationSettings.objects.update_or_create(
-                application_module_id=application_module_id,
-                defaults={"show_queue_settings": showQueueSettings}
-            )
+        models.ApplicationSettings.objects.update_or_create(
+            application_module_id=application_module_id, defaults=defaults
+        )
         return application_interface
 
     def get_userHasWriteAccess(self, appDeployment):
diff --git a/django_airavata/apps/api/static/django_airavata_api/js/index.js b/django_airavata/apps/api/static/django_airavata_api/js/index.js
index ca08f1d7..d57713d3 100644
--- a/django_airavata/apps/api/static/django_airavata_api/js/index.js
+++ b/django_airavata/apps/api/static/django_airavata_api/js/index.js
@@ -135,6 +135,9 @@ const services = {
 
   ParserService: ServiceFactory.service("Parsers"),
   ProjectService: ServiceFactory.service("Projects"),
+  QueueSettingsCalculatorService: ServiceFactory.service(
+    "QueueSettingsCalculators"
+  ),
   SCPDataMovementService,
   ServiceFactory,
   SettingsService: ServiceFactory.service("Settings"),
diff --git a/django_airavata/apps/api/static/django_airavata_api/js/models/ApplicationInterfaceDefinition.js b/django_airavata/apps/api/static/django_airavata_api/js/models/ApplicationInterfaceDefinition.js
index 008f3a0a..d75164cb 100644
--- a/django_airavata/apps/api/static/django_airavata_api/js/models/ApplicationInterfaceDefinition.js
+++ b/django_airavata/apps/api/static/django_airavata_api/js/models/ApplicationInterfaceDefinition.js
@@ -44,6 +44,11 @@ const FIELDS = [
     type: "boolean",
     default: true,
   },
+  {
+    name: "queueSettingsCalculatorId",
+    type: "string",
+    default: null,
+  },
 ];
 
 export default class ApplicationInterfaceDefinition extends BaseModel {
diff --git a/django_airavata/apps/api/static/django_airavata_api/js/models/QueueSettingsCalculator.js b/django_airavata/apps/api/static/django_airavata_api/js/models/QueueSettingsCalculator.js
new file mode 100644
index 00000000..4396d74f
--- /dev/null
+++ b/django_airavata/apps/api/static/django_airavata_api/js/models/QueueSettingsCalculator.js
@@ -0,0 +1,9 @@
+import BaseModel from "./BaseModel";
+
+const FIELDS = ["id", "name"];
+
+export default class QueueSettingsCalculator extends BaseModel {
+  constructor(data = {}) {
+    super(FIELDS, data);
+  }
+}
diff --git a/django_airavata/apps/api/static/django_airavata_api/js/service_config.js b/django_airavata/apps/api/static/django_airavata_api/js/service_config.js
index 95ee7ac8..0eb915c4 100644
--- a/django_airavata/apps/api/static/django_airavata_api/js/service_config.js
+++ b/django_airavata/apps/api/static/django_airavata_api/js/service_config.js
@@ -19,6 +19,7 @@ import LogRecord from "./models/LogRecord";
 import Notification from "./models/Notification";
 import Parser from "./models/Parser";
 import Project from "./models/Project";
+import QueueSettingsCalculator from "./models/QueueSettingsCalculator";
 import Settings from "./models/Settings";
 import SharedEntity from "./models/SharedEntity";
 import StoragePreference from "./models/StoragePreference";
@@ -341,6 +342,18 @@ export default {
     queryParams: ["limit", "offset"],
     modelClass: Project,
   },
+  QueueSettingsCalculators: {
+    url: "/api/queue-settings-calculators",
+    viewSet: ["retrieve", "list"],
+    methods: {
+      calculate: {
+        url: "/api/queue-settings-calculators/<lookup>/calculate/",
+        requestType: "post",
+        modelClass: QueueSettingsCalculator,
+      },
+    },
+    modelClass: QueueSettingsCalculator,
+  },
   Settings: {
     url: "/api/settings/",
     methods: {


[airavata-django-portal] 09/11: AIRAVATA-3649 Queue settings calculator docs

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

machristie pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git

commit cf64de703d9b3ea73bf729b96f3aef6ddfc35f13
Author: Marcus Christie <ma...@apache.org>
AuthorDate: Mon Aug 29 14:42:59 2022 -0400

    AIRAVATA-3649 Queue settings calculator docs
---
 docs/dev/queue_settings_calculator.md | 104 ++++++++++++++++++++++++++++++++++
 mkdocs.yml                            |   1 +
 2 files changed, 105 insertions(+)

diff --git a/docs/dev/queue_settings_calculator.md b/docs/dev/queue_settings_calculator.md
new file mode 100644
index 00000000..55073f8b
--- /dev/null
+++ b/docs/dev/queue_settings_calculator.md
@@ -0,0 +1,104 @@
+# Queue Settings Calculator
+
+A _Queue Settings Calculator_ is a Python function that computes the queue
+settings for an experiment. The function takes an instance of an experiment and
+returns:
+
+-   queue name
+-   core count
+-   node count
+-   walltime limit
+
+The Airavata Django Portal then uses these values to populate the Queue Settings
+fields when users are creating/editing experiments. This can greatly simplify
+experiment configuration for users and also lead to a better use of resources.
+
+## Getting started
+
+To add a Queue Settings Calculator function, you first will need a custom Django
+App. See the [Custom Django App](./custom_django_app.md) notes for how to create
+one.
+
+Next, add a `queue_settings_calculators.py` file to your custom Django app
+package if it doesn't already exist. You should add this file to the same folder
+that contains the `apps.py` file.
+
+In this file, you'll need to import the `@queue_settings_calculators` decorator.
+You'll use this to mark the queue settings calculator functions (you can define
+more than one).
+
+```python
+from airavata.model.experiment.ttypes import ExperimentModel
+from airavata_django_portal_sdk.decorators import queue_settings_calculator
+
+@queue_settings_calculator(
+    id="gateway_name-queue-settings-for-my-app", name="My Gateway: Queue Settings for My App"
+)
+def my_queue_settings_calculator(request, experiment_model: ExperimentModel):
+    # See https://airavata.apache.org/api-docs/master/experiment_model.html#Struct_ExperimentModel for ExperimentModel fields
+
+    total_core_count = 4
+    queue_name = "shared"
+    node_count = 1
+    walltime_limit = 30
+
+    # Return a dictionary with the queue settings values
+    result = {}
+    result["totalCPUCount"] = total_core_count
+    result["queueName"] = queue_name
+    result["nodeCount"] = node_count
+    result["wallTimeLimit"] = walltime_limit
+    return result
+```
+
+The `id` and `name` that are passed to the `@queue_settings_calculator`
+decorator are optional but highly recommended. The `id` will be used internally
+to associate applications with this function. Set the id to something that is
+unique to your gateway. The `name` is the value that will be displayed in the
+Settings UI for selecting this queue settings calculator.
+
+The queue settings calculator function is passed the Django `request` object and
+the `experiment_model`, an
+[ExperimentModel](https://airavata.apache.org/api-docs/master/experiment_model.html#Struct_ExperimentModel)
+instance.
+
+Primarily your function will inspect the `experiment_model` to determine the
+appropriate queue settings. For example you might look at one of the
+experiment's input files to determine how many cores are optimal for the job.
+
+Next, add the following import line to the `ready()` function of your apps.py
+AppConfig class. It will look something like this although the names of your
+AppConfig class and packages will be different:
+
+```python
+class CustomDjangoAppConfig(AppConfig):
+    name = 'custom_django_app'
+    label = name
+    verbose_name = "Custom Django App"
+    fa_icon_class = "fa-comment"
+    url_home = "custom_django_app:hello_world"
+
+    def ready(self) -> None:
+        from custom_django_app import queue_settings_calculators  # noqa
+```
+
+Add the `ready(self)` function to your AppConfig if it is missing. Then add an
+import of your queue_settings_calculators module. Importing the module will
+register the calculator functions at startup time.
+
+## Configuring an application's queue settings calculator
+
+To have one of your applications use your queue settings calculator, first make
+sure to install the custom Django app in your portal instance. Then, go to
+**Settings > Application Catalog** and click on your application. Select the
+**Interface** tab. Under _Queue Settings Calculator_, select your queue settings
+calculator from the drop down. It should be listed with the name that you gave
+to it in your function's decorator
+(`@queue_settings_calculator(id='...', name='...')`).
+
+When an application is configured to use a queue settings calculator, the Queue
+Settings UI in the Create/Edit Experiment views is disabled and made read only.
+
+## Examples
+
+-   <https://github.com/bio-miga/miga-autocomplete>
diff --git a/mkdocs.yml b/mkdocs.yml
index 3a0b01ec..9ecc3b0b 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -16,6 +16,7 @@ nav:
     - Custom UI Tutorial: tutorial/custom_ui_tutorial.md
     - Adding a Custom Django App: dev/custom_django_app.md
     - Adding a Custom Output View Provider: dev/custom_output_view_provider.md
+    - Adding a Queue Settings Calculator: dev/queue_settings_calculator.md
     - HOWTOs: dev/customization_howto.md
   - Administrator Guide:
     - Application Input Customization: admin/app_inputs.md


[airavata-django-portal] 03/11: AIRAVATA-3649 Apply queue settings calculator values in experiment editor

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

machristie pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git

commit 68e425fe5bd636677150a39938f44b91c1ef2d5c
Author: Marcus Christie <ma...@apache.org>
AuthorDate: Thu Aug 25 12:52:15 2022 -0400

    AIRAVATA-3649 Apply queue settings calculator values in experiment editor
---
 .../django_airavata_api/js/service_config.js       |  4 ++-
 .../js/components/experiment/ExperimentEditor.vue  | 40 ++++++++++++++++------
 2 files changed, 32 insertions(+), 12 deletions(-)

diff --git a/django_airavata/apps/api/static/django_airavata_api/js/service_config.js b/django_airavata/apps/api/static/django_airavata_api/js/service_config.js
index 0eb915c4..e12d6ce3 100644
--- a/django_airavata/apps/api/static/django_airavata_api/js/service_config.js
+++ b/django_airavata/apps/api/static/django_airavata_api/js/service_config.js
@@ -349,7 +349,9 @@ export default {
       calculate: {
         url: "/api/queue-settings-calculators/<lookup>/calculate/",
         requestType: "post",
-        modelClass: QueueSettingsCalculator,
+        bodyParams: {
+          name: "data",
+        },
       },
     },
     modelClass: QueueSettingsCalculator,
diff --git a/django_airavata/apps/workspace/static/django_airavata_workspace/js/components/experiment/ExperimentEditor.vue b/django_airavata/apps/workspace/static/django_airavata_workspace/js/components/experiment/ExperimentEditor.vue
index 79e04a25..57698d3b 100644
--- a/django_airavata/apps/workspace/static/django_airavata_workspace/js/components/experiment/ExperimentEditor.vue
+++ b/django_airavata/apps/workspace/static/django_airavata_workspace/js/components/experiment/ExperimentEditor.vue
@@ -1,6 +1,6 @@
 <template>
   <div>
-    <unsaved-changes-guard :dirty="dirty"/>
+    <unsaved-changes-guard :dirty="dirty" />
     <div class="row">
       <div class="col-auto mr-auto">
         <h1 class="h4 mb-4">
@@ -89,9 +89,13 @@
       </div>
       <div class="row">
         <div class="col">
-          <workspace-notices-management-container class="mt-2"
+          <workspace-notices-management-container
+            class="mt-2"
             v-if="appInterface && appInterface.applicationDescription"
-            :data="[{notificationMessage: appInterface.applicationDescription}]"/>
+            :data="[
+              { notificationMessage: appInterface.applicationDescription },
+            ]"
+          />
         </div>
       </div>
       <div class="row">
@@ -148,12 +152,8 @@
       </div>
       <div class="row">
         <div class="col">
-          <b-form-group
-            label="Email Settings"
-          >
-            <b-form-checkbox
-              v-model="localExperiment.enableEmailNotification"
-            >
+          <b-form-group label="Email Settings">
+            <b-form-checkbox v-model="localExperiment.enableEmailNotification">
               Receive email notification of experiment status
             </b-form-checkbox>
           </b-form-group>
@@ -186,8 +186,8 @@ import ComputationalResourceSchedulingEditor from "./ComputationalResourceSchedu
 import ExperimentDescriptionEditor from "./ExperimentDescriptionEditor.vue";
 import GroupResourceProfileSelector from "./GroupResourceProfileSelector.vue";
 import InputEditorContainer from "./input-editors/InputEditorContainer.vue";
-import {models, services} from "django-airavata-api";
-import {components, utils} from "django-airavata-common-ui";
+import { models, services } from "django-airavata-api";
+import { components, utils } from "django-airavata-common-ui";
 import WorkspaceNoticesManagementContainer from "../notices/WorkspaceNoticesManagementContainer";
 
 export default {
@@ -348,6 +348,23 @@ export default {
     inputValueChanged: function () {
       this.localExperiment.evaluateInputDependencies();
     },
+    async experimentChanged() {
+      if (this.appInterface.queueSettingsCalculatorId) {
+        const queueSettingsUpdate = await services.QueueSettingsCalculatorService.calculate(
+          {
+            lookup: this.appInterface.queueSettingsCalculatorId,
+            data: this.localExperiment,
+          }
+        );
+        // Override values in computationalResourceScheduling with the values
+        // returned from the queue settings calculator
+        Object.assign(
+          this.localExperiment.userConfigurationData
+            .computationalResourceScheduling,
+          queueSettingsUpdate
+        );
+      }
+    },
   },
   watch: {
     experiment: function (newValue) {
@@ -355,6 +372,7 @@ export default {
     },
     localExperiment: {
       handler() {
+        this.experimentChanged();
         this.edited = true;
       },
       deep: true,