You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dlab.apache.org by of...@apache.org on 2020/03/04 11:31:50 UTC

[incubator-dlab] branch DLAB-1571 updated (3713da8 -> 375d4b5)

This is an automated email from the ASF dual-hosted git repository.

ofuks pushed a change to branch DLAB-1571
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git.


    from 3713da8  Refactoring
     add 4e8eae2  DLAB-1517 added usage of ssn_nginx.conf file
     add 1345909  DLAB-1517 added nginx user creation
     add 2239cca  DLAB-1517 changed cp path for nginx.conf
     add 7ce3069  Merge pull request #619 from apache/DLAB-1517
     add 868e4b0  [DLAB-625] Created Jupyterlab template for Azure
     add acd8932  [DLAB-625] Bugfix
     add 04399b8  [DLAB-625] Fixed minor bug and updated jupyterlab version
     add 0e8c625  Merge pull request #620 from apache/DLAB-625
     add 5ac08a0  small style fix (#622)
     add b214927  [DLAB-1570]: Rarely notebook creation fails on stage of 'npm' installation fixed
     add f67b8b1  [DLAB-1570]: Rarely notebook creation fails on stage of 'npm' installation fixed
     add 68b9825  Merge pull request #623 from apache/DLAB-1570
     add bb069fe  [DLAB-1570]: returned name of the function
     add 8eb6f29  Merge pull request #624 from apache/DLAB-1570
     add 31b5cc2  [DLAB-1570]: 'npm' fixes updated
     add 8c2548f  Merge pull request #625 from apache/DLAB-1570
     add 5a644ac  'npm' install fixes
     add c031f1a  [DLAB-1468]: disables shared image enabled variable in dlab.ini and deploy dlab script
     add 1e5e8cc  [DLAB-1468]: disables shared image enabled variable in ssn_configure
     add 88572a5  [DLAB-1468]: removed usage of shared_image_enabled during ssn creation
     add 5b55d74  [DLAB-1468]: removed shared_image_enabled from README.md
     add e9400ca  Merge pull request #627 from apache/DLAB-1468
     add 76a0970  [DLAB-1595]: added features block
     add fd8506d  [DLAB-1595]: added endpoint network interface to security group association
     add dcbb778  [DLAB-1595]: added dependency for nif-sg association
     add 32f4456  [DLAB-1595]: changed nif-sg dependency
     add 8fb14dd  [DLAB-1595]: added additional dependencies for testing
     add 87cd4da3 [DLAB-1595]: removed dependencies
     add ec72a18  [DLAB-1595]: added dependency for testing
     add 6975cf0  Merge pull request #628 from apache/DLAB-1595
     add d2eb326  [DLAB-11485]: test commit for ssn fail
     add f51318f  [DLAB-11485]: test commit for ssn fail
     add e1a5989  [DLAB-11485]: [GCP] - SSN creation deletes predefined VPC/subnet in case of SSN creation fails fixed
     add 3b14af2  [DLAB-11485]: [GCP] - SSN fail for test
     add 01780c1  [DLAB-11485]: [GCP] - SSN fail for test 1
     add e3e9ace  [DLAB-11485]: [GCP] - SSN fail for test 2
     add 55a99a2  [DLAB-11485]: [GCP] - SSN fail for test 2
     add 017ddd0  [DLAB-11485]: [GCP] - SSN fail for test 3
     add 132ea90  [DLAB-11485]: [GCP] - SSN creation deletes predefined VPC/subnet in case of SSN creation fails fixed
     add 99adcb0  [DLAB-11485]: final changes
     add 8e734ea  Merge pull request #629 from apache/DLAB-1485
     add cfff604  [DLAB-1579]: Added confirmation dialog with information about termina… (#630)
     add 6129baf  [DLAB-1539]: changed user tag for notebooks
     add 907fcd6  [DLAB-1539]: added user tag and custom tag to notebook labels
     add ccdb2f6  [DLAB-1539]: removed prints from get_index... function in gcp meta_lib
     add c3a6c49  [DLAB-1539]: added user_tag for dataengine
     add 51afde3  Merge pull request #631 from apache/DLAB-1539
     new d383c85  Merge branch 'develop' into DLAB-1571
     new 375d4b5  Added Get API by dlabIds

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 README.md                                          |  3 -
 infrastructure-provisioning/scripts/deploy_dlab.py |  1 -
 .../src/general/conf/dlab.ini                      |  4 +-
 .../{jupyter_Dockerfile => jupyterlab_Dockerfile}  |  4 +-
 ...escription.json => jupyterlab_description.json} |  6 +-
 .../general/files/gcp/jupyterlab_description.json  |  2 +-
 .../src/general/lib/gcp/meta_lib.py                |  4 --
 .../src/general/lib/os/debian/common_lib.py        |  1 +
 .../src/general/lib/os/debian/ssn_lib.py           |  4 ++
 .../src/general/lib/os/fab.py                      | 27 +++++++-
 .../src/general/scripts/aws/ssn_configure.py       |  4 --
 ...upyter_configure.py => jupyterlab_configure.py} | 77 ++++++++++++++--------
 .../src/general/scripts/azure/ssn_configure.py     |  5 +-
 ...common_notebook_configure_dataengine-service.py | 12 +++-
 .../general/scripts/gcp/common_prepare_notebook.py | 18 +++--
 .../scripts/gcp/dataengine-service_prepare.py      | 13 +++-
 .../src/general/scripts/gcp/dataengine_prepare.py  | 15 ++++-
 .../src/general/scripts/gcp/ssn_configure.py       | 68 +++++++------------
 .../src/general/scripts/gcp/ssn_prepare.py         | 39 +++++------
 .../jenkins_jobs/create_notebook_image/config.xml  |  1 +
 .../jenkins_jobs/create_notebook_server/config.xml |  1 +
 .../jenkins_jobs/manage_git_credentials/config.xml |  1 +
 .../jenkins_jobs/start_notebook_server/config.xml  |  1 +
 .../jenkins_jobs/stop_notebook_server/config.xml   |  1 +
 .../terminate_notebook_image/config.xml            |  1 +
 .../terminate_notebook_server/config.xml           |  1 +
 .../src/jupyterlab/Dockerfile_jupyterlab           |  8 ++-
 .../scripts/configure_jupyterlab_node.py           |  7 --
 .../src/ssn/scripts/configure_ui.py                |  2 +-
 .../terraform/azure/endpoint/main/main.tf          |  1 +
 .../terraform/azure/endpoint/main/network.tf       |  7 +-
 .../billing/gcp/controller/BillingController.java  |  6 ++
 .../com/epam/dlab/billing/gcp/dao/BillingDAO.java  |  2 +
 .../billing/gcp/dao/impl/BigQueryBillingDAO.java   | 28 ++++++--
 .../project-list/project-list.component.html       |  2 +-
 .../app/administration/roles/roles.component.ts    | 50 +++++++++++---
 .../reporting-grid/reporting-grid.component.html   |  7 +-
 .../src/app/reporting/reporting.component.ts       |  4 +-
 .../resources-grid/resources-grid.component.scss   |  1 +
 .../confirmation-dialog-type.enum.ts               |  3 +-
 .../confirmation-dialog.component.html             |  9 +--
 .../webapp/src/assets/styles/_dialogs.scss         |  6 ++
 .../webapp/src/dictionary/azure.dictionary.ts      |  6 +-
 43 files changed, 294 insertions(+), 169 deletions(-)
 copy infrastructure-provisioning/src/general/files/azure/{jupyter_Dockerfile => jupyterlab_Dockerfile} (96%)
 copy infrastructure-provisioning/src/general/files/azure/{rstudio_description.json => jupyterlab_description.json} (84%)
 copy infrastructure-provisioning/src/general/scripts/azure/{jupyter_configure.py => jupyterlab_configure.py} (90%)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@dlab.apache.org
For additional commands, e-mail: commits-help@dlab.apache.org


[incubator-dlab] 02/02: Added Get API by dlabIds

Posted by of...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

ofuks pushed a commit to branch DLAB-1571
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git

commit 375d4b542f8ecafba1e7e7b0080e69d25474b3cf
Author: Oleh Fuks <ol...@gmail.com>
AuthorDate: Wed Mar 4 13:31:31 2020 +0200

    Added Get API by dlabIds
---
 .../billing/gcp/controller/BillingController.java  |  6 +++++
 .../com/epam/dlab/billing/gcp/dao/BillingDAO.java  |  2 ++
 .../billing/gcp/dao/impl/BigQueryBillingDAO.java   | 28 ++++++++++++++++++----
 3 files changed, 31 insertions(+), 5 deletions(-)

diff --git a/services/billing-gcp/src/main/java/com/epam/dlab/billing/gcp/controller/BillingController.java b/services/billing-gcp/src/main/java/com/epam/dlab/billing/gcp/controller/BillingController.java
index dbcfa10..525be72 100644
--- a/services/billing-gcp/src/main/java/com/epam/dlab/billing/gcp/controller/BillingController.java
+++ b/services/billing-gcp/src/main/java/com/epam/dlab/billing/gcp/controller/BillingController.java
@@ -24,6 +24,7 @@ import com.epam.dlab.dto.billing.BillingData;
 import org.springframework.http.HttpStatus;
 import org.springframework.http.ResponseEntity;
 import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.RequestParam;
 import org.springframework.web.bind.annotation.RestController;
 
 import java.util.List;
@@ -37,6 +38,11 @@ public class BillingController {
         this.billingDAO = billingDAO;
     }
 
+    @GetMapping
+    public ResponseEntity<List<BillingData>> getBilling(@RequestParam List<String> dlabIds) {
+        return new ResponseEntity<>(billingDAO.getBillingReport(dlabIds), HttpStatus.OK);
+    }
+
     @GetMapping("/report")
     public ResponseEntity<List<BillingData>> getBilling() {
         return new ResponseEntity<>(billingDAO.getBillingReport(), HttpStatus.OK);
diff --git a/services/billing-gcp/src/main/java/com/epam/dlab/billing/gcp/dao/BillingDAO.java b/services/billing-gcp/src/main/java/com/epam/dlab/billing/gcp/dao/BillingDAO.java
index 55c2c2d..7288614 100644
--- a/services/billing-gcp/src/main/java/com/epam/dlab/billing/gcp/dao/BillingDAO.java
+++ b/services/billing-gcp/src/main/java/com/epam/dlab/billing/gcp/dao/BillingDAO.java
@@ -29,4 +29,6 @@ public interface BillingDAO {
     List<GcpBillingData> getBillingData() throws InterruptedException;
 
     List<BillingData> getBillingReport();
+
+    List<BillingData> getBillingReport(List<String> dlabIds);
 }
diff --git a/services/billing-gcp/src/main/java/com/epam/dlab/billing/gcp/dao/impl/BigQueryBillingDAO.java b/services/billing-gcp/src/main/java/com/epam/dlab/billing/gcp/dao/impl/BigQueryBillingDAO.java
index 990399a..1df1ab2 100644
--- a/services/billing-gcp/src/main/java/com/epam/dlab/billing/gcp/dao/impl/BigQueryBillingDAO.java
+++ b/services/billing-gcp/src/main/java/com/epam/dlab/billing/gcp/dao/impl/BigQueryBillingDAO.java
@@ -36,6 +36,8 @@ import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.data.mongodb.core.MongoTemplate;
 import org.springframework.data.mongodb.core.aggregation.Aggregation;
 import org.springframework.data.mongodb.core.aggregation.GroupOperation;
+import org.springframework.data.mongodb.core.aggregation.MatchOperation;
+import org.springframework.data.mongodb.core.query.Criteria;
 import org.springframework.stereotype.Component;
 
 import java.math.BigDecimal;
@@ -108,8 +110,24 @@ public class BigQueryBillingDAO implements BillingDAO {
 				.sum("cost").as("cost");
 		Aggregation aggregation = newAggregation(groupOperation);
 
-		return mongoTemplate.aggregate(aggregation, "billing", GcpBillingData.class).getMappedResults().stream()
-				.map(this::toBillingReportDTO)
+		return mongoTemplate.aggregate(aggregation, "billing", GcpBillingData.class).getMappedResults()
+				.stream()
+				.map(this::toBillingData)
+				.collect(Collectors.toList());
+	}
+
+	@Override
+	public List<BillingData> getBillingReport(List<String> dlabIds) {
+		GroupOperation groupOperation = group("product", "currency", "usageType", "dlabId")
+				.min("from").as("from")
+				.max("to").as("to")
+				.sum("cost").as("cost");
+		MatchOperation matchOperation = Aggregation.match(Criteria.where("dlabId").in(dlabIds));
+		Aggregation aggregation = newAggregation(matchOperation, groupOperation);
+
+		return mongoTemplate.aggregate(aggregation, "billing", GcpBillingData.class).getMappedResults()
+				.stream()
+				.map(this::toBillingData)
 				.collect(Collectors.toList());
 	}
 
@@ -124,7 +142,7 @@ public class BigQueryBillingDAO implements BillingDAO {
 					.build();
 			final Stream<GcpBillingData> gcpBillingDataStream =
 					StreamSupport.stream(service.query(queryConfig).getValues().spliterator(), false)
-							.map(this::toBillingData);
+							.map(this::toGcpBillingData);
 			billingHistoryRepo.save(new BillingHistory(tableName, table.getLastModifiedTime()));
 			return gcpBillingDataStream;
 		} catch (InterruptedException e) {
@@ -132,7 +150,7 @@ public class BigQueryBillingDAO implements BillingDAO {
 		}
 	}
 
-	private GcpBillingData toBillingData(FieldValueList fields) {
+	private GcpBillingData toGcpBillingData(FieldValueList fields) {
 		return GcpBillingData.builder()
 				.usageDateFrom(toLocalDate(fields, "usage_date_from"))
 				.usageDateTo(toLocalDate(fields, "usage_date_to"))
@@ -150,7 +168,7 @@ public class BigQueryBillingDAO implements BillingDAO {
 				.atZone(ZoneId.systemDefault()));
 	}
 
-	private BillingData toBillingReportDTO(GcpBillingData billingData) {
+	private BillingData toBillingData(GcpBillingData billingData) {
 		return BillingData.builder()
 				.usageDateFrom(billingData.getUsageDateFrom())
 				.usageDateTo(billingData.getUsageDateTo())


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@dlab.apache.org
For additional commands, e-mail: commits-help@dlab.apache.org


[incubator-dlab] 01/02: Merge branch 'develop' into DLAB-1571

Posted by of...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

ofuks pushed a commit to branch DLAB-1571
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git

commit d383c85482e05fbf8dc8f6477c22a81be1aa69a7
Merge: 3713da8 51afde3
Author: Oleh Fuks <ol...@gmail.com>
AuthorDate: Wed Mar 4 11:22:52 2020 +0200

    Merge branch 'develop' into DLAB-1571

 README.md                                          |   3 -
 infrastructure-provisioning/scripts/deploy_dlab.py |   1 -
 .../src/general/conf/dlab.ini                      |   4 +-
 .../src/general/files/azure/jupyterlab_Dockerfile  |  51 +++
 .../files/azure/jupyterlab_description.json        |  25 ++
 .../general/files/gcp/jupyterlab_description.json  |   2 +-
 .../src/general/lib/gcp/meta_lib.py                |   4 -
 .../src/general/lib/os/debian/common_lib.py        |   1 +
 .../src/general/lib/os/debian/ssn_lib.py           |   4 +
 .../src/general/lib/os/fab.py                      |  27 +-
 .../src/general/scripts/aws/ssn_configure.py       |   4 -
 .../general/scripts/azure/jupyterlab_configure.py  | 398 +++++++++++++++++++++
 .../src/general/scripts/azure/ssn_configure.py     |   5 +-
 ...common_notebook_configure_dataengine-service.py |  12 +-
 .../general/scripts/gcp/common_prepare_notebook.py |  18 +-
 .../scripts/gcp/dataengine-service_prepare.py      |  13 +-
 .../src/general/scripts/gcp/dataengine_prepare.py  |  15 +-
 .../src/general/scripts/gcp/ssn_configure.py       |  68 ++--
 .../src/general/scripts/gcp/ssn_prepare.py         |  39 +-
 .../jenkins_jobs/create_notebook_image/config.xml  |   1 +
 .../jenkins_jobs/create_notebook_server/config.xml |   1 +
 .../jenkins_jobs/manage_git_credentials/config.xml |   1 +
 .../jenkins_jobs/start_notebook_server/config.xml  |   1 +
 .../jenkins_jobs/stop_notebook_server/config.xml   |   1 +
 .../terminate_notebook_image/config.xml            |   1 +
 .../terminate_notebook_server/config.xml           |   1 +
 .../src/jupyterlab/Dockerfile_jupyterlab           |   8 +-
 .../scripts/configure_jupyterlab_node.py           |   7 -
 .../src/ssn/scripts/configure_ui.py                |   2 +-
 .../terraform/azure/endpoint/main/main.tf          |   1 +
 .../terraform/azure/endpoint/main/network.tf       |   7 +-
 .../project-list/project-list.component.html       |   2 +-
 .../app/administration/roles/roles.component.ts    |  50 ++-
 .../reporting-grid/reporting-grid.component.html   |   7 +-
 .../src/app/reporting/reporting.component.ts       |   4 +-
 .../resources-grid/resources-grid.component.scss   |   1 +
 .../confirmation-dialog-type.enum.ts               |   3 +-
 .../confirmation-dialog.component.html             |   9 +-
 .../webapp/src/assets/styles/_dialogs.scss         |   6 +
 .../webapp/src/dictionary/azure.dictionary.ts      |   6 +-
 40 files changed, 681 insertions(+), 133 deletions(-)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@dlab.apache.org
For additional commands, e-mail: commits-help@dlab.apache.org