You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datalab.apache.org by my...@apache.org on 2021/07/05 09:04:53 UTC
[incubator-datalab] branch DATALAB-2372 updated (6123437 -> 8445020)
This is an automated email from the ASF dual-hosted git repository.
mykolabodnar pushed a change to branch DATALAB-2372
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git.
from 6123437 [DATALAB-2372] - [GCP] Deeplearning libs installation fixed
add fa0032b Merge pull request #1206 from apache/DATALAB-2372
add bb9465a [DATALAB] -- fixed bug with wrong count param for Spark
add dd3b031 [DATALAB-2429]: fixed rstudio dataproc
add dc417e8 [DATALAB-2429]: fixed zeppelin dataproc creation
add c17e6e3 Merge pull request #1208 from apache/DATALAB-2445
add 35c9b4c Merge branch 'develop' of https://github.com/apache/incubator-datalab into develop
add b11ed90 [DATALAB] -- fixed bug with wrong count param for Spark
add da8cf45 [DATALAB-2356] - Ungit commit executes with error fixed
add 01dfacc [DATALAB] -- fixed bug with wrong count param for Spark
add 7f164e5 [DATALAB] -- fixed bug with wrong count param for Spark
add 12718db [DATALAB-2457] made the image selection dropdown mandatory
add a907ec2 Merge pull request #1209 from apache/DATALAB-2457
add ccc4afb Merge remote-tracking branch 'origin/develop' into develop
add 605b932 [DATALAB-2456] changed params for Apache standalone cluster creation
add 0064ab0 Merge pull request #1210 from apache/DATALAB-2456
add 356fff4 [DATALAB] [GCP] Dataproc version downgrade
add f4ca2e1 [DATALAB] -- fixed bug with wrong count param for Spark
add 09e919a [DATALAB] -- fixed bug with wrong count param for Spark
add aab4fd0 minor ui fix
add f128725 [DATALAB] -- fixed bug with wrong count param for Spark
add af2dab8 [DATALAB] -- added licenses
add 9d452c3 [DATALAB] changed param name for Standalone cluster creation
add a15306f Merge branch 'develop' of https://github.com/apache/incubator-datalab into develop
add 621f0c9 [DATALAB] -- added licenses
add ad13c5a [DATALAB] -- fixed instance size selection for all clouds
add 9a9cf79 Merge pull request #1211 from apache/DATALAB-deeplearning
add bbea604 [DATALAB] -- fixed name for DP on AWS
add 8fc4d63 [DATALAB] -- added version in notebook template image param for deeplearning on aws
add 0ef7cfc Merge remote-tracking branch 'origin/develop' into develop
add a4fd5de [DATALAB] -- fixed name for DP on AWS
add ccd43cd [DATALAB-2441] removed gpu count with value 8 for all available instances/GPU type
add 9680d69 Merge pull request #1212 from apache/DATALAB-2441
add 73630a0 [DATALAB-2483] added none option for custom image
add cb0371d Merge pull request #1213 from apache/DATALAB-2483
add 4e24db1 Merge remote-tracking branch 'origin/develop' into develop
add dcb1818 [DATALAB] -- fixed bugs with statuses
add 45599a2 [DATALAB-2449] - [GCP] GPU type and count fixed for Jupyter and Spark cluster
add 706a001 Merge pull request #1214 from apache/DATALAB-2449
add 2084983 Merge remote-tracking branch 'origin/develop' into develop
add 1e13b38 [DATALAB-2486] -- hide user/userName param in yaml files
add dafba42 [DATALAB-2466] - [GCP] Deeplearning creation fro custom image fixed
add 60aa599 Merge pull request #1215 from apache/DATALAB-2466
add 12e1938 [DATALAB-2487]: removed emr 6.0.0
add 39ff5b6 Merge pull request #1216 from apache/DATALAB-2487
add cbd18d5 [DATALAB] -- fixed bugs with statuses
add 2900f41 Merge pull request #1217 from apache/DATALAB-2195
add a58182c [DATALAB-2464] - [GCP] Tensorboard links removed from deeplearning notebooks at UI
add ba10ae3 Merge pull request #1218 from apache/DATALAB-2464
add f9e96f2 [DATALAB-2442] added t-shirt size for gpu types, hid the option for recreate node
add 1ed6676 Merge pull request #1219 from apache/DATALAB-2442
add 5e2be7f Merge remote-tracking branch 'origin/develop' into develop
add 93de6c0 [DATALAB-2440] showed message about successfully self-service restarting
add b91988f Merge pull request #1220 from apache/DATALAB-2440
add 4d77abe [DATALAB-2448]: changed how devtools dependencies are installed
add 638ed5e [DATALAB-2465]: fixed permission denied to tmp file during lib install
add 5fceef5 [DATALAB-2448]: fixed R devtools installation issues for dataproc
add 3054f04 [DATALAB-2448]: changed how devtools is ensured for dataengine-service
add af69dd1 Merge pull request #1221 from apache/DATALAB-2448
new 8445020 [DATALAB-2372] - [AWS] DeepLearning notebook for using existing cloud images for deeplearning updated
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../files/aws/dataengine-service_description.json | 1 -
.../files/aws/deeplearning_description.json | 6 +-
.../files/azure/deeplearning_description.json | 4 +-
.../files/gcp/dataengine-service_description.json | 2 +-
.../src/general/lib/gcp/actions_lib.py | 10 +--
.../src/general/lib/os/debian/notebook_lib.py | 12 ++-
.../src/general/lib/os/fab.py | 71 ++++++++++++------
.../general/scripts/aws/common_prepare_notebook.py | 4 +-
.../general/scripts/gcp/common_create_instance.py | 4 +-
.../general/scripts/gcp/common_prepare_notebook.py | 22 ++++--
.../general/scripts/gcp/dataengine_configure.py | 32 ++++++++
.../src/general/scripts/gcp/dataengine_prepare.py | 37 +++++++---
.../general/scripts/gcp/deeplearning_configure.py | 14 ++--
.../src/general/scripts/gcp/jupyter_configure.py | 2 +-
.../rstudio_install_dataengine-service_kernels.py | 14 ++--
.../zeppelin_install_dataengine-service_kernels.py | 18 +++--
.../src/general/scripts/os/git_pre_commit.py | 2 +-
.../general/scripts/os/install_additional_libs.py | 7 --
.../templates/gcp/dataengine-service_cluster.json | 4 +-
.../gcp/dataengine-service_cluster_with_gpu.json | 4 +-
.../templates/gcp/dataengine-service_livy-env.sh | 21 ++++++
.../aws/computational/ComputationalCreateAws.java | 61 +++------------
.../SparkStandaloneClusterResource.java | 14 ++--
.../computational/SparkComputationalCreateGcp.java | 15 +---
.../ExploratoryEnvironmentImages.java | 19 +++++
.../datalab/model/exploratory/Exploratory.java | 2 +-
.../datalab/properties/ChangePropertiesConst.java | 9 ++-
.../properties/ChangePropertiesService.java | 16 +++-
.../com/epam/datalab/properties/RestartAnswer.java | 19 +++++
.../datalab/backendapi/core/DockerWarmuper.java | 27 ++++---
.../backendapi/core/commands/DockerCommands.java | 7 +-
.../response/handlers/ProjectCallbackHandler.java | 32 +++++---
.../response/handlers/ResourceCallbackHandler.java | 15 ++--
.../handlers/dao/FileSystemCallbackHandlerDao.java | 1 +
.../resources/base/ExploratoryService.java | 4 +
.../service/impl/ProjectServiceImpl.java | 86 +++++++++++-----------
.../backendapi/resources/ExploratoryResource.java | 1 +
.../backendapi/resources/ProjectResource.java | 7 ++
.../resources/aws/ComputationalResourceAws.java | 2 +-
.../resources/callback/ProjectCallback.java | 51 +++++++++----
.../dto/SparkStandaloneClusterCreateForm.java | 8 +-
.../dto/aws/AwsComputationalCreateForm.java | 2 -
.../epam/datalab/backendapi/roles/UserRoles.java | 17 +++--
.../service/impl/ComputationalServiceImpl.java | 6 +-
.../service/impl/ExploratoryServiceImpl.java | 14 ++++
.../datalab/backendapi/util/RequestBuilder.java | 12 +--
.../configuration/configuration.component.ts | 9 ++-
.../create-odahu-cluster.component.html | 19 +++++
.../create-odahu-cluster.component.scss | 19 +++++
.../app/administration/odahu/odahu-data.service.ts | 19 +++++
.../odahu/odahu-grid/odahu-grid.component.html | 18 +++++
.../odahu/odahu-grid/odahu-grid.component.scss | 19 +++++
.../odahu/odahu-grid/odahu-grid.component.ts | 19 +++++
.../app/administration/odahu/odahu.component.scss | 18 +++++
.../app/administration/odahu/odahu.component.ts | 19 +++++
.../project-list/project-list.component.html | 4 +-
.../app/core/services/odahu-deployment.service.ts | 19 +++++
.../webapp/src/app/core/util/helpUtils.ts | 36 +++++----
...utational-resource-create-dialog.component.html | 12 +--
...mputational-resource-create-dialog.component.ts | 21 +-----
.../computational-resource-create.model.ts | 3 +-
.../create-environment.component.html | 10 ++-
.../create-environment.component.ts | 37 ++++++++--
.../edge-action-dialog.component.ts | 2 +-
.../notification-dialog.component.ts | 2 +-
.../modal-dialog/odahu-action-dialog/index.ts | 19 +++++
.../odahu-action-dialog.component.ts | 19 +++++
.../service/impl/ComputationalServiceImplTest.java | 4 +-
68 files changed, 742 insertions(+), 343 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 01/01: [DATALAB-2372] - [AWS] DeepLearning
notebook for using existing cloud images for deeplearning updated
Posted by my...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mykolabodnar pushed a commit to branch DATALAB-2372
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 8445020ac92ee51b01e95475d8fa38801c3efc32
Author: bodnarmykola <bo...@gmail.com>
AuthorDate: Mon Jul 5 12:04:15 2021 +0300
[DATALAB-2372] - [AWS] DeepLearning notebook for using existing cloud images for deeplearning updated
---
.../src/general/files/aws/deeplearning_description.json | 2 +-
.../src/general/scripts/aws/common_prepare_notebook.py | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/infrastructure-provisioning/src/general/files/aws/deeplearning_description.json b/infrastructure-provisioning/src/general/files/aws/deeplearning_description.json
index 891cc5e..06f671a 100644
--- a/infrastructure-provisioning/src/general/files/aws/deeplearning_description.json
+++ b/infrastructure-provisioning/src/general/files/aws/deeplearning_description.json
@@ -8,7 +8,7 @@
"exploratory_environment_versions" :
[
{
- "template_name": "Deep Learning AMI (Ubuntu 18.04) Version 42.1",
+ "template_name": "Deep Learning AMI Version 42.1",
"description": "MXNet-1.8.0 & 1.7.0, TensorFlow-2.4.1, 2.1.3 & 1.15.5, PyTorch-1.4.0 & 1.8.0, Neuron, & others. NVIDIA CUDA, cuDNN, NCCL, Intel MKL-DNN, Docker, NVIDIA-Docker & EFA support. Uses Anaconda virtual environments, configured to keep the different framework installations separate and easy to switch between frameworks as Jupyter kernels.",
"environment_type": "exploratory",
"version": "Deep Learning AMI (Ubuntu 18.04) Version 42.1",
diff --git a/infrastructure-provisioning/src/general/scripts/aws/common_prepare_notebook.py b/infrastructure-provisioning/src/general/scripts/aws/common_prepare_notebook.py
index 270ae56..ad7f59d 100644
--- a/infrastructure-provisioning/src/general/scripts/aws/common_prepare_notebook.py
+++ b/infrastructure-provisioning/src/general/scripts/aws/common_prepare_notebook.py
@@ -80,7 +80,7 @@ if __name__ == "__main__":
notebook_config['project_name'],
notebook_config['endpoint_name'],
notebook_config['exploratory_name'], args.uuid)
- notebook_config['primary_disk_size'] = (lambda x: '30' if x == 'deeplearning' else '16')(
+ notebook_config['primary_disk_size'] = (lambda x: '100' if x == 'deeplearning' else '16')(
os.environ['application'])
notebook_config['role_profile_name'] = '{}-{}-{}-nb-de-profile'.format(
notebook_config['service_base_name'], notebook_config['project_name'], notebook_config['endpoint_name'])
@@ -105,7 +105,7 @@ if __name__ == "__main__":
os.environ['conf_os_family'])])
image_id = datalab.meta_lib.get_ami_id_by_name(notebook_config['notebook_image_name'], 'available')
if os.environ['conf_deeplearning_cloud_ami'] == 'true' and os.environ['application'] == 'deeplearning' and image_id == '':
- image_id = datalab.meta_lib.get_ami_id(notebook_config['notebook_image_name'])
+ image_id = datalab.meta_lib.get_ami_id(os.environ['notebook_image_name'])
if image_id != '':
notebook_config['ami_id'] = image_id
print('Pre-configured image found. Using: {}'.format(notebook_config['ami_id']))
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org