You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datalab.apache.org by lf...@apache.org on 2022/09/01 07:11:29 UTC
[incubator-datalab] branch DATALAB-1408 updated (673d9a876 -> 44d963997)
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a change to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
from 673d9a876 Merge pull request #1676 from apache/DATALAB-2982
add 16c89d52b DATALAB-2889: Update Tensorflow version from 2.5.0 to 2.9.1
add 1ae6687cd inital support for hdinsight
add 06a206c67 Merge branch 'develop' into DATALAB-2645
add 7234fbe3c add number of slave instances for hdinsight
add fc0a564de add termiante dto for hdinsight
add 77dcd610d removed condition
add a950a21f6 Merge pull request #1668 from apache/cluste-type-for-azure
add 50e5894d9 removed condition
add 40a16b81b Merge pull request #1669 from apache/cluste-type-for-azure
add a7a46127c changed create button style
add 568ae557c Merge pull request #1670 from apache/cluste-type-for-azure
add 615598d07 minor fix
add 76b9fff17 Merge pull request #1671 from apache/cluste-type-for-azure
add 047a34ede remove commented lines
add 1bab4739c filter instances with id null
add 90f987ee8 Merge pull request #1672 from apache/DATALAB-2838
add 6fabc5eac fixed scroll in firefox
add 7b939cc3f Merge pull request #1673 from apache/fix/DATALAB-2956/scroll-bar-width
add 4d40ba719 commented open terminal button
add 0aafdc5cc Merge pull request #1674 from apache/fix/DATALAB-2975/hide-open-terminal-button
add c7bcf124d minor refactor of the computational-resourse dialog component
add 74f83336d Merge pull request #1675 from apache/feat/DATALAB-2646/create-HDInside-on-Azure
add f6c148116 Merge branch 'develop' into DATALAB-2645
add 0e487eae2 add min and max instance count for hdinsoght to config
add 9dadd2d7b fix
add be86ed105 change limits field name
add cad0f84b4 change limits field name
add 6fa91441c fixed validation
add 522d01d75 initial commit
add 346efb18d minor fix
add 82b51ab38 Merge pull request #1678 from apache/feat/DATALAB-2646/create-HDInside-on-Azure
add d544bd8ad fixed limits naming
add 278da0e97 Merge pull request #1679 from apache/feat/DATALAB-2646/create-HDInside-on-Azure
add 96b72a47c Merge branch 'develop' into DATALAB-2645
add 19dd636cb Merge pull request #1680 from apache/DATALAB-2645
add 90a941e5a fixed scroll in action menu
add b76b3238a Merge pull request #1681 from apache/fix/DATALAB-2956/action-menu-scrollbar
add 6b1ac0beb Merge branch 'DATALAB-2889' into DATALAB-2889
add 7d671a702 Merge pull request #1682 from OleksandrRepnikov/DATALAB-2889
add bdb8503a5 Merge pull request #1683 from apache/DATALAB-2889
add 1a6f310ba added directive for validating
add f9132443d minor fix
add 0950350f5 Merge pull request #1685 from apache/fix/DATALAB-2863/group-name-validation
add a247f3344 minor fix
add 84000aabe Merge pull request #1686 from apache/fix/DATALAB-2863/group-name-validation
add 88f9b032e [DATALAB-2648]: added usage of full image name during image termination on aws and azure instead of notebook image name, now status terminated is returned if image already does not exist when it`s removal is executed
add 21fea4270 add support for terminating user image
add bb2122b20 Merge pull request #1687 from apache/DATALAB-2648
add 705c716cb update image image status after termination
add 41adeec38 remove prov.run.xml
add 948812d15 Merge pull request #1688 from apache/DATALAB-2648
add 5e514022f initial commit
add 87db970c8 minor fix
add 33f68a3f9 Merge pull request #1689 from apache/feat/DATALAB-2864/delete-one-image
add c444f433f minor fix
add 46463206f Merge pull request #1690 from apache/feat/DATALAB-2864/delete-one-image
add 7c4cfb07a minor fix
add 4c4bfad3e Merge pull request #1691 from apache/feat/DATALAB-2864/delete-one-image
new 2a4766739 added files and func to actions lib
new 727439e1d params and creds
new 488e44555 added hdinsight lib to reqs
new 9c613a6d7 add libs in base_Dockerfile
new 2bf412f82 add imports
new 008febb35 add class AzureActions
new dcac403ad delete hdinsight_client
new 8dffa5e89 modify hdinsight func
new a62b36011 modify func
new 189d2c0c4 modify func add self=
new 5379753b1 remove self
new dddfd288d add import self and add self to func
new d18321b36 remove self from func and from import
new c3be8d16b add self before client creation
new c91deaf60 changes in def hdinsight
new 17fd0c1fa some changes
new b1cf6e39a revert changes
new 3dc6e239c edit actions_lib
new bd88e30d7 changes
new fde2b5d73 revert changes
new 6fa45158d changes
new b56086d20 revert changes
new d0543dfb3 changes
new b2df54a2d some changes
new 9e4e3ca79 changes
new 5cb6feb91 change func hdinsight
new 2fc07eccc change cluster name
new 05b2d825b delete unused statement
new 42ed96c99 changes
new c6b536591 refactor dataengine-service_prepare
new fa7e32fc6 fix errors
new 23223bd7e add print to def_terminate
new c8092d2e2 [DATALAB-2982]: added new files for hdinsight
new 5794a7399 [DATALAB-2982]: changed cluster name variable
new 14f36c05f [DATALAB-1408]: added some creation changes
new 44d963997 fixed conflict
The 36 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../src/general/conf/datalab.ini | 2 +-
.../files/aws/tensor-jupyterlab_description.json | 4 +-
.../files/aws/tensor-rstudio_description.json | 4 +-
.../src/general/files/aws/tensor_description.json | 4 +-
.../general/files/azure/tensor_description.json | 4 +-
.../src/general/files/gcp/tensor_description.json | 4 +-
.../scripts/aws/common_terminate_notebook_image.py | 23 +++--
.../azure/common_terminate_notebook_image.py | 20 ++--
.../scripts/azure/dataengine-service_prepare.py | 107 ++++++++++++++++++++-
.../scripts/gcp/common_terminate_notebook_image.py | 14 +--
.../scripts/gcp/dataengine-service_create.py | 2 +-
.../scripts/configure_tensor-jupyterlab_node.py | 6 +-
.../src/tensor/scripts/configure_tensor_node.py | 6 +-
.../computational/AzureComputationalResource.java | 50 ++++++++++
.../AzureComputationalTerminateDTO.java | 14 +++
.../computational/ComputationalConfigAzure.java | 5 +
.../computational/ComputationalCreateAzure.java | 19 ++++
.../computational/UserComputationalResource.java | 2 +
.../epam/datalab/dto/exploratory/ImageStatus.java | 2 +
.../datalab/rest/contracts/ExploratoryAPI.java | 1 +
.../backendapi/core/commands/DockerAction.java | 1 +
.../backendapi/resources/ImageResource.java | 13 +++
.../azure/ComputationalResourceAzure.java | 107 ++++++++++++++++++++-
services/self-service/self-service.yml | 5 +
.../conf/SelfServiceApplicationConfiguration.java | 17 ++++
.../backendapi/dao/ImageExploratoryDAO.java | 3 +
.../backendapi/dao/ImageExploratoryDAOImpl.java | 12 +++
.../backendapi/domain/NotebookTemplate.java | 6 +-
.../resources/ImageExploratoryResource.java | 22 +++--
.../azure/ComputationalResourceAzure.java | 64 +++++++++++-
.../resources/callback/ImageCallback.java | 9 +-
.../dto/azure/AzureComputationalCreateForm.java | 32 ++++++
.../dto/azure/AzureHDInsightConfiguration.java | 19 ++++
.../CheckInfrastructureStatusScheduler.java | 3 +-
.../service/ImageExploratoryService.java | 4 +
.../service/impl/ImageExploratoryServiceImpl.java | 33 ++++++-
.../impl/InfrastructureTemplateServiceImpl.java | 19 +++-
.../main/resources/mongo/general/mongo_roles.json | 12 +++
.../app/administration/administration.module.ts | 5 +-
.../webapp/src/app/administration/roles/index.ts | 14 +--
.../app/administration/roles/roles.component.html | 1 +
.../app/administration/roles/roles.component.ts | 18 ++--
.../src/app/core/configs/docker-image-name.ts} | 9 +-
.../configs/image-template-name.ts} | 5 +-
.../app/core/{models => configs}/imageType.enum.ts | 0
.../imageType.enum.ts => configs/providers.ts} | 7 +-
.../core/{models => configs}/statusTypes.enum.ts | 0
.../webapp/src/app/core/directives/index.ts | 19 +++-
.../directives/is-group-name-unique.directive.ts | 54 +++++++++++
.../models/computationalResourceImage.model.ts | 8 +-
.../models/exploratoryEnvironmentVersion.model.ts | 2 +-
.../resources/webapp/src/app/core/models/index.ts | 6 +-
.../project.model.ts => core/models/role.model.ts} | 34 ++++---
.../webapp/src/app/core/models/templates-name.ts | 22 -----
.../is-element-available.pipe.ts | 4 +-
.../services/applicationServiceFacade.service.ts | 10 +-
.../app/core/services/rolesManagement.service.ts | 11 +--
.../app/core/services/user-images-page.service.ts | 18 +++-
.../src/app/core/services/userResource.service.ts | 6 +-
...utational-resource-create-dialog.component.html | 18 ++--
...mputational-resource-create-dialog.component.ts | 50 +++++-----
.../computational.resource.model.ts} | 59 ++++++++----
.../computational-resource-create-dialog/index.ts | 5 +-
.../create-environment.component.ts | 4 +-
.../image-action-dialog.component.html} | 16 ++-
.../image-action-dialog.component.scss} | 36 ++++++-
.../image-action-dialog.component.ts | 92 ++++++++++++++++++
.../image-action-dialog.module.ts} | 13 +--
.../image-detail-dialog.component.ts | 10 +-
.../library-info-modal.component.ts | 4 +-
.../share-image-dialog.component.ts | 52 ----------
.../src/app/resources/images/images.component.html | 6 +-
.../src/app/resources/images/images.component.ts | 31 ++++--
.../src/app/resources/images/images.config.ts | 15 ++-
.../src/app/resources/images/images.model.ts | 11 ++-
.../src/app/resources/images/images.service.ts | 20 ++--
.../resources-grid/resources-grid.component.html | 26 ++---
.../resources-grid/resources-grid.component.scss | 2 +-
.../src/app/resources/resources.component.ts | 2 -
.../webapp/src/app/resources/resources.module.ts | 6 +-
.../webapp/src/dictionary/azure.dictionary.ts | 6 +-
.../src/main/resources/webapp/src/styles.scss | 3 +-
82 files changed, 1077 insertions(+), 337 deletions(-)
create mode 100644 services/datalab-model/src/main/java/com/epam/datalab/dto/azure/computational/AzureComputationalResource.java
create mode 100644 services/datalab-model/src/main/java/com/epam/datalab/dto/azure/computational/AzureComputationalTerminateDTO.java
create mode 100644 services/datalab-model/src/main/java/com/epam/datalab/dto/azure/computational/ComputationalConfigAzure.java
create mode 100644 services/datalab-model/src/main/java/com/epam/datalab/dto/azure/computational/ComputationalCreateAzure.java
create mode 100644 services/self-service/src/main/java/com/epam/datalab/backendapi/resources/dto/azure/AzureComputationalCreateForm.java
create mode 100644 services/self-service/src/main/java/com/epam/datalab/backendapi/resources/dto/azure/AzureHDInsightConfiguration.java
copy services/{datalab-mongo-migration/src/main/java/com/epam/datalab/migration/DbMigration.java => self-service/src/main/resources/webapp/src/app/core/configs/docker-image-name.ts} (85%)
copy services/self-service/src/main/resources/webapp/src/app/{shared/checkbox/checkbox.component.css => core/configs/image-template-name.ts} (92%)
copy services/self-service/src/main/resources/webapp/src/app/core/{models => configs}/imageType.enum.ts (100%)
rename services/self-service/src/main/resources/webapp/src/app/core/{models/imageType.enum.ts => configs/providers.ts} (91%)
rename services/self-service/src/main/resources/webapp/src/app/core/{models => configs}/statusTypes.enum.ts (100%)
create mode 100644 services/self-service/src/main/resources/webapp/src/app/core/directives/is-group-name-unique.directive.ts
copy services/self-service/src/main/resources/webapp/src/app/{administration/project/project.model.ts => core/models/role.model.ts} (61%)
delete mode 100644 services/self-service/src/main/resources/webapp/src/app/core/models/templates-name.ts
copy services/self-service/src/main/resources/webapp/src/app/{core/models/exploratoryEnvironmentVersion.model.ts => resources/computational/computational-resource-create-dialog/computational.resource.model.ts} (50%)
rename services/self-service/src/main/resources/webapp/src/app/resources/exploratory/{share-image-dialog/share-image-dialog.component.html => image-action-dialog/image-action-dialog.component.html} (69%)
rename services/self-service/src/main/resources/webapp/src/app/resources/exploratory/{share-image-dialog/share-image-dialog.component.scss => image-action-dialog/image-action-dialog.component.scss} (66%)
create mode 100644 services/self-service/src/main/resources/webapp/src/app/resources/exploratory/image-action-dialog/image-action-dialog.component.ts
rename services/self-service/src/main/resources/webapp/src/app/resources/exploratory/{share-image-dialog/share-image-dialog.module.ts => image-action-dialog/image-action-dialog.module.ts} (73%)
delete mode 100644 services/self-service/src/main/resources/webapp/src/app/resources/exploratory/share-image-dialog/share-image-dialog.component.ts
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 11/36: remove self
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 5379753b1970d218a924114cd2e170a6ca0b25b3
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Wed May 25 17:04:56 2022 +0300
remove self
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index cac66816b..8f0d31750 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1503,7 +1503,7 @@ def find_des_jars(all_jars, des_path):
def create_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__(self=).hdinsight_client
+ client_1 = AzureActions.__init__().hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
@@ -1519,7 +1519,7 @@ def create_hdinsight_cluster(resource_group_name, instance_name, cluster_paramet
def terminate_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__(self=).hdinsight_client
+ client_1 = AzureActions.__init__().hdinsight_client
client_1.clusters.begin_delete(resource_group_name, instance_name, cluster_parameters)
except Exception as err:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 17/36: revert changes
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit b1cf6e39ac1be73765572ed64c24fb6324211a87
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Thu May 26 13:47:47 2022 +0300
revert changes
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index c270f5c23..8f0d31750 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1501,9 +1501,9 @@ def find_des_jars(all_jars, des_path):
print('Error:', str(err))
sys.exit(1)
-def create_hdinsight_cluster(self, resource_group_name, instance_name, cluster_parameters):
+def create_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__(self).hdinsight_client
+ client_1 = AzureActions.__init__().hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 36/36: fixed conflict
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 44d963997a0a3fd6a123a1d2c6946a5339d5180c
Merge: 14f36c05f 673d9a876
Author: leonidfrolov <fr...@gmail.com>
AuthorDate: Thu Sep 1 10:11:15 2022 +0300
fixed conflict
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 03/36: added hdinsight lib to reqs
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 488e44555221f243a57f335dde8a35ab6568a82d
Author: owlleg6 <ow...@gmail.com>
AuthorDate: Thu May 19 12:15:20 2022 +0300
added hdinsight lib to reqs
---
infrastructure-provisioning/src/base/scripts/install_prerequisites.py | 2 +-
infrastructure-provisioning/src/general/files/azure/base_Dockerfile | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/infrastructure-provisioning/src/base/scripts/install_prerequisites.py b/infrastructure-provisioning/src/base/scripts/install_prerequisites.py
index 06135eeff..8150affab 100644
--- a/infrastructure-provisioning/src/base/scripts/install_prerequisites.py
+++ b/infrastructure-provisioning/src/base/scripts/install_prerequisites.py
@@ -39,7 +39,7 @@ parser.add_argument('--pip_packages', type=str,
default='boto3 argparse fabric awscli google-api-python-client google-auth-httplib2 '
'google-cloud-storage pycryptodome azure-storage-blob azure-datalake-store '
'azure-mgmt-compute azure-mgmt-network azure-mgmt-resource azure-mgmt-storage '
- 'azure-mgmt-datalake-store azure-identity')
+ 'azure-mgmt-datalake-store azure-identity azure-mgmt-hdinsight')
parser.add_argument('--additional_config', type=str, default='{"empty":"string"}')
parser.add_argument('--user', type=str, default='')
parser.add_argument('--edge_private_ip', type=str, default='')
diff --git a/infrastructure-provisioning/src/general/files/azure/base_Dockerfile b/infrastructure-provisioning/src/general/files/azure/base_Dockerfile
index 7c016ae1b..7653d7dbd 100644
--- a/infrastructure-provisioning/src/general/files/azure/base_Dockerfile
+++ b/infrastructure-provisioning/src/general/files/azure/base_Dockerfile
@@ -33,7 +33,7 @@ RUN apt-get update && \
RUN python3 -m pip install -UI qtconsole==5.1.1 pip==21.1.2 && \
python3 -m pip install backoff patchwork fabric fabvenv argparse requests ujson jupyter pycryptodome azure-common pyyaml && \
python3 -m pip install azure-mgmt-storage azure-mgmt-compute azure-mgmt-core azure-mgmt-network azure-storage-blob azure-mgmt-resource && \
- python3 -m pip install azure-storage-file-datalake azure-datalake-store azure-identity azure-mgmt-datalake-store
+ python3 -m pip install azure-storage-file-datalake azure-datalake-store azure-identity azure-mgmt-datalake-store azure-mgmt-hdinsight
# Configuring ssh for user
RUN mkdir -p /root/.ssh; echo "Host *" > /root/.ssh/config; \
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 28/36: delete unused statement
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 05b2d825ba5029662bce68dec5ec4db88fd5bd79
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Thu May 26 16:46:26 2022 +0300
delete unused statement
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 2 --
1 file changed, 2 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index aa6d02ec4..800a57b0d 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1505,7 +1505,6 @@ def create_hdinsight_cluster(resource_group_name, instance_name, cluster_paramet
try:
azure_action = AzureActions()
client_1 = azure_action.hdinsight_client
- #client_1 = AzureActions.__init__()hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
@@ -1523,7 +1522,6 @@ def terminate_hdinsight_cluster(resource_group_name, instance_name, cluster_para
try:
azure_action = AzureActions()
client_1 = azure_action.hdinsight_client
- #client_1 = AzureActions.__init__().hdinsight_client
client_1.clusters.begin_delete(resource_group_name, instance_name, cluster_parameters)
except Exception as err:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 09/36: modify func
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit a62b36011dabe25f98b28321b21c2969edc37596
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Wed May 25 16:52:01 2022 +0300
modify func
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 021d0accc..d7b89f8ab 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1503,7 +1503,7 @@ def find_des_jars(all_jars, des_path):
def create_hdinsight_cluster(self, resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__(self=).hdinsight_client
+ client_1 = AzureActions.__init__().hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
@@ -1519,7 +1519,7 @@ def create_hdinsight_cluster(self, resource_group_name, instance_name, cluster_p
def terminate_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__(self=).hdinsight_client
+ client_1 = AzureActions.__init__().hdinsight_client
client_1.clusters.begin_delete(resource_group_name, instance_name, cluster_parameters)
except Exception as err:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 26/36: change func hdinsight
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 5cb6feb91199a7705ac1bf3326f9dc4aef82c690
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Thu May 26 15:53:38 2022 +0300
change func hdinsight
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 70e305f64..aa6d02ec4 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1521,7 +1521,9 @@ def create_hdinsight_cluster(resource_group_name, instance_name, cluster_paramet
def terminate_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__().hdinsight_client
+ azure_action = AzureActions()
+ client_1 = azure_action.hdinsight_client
+ #client_1 = AzureActions.__init__().hdinsight_client
client_1.clusters.begin_delete(resource_group_name, instance_name, cluster_parameters)
except Exception as err:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 33/36: [DATALAB-2982]: added new files for hdinsight
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit c8092d2e2f0c552a702ba285c3bd907b43311183
Author: leonidfrolov <fr...@gmail.com>
AuthorDate: Tue Aug 16 11:15:32 2022 +0300
[DATALAB-2982]: added new files for hdinsight
---
.../files/azure/dataengine-service_Dockerfile | 39 +++++++++
.../azure/dataengine-service_description.json | 27 ++++++
.../scripts/azure/dataengine-service_configure.py | 60 +++++++++++++
.../scripts/azure/dataengine-service_prepare.py | 4 +-
.../scripts/azure/dataengine-service_terminate.py | 99 ++++++++++++++++++++++
5 files changed, 227 insertions(+), 2 deletions(-)
diff --git a/infrastructure-provisioning/src/general/files/azure/dataengine-service_Dockerfile b/infrastructure-provisioning/src/general/files/azure/dataengine-service_Dockerfile
new file mode 100644
index 000000000..2b443239b
--- /dev/null
+++ b/infrastructure-provisioning/src/general/files/azure/dataengine-service_Dockerfile
@@ -0,0 +1,39 @@
+# *****************************************************************************
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# ******************************************************************************
+
+FROM docker.datalab-base:latest
+
+ARG OS
+
+COPY dataengine-service/fabfile.py /root/
+COPY dataengine-service/description.json /root/
+COPY general/scripts/azure/dataengine-service_* /root/scripts/
+COPY general/lib/os/${OS}/notebook_lib.py /usr/lib/python3.8/datalab/notebook_lib.py
+COPY general/scripts/os/common_* /root/scripts/
+COPY general/scripts/os/install_additional_libs.py /root/scripts/install_additional_libs.py
+COPY general/scripts/os/get_list_available_pkgs.py /root/scripts/get_list_available_pkgs.py
+COPY general/templates/os/inactive.sh /root/templates/
+COPY general/templates/os/inactive.service /root/templates/
+COPY general/templates/os/inactive.timer /root/templates/
+
+RUN chmod a+x /root/fabfile.py; \
+ chmod a+x /root/scripts/*
+
diff --git a/infrastructure-provisioning/src/general/files/azure/dataengine-service_description.json b/infrastructure-provisioning/src/general/files/azure/dataengine-service_description.json
new file mode 100644
index 000000000..d8ad248e5
--- /dev/null
+++ b/infrastructure-provisioning/src/general/files/azure/dataengine-service_description.json
@@ -0,0 +1,27 @@
+{
+ "template_name": "HDInsight cluster",
+ "description": "HDInsight cluster",
+ "environment_type": "computational",
+ "computation_resources_shapes":
+ {
+ "For testing" : [
+ {"Size": "S", "Description": "Standard_F4s", "Type": "Standard_F4s","Ram": "8.0 GB","Cpu": "4"}
+ ],
+ "Memory optimized" : [
+ {"Size": "S", "Description": "Standard_E4s_v3", "Type": "Standard_E4s_v3","Ram": "32 GB","Cpu": "4"},
+ {"Size": "M", "Description": "Standard_E16s_v3", "Type": "Standard_E16s_v3","Ram": "128 GB","Cpu": "16"},
+ {"Size": "L", "Description": "Standard_E32s_v3", "Type": "Standard_E32s_v3","Ram": "256 GB","Cpu": "32"}
+ ],
+ "Compute optimized": [
+ {"Size": "S", "Description": "Standard_F4s", "Type": "Standard_F4s","Ram": "8.0 GB","Cpu": "4"},
+ {"Size": "M", "Description": "Standard_F8s", "Type": "Standard_F8s","Ram": "16.0 GB","Cpu": "8"},
+ {"Size": "L", "Description": "Standard_F16s", "Type": "Standard_F16s","Ram": "32.0 GB","Cpu": "16"}
+ ],
+ "GPU optimized": [
+ {"Size": "S", "Description": "Standard_NC6", "Type": "Standard_NC6","Ram": "56.0 GB","Cpu": "6"}
+ ]
+ },
+ "templates":
+ [
+ ]
+}
\ No newline at end of file
diff --git a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_configure.py b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_configure.py
index e69de29bb..9973c3d5c 100644
--- a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_configure.py
+++ b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_configure.py
@@ -0,0 +1,60 @@
+#!/usr/bin/python3
+
+# *****************************************************************************
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# ******************************************************************************
+
+if __name__ == "__main__":
+ try:
+ data_engine['service_base_name'] = os.environ['conf_service_base_name']
+ data_engine['resource_group_name'] = os.environ['azure_resource_group_name']
+ data_engine['region'] = os.environ['azure_region']
+ data_engine['key_name'] = os.environ['conf_key_name']
+ data_engine['vpc_name'] = os.environ['azure_vpc_name']
+ data_engine['user_name'] = os.environ['edge_user_name']
+ data_engine['project_name'] = os.environ['project_name']
+ data_engine['project_tag'] = data_engine['project_name']
+ data_engine['endpoint_name'] = os.environ['endpoint_name']
+ data_engine['endpoint_tag'] = data_engine['endpoint_name']
+ data_engine['master_node_name'] = '{}-m'.format(data_engine['cluster_name'])
+ data_engine['key_name'] = os.environ['conf_key_name']
+ if 'computational_name' in os.environ:
+ data_engine['computational_name'] = os.environ['computational_name']
+ else:
+ data_engine['computational_name'] = ''
+ data_engine['cluster_name'] = '{}-{}-{}-des-{}'.format(data_engine['service_base_name'],
+ data_engine['project_name'],
+ data_engine['endpoint_name'],
+ data_engine['computational_name'])
+ with open("/root/result.json", 'w') as result:
+ res = {"hostname": data_engine['cluster_name'],
+ "instance_id": data_engine['master_node_name'],
+ "key_name": data_engine['key_name'],
+ "Action": "Create new HDInsight cluster",
+ "computational_url": [
+ {"description": "HDInsight cluster",
+ "url": "spark_master_access_url"}
+ # {"description": "Apache Spark Master (via tunnel)",
+ # "url": spark_master_url}
+ ]
+ }
+ result.write(json.dumps(res))
+ except:
+ pass
\ No newline at end of file
diff --git a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
index 140495368..177275f78 100644
--- a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
+++ b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
@@ -103,5 +103,5 @@ def create_cluster_parameters():
)
if __name__ == "__main__":
- params = create_cluster_parameters()
- create_hdinsight_cluster(RESOURCE_GROUP_NAME,CLUSTER_NAME, params)
\ No newline at end of file
+ #params = create_cluster_parameters()
+ #create_hdinsight_cluster(RESOURCE_GROUP_NAME,CLUSTER_NAME, params)
\ No newline at end of file
diff --git a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_terminate.py b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_terminate.py
new file mode 100644
index 000000000..02466e026
--- /dev/null
+++ b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_terminate.py
@@ -0,0 +1,99 @@
+#!/usr/bin/python3
+
+# *****************************************************************************
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# ******************************************************************************
+
+import boto3
+import datalab.actions_lib
+import datalab.fab
+import datalab.meta_lib
+import json
+import os
+import sys
+import traceback
+from datalab.logger import logging
+
+
+def terminate_hdin_cluster(hdin_name, bucket_name, tag_name, nb_tag_value, ssh_user, key_path):
+ logging.info('Terminating hdin cluster and cleaning hdin config from S3 bucket')
+ # try:
+ # clusters_list = datalab.meta_lib.get_hdin_list(hdin_name, 'Value')
+ # if clusters_list:
+ # for cluster_id in clusters_list:
+ # computational_name = ''
+ # client = boto3.client('hdin')
+ # cluster = client.describe_cluster(ClusterId=cluster_id)
+ # cluster = cluster.get("Cluster")
+ # hdin_name = cluster.get('Name')
+ # hdin_version = cluster.get('ReleaseLabel')
+ # for tag in cluster.get('Tags'):
+ # if tag.get('Key') == 'ComputationalName':
+ # computational_name = tag.get('Value')
+ # datalab.actions_lib.s3_cleanup(bucket_name, hdin_name, os.environ['project_name'])
+ # print("The bucket {} has been cleaned successfully".format(bucket_name))
+ # datalab.actions_lib.terminate_hdin(cluster_id)
+ # print("The hdin cluster {} has been terminated successfully".format(hdin_name))
+ # print("Removing hdin kernels from notebook")
+ # datalab.actions_lib.remove_kernels(hdin_name, tag_name, nb_tag_value, ssh_user, key_path,
+ # hdin_version, computational_name)
+ # else:
+ # logging.info("There are no hdin clusters to terminate.")
+ except:
+ sys.exit(1)
+
+
+if __name__ == "__main__":
+ # generating variables dictionary
+ datalab.actions_lib.create_aws_config_files()
+ logging.info('Generating infrastructure names and tags')
+ hdin_conf = dict()
+ hdin_conf['service_base_name'] = (os.environ['conf_service_base_name'])
+ hdin_conf['hdin_name'] = os.environ['computational_name']
+ hdin_conf['notebook_name'] = os.environ['notebook_instance_name']
+ hdin_conf['project_name'] = os.environ['project_name']
+ hdin_conf['endpoint_name'] = os.environ['endpoint_name']
+ hdin_conf['bucket_name'] = '{0}-{1}-{2}-bucket'.format(hdin_conf['service_base_name'], hdin_conf['project_name'],
+ hdin_conf['endpoint_name']).lower().replace('_', '-')
+ hdin_conf['key_path'] = os.environ['conf_key_dir'] + '/' + os.environ['conf_key_name'] + '.pem'
+ hdin_conf['tag_name'] = hdin_conf['service_base_name'] + '-tag'
+
+ # try:
+ # logging.info('[TERMINATE hdin CLUSTER]')
+ # try:
+ # terminate_hdin_cluster(hdin_conf['hdin_name'], hdin_conf['bucket_name'], hdin_conf['tag_name'],
+ # hdin_conf['notebook_name'], os.environ['conf_os_user'], hdin_conf['key_path'])
+ # except Exception as err:
+ # traceback.print_exc()
+ # datalab.fab.append_result("Failed to terminate hdin cluster.", str(err))
+ # raise Exception
+ # except:
+ # sys.exit(1)
+
+ try:
+ with open("/root/result.json", 'w') as result:
+ res = {"dataengine-service_name": hdin_conf['hdin_name'],
+ "notebook_name": hdin_conf['notebook_name'],
+ "Action": "Terminate HDInsight cluster"}
+ print(json.dumps(res))
+ result.write(json.dumps(res))
+ except Exception as err:
+ datalab.fab.append_result("Error with writing results", str(err))
+ sys.exit(1)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 20/36: revert changes
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit fde2b5d738f89c276e61abe47adf5b0e2e3bdebf
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Thu May 26 14:58:44 2022 +0300
revert changes
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index e9f080873..4238ece7d 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1501,9 +1501,9 @@ def find_des_jars(all_jars, des_path):
print('Error:', str(err))
sys.exit(1)
-def create_hdinsight_cluster(self, resource_group_name, instance_name, cluster_parameters):
+def create_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__()self.hdinsight_client
+ client_1 = AzureActions.__init__()hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 01/36: added files and func to actions lib
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 2a476673982675ad11e44959aaa881d274c599d7
Author: owlleg6 <ow...@gmail.com>
AuthorDate: Tue May 17 17:50:37 2022 +0300
added files and func to actions lib
---
.../src/general/lib/azure/actions_lib.py | 29 +++++++++
.../scripts/azure/dataengine-service_configure.py | 0
.../scripts/azure/dataengine-service_create.py | 0
.../scripts/azure/dataengine-service_prepare.py | 74 ++++++++++++++++++++++
4 files changed, 103 insertions(+)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 59acc388f..f68e2de48 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1494,3 +1494,32 @@ def find_des_jars(all_jars, des_path):
except Exception as err:
print('Error:', str(err))
sys.exit(1)
+
+def create_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
+ try:
+ client_1 = AzureActions.__init__().hdinsight_client
+ print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
+ result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
+
+ return result
+ except Exception as err:
+ logging.info(
+ "Unable to create HDInsight Spark cluster: " + str(err) + "\n Traceback: " + traceback.print_exc(file=sys.stdout))
+ append_result(str({"error": "Unable to create HDInsight Spark cluster",
+ "error_message": str(err) + "\n Traceback: " + traceback.print_exc(
+ file=sys.stdout)}))
+ traceback.print_exc(file=sys.stdout)
+
+
+def terminate_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
+ try:
+ client_1 = AzureActions.__init__().hdinsight_client
+ client_1.clusters.begin_delete(resource_group_name, instance_name, cluster_parameters)
+
+ except Exception as err:
+ logging.info(
+ "Unable to delete HDInsight Spark cluster: " + str(err) + "\n Traceback: " + traceback.print_exc(file=sys.stdout))
+ append_result(str({"error": "Unable to create HDInsight Spark cluster",
+ "error_message": str(err) + "\n Traceback: " + traceback.print_exc(
+ file=sys.stdout)}))
+ traceback.print_exc(file=sys.stdout)
\ No newline at end of file
diff --git a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_configure.py b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_configure.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_create.py b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_create.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
new file mode 100644
index 000000000..87ac3ab88
--- /dev/null
+++ b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
@@ -0,0 +1,74 @@
+import datalab.actions_lib
+import datalab.fab
+import datalab.meta_lib
+import json
+from datalab.logger import logging
+import multiprocessing
+import os
+import sys
+import traceback
+import subprocess
+from Crypto.PublicKey import RSA
+from fabric import *
+from azure.mgmt.hdinsight.models import *
+
+
+
+def create_cluster_parameters():
+ # Returns cluster parameters
+
+ return ClusterCreateParametersExtended(
+ location=LOCATION,
+ tags={},
+ properties=ClusterCreateProperties(
+ cluster_version="4.0",
+ os_type=OSType.linux,
+ tier=Tier.standard,
+ cluster_definition=ClusterDefinition(
+ kind="Spark",
+ configurations={
+ "gateway": {
+ "restAuthCredential.isEnabled": "true",
+ "restAuthCredential.username": CLUSTER_LOGIN_USER_NAME,
+ "restAuthCredential.password": PASSWORD
+ }
+ }
+ ),
+ compute_profile=ComputeProfile(
+ roles=[
+ Role(
+ name="headnode",
+ target_instance_count=2,
+ hardware_profile=HardwareProfile(vm_size="Standard_E8_V3"),
+ os_profile=OsProfile(
+ linux_operating_system_profile=LinuxOperatingSystemProfile(
+ username=SSH_USER_NAME,
+ password=PASSWORD
+ )
+ )
+ ),
+ Role(
+ name="workernode",
+ target_instance_count=2,
+ hardware_profile=HardwareProfile(vm_size="Standard_E8_V3"),
+ os_profile=OsProfile(
+ linux_operating_system_profile=LinuxOperatingSystemProfile(
+ username=SSH_USER_NAME,
+ password=PASSWORD
+ )
+ )
+ )
+ ]
+ ),
+ storage_profile=StorageProfile(
+ storageaccounts=[
+ StorageAccount(
+ name=STORAGE_ACCOUNT_NAME + BLOB_ENDPOINT_SUFFIX,
+ key=STORAGE_ACCOUNT_KEY,
+ container=CONTAINER_NAME.lower(),
+ is_default=True
+ )
+ ]
+ )
+ )
+ )
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 12/36: add import self and add self to func
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit dddfd288dad64eadad756f65198df518ce875639
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Wed May 25 17:19:29 2022 +0300
add import self and add self to func
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 8f0d31750..05d755d0c 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -31,6 +31,8 @@ import os
import sys
import time
import traceback
+
+import self as self
import urllib3
import urllib.request
import subprocess
@@ -1503,7 +1505,7 @@ def find_des_jars(all_jars, des_path):
def create_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__().hdinsight_client
+ client_1 = AzureActions.__init__(self).hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
@@ -1519,7 +1521,7 @@ def create_hdinsight_cluster(resource_group_name, instance_name, cluster_paramet
def terminate_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__().hdinsight_client
+ client_1 = AzureActions.__init__(self).hdinsight_client
client_1.clusters.begin_delete(resource_group_name, instance_name, cluster_parameters)
except Exception as err:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 15/36: changes in def hdinsight
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit c91deaf604b2316c5444269e3024c6b557f21444
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Thu May 26 13:40:49 2022 +0300
changes in def hdinsight
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index d4cbf83cd..6794acaf5 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1505,7 +1505,7 @@ def create_hdinsight_cluster(self, resource_group_name, instance_name, cluster_p
try:
client_1 = AzureActions.__init__().hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
- result = self, client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
+ result = self.client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
return result
except Exception as err:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 27/36: change cluster name
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 2fc07eccc14fc11c22e68a314e4a306485876678
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Thu May 26 16:42:28 2022 +0300
change cluster name
---
.../src/general/scripts/azure/dataengine-service_prepare.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
index de956ec8c..87d99815c 100644
--- a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
+++ b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
@@ -22,7 +22,7 @@ from azure.core import *
from datalab.actions_lib import *
-CLUSTER_NAME = 'hdinsight_1'
+CLUSTER_NAME = 'hdinsight-test'
# The name of your existing Resource Group
RESOURCE_GROUP_NAME = 'dlab-resource-group'
# Choose a region. i.e. "East US 2".
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 18/36: edit actions_lib
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 3dc6e239cda86e34c09ddb08752d4bad8cec37f7
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Thu May 26 14:52:45 2022 +0300
edit actions_lib
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 8f0d31750..9c33132da 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1501,9 +1501,9 @@ def find_des_jars(all_jars, des_path):
print('Error:', str(err))
sys.exit(1)
-def create_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
+def create_hdinsight_cluster(self, resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__().hdinsight_client
+ self.client_1 = AzureActions.__init__().hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 22/36: revert changes
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit b56086d20bace6bde91c445e04e45effa7486c27
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Thu May 26 15:12:58 2022 +0300
revert changes
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 3675c92b1..4238ece7d 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1501,11 +1501,11 @@ def find_des_jars(all_jars, des_path):
print('Error:', str(err))
sys.exit(1)
-def create_hdinsight_cluster(self, resource_group_name, instance_name, cluster_parameters):
+def create_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
try:
- self.client_1 = AzureActions.__init__()hdinsight_client
+ client_1 = AzureActions.__init__()hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
- result = self.client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
+ result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
return result
except Exception as err:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 23/36: changes
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit d0543dfb3c0f5977cab97ae11ecd13c8fb1a777b
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Thu May 26 15:21:50 2022 +0300
changes
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 4238ece7d..b1d951f76 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1503,7 +1503,8 @@ def find_des_jars(all_jars, des_path):
def create_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__()hdinsight_client
+ client_1 = AzureActions()
+ #client_1 = AzureActions.__init__()hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 30/36: refactor dataengine-service_prepare
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit c6b536591cdc30d796be7402186dda89664a85f1
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Tue May 31 14:23:01 2022 +0300
refactor dataengine-service_prepare
---
.../src/general/scripts/azure/dataengine-service_prepare.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
index 2011b4aff..140495368 100644
--- a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
+++ b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
@@ -43,6 +43,7 @@ CONTAINER_NAME = 'hdinsight'
BLOB_ENDPOINT_SUFFIX = '.blob.core.windows.net'
def create_cluster_parameters():
+
# Returns cluster parameters
return ClusterCreateParametersExtended(
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 07/36: delete hdinsight_client
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit dcac403ad9f5c505050c0d8af8b7ee8ac0c2589e
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Wed May 25 16:05:19 2022 +0300
delete hdinsight_client
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 2 --
1 file changed, 2 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 0e30da5d2..8f0d31750 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -49,8 +49,6 @@ from patchwork import files
class AzureActions:
- hdinsight_client: HDInsightManagementClient
-
def __init__(self):
os.environ['AZURE_AUTH_LOCATION'] = '/root/azure_auth.json'
with open('/root/azure_auth.json') as json_file:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 02/36: params and creds
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 727439e1d9d014d248b58f6a689695901d7fe721
Author: owlleg6 <ow...@gmail.com>
AuthorDate: Wed May 18 16:42:02 2022 +0300
params and creds
---
.../src/general/lib/azure/actions_lib.py | 6 +++++
.../scripts/azure/dataengine-service_prepare.py | 27 ++++++++++++++++++++--
2 files changed, 31 insertions(+), 2 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index f68e2de48..8f0d31750 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -42,6 +42,7 @@ from azure.mgmt.storage import StorageManagementClient
from azure.storage.blob import BlobServiceClient
from azure.identity import ClientSecretCredential
from azure.mgmt.datalake.store import DataLakeStoreAccountManagementClient
+from azure.mgmt.hdinsight import HDInsightManagementClient
from fabric import *
from patchwork.files import exists
from patchwork import files
@@ -89,6 +90,11 @@ class AzureActions:
json_dict["subscriptionId"],
base_url=json_dict["resourceManagerEndpointUrl"]
)
+ self.hdinsight_client = HDInsightManagementClient(
+ self.credential,
+ json_dict["subscriptionId"],
+ base_url=json_dict["resourceManagerEndpointUrl"]
+ )
self.sp_creds = json.loads(open(os.environ['AZURE_AUTH_LOCATION']).read())
self.dl_filesystem_creds = lib.auth(tenant_id=json.dumps(self.sp_creds['tenantId']).replace('"', ''),
client_secret=json.dumps(self.sp_creds['clientSecret']).replace('"', ''),
diff --git a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
index 87ac3ab88..05260a0cd 100644
--- a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
+++ b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
@@ -1,4 +1,4 @@
-import datalab.actions_lib
+from datalab.actions_lib import *
import datalab.fab
import datalab.meta_lib
import json
@@ -13,6 +13,25 @@ from fabric import *
from azure.mgmt.hdinsight.models import *
+CLUSTER_NAME = 'hdinsight_1'
+# The name of your existing Resource Group
+RESOURCE_GROUP_NAME = 'dlab-resource-group'
+# Choose a region. i.e. "East US 2".
+LOCATION = 'West US 2'
+# Cluster login username
+CLUSTER_LOGIN_USER_NAME = 'datalab-user'
+# (SSH) user username
+SSH_USER_NAME = 'datalab-user'
+# Cluster admin password
+PASSWORD = ''
+# The name of blob storage account
+STORAGE_ACCOUNT_NAME = 'hdinsight'
+# Blob storage account key
+STORAGE_ACCOUNT_KEY = ''
+# Blob storage account container name
+CONTAINER_NAME = 'hdinsight'
+# Blob Storage endpoint suffix.
+BLOB_ENDPOINT_SUFFIX = '.blob.core.windows.net'
def create_cluster_parameters():
# Returns cluster parameters
@@ -71,4 +90,8 @@ def create_cluster_parameters():
]
)
)
- )
\ No newline at end of file
+ )
+
+if __name__ == "__main__":
+ params = create_cluster_parameters()
+ create_hdinsight_cluster(RESOURCE_GROUP_NAME,CLUSTER_NAME, params)
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 05/36: add imports
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 2bf412f826bac4d13a1f1818ce131edefa12c7e4
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Wed May 25 13:14:32 2022 +0300
add imports
---
.../general/scripts/azure/dataengine-service_prepare.py | 17 +++++++++++++----
1 file changed, 13 insertions(+), 4 deletions(-)
diff --git a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
index 05260a0cd..de956ec8c 100644
--- a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
+++ b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
@@ -1,16 +1,25 @@
-from datalab.actions_lib import *
+#!/usr/bin/python3
+
+import datalab.actions_lib
import datalab.fab
import datalab.meta_lib
import json
-from datalab.logger import logging
+import datalab.logger
import multiprocessing
import os
import sys
import traceback
import subprocess
-from Crypto.PublicKey import RSA
-from fabric import *
+import Crypto.PublicKey
+import fabric
+import azure.mgmt.hdinsight.models
+#from Crypto.PublicKey import RSA
+#from fabric import *
from azure.mgmt.hdinsight.models import *
+from azure.mgmt.core import *
+from azure.common import *
+from azure.core import *
+from datalab.actions_lib import *
CLUSTER_NAME = 'hdinsight_1'
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 24/36: some changes
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit b2df54a2d9c5f1fde5bdb1990134bd58f4a1db63
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Thu May 26 15:24:46 2022 +0300
some changes
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index b1d951f76..6cd7de874 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1503,7 +1503,7 @@ def find_des_jars(all_jars, des_path):
def create_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions()
+ client_1 = AzureActions().hdinsight_client
#client_1 = AzureActions.__init__()hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 10/36: modify func add self=
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 189d2c0c4e8c317cf6525f024a8de9246111b3dc
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Wed May 25 17:01:36 2022 +0300
modify func add self=
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index d7b89f8ab..cac66816b 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1501,9 +1501,9 @@ def find_des_jars(all_jars, des_path):
print('Error:', str(err))
sys.exit(1)
-def create_hdinsight_cluster(self, resource_group_name, instance_name, cluster_parameters):
+def create_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__().hdinsight_client
+ client_1 = AzureActions.__init__(self=).hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
@@ -1519,7 +1519,7 @@ def create_hdinsight_cluster(self, resource_group_name, instance_name, cluster_p
def terminate_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__().hdinsight_client
+ client_1 = AzureActions.__init__(self=).hdinsight_client
client_1.clusters.begin_delete(resource_group_name, instance_name, cluster_parameters)
except Exception as err:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 31/36: fix errors
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit fa7e32fc6be486956131a65b998e001623e63a77
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Wed Jun 8 17:07:40 2022 +0300
fix errors
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 800a57b0d..17e9ff66b 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1526,8 +1526,8 @@ def terminate_hdinsight_cluster(resource_group_name, instance_name, cluster_para
except Exception as err:
logging.info(
- "Unable to delete HDInsight Spark cluster: " + str(err) + "\n Traceback: " + traceback.print_exc(file=sys.stdout))
- append_result(str({"error": "Unable to create HDInsight Spark cluster",
+ "Unable to terminate HDInsight Spark cluster: " + str(err) + "\n Traceback: " + traceback.print_exc(file=sys.stdout))
+ append_result(str({"error": "Unable to terminate HDInsight Spark cluster",
"error_message": str(err) + "\n Traceback: " + traceback.print_exc(
file=sys.stdout)}))
traceback.print_exc(file=sys.stdout)
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 32/36: add print to def_terminate
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 23223bd7e91971b446cc1d770c7340abc3d52eea
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Wed Jun 8 18:12:09 2022 +0300
add print to def_terminate
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 17e9ff66b..cdd3dc8f9 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1522,6 +1522,7 @@ def terminate_hdinsight_cluster(resource_group_name, instance_name, cluster_para
try:
azure_action = AzureActions()
client_1 = azure_action.hdinsight_client
+ print('Starting to terminate HDInsight Spark cluster {}'.format('hdinsight'))
client_1.clusters.begin_delete(resource_group_name, instance_name, cluster_parameters)
except Exception as err:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 04/36: add libs in base_Dockerfile
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 9c613a6d7745cc02c069ecb98176b697bdb0bf04
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Wed May 25 13:09:05 2022 +0300
add libs in base_Dockerfile
---
infrastructure-provisioning/src/general/files/azure/base_Dockerfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/infrastructure-provisioning/src/general/files/azure/base_Dockerfile b/infrastructure-provisioning/src/general/files/azure/base_Dockerfile
index 7653d7dbd..03a97dda1 100644
--- a/infrastructure-provisioning/src/general/files/azure/base_Dockerfile
+++ b/infrastructure-provisioning/src/general/files/azure/base_Dockerfile
@@ -33,7 +33,7 @@ RUN apt-get update && \
RUN python3 -m pip install -UI qtconsole==5.1.1 pip==21.1.2 && \
python3 -m pip install backoff patchwork fabric fabvenv argparse requests ujson jupyter pycryptodome azure-common pyyaml && \
python3 -m pip install azure-mgmt-storage azure-mgmt-compute azure-mgmt-core azure-mgmt-network azure-storage-blob azure-mgmt-resource && \
- python3 -m pip install azure-storage-file-datalake azure-datalake-store azure-identity azure-mgmt-datalake-store azure-mgmt-hdinsight
+ python3 -m pip install azure-storage-file-datalake azure-datalake-store azure-identity azure-mgmt-datalake-store azure-mgmt-hdinsight azure-common azure-core
# Configuring ssh for user
RUN mkdir -p /root/.ssh; echo "Host *" > /root/.ssh/config; \
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 14/36: add self before client creation
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit c3be8d16bd175b000e355afe5021d05aa9e365db
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Thu May 26 13:36:20 2022 +0300
add self before client creation
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 8f0d31750..d4cbf83cd 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1501,11 +1501,11 @@ def find_des_jars(all_jars, des_path):
print('Error:', str(err))
sys.exit(1)
-def create_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
+def create_hdinsight_cluster(self, resource_group_name, instance_name, cluster_parameters):
try:
client_1 = AzureActions.__init__().hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
- result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
+ result = self, client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
return result
except Exception as err:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 19/36: changes
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit bd88e30d740ea4898e9d4c2a66f87a37fdeccb86
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Thu May 26 14:56:00 2022 +0300
changes
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 9c33132da..e9f080873 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1503,7 +1503,7 @@ def find_des_jars(all_jars, des_path):
def create_hdinsight_cluster(self, resource_group_name, instance_name, cluster_parameters):
try:
- self.client_1 = AzureActions.__init__().hdinsight_client
+ client_1 = AzureActions.__init__()self.hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 13/36: remove self from func and from import
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit d18321b3634724ee728564b804db59fc6d213b37
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Wed May 25 17:23:25 2022 +0300
remove self from func and from import
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 05d755d0c..8f0d31750 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -31,8 +31,6 @@ import os
import sys
import time
import traceback
-
-import self as self
import urllib3
import urllib.request
import subprocess
@@ -1505,7 +1503,7 @@ def find_des_jars(all_jars, des_path):
def create_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__(self).hdinsight_client
+ client_1 = AzureActions.__init__().hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
@@ -1521,7 +1519,7 @@ def create_hdinsight_cluster(resource_group_name, instance_name, cluster_paramet
def terminate_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__(self).hdinsight_client
+ client_1 = AzureActions.__init__().hdinsight_client
client_1.clusters.begin_delete(resource_group_name, instance_name, cluster_parameters)
except Exception as err:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 25/36: changes
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 9e4e3ca79c89abb6f3b704eba090d6627f444921
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Thu May 26 15:28:10 2022 +0300
changes
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 6cd7de874..70e305f64 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1503,7 +1503,8 @@ def find_des_jars(all_jars, des_path):
def create_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions().hdinsight_client
+ azure_action = AzureActions()
+ client_1 = azure_action.hdinsight_client
#client_1 = AzureActions.__init__()hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 29/36: changes
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 42ed96c990e7ff45272f9476de2fd1527e68b8fa
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Mon May 30 12:18:41 2022 +0300
changes
---
.../src/general/scripts/azure/dataengine-service_prepare.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
index 87d99815c..2011b4aff 100644
--- a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
+++ b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
@@ -67,7 +67,7 @@ def create_cluster_parameters():
Role(
name="headnode",
target_instance_count=2,
- hardware_profile=HardwareProfile(vm_size="Standard_E8_V3"),
+ hardware_profile=HardwareProfile(vm_size="Standard_A4_v2"),
os_profile=OsProfile(
linux_operating_system_profile=LinuxOperatingSystemProfile(
username=SSH_USER_NAME,
@@ -78,7 +78,7 @@ def create_cluster_parameters():
Role(
name="workernode",
target_instance_count=2,
- hardware_profile=HardwareProfile(vm_size="Standard_E8_V3"),
+ hardware_profile=HardwareProfile(vm_size="Standard_A4_v2"),
os_profile=OsProfile(
linux_operating_system_profile=LinuxOperatingSystemProfile(
username=SSH_USER_NAME,
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 34/36: [DATALAB-2982]: changed cluster name variable
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 5794a7399ccbf2906803a05f90276fd8583bf429
Author: leonidfrolov <fr...@gmail.com>
AuthorDate: Tue Aug 16 11:24:10 2022 +0300
[DATALAB-2982]: changed cluster name variable
---
.../src/general/scripts/azure/dataengine-service_terminate.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_terminate.py b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_terminate.py
index 02466e026..bde44edb7 100644
--- a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_terminate.py
+++ b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_terminate.py
@@ -66,7 +66,7 @@ if __name__ == "__main__":
logging.info('Generating infrastructure names and tags')
hdin_conf = dict()
hdin_conf['service_base_name'] = (os.environ['conf_service_base_name'])
- hdin_conf['hdin_name'] = os.environ['computational_name']
+ hdin_conf['hdin_name'] = os.environ['hdinsight_cluster_name']
hdin_conf['notebook_name'] = os.environ['notebook_instance_name']
hdin_conf['project_name'] = os.environ['project_name']
hdin_conf['endpoint_name'] = os.environ['endpoint_name']
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 35/36: [DATALAB-1408]: added some creation changes
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 14f36c05fd0ce5f2c92589329649f382f41f399c
Author: leonidfrolov <fr...@gmail.com>
AuthorDate: Tue Aug 23 16:01:19 2022 +0300
[DATALAB-1408]: added some creation changes
---
.../scripts/azure/dataengine-service_prepare.py | 107 ++++++++++++++++++++-
.../scripts/gcp/dataengine-service_create.py | 2 +-
2 files changed, 105 insertions(+), 4 deletions(-)
diff --git a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
index 177275f78..7bd369910 100644
--- a/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
+++ b/infrastructure-provisioning/src/general/scripts/azure/dataengine-service_prepare.py
@@ -1,5 +1,26 @@
#!/usr/bin/python3
+# *****************************************************************************
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# ******************************************************************************
+
import datalab.actions_lib
import datalab.fab
import datalab.meta_lib
@@ -42,7 +63,7 @@ CONTAINER_NAME = 'hdinsight'
# Blob Storage endpoint suffix.
BLOB_ENDPOINT_SUFFIX = '.blob.core.windows.net'
-def create_cluster_parameters():
+def create_cluster_parameters(LOCATION, CLUSTER_LOGIN_USER_NAME, PASSWORD, SSH_USER_NAME):
# Returns cluster parameters
@@ -92,7 +113,7 @@ def create_cluster_parameters():
storage_profile=StorageProfile(
storageaccounts=[
StorageAccount(
- name=STORAGE_ACCOUNT_NAME + BLOB_ENDPOINT_SUFFIX,
+ name=STORAGE_ACCOUNT_NAME + ".blob.core.windows.net",
key=STORAGE_ACCOUNT_KEY,
container=CONTAINER_NAME.lower(),
is_default=True
@@ -104,4 +125,84 @@ def create_cluster_parameters():
if __name__ == "__main__":
#params = create_cluster_parameters()
- #create_hdinsight_cluster(RESOURCE_GROUP_NAME,CLUSTER_NAME, params)
\ No newline at end of file
+ #create_hdinsight_cluster(RESOURCE_GROUP_NAME,CLUSTER_NAME, params)
+
+ try:
+ AzureMeta = datalab.meta_lib.AzureMeta()
+ AzureActions = datalab.actions_lib.AzureActions()
+ logging.info('Generating infrastructure names and tags')
+ hdinsight_conf = dict()
+ if 'exploratory_name' in os.environ:
+ hdinsight_conf['exploratory_name'] = os.environ['exploratory_name'].replace('_', '-').lower()
+ else:
+ hdinsight_conf['exploratory_name'] = ''
+ if 'computational_name' in os.environ:
+ hdinsight_conf['computational_name'] = os.environ['computational_name'].replace('_', '-').lower()
+ else:
+ hdinsight_conf['computational_name'] = ''
+
+ hdinsight_conf['service_base_name'] = (os.environ['conf_service_base_name'])
+ hdinsight_conf['edge_user_name'] = (os.environ['edge_user_name'])
+ hdinsight_conf['project_name'] = (os.environ['project_name']).replace('_', '-').lower()
+ hdinsight_conf['project_tag'] = hdinsight_conf['project_name']
+ hdinsight_conf['endpoint_name'] = (os.environ['endpoint_name']).replace('_', '-').lower()
+ hdinsight_conf['endpoint_tag'] = hdinsight_conf['endpoint_name']
+ hdinsight_conf['key_name'] = os.environ['conf_key_name']
+ hdinsight_conf['key_path'] = '{0}{1}.pem'.format(os.environ['conf_key_dir'], os.environ['conf_key_name'])
+ hdinsight_conf['zone'] = os.environ['gcp_zone']
+ hdinsight_conf['resource_group_name'] = os.environ['azure_resource_group_name']
+ hdinsight_conf['region'] = os.environ['azure_region']
+ data_engine['vpc_name'] = os.environ['azure_vpc_name']
+ data_engine['private_subnet_name'] = '{}-{}-{}-subnet'.format(data_engine['service_base_name'],
+ data_engine['project_name'],
+ data_engine['endpoint_name'])
+ data_engine['private_subnet_cidr'] = AzureMeta.get_subnet(data_engine['resource_group_name'],
+ data_engine['vpc_name'],
+ data_engine['private_subnet_name']).address_prefix
+ data_engine['cluster_name'] = '{}-{}-{}-des-{}'.format(data_engine['service_base_name'],
+ data_engine['project_name'],
+ data_engine['endpoint_name'],
+ data_engine['computational_name'])
+
+
+
+ hdinsight_conf['subnet'] = '{0}-{1}-{2}-subnet'.format(hdinsight_conf['service_base_name'],
+ hdinsight_conf['project_name'],
+ hdinsight_conf['endpoint_name'])
+ hdinsight_conf['cluster_name'] = '{0}-{1}-{2}-des-{3}'.format(hdinsight_conf['service_base_name'],
+ hdinsight_conf['project_name'],
+ hdinsight_conf['endpoint_name'],
+ hdinsight_conf['computational_name'])
+ hdinsight_conf['cluster_tag'] = '{0}-{1}-{2}-ps'.format(hdinsight_conf['service_base_name'],
+ hdinsight_conf['project_name'],
+ hdinsight_conf['endpoint_name'])
+ hdinsight_conf['bucket_name'] = '{0}-{1}-{2}-bucket'.format(hdinsight_conf['service_base_name'],
+ hdinsight_conf['project_name'],
+ hdinsight_conf['endpoint_name'])
+
+ hdinsight_conf['edge_instance_hostname'] = '{0}-{1}-{2}-edge'.format(hdinsight_conf['service_base_name'],
+ hdinsight_conf['project_name'],
+ hdinsight_conf['endpoint_name'])
+ hdinsight_conf['datalab_ssh_user'] = os.environ['conf_os_user']
+ except Exception as err:
+ datalab.fab.append_result("Failed to generate variables dictionary. Exception:" + str(err))
+ sys.exit(1)
+
+ try:
+ logging.info('[Creating HDInsight Cluster]')
+ params = "--region {0} --bucket {1} --params '{2}'".format(hdinsight_conf['region'],
+ hdinsight_conf['bucket_name'],
+ json.dumps(hdinsight_cluster))
+
+ try:
+ subprocess.run("~/scripts/{}.py {}".format('dataengine-service_create', params), shell=True, check=True)
+ except:
+ traceback.print_exc()
+ raise Exception
+
+ keyfile_name = "/root/keys/{}.pem".format(hdinsight_conf['key_name'])
+ subprocess.run('rm /response/.hdinsight_creating_{}'.format(os.environ['exploratory_name']), shell=True, check=True)
+ except Exception as err:
+ datalab.fab.append_result("Failed to create hdinsight Cluster.", str(err))
+ subprocess.run('rm /response/.hdinsight_creating_{}'.format(os.environ['exploratory_name']), shell=True, check=True)
+ sys.exit(1)
diff --git a/infrastructure-provisioning/src/general/scripts/gcp/dataengine-service_create.py b/infrastructure-provisioning/src/general/scripts/gcp/dataengine-service_create.py
index 978d2d4fe..2ea506d7b 100644
--- a/infrastructure-provisioning/src/general/scripts/gcp/dataengine-service_create.py
+++ b/infrastructure-provisioning/src/general/scripts/gcp/dataengine-service_create.py
@@ -81,4 +81,4 @@ if __name__ == "__main__":
build_dataproc_cluster(args, cluster_name)
send_parser_job(args, cluster_name, cluster_version)
- sys.exit(0)
\ No newline at end of file
+ sys.exit(0)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 08/36: modify hdinsight func
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 8dffa5e89b04c58be178d36c84cbf704d6f8afbb
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Wed May 25 16:46:51 2022 +0300
modify hdinsight func
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 8f0d31750..021d0accc 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1501,9 +1501,9 @@ def find_des_jars(all_jars, des_path):
print('Error:', str(err))
sys.exit(1)
-def create_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
+def create_hdinsight_cluster(self, resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__().hdinsight_client
+ client_1 = AzureActions.__init__(self=).hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
@@ -1519,7 +1519,7 @@ def create_hdinsight_cluster(resource_group_name, instance_name, cluster_paramet
def terminate_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__().hdinsight_client
+ client_1 = AzureActions.__init__(self=).hdinsight_client
client_1.clusters.begin_delete(resource_group_name, instance_name, cluster_parameters)
except Exception as err:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 21/36: changes
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 6fa45158dfd11e7539ff862dd3c792d5b239aac8
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Thu May 26 15:11:22 2022 +0300
changes
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 4238ece7d..3675c92b1 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1501,11 +1501,11 @@ def find_des_jars(all_jars, des_path):
print('Error:', str(err))
sys.exit(1)
-def create_hdinsight_cluster(resource_group_name, instance_name, cluster_parameters):
+def create_hdinsight_cluster(self, resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__()hdinsight_client
+ self.client_1 = AzureActions.__init__()hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
- result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
+ result = self.client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
return result
except Exception as err:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 16/36: some changes
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 17fd0c1fa8653450fad4ea13cf1d305159a27004
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Thu May 26 13:45:32 2022 +0300
some changes
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 6794acaf5..c270f5c23 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -1503,9 +1503,9 @@ def find_des_jars(all_jars, des_path):
def create_hdinsight_cluster(self, resource_group_name, instance_name, cluster_parameters):
try:
- client_1 = AzureActions.__init__().hdinsight_client
+ client_1 = AzureActions.__init__(self).hdinsight_client
print('Starting to create HDInsight Spark cluster {}'.format('hdinsight'))
- result = self.client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
+ result = client_1.clusters.begin_create(resource_group_name, instance_name, cluster_parameters)
return result
except Exception as err:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 06/36: add class AzureActions
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-1408
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 008febb35b6205b61abff63dbadfca28cf4b7422
Author: orepnikov <60...@users.noreply.github.com>
AuthorDate: Wed May 25 15:45:50 2022 +0300
add class AzureActions
---
infrastructure-provisioning/src/general/lib/azure/actions_lib.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 8f0d31750..0e30da5d2 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -49,6 +49,8 @@ from patchwork import files
class AzureActions:
+ hdinsight_client: HDInsightManagementClient
+
def __init__(self):
os.environ['AZURE_AUTH_LOCATION'] = '/root/azure_auth.json'
with open('/root/azure_auth.json') as json_file:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org