You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dlab.apache.org by om...@apache.org on 2019/11/01 14:50:55 UTC
[incubator-dlab] branch master updated: DLAB documentation for
v.2.1.1 (Ldap parameters as mandatory marked)
This is an automated email from the ASF dual-hosted git repository.
omartushevskyi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git
The following commit(s) were added to refs/heads/master by this push:
new 55362b7 DLAB documentation for v.2.1.1 (Ldap parameters as mandatory marked)
new 842f8e4 Merge pull request #378 from bodnarmykola/DLAB_v.2.1.1_documentation
55362b7 is described below
commit 55362b7aa47db9a62ed63410275e14673f043bcd
Author: Mykola_Bodnar1 <bo...@gmail.com>
AuthorDate: Fri Nov 1 16:46:12 2019 +0200
DLAB documentation for v.2.1.1 (Ldap parameters as mandatory marked)
---
README.md | 930 ++++++++++++++++++++++++++++++++++++++++----------------------
1 file changed, 602 insertions(+), 328 deletions(-)
diff --git a/README.md b/README.md
index 8a3db02..a175e2d 100644
--- a/README.md
+++ b/README.md
@@ -15,19 +15,21 @@ CONTENTS
[DLab Deployment](#DLab_Deployment)
- [Preparing environment for DLab deployment](#Env_for_DLab)
-
[Structure of main DLab directory](#DLab_directory)
[Structure of log directory](#log_directory)
+ [Preparing environment for DLab deployment](#Env_for_DLab)
+
[Self-Service Node](#Self_Service_Node)
[Edge Node](#Edge_Node)
[Notebook node](#Notebook_node)
- [EMR cluster](#EMR_cluster)
+ [Dataengine-service cluster](#Dataengine-service_cluster)
+
+ [Dataengine cluster](#Dataengine_cluster)
[Configuration files](#Configuration_files)
@@ -103,12 +105,14 @@ Database serves as a storage with description of user infrastructure, user’s s
-----------------------------
# Physical architecture <a name="Physical_architecture"></a>
-The following diagrams demonstrate high-level physical architecture of DLab in AWS and Azure.
+The following diagrams demonstrate high-level physical architecture of DLab in AWS, Azure and GCP.
+
+
## Main components
- Self-service node (SSN)
@@ -155,32 +159,7 @@ That simplifies running big data frameworks, such as Apache Hadoop and Apache Sp
----------------------
# DLab Deployment <a name="DLab_Deployment"></a>
-## Preparing environment for DLab deployment <a name="Env_for_DLab"></a>
-
-#### In Amazon cloud
-If you want to deploy DLab from inside of your AWS account, you can use the following instruction:
-
-- Create an EC2 instance with the following settings:
- - Shape of the instance shouldn't be less than t2.medium
- - The instance should have access to Internet in order to install required prerequisites
- - The instance should have access to further DLab installation
- - AMI - Ubuntu 16.04
- - IAM role with [policy](#AWS_SSN_policy) should be assigned to the instance
-- Connect to the instance via SSH and run the following commands:
-```
- sudo su
- apt-get update
- curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
- add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
- apt-get update
- apt-cache policy docker-ce
- apt-get install -y docker-ce=17.06.2~ce-0~ubuntu
- usermod -a -G docker ubuntu
- pip install fabric==1.14.0
-```
-- Clone DLab repository and run deploy script.
-
-## Structure of main DLab directory <a name="DLab_directory"></a>
+### Structure of main DLab directory <a name="DLab_directory"></a>
DLab’s SSN node main directory structure is as follows:
@@ -201,7 +180,7 @@ DLab’s SSN node main directory structure is as follows:
- webapp – contains all .jar files for DLab Web UI and back-end
services.
-## Structure of log directory <a name="log_directory"></a>
+### Structure of log directory <a name="log_directory"></a>
SSN node structure of log directory is as follows:
@@ -224,112 +203,324 @@ These directories contain the log files for each template and for DLab back-end
## Self-Service Node <a name="Self_Service_Node"></a>
-### Create
+### Preparing environment for DLab deployment <a name="Env_for_DLab"></a>
-Deployment of DLab starts from creating Self-Service(SSN) node. DLab can be deployed in AWS, Azure and Google cloud.
+Deployment of DLab starts from creating Self-Service(SSN) node. DLab can be deployed in AWS, Azure and Google cloud.
For each cloud provider, prerequisites are different.
-#### In Amazon cloud
+<details><summary>In Amazon cloud <i>(click to expand)</i></summary>
+
+**Prerequisites:**
-Prerequisites:
+DLab can be deployed using one of the two options:
+ - **First option: using IAM user.** DLab deployment script is executed on local machine and uses IAM user permissions to create resources in AWS.
+ - **Second option (preferred) - using EC2 instance.** DLab deployment script is executed on EC2 instance prepared in advance and with attached IAM role. Deployment script uses the attached IAM role to create resources in AWS.
+
+**Requirements for the first option:**
+
+ - IAM user with created AWS access key ID and secret access key. These keys are provided as arguments for the deployment script and are used to create resources in AWS.
+ - Amazon EC2 Key Pair. This key is system and is used for configuring DLab instances.
+ - The following IAM [policy](#AWS_SSN_policy) should be attached to the IAM user in order to deploy DLab.
+
+ **Requirements for the second option:**
+
+ - Amazon EC2 Key Pair. This key is system and is used for configuring DLab instances.
+ - EC2 instance where DLab deployment script is executed.
+ - IAM role with the following IAM [policy](#AWS_SSN_policy) should be attached to the EC2 instance.
+
+ **Requirements for both options(optional):**
+
+ - VPC ID. If VPC where DLab should be deployed is already in place, then "VPC ID" should be provided for deployment script. DLab instances are deployed in this VPC.
+ - Subnet ID. If Subnet where DLab should be deployed is already in place, then "Subnet ID" should be provided for deployment script. DLab SSN node and users' Edge nodes are deployed in this Subnet.
- - SSH key for EC2 instances. This key could be created through Amazon Console.
- - IAM user
- - AWS access key ID and secret access key
- - The following permissions should be assigned for IAM user:
+ **DLab IAM Policy**
<a name="AWS_SSN_policy"></a>
```
{
- "Version": "2012-10-17",
- "Statement": [
- {
- "Action": [
- "iam:ListRoles",
- "iam:CreateRole",
- "iam:CreateInstanceProfile",
- "iam:PutRolePolicy",
- "iam:AddRoleToInstanceProfile",
- "iam:PassRole",
- "iam:GetInstanceProfile",
- "iam:ListInstanceProfilesForRole"
- "iam:RemoveRoleFromInstanceProfile",
- "iam:DeleteInstanceProfile"
- ],
- "Effect": "Allow",
- "Resource": "*"
- },
- {
- "Action": [
- "ec2:DescribeImages",
- "ec2:CreateTags",
- "ec2:DescribeRouteTables",
- "ec2:CreateRouteTable",
- "ec2:AssociateRouteTable",
- "ec2:DescribeVpcEndpoints",
- "ec2:CreateVpcEndpoint",
- "ec2:ModifyVpcEndpoint",
- "ec2:DescribeInstances",
- "ec2:RunInstances",
- "ec2:DescribeAddresses",
- "ec2:AllocateAddress",
- "ec2:DescribeInstances",
- "ec2:AssociateAddress",
- "ec2:DisassociateAddress",
- "ec2:ReleaseAddress",
- "ec2:TerminateInstances"
- ],
- "Effect": "Allow",
- "Resource": "*"
- },
- {
- "Action": [
- "s3:ListAllMyBuckets",
- "s3:CreateBucket",
- "s3:PutBucketTagging",
- "s3:GetBucketTagging"
- ],
- "Effect": "Allow",
- "Resource": "*"
- }
- ]
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Action": [
+ "iam:CreatePolicy",
+ "iam:AttachRolePolicy",
+ "iam:DetachRolePolicy",
+ "iam:DeletePolicy",
+ "iam:DeleteRolePolicy",
+ "iam:GetRolePolicy",
+ "iam:GetPolicy",
+ "iam:GetUser",
+ "iam:ListUsers",
+ "iam:ListAccessKeys",
+ "iam:ListUserPolicies",
+ "iam:ListAttachedRolePolicies",
+ "iam:ListPolicies",
+ "iam:ListRolePolicies",
+ "iam:ListRoles",
+ "iam:CreateRole",
+ "iam:CreateInstanceProfile",
+ "iam:PutRolePolicy",
+ "iam:AddRoleToInstanceProfile",
+ "iam:PassRole",
+ "iam:GetInstanceProfile",
+ "iam:ListInstanceProfilesForRole",
+ "iam:RemoveRoleFromInstanceProfile",
+ "iam:DeleteInstanceProfile",
+ "iam:ListInstanceProfiles",
+ "iam:DeleteRole",
+ "iam:GetRole"
+ ],
+ "Effect": "Allow",
+ "Resource": "*"
+ },
+ {
+ "Action": [
+ "ec2:AuthorizeSecurityGroupEgress",
+ "ec2:AuthorizeSecurityGroupIngress",
+ "ec2:DeleteRouteTable",
+ "ec2:DeleteSubnet",
+ "ec2:DeleteTags",
+ "ec2:DescribeSubnets",
+ "ec2:DescribeVpcs",
+ "ec2:DescribeInstanceStatus",
+ "ec2:ModifyInstanceAttribute",
+ "ec2:RevokeSecurityGroupIngress",
+ "ec2:DescribeImages",
+ "ec2:CreateTags",
+ "ec2:DescribeRouteTables",
+ "ec2:CreateRouteTable",
+ "ec2:AssociateRouteTable",
+ "ec2:DescribeVpcEndpoints",
+ "ec2:CreateVpcEndpoint",
+ "ec2:ModifyVpcEndpoint",
+ "ec2:DescribeInstances",
+ "ec2:RunInstances",
+ "ec2:DescribeAddresses",
+ "ec2:AllocateAddress",
+ "ec2:AssociateAddress",
+ "ec2:DisassociateAddress",
+ "ec2:ReleaseAddress",
+ "ec2:TerminateInstances",
+ "ec2:AuthorizeSecurityGroupIngress",
+ "ec2:AuthorizeSecurityGroupEgress",
+ "ec2:DescribeSecurityGroups",
+ "ec2:CreateSecurityGroup",
+ "ec2:DeleteSecurityGroup",
+ "ec2:RevokeSecurityGroupEgress"
+
+ ],
+ "Effect": "Allow",
+ "Resource": "*"
+ },
+ {
+ "Action": [
+ "s3:GetBucketLocation",
+ "s3:PutBucketPolicy",
+ "s3:GetBucketPolicy",
+ "s3:DeleteBucket",
+ "s3:DeleteObject",
+ "s3:GetObject",
+ "s3:ListBucket",
+ "s3:PutEncryptionConfiguration"
+ "s3:ListAllMyBuckets",
+ "s3:CreateBucket",
+ "s3:PutBucketTagging",
+ "s3:GetBucketTagging"
+ ],
+ "Effect": "Allow",
+ "Resource": "*"
+ }
+ ]
}
```
-To build SSN node, following steps should be executed:
+**Preparation steps for preferred deployment option:**
-1. Clone Git repository and make sure that all following [pre-requisites](#Pre-requisites) are installed.
-2. Go to *dlab* directory.
-3. Execute following script:
-```
-/usr/bin/python infrastructure-provisioning/scripts/deploy_dlab.py --conf_service_base_name dlab_test --aws_access_key XXXXXXX --aws_secret_access_key XXXXXXXXXX --aws_region us-west-2 --conf_os_family debian --conf_cloud_provider aws --aws_vpc_id vpc-xxxxx --aws_subnet_id subnet-xxxxx --aws_security_groups_ids sg-xxxxx,sg-xxxx --key_path /root/ --conf_key_name Test --conf_tag_resource_id dlab --aws_account_id xxxxxxxx --aws_billing_bucket billing_bucket --aws_report_path /billing/direct [...]
-```
+- Create an EC2 instance with the following settings:
+ - The instance should have access to Internet in order to install required prerequisites
+ - The instance should have access to further DLab installation
+ - AMI - Ubuntu 16.04
+ - IAM role with [policy](#AWS_SSN_policy) should be assigned to the instance
+- Put SSH key file created through Amazon Console on the instance with the same name</details>
-This python script will build front-end and back-end part of DLab, create SSN docker image and run Docker container for creating SSN node.
+<details><summary>In Azure cloud <i>(click to expand)</i></summary>
-List of parameters for SSN node deployment:
-| Parameter | Description/Value |
-|---------------------------|-----------------------------------------------------------------------------------------|
-| conf\_service\_base\_name | Any infrastructure value (should be unique if multiple SSN’s have been deployed before) |
-| aws\_access\_key | AWS user access key |
-| aws\_secret\_access\_key | AWS user secret access key |
-| aws\_region | AWS region |
-| conf\_os\_family | Name of the Linux distributive family, which is supported by DLab (Debian/RedHat) |
-| conf\_cloud\_provider | Name of the cloud provider, which is supported by DLab (AWS)
-| conf\_duo\_vpc\_enable | "true" - for installing DLab into two Virtual Private Clouds (VPCs) or "false" - for installing DLab into one VPC. Also this parameter isn't required when deploy DLab in one VPC
-| aws\_vpc\_id | ID of the VPC |
-| aws\_subnet\_id | ID of the public subnet |
-| aws\_security\_groups\_ids| One or more ID\`s of AWS Security Groups, which will be assigned to SSN node |
-| key\_path | Path to admin key (without key name) |
-| conf\_key\_name | Name of the uploaded SSH key file (without “.pem” extension) |
-| conf\_tag\_resource\_id | The name of tag for billing reports |
-| aws\_account\_id | The The ID of Amazon account |
-| aws\_billing\_bucket | The name of S3 bucket where billing reports will be placed |
-| aws\_report\_path | The path to billing reports directory in S3 bucket. This parameter isn't required when billing reports are placed in the root of S3 bucket. |
-| action | In case of SSN node creation, this parameter should be set to “create”
-| workspace\_path | Path to DLab sources root
+**Prerequisites:**
-**Note:** If the following parameters are not specified, they will be created automatically:
+- IAM user with Contributor permissions.
+- Service principal and JSON based auth file with clientId, clientSecret and tenantId.
+
+**Note:** The following permissions should be assigned to the service principal:
+
+- Windows Azure Active Directory
+- Microsoft Graph
+- Windows Azure Service Management API
+
+**Preparation steps for deployment:**
+
+- Create a VM instance with the following settings:
+ - The instance should have access to Internet in order to install required prerequisites
+ - Image - Ubuntu 16.04
+- Generate SSH key pair and rename private key with .pem extension
+- Put JSON auth file to users home directory</details>
+
+<details><summary>In Google cloud (GCP) <i>(click to expand)</i></summary>
+
+**Prerequisites:**
+
+- Service account and JSON auth file for it. In order to get JSON auth file, Key should be created for service account through Google cloud console.
+- Google Cloud Storage JSON API should be enabled
+
+**Preparation steps for deployment:**
+
+- Create a VM instance with the following settings:
+ - The instance should have access to Internet in order to install required prerequisites
+ - Boot disk OS Image - Ubuntu 16.04
+- Generate SSH key pair and rename private key with .pem extension
+- Put JSON auth file created through Google cloud console to users home directory</details>
+
+### Executing deployment script
+
+To build SSN node using instance in the specific cloud platform, following steps should be executed:
+
+- Connect to created at previous step instance via SSH and run the following commands:
+
+```
+sudo su
+apt-get update
+apt-get install git
+git clone https://github.com/apache/incubator-dlab.git -b v2.1.1
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
+apt-get update
+apt-cache policy docker-ce
+apt-get install -y docker-ce=17.06.2~ce-0~ubuntu
+usermod -a -G docker *username*
+apt-get install python-pip
+pip install fabric==1.14.0
+cd incubator-dlab
+```
+
+- Run deployment script, which builds front-end and back-end part of DLab, creates SSN docker image and runs Docker container for creating SSN node:
+
+```
+/usr/bin/python infrastructure-provisioning/scripts/deploy_dlab.py --action create
+ --conf_service_base_name <value>
+ --conf_os_family <value>
+ --conf_cloud_provider <value>
+ --key_path <value>
+ --conf_key_name <value>
+ --ldap_dn <value> --ldap_ou <value> --ldap_service_username <value> --ldap_service_password <value>
+ [--conf_tag_resource_id <value>]
+ [--workspace_path <value>]
+ [--conf_network_type <value>]
+ [--conf_vpc_cidr <value>]
+ [--conf_vpc2_cidr <value>]
+ [--conf_allowed_ip_cidr <value>]
+ [--conf_user_subnets_range <value>]
+ [--conf_additional_tags <value>]
+ [--ssn_hosted_zone_name <value>]
+ [--ssn_hosted_zone_id <value>]
+ [--ssn_subdomain <value>]
+ [--ssn_assume_role_arn <value>]
+ [--ssl_cert_path <value> --ssl_key_path <value>]
+ [--aws_access_key <value> --aws_secret_access_key <value>]
+ [--aws_region', type=str <value> | --azure_region <value> | --gcp_region <value>]
+ [--aws_zone <value> | --gcp_zone <value>]
+ [--aws_vpc_id <value> | --azure_vpc_name <value> | --gcp_vpc_name <value>]
+ [--conf_duo_vpc_enable <value>]
+ [--aws_vpc2_id <value>]
+ [--aws_user_predefined_s3_policies <value>]
+ [--aws_peering_id <value>]
+ [--aws_subnet_id <value> | --azure_subnet_name <value> | --gcp_subnet_name <value>]
+ [--aws_security_groups_ids <value> | --azure_security_group_name <value> | --gcp_firewall_name <value>]
+ [--aws_ssn_instance_size <value> | --azure_ssn_instance_size <value> | --gcp_ssn_instance_size <value>]
+ [--aws_account_id <value>]
+ [--aws_billing_bucket <value>]
+ [--aws_report_path <value>]
+ [--azure_resource_group_name <value>]
+ [--azure_auth_path <value>]
+ [--azure_datalake_enable <value>]
+ [--azure_ad_group_id <value>]
+ [--azure_offer_number <value>]
+ [--azure_currency <value>]
+ [--azure_locale <value>
+ [--azure_application_id <value>]
+ [--azure_validate_permission_scope <value>]
+ [--azure_oauth2_enabled <value>]
+ [--azure_region_info <value>]
+ [--azure_source_vpc_name <value>]
+ [--azure_source_resource_group_name <value>]
+ [--gcp_project_id --gcp_service_account_path <value>]
+ [--dlab_id <value>]
+ [--usage_date <value>]
+ [--product <value>]
+ [--usage_type <value>]
+ [--usage <value>]
+ [--cost <value>]
+ [--resource_id <value>]
+ [--tags <value>]
+```
+
+<details><summary>List of common options and their descriptions: <i>(click to expand)</i></summary>
+
+
+| Option | Description |
+| :--- | :--- |
+| --conf_service_base_name (string) | Any infrastructure value (should be unique if multiple SSN’s have been deployed before) |
+| --conf\_os\_family (string) | Name of the Linux distributive family, which is supported by DLab |
+| --conf\_cloud\_provider (string) | Name of the cloud provider, which is supported by DLab (AWS, GCP or Azure)|
+| --key\_path (string) | Path to admin key (without key name) |
+| --conf\_key\_name (string) | Name of the uploaded SSH key file (without “.pem” extension) |
+| --conf\_tag\_resource\_id (string) | The name of tag for billing reports |
+| --workspace\_path (string) | Path to DLab sources root |
+| --conf_network_type (string) | Define in which network DLab will be deployed. Possible options: public \| private |
+| --conf_vpc_cidr (string) | CIDR of VPC |
+| --conf_vpc2_cidr (string) | CIDR of secondary VPC |
+| --conf_allowed_ip_cidr (string) | Comma-separated CIDR of IPs which will have access to SSN |
+| --conf_user_subnets_range (string) | Range of subnets which will be using for users environments. For example: 10.10.0.0/24 - 10.10.10.0/24 |
+| --ssl_cert_path (string) | Full path to SSL certificate |
+| --ssl_key_path (string) | Full path to key for SSL certificate |
+| --ssn_hosted_zone_name (string) | Name of hosted zone |
+| --ssn_hosted_zone_id (string) | ID of hosted zone |
+| --ssn_subdomain (string) | Subdomain name |
+| --ssn_assume_role_arn (string) | Role ARN for creating Route53 record in different AWS account |
+| --dlab_id (string) | Column name in report file that contains dlab id tag |
+| --usage_date (string) | Column name in report file that contains usage date tag |
+| --product (string) | Column name in report file that contains product name tag |
+| --usage_type (string) | Column name in report file that contains usage type tag |
+| --usage (string) | Column name in report file that contains usage tag |
+| --cost (string) | Column name in report file that contains cost tag |
+| --resource_id (string) | Column name in report file that contains dlab resource id tag |
+| --tags (string) | Column name in report file that contains dlab tags |
+| --ldap_dn (string) | Ldap distinguished name |
+| --ldap_ou (string) | Ldap organisation unit |
+| --ldap_service_username (string) | Ldap service user name |
+| --ldap_service_password (string) | Ldap password for admin user |
+</details>
+
+<details><summary>List of options and expample for AWS: <i>(click to expand)</i></summary>
+
+
+| Option | Description |
+| :--- | :--- |
+| --aws_access_key (string) | AWS user access key |
+| --aws_secret_access_key (string) | AWS user secret access key |
+| --aws_region (string) | AWS region |
+| --aws_zone (string) | AWS zone |
+| --aws\_vpc\_id (string) | ID of the VPC |
+| --conf\_duo\_vpc\_enable (boolean) | "true" - for installing DLab into two Virtual Private Clouds (VPCs) or "false" - for installing DLab into one VPC. Also this parameter isn't required when deploy DLab in one VPC |
+| --aws_vpc2_id (string) | Secondary AWS VPC ID |
+| --aws\_subnet\_id (string) | ID of the public subnet |
+| --aws\_security\_groups\_ids (list) | One or more ID`s of AWS Security Groups, which will be assigned to SSN node |
+| --aws\_account\_id (string) | The The ID of Amazon account |
+| --aws\_billing\_bucket (string) | The name of S3 bucket where billing reports will be placed |
+| --aws\_report\_path (string) | The path to billing reports directory |
+| --aws_peering_id (string) | Amazon peering connection id |
+| --aws_ssn_instance_size (string) | The SSN instance shape |
+| --conf_additional_tags (list) | Additional tags in format "Key1:Value1;Key2:Value2" |
+
+**Note:** If the following parameters are not specified, they creates automatically:
- aws\_vpc\_id
- aws\_subnet\_id
- aws\_sg\_ids
@@ -339,7 +530,7 @@ List of parameters for SSN node deployment:
- aws\_billing\_bucket
- aws\_report\_path
-After SSN node deployment following AWS resources will be created:
+**SSN deployment creates following AWS resources:**
- SSN EC2 instance
- Elastic IP for SSN instance
@@ -349,60 +540,48 @@ After SSN node deployment following AWS resources will be created:
- S3 bucket – its name will be \<service\_base\_name\>-ssn-bucket. This bucket will contain necessary dependencies and configuration files for Notebook nodes (such as .jar files, YARN configuration, etc.)
- S3 bucket for for collaboration between Dlab users. Its name will be \<service\_base\_name\>-shared-bucket
-#### In Azure cloud
-
-Prerequisites:
-
-- IAM user with Contributor permissions.
-- Service principal and JSON based auth file with clientId, clientSecret and tenantId.
-
-**Note:** The following permissions should be assigned to the service principal:
-
-- Windows Azure Active Directory
-- Microsoft Graph
-- Windows Azure Service Management API
-
-To build SSN node, following steps should be executed:
-
-1. Clone Git repository and make sure that all following [pre-requisites](#Pre-requisites) are installed
-2. Go to *dlab* directory
-3. To have working billing functionality please review Billing configuration note and use proper parameters for SSN node deployment
-4. To use Data Lake Store please review Azure Data Lake usage pre-requisites note and use proper parameters for SSN node deployment
-5. Execute following deploy_dlab.py script:
-```
-/usr/bin/python infrastructure-provisioning/scripts/deploy_dlab.py --conf_service_base_name dlab_test --azure_region westus2 --conf_os_family debian --conf_cloud_provider azure --azure_vpc_name vpc-test --azure_subnet_name subnet-test --azure_security_group_name sg-test1,sg-test2 --key_path /root/ --conf_key_name Test --azure_auth_path /dir/file.json --action create
-```
-
-This python script will build front-end and back-end part of DLab, create SSN docker image and run Docker container for creating SSN node.
-
-List of parameters for SSN node deployment:
-
-| Parameter | Description/Value |
-|-----------------------------------|-----------------------------------------------------------------------------------------|
-| conf\_service\_base\_name | Any infrastructure value (should be unique if multiple SSN’s have been deployed before) |
-| azure\_resource\_group\_name | Resource group name (could be the same as service base name |
-| azure\_region | Azure region |
-| conf\_os\_family | Name of the Linux distributive family, which is supported by DLab (Debian/RedHat) |
-| conf\_cloud\_provider | Name of the cloud provider, which is supported by DLab (Azure) |
-| azure\_vpc\_name | Name of the Virtual Network (VN) |
-| azure\_subnet\_name | Name of the Azure subnet |
-| azure\_security\_groups\_name | One or more Name\`s of Azure Security Groups, which will be assigned to SSN node |
-| azure\_ssn\_instance\_size | Instance size of SSN instance in Azure |
-| key\_path | Path to admin key (without key name) |
-| conf\_key\_name | Name of the uploaded SSH key file (without “.pem” extension) |
-| azure\_auth\_path | Full path to auth json file |
-| azure\_offer\_number | Azure offer id number |
-| azure\_currency | Currency that is used for billing information(e.g. USD) |
-| azure\_locale | Locale that is used for billing information(e.g. en-US) |
-| azure\_region\_info | Region info that is used for billing information(e.g. US) |
-| azure\_datalake\_enable | Support of Azure Data Lake (true/false) |
-| azure\_oauth2\_enabled | Defines if Azure OAuth2 authentication mechanisms is enabled(true/false) |
-| azure\_validate\_permission\_scope| Defines if DLab verifies user's permission to the configured resource(scope) during login with OAuth2 (true/false). If Data Lake is enabled default scope is Data Lake Store Account, else Resource Group, where DLab is deployed, is default scope. If user does not have any role in scope he/she is forbidden to log in
-| azure\_application\_id | Azure application ID that is used to log in users in DLab |
-| azure\_ad\_group\_id | ID of group in Active directory whose members have full access to shared folder in Azure Data Lake Store |
-| action | In case of SSN node creation, this parameter should be set to “create” |
-
-**Note:** If the following parameters are not specified, they will be created automatically:
+**Example for preferred deployment option:**
+```
+/usr/bin/python infrastructure-provisioning/scripts/deploy_dlab.py --action create
+ --conf_service_base_name dlab-sample
+ --aws_region us-west-2
+ --conf_os_family debian
+ --conf_cloud_provider aws
+ --aws_vpc_id vpc-xxxxxxxx
+ --aws_subnet_id subnet-xxxxxxxx
+ --aws_security_groups_ids sg-xxxxxxxx,sg-xxxxxxxx
+ --key_path /home/ubuntu/key/
+ --conf_key_name dlab_key
+ --conf_tag_resource_id user:tag
+```
+</details>
+
+<details><summary>List of options and example for Azure: <i>(click to expand)</i></summary>
+
+
+| Option | Description |
+| --- | --- |
+| --azure\_resource\_group\_name (string) | Resource group name (could be the same as service base name) |
+| --azure\_region (string) | Azure region |
+| --azure\_vpc\_name (string) | Name of the Virtual Network (VN) |
+| --azure\_subnet\_name (string) | Name of the Azure subnet |
+| --azure\_security\_groups\_name (list) | One or more Name\`s of Azure Security Groups, which will be assigned to SSN node |
+| --azure\_ssn\_instance\_size (string) | Instance size of SSN instance in Azure |
+| --azure\_auth\_path (string) | Full path to auth json file |
+| --azure\_offer\_number (string) | Azure offer id number |
+| --azure\_currency (string) | Currency that is used for billing information(e.g. USD) |
+| --azure\_locale (string) | Locale that is used for billing information(e.g. en-US) |
+| --azure\_region\_info (string) | Region info that is used for billing information(e.g. US) |
+| --azure\_datalake\_enable (boolean) | Support of Azure Data Lake (true/false) |
+| --azure\_oauth2\_enabled (string) | Defines if Azure OAuth2 authentication mechanisms is enabled(true/false) |
+| --azure\_validate\_permission\_scope (string)| Defines if DLab verifies user's permission to the configured resource(scope) during login with OAuth2 (true/false). If Data Lake is enabled default scope is Data Lake Store Account, else Resource Group, where DLab is deployed, is default scope. If user does not have any role in scope he/she is forbidden to log in |
+| --azure\_application\_id (string) | Azure application ID that is used to log in users in DLab |
+| --azure\_ad\_group\_id (string) | ID of group in Active directory whose members have full access to shared folder in Azure Data Lake Store |
+| --azure_oauth2_enabled (boolean) | Using OAuth2 for logging in DLab |
+| --azure_source_vpc_name (string) | Azure VPC source Name |
+| --azure_source_resource_group_name (string) | Azure source resource group |
+
+**Note:** If the following parameters are not specified, they creates automatically:
- azure\_vpc\_nam
- azure\_subnet\_name
- azure\_security\_groups\_name
@@ -416,17 +595,21 @@ To know azure\_offer\_number open [Azure Portal](https://portal.azure.com), go t
Please see [RateCard API](https://msdn.microsoft.com/en-us/library/mt219004.aspx) to get more details about azure\_offer\_number,
azure\_currency, azure\_locale, azure\_region_info. These DLab deploy properties correspond to RateCard API request parameters.
+To have working billing functionality please review Billing configuration note and use proper parameters for SSN node deployment.
+
+To use Data Lake Store please review Azure Data Lake usage pre-requisites note and use proper parameters for SSN node deployment.
+
**Note:** Azure Data Lake usage pre-requisites:
1. Configure application in Azure portal and grant proper permissions to it.
- Open *Azure Active Directory* tab, then *App registrations* and click *New application registration*
- Fill in ui form with the following parameters *Name* - put name of the new application, *Application type* - select Native, *Sign-on URL* put any valid url as it will be updated later
- Grant proper permissions to the application. Select the application you just created on *App registration* view, then click *Required permissions*, then *Add->Select an API-> In search field type MicrosoftAzureQueryService* and press *Select*, then check the box *Have full access to the Azure Data Lake service* and save the changes. Repeat the same actions for *Windows Azure Active Directory* API (available on *Required permissions->Add->Select an API*) and the box *Sign in and read us [...]
-- Get *Application ID* from application properties it will be used as azure_application_id for deploy_dlap.py script
+- Get *Application ID* from application properties it will be used as azure_application_id for deploy_dlap.py script
2. Usage of Data Lake resource predicts shared folder where all users can write or read any data. To manage access to this folder please create ot use existing group in Active Directory. All users from this group will have RW access to the shared folder. Put ID(in Active Directory) of the group as *azure_ad_group_id* parameter to deploy_dlab.py script
-3. After execution of deploy_dlab.py script go to the application created in step 1 and change *Redirect URIs* value to the https://SSN_HOSTNAME/ where SSN_HOSTNAME - SSN node hostname
+3. After execution of deploy_dlab.py script go to the application created in step 1 and change *Redirect URIs* value to the https://SSN_HOSTNAME/ where SSN_HOSTNAME - SSN node hostname
-After SSN node deployment following Azure resources will be created:
+**SSN deployment creates following Azure resources:**
- Resource group where all DLAb resources will be provisioned
- SSN Virtual machine
@@ -436,49 +619,41 @@ After SSN node deployment following Azure resources will be created:
- Virtual network and Subnet (if they have not been specified) for SSN and EDGE nodes
- Storage account and blob container for necessary further dependencies and configuration files for Notebook nodes (such as .jar files, YARN configuration, etc.)
- Storage account and blob container for collaboration between Dlab users
-- If support of Data Lake is enabled: Data Lake and shared directory will be created
-
-#### In Google cloud (GCP)
-
-Prerequisites:
-
-- IAM user
-- Service account and JSON auth file for it. In order to get JSON auth file, Key should be created for service account through Google cloud console.
-
-To build SSN node, following steps should be executed:
-
-1. Clone Git repository and make sure that all following [pre-requisites](#Pre-requisites) are installed.
-2. Go to *dlab* directory.
-3. Execute following script:
-```
-/usr/bin/python infrastructure-provisioning/scripts/deploy_dlab.py --conf_service_base_name dlab --gcp_region us-west1 --gcp_zone us-west1-a --conf_os_family debian --conf_cloud_provider gcp --key_path /key/path/ --conf_key_name key_name --gcp_ssn_instance_size n1-standard-1 --gcp_project_id project_id --gcp_service_account_path /path/to/auth/file.json --action create
-```
-
-This python script will build front-end and back-end part of DLab, create SSN docker image and run Docker container for creating SSN node.
-
-List of parameters for SSN node deployment:
-
-| Parameter | Description/Value |
-|------------------------------|-----------------------------------------------------------------------------------------|
-| conf\_service\_base\_name | Any infrastructure value (should be unique if multiple SSN’s have been deployed before) |
-| gcp\_region | GCP region |
-| gcp\_zone | GCP zone |
-| conf\_os\_family | Name of the Linux distributive family, which is supported by DLab (Debian/RedHat) |
-| conf\_cloud\_provider | Name of the cloud provider, which is supported by DLab (GCP) |
-| gcp\_vpc\_name | Name of the Virtual Network (VN) |
-| gcp\_subnet\_name | Name of the GCP subnet |
-| gcp\_firewall\_name | One or more Name\`s of GCP Security Groups, which will be assigned to SSN node |
-| key\_path | Path to admin key (without key name) |
-| conf\_key\_name | Name of the uploaded SSH key file (without “.pem” extension) |
-| gcp\_service\_account\_path | Full path to auth json file |
-| gcp\_ssn\_instance\_size | Instance size of SSN instance in GCP |
-| gcp\_project\_id | ID of GCP project |
-| action | In case of SSN node creation, this parameter should be set to “create” |
-
+- If support of Data Lake is enabled: Data Lake and shared directory will be created
+
+**Example:**
+```
+/usr/bin/python infrastructure-provisioning/scripts/deploy_dlab.py --action create
+ --conf_service_base_name dlab-sample
+ --azure_region westus2
+ --conf_os_family debian
+ --conf_cloud_provider azure
+ --azure_vpc_name dlab-sample-vpc
+ --azure_subnet_name dlab-sample-subnet
+ --azure_security_group_name dlab-sample1-sg,dlab-sample2-sg
+ --key_path /root/
+ --conf_key_name dlab_key
+ --azure_auth_path /home/ubuntu/dlab.json
+```
+</details>
+
+<details><summary>List of options and example for GCP: <i>(click to expand)</i></summary>
+
+
+| Option | Description |
+| --- | --- |
+| --gcp\_region (string) | GCP region |
+| --gcp\_zone (string) | GCP zone |
+| --gcp\_vpc\_name (string) | Name of the Virtual Network (VN) |
+| --gcp\_subnet\_name (string) | Name of the GCP subnet |
+| --gcp\_firewall\_name (list) | One or more Name\`s of GCP Security Groups, which will be assigned to SSN node |
+| --gcp\_service\_account\_path (string) | Full path to auth json file |
+| --gcp\_ssn\_instance\_size (string) | Instance size of SSN instance in GCP |
+| --gcp\_project\_id (string) | ID of GCP project |
**Note:** If you gonna use Dataproc cluster, be aware that Dataproc has limited availability in GCP regions. [Cloud Dataproc availability by Region in GCP](https://cloud.google.com/about/locations/)
-After SSN node deployment following GCP resources will be created:
+**SSN deployment creates following GCP resources:**
- SSN VM instance
- External IP address for SSN instance
@@ -488,16 +663,30 @@ After SSN node deployment following GCP resources will be created:
- Bucket – its name will be \<service\_base\_name\>-ssn-bucket. This bucket will contain necessary dependencies and configuration files for Notebook nodes (such as .jar files, YARN configuration, etc.)
- Bucket for for collaboration between Dlab users. Its name will be \<service\_base\_name\>-shared-bucket
-### Terminate
-
-Terminating SSN node will also remove all nodes and components related to it. Basically, terminating Self-service node will terminate all DLab’s infrastructure.
-Example of command for terminating DLab environment:
-
-#### In Amazon
+**Example:**
```
-/usr/bin/python infrastructure-provisioning/scripts/deploy_dlab.py --conf_service_base_name dlab-test --aws_access_key XXXXXXX --aws_secret_access_key XXXXXXXX --aws_region us-west-2 --key_path /root/ --conf_key_name Test --conf_os_family debian --conf_cloud_provider aws --action terminate
+/usr/bin/python infrastructure-provisioning/scripts/deploy_dlab.py --action create
+ --conf_service_base_name dlab-sample
+ --gcp_region us-west1
+ --gcp_zone us-west1-a
+ --conf_os_family debian
+ --conf_cloud_provider gcp
+ --key_path /root/
+ --conf_key_name dlab_key
+ --gcp_ssn_instance_size n1-standard-1
+ --gcp_project_id xxx-xxx-xxxx-xxxx-xxxxxx
+ --gcp_service_account_path /home/ubuntu/gcp/service_account.json
```
-List of parameters for SSN node termination:
+</details>
+
+### Terminating Self-Service Node
+
+Terminating SSN node also removes all nodes and components related to it. Basically, terminating Self-service node terminates all DLab’s infrastructure.
+
+**Example of command for terminating DLab environment:**
+
+<details><summary>In Amazon <i>(click to expand)</i></summary>
+
| Parameter | Description/Value |
|----------------------------|------------------------------------------------------------------------------------|
@@ -511,11 +700,22 @@ List of parameters for SSN node termination:
| conf\_cloud\_provider | Name of the cloud provider, which is supported by DLab (AWS) |
| action | terminate |
-#### In Azure
+**Example:**
```
-/usr/bin/python infrastructure-provisioning/scripts/deploy_dlab.py --conf_service_base_name dlab-test --azure_vpc_name vpc-test --azure_resource_group_name resource-group-test --azure_region westus2 --key_path /root/ --conf_key_name Test --conf_os_family debian --conf_cloud_provider azure --azure_auth_path /dir/file.json --action terminate
+/usr/bin/python infrastructure-provisioning/scripts/deploy_dlab.py --action terminate
+ --aws_access_key=XXXXXXXXXXXXXXXXXXXX
+ --aws_secret_access_key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+ --conf_service_base_name dlab-sample
+ --aws_region us-west-2
+ --key_path /home/ubuntu/key/
+ --conf_key_name dlab_key
+ --conf_os_family debian
+ --conf_cloud_provider aws
```
-List of parameters for SSN node termination:
+</details>
+
+<details><summary>In Azure <i>(click to expand)</i></summary>
+
| Parameter | Description/Value |
|----------------------------|------------------------------------------------------------------------------------|
@@ -529,12 +729,23 @@ List of parameters for SSN node termination:
| azure\_auth\_path | Full path to auth json file |
| action | terminate |
-
-#### In Google cloud
+**Example:**
```
-/usr/bin/python infrastructure-provisioning/scripts/deploy_dlab.py --gcp_project_id project_id --conf_service_base_name dlab --gcp_region us-west1 --gcp_zone us-west1-a --key_path /root/ --conf_key_name key_name --conf_os_family debian --conf_cloud_provider gcp --gcp_service_account_path /path/to/auth/file.json --action terminate
+/usr/bin/python infrastructure-provisioning/scripts/deploy_dlab.py --action terminate
+ --conf_service_base_name dlab-sample
+ --azure_vpc_name dlab-sample-vpc
+ --azure_resource_group_name dlab-sample
+ --azure_region westus2
+ --key_path /root/
+ --conf_key_name dlab_key
+ --conf_os_family debian
+ --conf_cloud_provider azure
+ --azure_auth_path /home/ubuntu/dlab.json
```
-List of parameters for SSN node termination:
+</details>
+
+<details><summary>In Google cloud <i>(click to expand)</i></summary>
+
| Parameter | Description/Value |
|------------------------------|-----------------------------------------------------------------------------------------|
@@ -549,6 +760,20 @@ List of parameters for SSN node termination:
| gcp\_project\_id | ID of GCP project |
| action | In case of SSN node termination, this parameter should be set to “terminate” |
+**Example:**
+```
+/usr/bin/python infrastructure-provisioning/scripts/deploy_dlab.py --action terminate
+ --gcp_project_id xxx-xxx-xxxx-xxxx-xxxxxx
+ --conf_service_base_name dlab-sample
+ --gcp_region us-west1
+ --gcp_zone us-west1-a
+ --key_path /root/
+ --conf_key_name dlab_key
+ --conf_os_family debian
+ --conf_cloud_provider gcp
+ --gcp_service_account_path /home/ubuntu/gcp/service_account.json
+```
+</details>
## Edge Node <a name="Edge_Node"></a>
@@ -558,7 +783,7 @@ Gateway node (or an Edge node) is an instance(virtual machine) provisioned in a
In order to create Edge node using DLab Web UI – login and, click on the button “Upload” (Depending on authorization provider that was chosen on deployment stage, user may be taken from [LDAP](#LDAP_Authentication) or from [Azure AD (Oauth2)](#Azure_OAuth2_Authentication)). Choose user’s SSH public key and after that click on the button “Create”. Edge node will be deployed and corresponding instance (virtual machine) will be started.
-#### In Amazon
+<details><summary>In Amazon <i>(click to expand)</i></summary>
The following AWS resources will be created:
- Edge EC2 instance
@@ -588,8 +813,9 @@ List of parameters for Edge node creation:
| aws\_private\_subnet\_prefix | Prefix of the private subnet |
| conf\_tag\_resource\_id | The name of tag for billing reports |
| action | create |
+</details>
-#### In Azure
+<details><summary>In Azure <i>(click to expand)</i></summary>
The following Azure resources will be created:
- Edge virtual machine
@@ -600,7 +826,7 @@ The following Azure resources will be created:
- Security Groups for all further user's master nodes of data engine cluster
- Security Groups for all further user's slave nodes of data engine cluster
- User's private subnet. All further nodes (Notebooks, data engine clusters) will be provisioned in different subnet than SSN.
-- User's storage account and blob container
+- User's storage account and blob container
List of parameters for Edge node creation:
@@ -616,8 +842,9 @@ List of parameters for Edge node creation:
| azure\_vpc\_name | Name of Azure Virtual network where all infrastructure is being deployed |
| azure\_subnet\_name | Name of the Azure public subnet where Edge will be deployed |
| action | create |
+</details>
-#### In Google cloud
+<details><summary>In Google cloud <i>(click to expand)</i></summary>
The following GCP resources will be created:
- Edge VM instance
@@ -627,7 +854,7 @@ The following GCP resources will be created:
- Security Groups for all further user's master nodes of data engine cluster
- Security Groups for all further user's slave nodes of data engine cluster
- User's private subnet. All further nodes (Notebooks, data engine clusters) will be provisioned in different subnet than SSN.
-- User's bucket
+- User's bucket
List of parameters for Edge node creation:
@@ -644,12 +871,13 @@ List of parameters for Edge node creation:
| gcp\_subnet\_name | Name of the Azure public subnet where Edge will be deployed |
| gcp\_project\_id | ID of GCP project |
| action | create |
+</details>
### Start/Stop <a name=""></a>
To start/stop Edge node, click on the button which looks like a cycle on the top right corner, then click on the button which is located in “Action” field and in the drop-down menu click on the appropriate action.
-#### In Amazon
+<details><summary>In Amazon <i>(click to expand)</i></summary>
List of parameters for Edge node starting/stopping:
@@ -660,8 +888,9 @@ List of parameters for Edge node starting/stopping:
| edge\_user\_name | Name of the user |
| aws\_region | AWS region where infrastructure was deployed |
| action | start/stop |
+</details>
-#### In Azure
+<details><summary>In Azure <i>(click to expand)</i></summary>
List of parameters for Edge node starting:
@@ -683,8 +912,9 @@ List of parameters for Edge node stopping:
| edge\_user\_name | Name of the user |
| azure\_resource\_group\_name | Name of the resource group where all DLAb resources are being provisioned |
| action | stop |
+</details>
-#### In Google cloud
+<details><summary>In Google cloud <i>(click to expand)</i></summary>
List of parameters for Edge node starting/stopping:
@@ -697,6 +927,7 @@ List of parameters for Edge node starting/stopping:
| gcp\_zone | GCP zone where infrastructure was deployed |
| gcp\_project\_id | ID of GCP project |
| action | start/stop |
+</details>
### Recreate <a name=""></a>
@@ -706,7 +937,7 @@ If Edge node was removed for some reason, to re-create it, click on the status b
List of parameters for Edge node recreation:
-#### In Amazon
+<details><summary>In Amazon <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|----------------------------|-----------------------------------------------------------------------------------|
@@ -722,8 +953,9 @@ List of parameters for Edge node recreation:
| edge\_elastic\_ip | AWS Elastic IP address which was associated to Edge node |
| conf\_tag\_resource\_id | The name of tag for billing reports |
| action | Create |
+</details>
-#### In Azure
+<details><summary>In Azure <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|------------------------------|-----------------------------------------------------------------------------------|
@@ -737,8 +969,10 @@ List of parameters for Edge node recreation:
| azure\_resource\_group\_name | Name of the resource group where all DLAb resources are being provisioned |
| azure\_subnet\_name | Name of the Azure public subnet where Edge was deployed |
| action | Create |
+</details>
+
+<details><summary>In Google cloud <i>(click to expand)</i></summary>
-#### In Google cloud
| Parameter | Description/Value |
|--------------------------------|-----------------------------------------------------------------------------------|
| conf\_resource | edge |
@@ -752,6 +986,7 @@ List of parameters for Edge node recreation:
| gcp\_subnet\_name | Name of the Azure public subnet where Edge will be deployed |
| gcp\_project\_id | ID of GCP project |
| action | create |
+</details>
## Notebook node <a name="Notebook_node"></a>
@@ -763,7 +998,7 @@ To create Notebook node, click on the “Create new” button. Then, in drop-dow
List of parameters for Notebook node creation:
-#### In Amazon
+<details><summary>In Amazon <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|-------------------------------|-----------------------------------------------------------------------------------|
@@ -781,8 +1016,9 @@ List of parameters for Notebook node creation:
| action | Create |
**Note:** For format of git_creds see "Manage git credentials" lower.
+</details>
-#### In Azure
+<details><summary>In Azure <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|---------------------------------|-----------------------------------------------------------------------------------|
@@ -798,8 +1034,9 @@ List of parameters for Notebook node creation:
| application | Type of the notebook template (jupyter/rstudio/zeppelin/tensor/deeplearning) |
| git\_creds | User git credentials in JSON format |
| action | Create |
+</details>
-#### In Google cloud
+<details><summary>In Google cloud <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|-------------------------------|-----------------------------------------------------------------------------------|
@@ -816,6 +1053,7 @@ List of parameters for Notebook node creation:
| application | Type of the notebook template (jupyter/rstudio/zeppelin/tensor/deeplearning) |
| git\_creds | User git credentials in JSON format |
| action | Create |
+</details>
### Stop
@@ -823,7 +1061,7 @@ In order to stop Notebook node, click on the “gear” button in Actions column
List of parameters for Notebook node stopping:
-#### In Amazon
+<details><summary>In Amazon <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|---------------------------|--------------------------------------------------------------|
@@ -834,8 +1072,9 @@ List of parameters for Notebook node stopping:
| notebook\_instance\_name | Name of the Notebook instance to terminate |
| aws\_region | AWS region where infrastructure was deployed |
| action | Stop |
+</details>
-#### In Azure
+<details><summary>In Azure <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|---------------------------------|-----------------------------------------------------------------------------------|
@@ -846,8 +1085,9 @@ List of parameters for Notebook node stopping:
| notebook\_instance\_name | Name of the Notebook instance to terminate |
| azure\_resource\_group\_name | Name of the resource group where all DLAb resources are being provisioned |
| action | Stop |
+</details>
-#### In Google cloud
+<details><summary>In Google cloud <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|---------------------------|--------------------------------------------------------------|
@@ -860,6 +1100,7 @@ List of parameters for Notebook node stopping:
| gcp\_zone | GCP zone where infrastructure was deployed |
| gcp\_project\_id | ID of GCP project |
| action | Stop |
+</details>
### Start
@@ -867,7 +1108,7 @@ In order to start Notebook node, click on the button, which looks like gear in
List of parameters for Notebook node start:
-#### In Amazon
+<details><summary>In Amazon <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|---------------------------|--------------------------------------------------------------|
@@ -881,8 +1122,9 @@ List of parameters for Notebook node start:
| action | start |
**Note:** For format of git_creds see "Manage git credentials" lower.
+</details>
-#### In Azure
+<details><summary>In Azure <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|---------------------------------|-----------------------------------------------------------------------------------|
@@ -895,8 +1137,9 @@ List of parameters for Notebook node start:
| azure\_region | Azure region where infrastructure was deployed |
| git\_creds | User git credentials in JSON format |
| action | start |
+</details>
-#### In Google cloud
+<details><summary>In Google cloud <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|---------------------------|--------------------------------------------------------------|
@@ -910,6 +1153,7 @@ List of parameters for Notebook node start:
| gcp\_project\_id | ID of GCP project |
| git\_creds | User git credentials in JSON format |
| action | Stop |
+</details>
### Terminate
@@ -917,7 +1161,7 @@ In order to terminate Notebook node, click on the button, which looks like gear
List of parameters for Notebook node termination:
-#### In Amazon
+<details><summary>In Amazon <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|---------------------------|--------------------------------------------------------------|
@@ -930,8 +1174,9 @@ List of parameters for Notebook node termination:
| action | terminate |
**Note:** If terminate action is called, all connected data engine clusters will be removed.
+</details>
-#### In Azure
+<details><summary>In Azure <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|---------------------------------|-----------------------------------------------------------------------------------|
@@ -941,8 +1186,9 @@ List of parameters for Notebook node termination:
| notebook\_instance\_name | Name of the Notebook instance to terminate |
| azure\_resource\_group\_name | Name of the resource group where all DLAb resources are being provisioned |
| action | terminate |
+</details>
-#### In Google cloud
+<details><summary>In Google cloud <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|---------------------------|--------------------------------------------------------------|
@@ -955,12 +1201,13 @@ List of parameters for Notebook node termination:
| gcp\_project\_id | ID of GCP project |
| git\_creds | User git credentials in JSON format |
| action | Stop |
+</details>
### List/Install additional libraries
In order to list available libraries (OS/Python2/Python3/R/Others) on Notebook node, click on the button, which looks like gear in “Action” field. Then in drop-down menu choose “Manage libraries” action.
-#### In Amazon
+<details><summary>In Amazon <i>(click to expand)</i></summary>
List of parameters for Notebook node to **get list** of available libraries:
@@ -985,7 +1232,7 @@ List of parameters for Notebook node to **get list** of available libraries:
"pip2": {"requests": "N/A", "configparser": "N/A"},
"pip3": {"configparser": "N/A"},
"r_pkg": {"rmarkdown": "1.5"},
- "others": {"Keras": "N/A"}
+ "others": {"Keras": "N/A"}
}
```
@@ -1020,8 +1267,9 @@ List of parameters for Notebook node to **install** additional libraries:
...
}
```
+</details>
-#### In Azure
+<details><summary>In Azure <i>(click to expand)</i></summary>
List of parameters for Notebook node to **get list** of available libraries:
@@ -1049,9 +1297,9 @@ List of parameters for Notebook node to **install** additional libraries:
| application | Type of the notebook template (jupyter/rstudio/zeppelin/tensor/deeplearning) |
| libs | List of additional libraries in JSON format with type (os_pkg/pip2/pip3/r_pkg/others)|
| action | lib_install |
+</details>
-
-#### In Google cloud
+<details><summary>In Google cloud <i>(click to expand)</i></summary>
List of parameters for Notebook node to **get list** of available libraries:
@@ -1081,12 +1329,13 @@ List of parameters for Notebook node to **install** additional libraries:
| application | Type of the notebook template (jupyter/rstudio/zeppelin/tensor/deeplearning) |
| libs | List of additional libraries in JSON format with type (os_pkg/pip2/pip3/r_pkg/others)|
| action | lib_install |
+</details>
### Manage git credentials
In order to manage git credentials on Notebook node, click on the button “Git credentials”. Then in menu you can add or edit existing credentials.
-#### In Amazon
+<details><summary>In Amazon <i>(click to expand)</i></summary>
List of parameters for Notebook node to **manage git credentials**:
@@ -1118,8 +1367,9 @@ List of parameters for Notebook node to **manage git credentials**:
**Note:** Leave "hostname" field empty to apply login/password by default for all services.
**Note:** Also your can use "Personal access tokens" against passwords.
+</details>
-#### In Azure
+<details><summary>In Azure <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|-------------------------------|-----------------------------------------------------------------------------------|
@@ -1131,8 +1381,9 @@ List of parameters for Notebook node to **manage git credentials**:
| azure\_resource\_group\_name | Name of the resource group where all DLAb resources are being provisioned |
| git\_creds | User git credentials in JSON format |
| action | git\_creds |
+</details>
-#### In Google cloud
+<details><summary>In Google cloud <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|-------------------------------|-----------------------------------------------------------------------------------|
@@ -1146,8 +1397,9 @@ List of parameters for Notebook node to **manage git credentials**:
| notebook\_instance\_name | Name of the Notebook instance to terminate |
| git\_creds | User git credentials in JSON format |
| action | git\_creds |
+</details>
-## Dataengine-service cluster <a name="Dataengine-service cluster"></a>
+## Dataengine-service cluster <a name="Dataengine-service_cluster"></a>
Dataengine-service is a cluster provided by cloud as a service (EMR on AWS) can be created if more computational resources are needed for executing analytical algorithms and models, triggered from analytical tools. Jobs execution will be scaled to a cluster mode increasing the performance and decreasing execution time.
@@ -1157,7 +1409,7 @@ To create dataengine-service cluster click on the “gear” button in Actions c
List of parameters for dataengine-service cluster creation:
-#### In Amazon
+<details><summary>In Amazon <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|-----------------------------|--------------------------------------------------------------------------|
@@ -1176,8 +1428,9 @@ List of parameters for dataengine-service cluster creation:
| action | create |
**Note:** If “Spot instances” is enabled, dataengine-service Slave nodes will be created as EC2 Spot instances.
+</details>
-#### In Google cloud
+<details><summary>In Google cloud <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|---------------------------------|--------------------------------------------------------------------------|
@@ -1198,6 +1451,7 @@ List of parameters for dataengine-service cluster creation:
| gcp\_zone | GCP zone name |
| conf\_tag\_resource\_id | The name of tag for billing reports |
| action | create |
+</details>
### Terminate
@@ -1205,7 +1459,7 @@ In order to terminate dataengine-service cluster, click on “x” button which
List of parameters for dataengine-service cluster termination:
-#### In Amazon
+<details><summary>In Amazon <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|---------------------------|---------------------------------------------------------------------|
@@ -1217,8 +1471,9 @@ List of parameters for dataengine-service cluster termination:
| notebook\_instance\_name | Name of the Notebook instance which dataengine-service is linked to |
| aws\_region | AWS region where infrastructure was deployed |
| action | Terminate |
+</details>
-#### In Google cloud
+<details><summary>In Google cloud <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|---------------------------|---------------------------------------------------------------------|
@@ -1232,12 +1487,13 @@ List of parameters for dataengine-service cluster termination:
| gcp\_zone | GCP zone name |
| dataproc\_cluster\_name | Dataproc cluster name |
| action | Terminate |
+</details>
### List/Install additional libraries
In order to list available libraries (OS/Python2/Python3/R/Others) on Dataengine-service, click on the button, which looks like gear in “Action” field. Then in drop-down menu choose “Manage libraries” action.
-#### In Amazon
+<details><summary>In Amazon <i>(click to expand)</i></summary>
List of parameters for Dataengine-service node to **get list** of available libraries:
@@ -1261,7 +1517,7 @@ List of parameters for Dataengine-service node to **get list** of available libr
"pip2": {"requests": "N/A", "configparser": "N/A"},
"pip3": {"configparser": "N/A"},
"r_pkg": {"rmarkdown": "1.5"},
- "others": {"Keras": "N/A"}
+ "others": {"Keras": "N/A"}
}
```
@@ -1295,8 +1551,9 @@ List of parameters for Dataengine-service to **install** additional libraries:
...
}
```
+</details>
-#### In Google cloud
+<details><summary>In Google cloud <i>(click to expand)</i></summary>
List of parameters for Dataengine-service node to **get list** of available libraries:
@@ -1323,9 +1580,9 @@ List of parameters for Dataengine-service node to **install** additional librari
| gcp\_region | GCP region name |
| gcp\_zone | GCP zone name |
| action | lib_install |
+</details>
-
-## Dataengine cluster <a name="Dataengine cluster"></a>
+## Dataengine cluster <a name="Dataengine_cluster"></a>
Dataengine is cluster based on Standalone Spark framework can be created if more computational resources are needed for executing analytical algorithms, but without additional expenses for cloud provided service.
@@ -1335,7 +1592,7 @@ To create Spark standalone cluster click on the “gear” button in Actions col
List of parameters for dataengine cluster creation:
-#### In Amazon
+<details><summary>In Amazon <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|--------------------------------|-----------------------------------------------------------------------------------|
@@ -1350,8 +1607,9 @@ List of parameters for dataengine cluster creation:
| aws\_dataengine\_master\_size | Size of master node |
| aws\_dataengine\_slave\_size | Size of slave node |
| action | create |
+</details>
-#### In Azure
+<details><summary>In Azure <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|--------------------------------|-----------------------------------------------------------------------------------|
@@ -1369,8 +1627,9 @@ List of parameters for dataengine cluster creation:
| azure\_resource\_group\_name | Name of the resource group where all DLAb resources are being provisioned |
| azure\_subnet\_name | Name of the Azure public subnet where Edge was deployed |
| action | create |
+</details>
-#### In Google cloud
+<details><summary>In Google cloud <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|------------------------------|-----------------------------------------------------------------------------------|
@@ -1389,7 +1648,7 @@ List of parameters for dataengine cluster creation:
| gcp\_zone | GCP zone name |
| edge\_user\_name | Value that previously was used when Edge being provisioned |
| action | create |
-
+</details>
### Terminate
@@ -1397,7 +1656,7 @@ In order to terminate dataengine cluster, click on “x” button which is locat
List of parameters for dataengine cluster termination:
-#### In Amazon
+<details><summary>In Amazon <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|------------------------------|--------------------------------------------------------------------------|
@@ -1409,8 +1668,9 @@ List of parameters for dataengine cluster termination:
| computational\_name | Name of cluster |
| aws\_region | AWS region where infrastructure was deployed |
| action | Terminate |
+</details>
-#### In Azure
+<details><summary>In Azure <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|------------------------------|--------------------------------------------------------------------------|
@@ -1423,8 +1683,9 @@ List of parameters for dataengine cluster termination:
| azure\_region | Azure region where infrastructure was deployed |
| azure\_resource\_group\_name | Name of the resource group where all DLAb resources are being provisioned|
| action | Terminate |
+</details>
-#### In Google cloud
+<details><summary>In Google cloud <i>(click to expand)</i></summary>
| Parameter | Description/Value |
|------------------------------|--------------------------------------------------------------------------|
@@ -1438,12 +1699,13 @@ List of parameters for dataengine cluster termination:
| gcp\_region | GCP region where infrastructure was deployed |
| gcp\_zone | GCP zone name |
| action | Terminate |
+</details>
### List/Install additional libraries
In order to list available libraries (OS/Python2/Python3/R/Others) on Dataengine, click on the button, which looks like gear in “Action” field. Then in drop-down menu choose “Manage libraries” action.
-#### In Amazon
+<details><summary>In Amazon <i>(click to expand)</i></summary>
List of parameters for Dataengine node to **get list** of available libraries:
@@ -1467,7 +1729,7 @@ List of parameters for Dataengine node to **get list** of available libraries:
"pip2": {"requests": "N/A", "configparser": "N/A"},
"pip3": {"configparser": "N/A"},
"r_pkg": {"rmarkdown": "1.5"},
- "others": {"Keras": "N/A"}
+ "others": {"Keras": "N/A"}
}
```
@@ -1500,8 +1762,9 @@ List of parameters for Dataengine node to **install** additional libraries:
...
}
```
+</details>
-#### In Azure
+<details><summary>In Azure <i>(click to expand)</i></summary>
List of parameters for Dataengine node to **get list** of available libraries:
@@ -1528,9 +1791,9 @@ List of parameters for Dataengine node to **install** additional libraries:
| computational\_id | Name of cluster |
| application | Type of the notebook template (jupyter/rstudio/zeppelin/tensor/deeplearning) |
| action | lib_install |
+</details>
-
-#### In Google cloud
+<details><summary>In Google cloud <i>(click to expand)</i></summary>
List of parameters for Dataengine node to **get list** of available libraries:
@@ -1559,7 +1822,7 @@ List of parameters for Dataengine node to **install** additional libraries:
| gcp\_zone | GCP zone name |
| computational\_id | Name of cluster |
| action | lib_install |
-
+</details>
## Configuration files <a name="Configuration_files"></a>
@@ -1604,7 +1867,7 @@ To use your own certificate please do the following:
## Billing report <a name="Billing_Report"></a>
-### AWS
+<details><summary>AWS <i>(click to expand)</i></summary>
Billing module is implemented as a separate jar file and can be running in the follow modes:
@@ -1634,8 +1897,9 @@ If you want billing to work as a separate process from the Self-Service use foll
```
java -cp /opt/dlab/webapp/lib/billing/billing-aws.x.y.jar com.epam.dlab.BillingScheduler --conf /opt/dlab/conf/billing.yml
```
+</details>
-### Azure
+<details><summary>Azure <i>(click to expand)</i></summary>
Billing module is implemented as a separate jar file and can be running in the follow modes:
@@ -1646,6 +1910,7 @@ If you want to start billing module as a separate process use the following comm
```
java -jar /opt/dlab/webapp/lib/billing/billing-azure.x.y.jar /opt/dlab/conf/billing.yml
```
+</details>
## Backup and Restore <a name="Backup_and_Restore"></a>
@@ -1700,7 +1965,7 @@ To deploy Gitlab server, set all needed parameters in ```gitlab.ini``` and run s
**Note:** Terminate process uses ```node_name``` to find instance.
-**Note:** GitLab wouldn't be terminated with all environment termination process.
+**Note:** GitLab wouldn't be terminated with all environment termination process.
## Troubleshooting <a name="Troubleshooting"></a>
@@ -1719,7 +1984,17 @@ a85d0d3c27aa docker.dlab-dataengine:latest "/root/entrypoint...." 2 hours ago Ex
...
docker logs <container_id> – to get log for particular Docker container.
```
-To change Docker images on existing environment, execute following steps:
+
+To change Docker images on existing environment, you can run script on SSN node that rebuilds docker images with the command:
+```
+docker-build all #to rebuild all images
+```
+or
+```
+docker-build <notebook_name> #to rebuild certain images
+```
+
+You can also rebuild images manually by executing the following steps:
1. SSH to SSN instance
2. go to */opt/dlab/sources/*
@@ -1825,12 +2100,12 @@ Some class names may have endings like Aws or Azure(e.g. ComputationalResourceAw
#### Security service
-Security service is REST based service for user authentication against LDAP/LDAP + AWS/Azure OAuth2 depending on module configuration and cloud provider.
-LDAP only provides with authentication end point that allows to verify authenticity of users against LDAP instance.
+Security service is REST based service for user authentication against LDAP/LDAP + AWS/Azure OAuth2 depending on module configuration and cloud provider.
+LDAP only provides with authentication end point that allows to verify authenticity of users against LDAP instance.
If you use AWS cloud provider LDAP + AWS authentication could be useful as it allows to combine LDAP authentication and verification if user has any role in AWS account
DLab provides OAuth2(client credentials and authorization code flow) security authorization mechanism for Azure users. This kind of authentication is required when you are going to use Data Lake. If Data Lake is not enabled you have two options LDAP or OAuth2
-If OAuth2 is in use security-service validates user's permissions to configured permission scope(resource in Azure).
+If OAuth2 is in use security-service validates user's permissions to configured permission scope(resource in Azure).
If Data Lake is enabled default permission scope(can be configured manually after deploy DLab) is Data Lake Store account so only if user has any role in scope of Data Lake Store Account resource he/she will be allowed to log in
If Data Lake is disabled but Azure OAuth2 is in use default permission scope will be Resource Group where DLab is created and only users who have any roles in the resource group will be allowed to log in.
@@ -1849,7 +2124,7 @@ Sources are located in dlab/services/self-service/src/main/resources/webapp
| Home page (list of resources) | HomeComponent<br>nested several main components like ResourcesGrid for notebooks data rendering and filtering, using custom MultiSelectDropdown component;<br>multiple modal dialogs components used for new instances creation, displaying detailed info and actions confirmation. |
| Health Status page | HealthStatusComponent<br>*HealthStatusGridComponent* displays list of instances, their types, statutes, ID’s and uses *healthStatusService* for handling main actions. |
| Help pages | Static pages that contains information and instructions on how to access Notebook Server and generate SSH key pair. Includes only *NavbarComponent*. |
-| Error page | Simple static page letting users know that opened page does not exist. Includes only *NavbarComponent*. |
+| Error page | Simple static page letting users know that opened page does not exist. Includes only *NavbarComponent*. |
| Reporting page | ReportingComponent<br>ReportingGridComponent displays billing detailed info with built-in filtering and DateRangePicker component for custom range filtering;<br>uses *BillingReportService* for handling main actions and exports report data to .csv file. |
## How to setup local development environment <a name="setup_local_environment"></a>
@@ -1913,7 +2188,7 @@ mongo:
*Unix*
```
-ln -s ../../infrastructure-provisioning/src/ssn/templates/ssn.yml ssn.yml
+ln -s ../../infrastructure-provisioning/src/ssn/templates/ssn.yml ssn.yml
```
*Windows*
@@ -1960,7 +2235,7 @@ export * from './(aws|azure).dictionary';
npm run build.prod
```
-### Prepare HTTPS prerequisites
+### Prepare HTTPS prerequisites
To enable a SSL connection the web server should have a Digital Certificate.
To create a server certificate, follow these steps:
@@ -1975,7 +2250,7 @@ To create a server certificate, follow these steps:
Please find below set of commands to create certificate, depending on OS.
-#### Create Unix/Ubuntu server certificate
+#### Create Unix/Ubuntu server certificate
Pay attention that the last command has to be executed with administrative permissions.
```
@@ -2369,7 +2644,7 @@ Also depending on customization, there might be differences in attributes config
**CN** vs **UID**.
-The relation between users and groups also varies from vendor to vendor.
+The relation between users and groups also varies from vendor to vendor.
For example, in Open LDAP the group object may contain set (from 0 to many) attributes **"memberuid"** with values equal to user`s attribute **“uid”**.
@@ -2378,8 +2653,8 @@ On a group size there is attribute **"member"** (from 0 to many values) and its
To fit the unified way of LDAP usage, we introduced configuration file with set of properties and customized scripts (python and JavaScript based).
-On backend side, all valuable attributes are further collected and passed to these scripts.
-To apply some customization it is required to update a few properties in **security.yml** and customize the scripts.
+On backend side, all valuable attributes are further collected and passed to these scripts.
+To apply some customization it is required to update a few properties in **security.yml** and customize the scripts.
### Properties overview
@@ -2399,14 +2674,14 @@ Additional parameters that are populated during deployment and may be changed in
- **ldapConnectionConfig.ldapHost: ldap host**
- **ldapConnectionConfig.ldapPort: ldap port**
- **ldapConnectionConfig.credentials: ldap credentials**
-
+
### Scripts overview
There are 3 scripts in security.yml:
-- **userLookUp** (python based) - responsible for user lookup in LDap and returns additional user`s attributes;
+- **userLookUp** (python based) - responsible for user lookup in LDap and returns additional user`s attributes;
- **userInfo** (python based) - enriches user with additional data;
- **groupInfo** (javascript based) – responsible for mapping between users and groups;
-
+
### Script structure
The scripts above were created to flexibly manage user`s security configuration. They all are part of **security.yml** configuration. All scripts have following structure:
@@ -2421,14 +2696,14 @@ The scripts above were created to flexibly manage user`s security configuration.
- **searchResultProcessor:**
- **language**
- **code**
-
-Major properties are:
+
+Major properties are:
- **attributes** - list of attributes that will be retrieved from LDAP (-name, -cn, -uid, -member, etc);
-- **filter** - the filter, based on which the object will be retrieved from LDAP;
+- **filter** - the filter, based on which the object will be retrieved from LDAP;
- **searchResultProcessor** - optional. If only LDAP object attributes retrieving is required, this property should be empty. For example, “userLookup” script only retrieves list of "attributes". Otherwise, code customization (like user enrichment, user to groups matching, etc.) should be added into sub-properties below:
- - **language** - the script language - "python" or "JavaScript"
+ - **language** - the script language - "python" or "JavaScript"
- **code** - the script code.
-
+
### "userLookUp" script
@@ -2445,34 +2720,34 @@ Script code:
expirationTimeMsec: 600000
scope: SUBTREE
attributes:
- - cn
+ - cn
- gidNumber
- mail
- memberOf
timeLimit: 0
base: ou=users,ou=alxn,dc=alexion,dc=cloud
filter: "(&(objectCategory=person)(objectClass=user)(mail=%mail%))"
-
+
In the example above, the user login passed from GUI is a mail (**ldapSearchAttribute: mail**) and based on the filer (**filter: "(&(objectCategory=person)(objectClass=user)(mail=%mail%))")** so, the service would search user by its **“mail”**.
If corresponding users are found - the script will return additional user`s attributes:
- cn
- gidNumber
- mail
- memberOf
-
+
User`s authentication into LDAP would be done for DN with following template **ldapBindTemplate: 'cn=%s,ou=users,ou=alxn,dc=alexion,dc=cloud'**, where CN is attribute retrieved by **“userLookUp”** script.
## Azure OAuth2 Authentication <a name="Azure_OAuth2_Authentication"></a>
-DLab supports OAuth2 authentication that is configured automatically in Security Service and Self Service after DLab deployment.
-Please see explanation details about configuration parameters for Self Service and Security Service below.
-DLab supports client credentials(username + password) and authorization code flow for authentication.
+DLab supports OAuth2 authentication that is configured automatically in Security Service and Self Service after DLab deployment.
+Please see explanation details about configuration parameters for Self Service and Security Service below.
+DLab supports client credentials(username + password) and authorization code flow for authentication.
### Azure OAuth2 Self Service configuration
azureLoginConfiguration:
useLdap: false
- tenant: xxxx-xxxx-xxxx-xxxx
+ tenant: xxxx-xxxx-xxxx-xxxx
authority: https://login.microsoftonline.com/
clientId: xxxx-xxxx-xxxx-xxxx
redirectUrl: https://dlab.azure.cloudapp.azure.com/
@@ -2481,7 +2756,7 @@ DLab supports client credentials(username + password) and authorization code flo
silent: true
loginPage: https://dlab.azure.cloudapp.azure.com/
maxSessionDurabilityMilliseconds: 288000000
-
+
where:
- **useLdap** - defines if LDAP authentication is enabled(true/false). If false Azure OAuth2 takes place with configuration properties below
- **tenant** - tenant id of your company
@@ -2490,25 +2765,25 @@ where:
- **redirectUrl** - redirect URL to DLab application after try to login to Azure using OAuth2
- **responseMode** - defines how Azure sends authorization code or error information to DLab during log in procedure
- **prompt** - defines kind of prompt during Oauth2 login
-- **silent** - defines if DLab tries to log in user without interaction(true/false), if false DLab tries to login user with configured prompt
+- **silent** - defines if DLab tries to log in user without interaction(true/false), if false DLab tries to login user with configured prompt
- **loginPage** - start page of DLab application
-- **maxSessionDurabilityMilliseconds** - max user session durability. user will be asked to login after this period of time and when he/she creates ot starts notebook/cluster. This operation is needed to update refresh_token that is used by notebooks to access Data Lake Store
-
+- **maxSessionDurabilityMilliseconds** - max user session durability. user will be asked to login after this period of time and when he/she creates ot starts notebook/cluster. This operation is needed to update refresh_token that is used by notebooks to access Data Lake Store
+
To get more info about *responseMode*, *prompt* parameters please visit [Authorize access to web applications using OAuth 2.0 and Azure Active Directory](https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-protocols-oauth-code)
-
-
+
+
### Azure OAuth2 Security Service configuration
azureLoginConfiguration:
useLdap: false
- tenant: xxxx-xxxx-xxxx-xxxx
+ tenant: xxxx-xxxx-xxxx-xxxx
authority: https://login.microsoftonline.com/
clientId: xxxx-xxxx-xxxx-xxxx
redirectUrl: https://dlab.azure.cloudapp.azure.com/
- validatePermissionScope: true
+ validatePermissionScope: true
permissionScope: subscriptions/xxxx-xxxx-xxxx-xxxx/resourceGroups/xxxx-xxxx/providers/Microsoft.DataLakeStore/accounts/xxxx/providers/Microsoft.Authorization/
managementApiAuthFile: /dlab/keys/azure_authentication.json
-
+
where:
- **useLdap** - defines if LDAP authentication is enabled(true/false). If false Azure OAuth2 takes place with configuration properties below
- **tenant** - tenant id of your company
@@ -2517,5 +2792,4 @@ where:
- **redirectUrl** - redirect URL to DLab application after try to login to Azure using OAuth2
- **validatePermissionScope** - defines(true/false) if user's permissions should be validated to resource that is provided in permissionScope parameter. User will be logged in onlu in case he/she has any role in resource IAM described with permissionScope parameter
- **permissionScope** - describes Azure resource where user should have any role to pass authentication. If user has no role in resource IAM he/she will not be logged in
-- **managementApiAuthFile** - authentication file that is used to query Microsoft Graph API to check user roles in resource described in permissionScope
-
+- **managementApiAuthFile** - authentication file that is used to query Microsoft Graph API to check user roles in resource described in permissionScope
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@dlab.apache.org
For additional commands, e-mail: commits-help@dlab.apache.org