You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dlab.apache.org by lf...@apache.org on 2020/04/02 14:35:25 UTC
[incubator-dlab] 05/05: [DLAB-1680]: removed paragraph about edge
recreation, made some minor changes, made text more readable
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DLAB-1680
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git
commit d1557490e1628103259410f08ebf176d1dff9f25
Author: leonidfrolov <fr...@gmail.com>
AuthorDate: Thu Apr 2 17:34:05 2020 +0300
[DLAB-1680]: removed paragraph about edge recreation, made some minor changes, made text more readable
---
README.md | 469 ++++++++++++++++++++++++++++++++++++--------------------------
1 file changed, 273 insertions(+), 196 deletions(-)
diff --git a/README.md b/README.md
index 6722e59..6db2dcc 100644
--- a/README.md
+++ b/README.md
@@ -68,7 +68,10 @@ CONTENTS
---------------
# What is DLAB? <a name="What_is_DLAB"></a>
-DLab is an essential toolset for analytics. It is a self-service Web Console, used to create and manage exploratory environments. It allows teams to spin up analytical environments with best of breed open-source tools just with a single click of the mouse. Once established, environment can be managed by an analytical team itself, leveraging simple and easy-to-use Web Interface.
+DLab is an essential toolset for analytics. It is a self-service Web Console, used to create and manage exploratory
+environments. It allows teams to spin up analytical environments with best of breed open-source tools just with a
+single click of the mouse. Once established, environment can be managed by an analytical team itself, leveraging simple
+and easy-to-use Web Interface.
<p>See more at <a href="https://dlab.apache.org/" rel="nofollow">dlab.apache.org</a>.</p>
----------------------------
@@ -78,19 +81,28 @@ The following diagram demonstrate high-level logical architecture.
-The diagram shows main components of DLab, which is a self-service for the infrastructure deployment and interaction with it. The purpose of each component is described below.
+The diagram shows main components of DLab, which is a self-service for the infrastructure deployment and interaction
+with it. The purpose of each component is described below.
## Self-Service
-Self-Service is a service, which provides RESTful user API with Web User Interface for data scientist. It tightly interacts with Provisioning Service and Database. Self-Service delegates all user\`s requests to Provisioning Service. After execution of certain request from Self-service, Provisioning Service returns response about corresponding action happened with particular resource. Self-service, then, saves this response into Database. So, each time Self-Service receives request about [...]
+Self-Service is a service, which provides RESTful user API with Web User Interface for data scientist. It tightly
+interacts with Provisioning Service and Database. Self-Service delegates all user\`s requests to Provisioning Service.
+After execution of certain request from Self-service, Provisioning Service returns response about corresponding action
+happened with particular resource. Self-service, then, saves this response into Database. So, each time Self-Service
+receives request about status of provisioned infrastructure resources – it loads it from Database and propagates to Web UI.
## Billing
-Billing is a module, which provides a loading of the billing report for the environment to the database. It can be running as part of the Self-Service or a separate process.
+Billing is a module, which provides a loading of the billing report for the environment to the database. It can be
+running as part of the Self-Service or a separate process.
## Provisioning Service
-The Provisioning Service is a RESTful service, which provides APIs for provisioning of the user’s infrastructure. Provisioning Service receives the request from Self-Service, afterwards it forms and sends a command to the docker to execute requested action. Docker executes the command and generates a response.json file. Provisioning service analyzes response.json and responds to initial request of Self-Service, providing status-related information of the instance.
+The Provisioning Service is a RESTful service, which provides APIs for provisioning of the user’s infrastructure.
+Provisioning Service receives the request from Self-Service, afterwards it forms and sends a command to the docker
+to execute requested action. Docker executes the command and generates a response.json file. Provisioning service
+analyzes response.json and responds to initial request of Self-Service, providing status-related information of the instance.
## Security service
@@ -125,37 +137,49 @@ The following diagrams demonstrate high-level physical architecture of DLab in A
Creation of self-service node – is the first step for deploying DLab. SSN is a main server with following pre-installed services:
-- DLab Web UI – is Web user interface for managing/deploying all components of DLab. It is accessible by the following URL: http[s]://SSN\_Public\_IP\_or\_Public\_DNS
-- MongoDB – is a database, which contains part of DLab’s configuration, user’s exploratory environments description as well as user’s preferences.
+- DLab Web UI – is Web user interface for managing/deploying all components of DLab. It is accessible by the
+ following URL: http[s]://SSN\_Public\_IP\_or\_Public\_DNS
+- MongoDB – is a database, which contains part of DLab’s configuration, user’s exploratory environments description
+ as well as user’s preferences.
- Docker – used for building DLab Docker containers, which will be used for provisioning other components.
- Jenkins – is an alternative to Web UI. It is accessible by the following link: http[s]://SSN\_Public\_IP\_or\_Public\_DNS/jenkins
-Elastic(Static) IP address is assigned to an SSN Node, so you are free to stop|start it and and SSN node's IP address won’t change.
+Elastic(Static) IP address is assigned to an SSN Node, so you are free to stop|start it and and SSN node's IP address
+won’t change.
## Edge node
-Setting up Edge node is the first step that user is asked to do once logged into DLab. This node is used as proxy server and SSH gateway for the user. Through Edge node users can access Notebook via HTTP and SSH. Edge Node has a Squid HTTP web proxy pre-installed.
+Setting up Edge node is the first step that user is asked to do once logged into DLab. This node is used as proxy
+server and SSH gateway for the user. Through Edge node users can access Notebook via HTTP and SSH. Edge Node has a
+Squid HTTP web proxy pre-installed.
## Notebook node
-The next step is setting up a Notebook node (or a Notebook server). It is a server with pre-installed applications and libraries for data processing, data cleaning and transformations, numerical simulations, statistical modeling, machine learning, etc. Following analytical tools are currently supported in DLab and can be installed on a Notebook node:
+The next step is setting up a Notebook node (or a Notebook server). It is a server with pre-installed applications and
+libraries for data processing, data cleaning and transformations, numerical simulations, statistical modeling, machine
+learning, etc. Following analytical tools are currently supported in DLab and can be installed on a Notebook node:
- Jupyter
+- Jupyterlab
- RStudio
- Apache Zeppelin
- TensorFlow + Jupyter
+- TensorFlow + RStudio
- Deep Learning + Jupyter
Apache Spark is also installed for each of the analytical tools above.
-**Note:** terms 'Apache Zeppelin' and 'Apache Spark' hereinafter may be referred to as 'Zeppelin' and 'Spark' respectively or may have original reference.
+**Note:** terms 'Apache Zeppelin' and 'Apache Spark' hereinafter may be referred to as 'Zeppelin' and 'Spark'
+respectively or may have original reference.
## Data engine cluster
After deploying Notebook node, user can create one of the cluster for it:
- Data engine - Spark standalone cluster
- Data engine service - cloud managed cluster platform (EMR for AWS or Dataproc for GCP)
-That simplifies running big data frameworks, such as Apache Hadoop and Apache Spark to process and analyze vast amounts of data. Adding cluster is not mandatory and is only needed in case additional computational resources are required for job execution.
+That simplifies running big data frameworks, such as Apache Hadoop and Apache Spark to process and analyze vast amounts
+of data. Adding cluster is not mandatory and is only needed in case additional computational resources are required for
+job execution.
----------------------
# DLab Deployment <a name="DLab_Deployment"></a>
@@ -192,6 +216,7 @@ SSN node structure of log directory is as follows:
├───dateengine-service
├───edge
├───notebook
+ ├───project
└───ssn
These directories contain the log files for each template and for DLab back-end services.
@@ -203,7 +228,8 @@ These directories contain the log files for each template and for DLab back-end
## Keycloak server <a name="Keycloak_server"></a>
-**Keycloak** is used to manage user authentication instead of the aplication. To use existing server following parameters must be specified either when running *Dlab* deployment script or in
+**Keycloak** is used to manage user authentication instead of the aplication. To use existing server following
+ parameters must be specified either when running *Dlab* deployment script or in
*/opt/dlab/conf/self-service.yml* and */opt/dlab/conf/provisioning.yml* files on SSN node.
| Parameter | Description/Value |
@@ -223,7 +249,8 @@ Preparation steps for deployment:
- Create an VM instance with the following settings:
- The instance should have access to Internet in order to install required prerequisites
- Boot disk OS Image - Ubuntu 18.04
-- Put private key that is used to connect to instance where Keycloak will be deployed somewhere on the instance where deployment script will be executed.
+- Put private key that is used to connect to instance where Keycloak will be deployed somewhere on the instance where
+ deployment script will be executed.
- Install Git and clone DLab repository</details>
### Executing deployment script
To build Keycloak node, following steps should be executed:
@@ -267,11 +294,13 @@ Prerequisites:
DLab can be deployed using the following two methods:
- IAM user: DLab deployment script is executed on local machine and uses IAM user permissions to create resources in AWS.
- - EC2 instance: DLab deployment script is executed on EC2 instance prepared in advance and with attached IAM role. Deployment script uses the attached IAM role to create resources in AWS.
+ - EC2 instance: DLab deployment script is executed on EC2 instance prepared in advance and with attached IAM role.
+ Deployment script uses the attached IAM role to create resources in AWS.
**'IAM user' method prerequisites:**
- - IAM user with created AWS access key ID and secret access key. These keys are provided as arguments for the deployment script and are used to create resources in AWS.
+ - IAM user with created AWS access key ID and secret access key. These keys are provided as arguments for the
+ deployment script and are used to create resources in AWS.
- Amazon EC2 Key Pair. This key is system and is used for configuring DLab instances.
- The following IAM [policy](#AWS_SSN_policy) should be attached to the IAM user in order to deploy DLab.
@@ -283,8 +312,10 @@ DLab can be deployed using the following two methods:
**Optional prerequisites for both methods:**
- - VPC ID. If VPC where DLab should be deployed is already in place, then "VPC ID" should be provided for deployment script. DLab instances are deployed in this VPC.
- - Subnet ID. If Subnet where DLab should be deployed is already in place, then "Subnet ID" should be provided for deployment script. DLab SSN node and users' Edge nodes are deployed in this Subnet.
+ - VPC ID. If VPC where DLab should be deployed is already in place, then "VPC ID" should be provided for deployment
+ script. DLab instances are deployed in this VPC.
+ - Subnet ID. If Subnet where DLab should be deployed is already in place, then "Subnet ID" should be provided for
+ deployment script. DLab SSN node and users' Edge nodes are deployed in this Subnet.
DLab IAM Policy
<a name="AWS_SSN_policy"></a>
@@ -414,7 +445,8 @@ Prerequisites:
Prerequisites:
- IAM user
-- Service account and JSON auth file for it. In order to get JSON auth file, Key should be created for service account through Google cloud console.
+- Service account and JSON auth file for it. In order to get JSON auth file, Key should be created for service account
+through Google cloud console.
- Google Cloud Storage JSON API should be enabled
Preparation steps for deployment:
@@ -447,7 +479,8 @@ pip install fabric==1.14.0
- Go to *dlab* directory
- Run *infrastructure-provisioning/scripts/deploy_dlab.py* deployment script:
-This python script will build front-end and back-end part of DLab, create SSN docker image and run Docker container for creating SSN node.
+This python script will build front-end and back-end part of DLab, create SSN docker image and run Docker container
+for creating SSN node.
<details><summary>In Amazon cloud <i>(click to expand)</i></summary>
@@ -484,7 +517,7 @@ List of parameters for SSN node deployment:
- aws\_subnet\_id
- aws\_sg\_ids
-**Note:** If billing won't be using, the following parameters are not required:
+**Note:** If billing won't be used, the following parameters are not required:
- aws\_account\_id
- aws\_billing\_bucket
- aws\_report\_path
@@ -496,8 +529,7 @@ After SSN node deployment following AWS resources will be created:
- IAM role and EC2 Instance Profile for SSN
- Security Group for SSN node (if it was specified, script will attach the provided one)
- VPC, Subnet (if they have not been specified) for SSN and EDGE nodes
-- S3 bucket – its name will be \<service\_base\_name\>-ssn-bucket. This bucket will contain necessary dependencies and configuration files for Notebook nodes (such as .jar files, YARN configuration, etc.)
-- S3 bucket for for collaboration between Dlab users. Its name will be \<service\_base\_name\>-shared-bucket</details>
+- S3 bucket for for collaboration between Dlab users. Its name will be \<service\_base\_name\>-\<endpoint\_name\>-shared-bucket</details>
<details><summary>In Azure cloud <i>(click to expand)</i></summary>
@@ -541,26 +573,38 @@ List of parameters for SSN node deployment:
**Note:** Billing configuration:
-To know azure\_offer\_number open [Azure Portal](https://portal.azure.com), go to Subscriptions and open yours, then click Overview and you should see it under Offer ID property:
+To know azure\_offer\_number open [Azure Portal](https://portal.azure.com), go to Subscriptions and open yours, then
+click Overview and you should see it under Offer ID property:
-Please see [RateCard API](https://msdn.microsoft.com/en-us/library/mt219004.aspx) to get more details about azure\_offer\_number,
-azure\_currency, azure\_locale, azure\_region_info. These DLab deploy properties correspond to RateCard API request parameters.
+Please see [RateCard API](https://msdn.microsoft.com/en-us/library/mt219004.aspx) to get more details about
+azure\_offer\_number, azure\_currency, azure\_locale, azure\_region_info. These DLab deploy properties correspond to
+RateCard API request parameters.
-To have working billing functionality please review Billing configuration note and use proper parameters for SSN node deployment.
+To have working billing functionality please review Billing configuration note and use proper parameters for SSN node
+deployment.
-To use Data Lake Store please review Azure Data Lake usage pre-requisites note and use proper parameters for SSN node deployment.
+To use Data Lake Store please review Azure Data Lake usage pre-requisites note and use proper parameters for SSN node
+deployment.
**Note:** Azure Data Lake usage pre-requisites:
1. Configure application in Azure portal and grant proper permissions to it.
- Open *Azure Active Directory* tab, then *App registrations* and click *New application registration*
-- Fill in ui form with the following parameters *Name* - put name of the new application, *Application type* - select Native, *Sign-on URL* put any valid url as it will be updated later
-- Grant proper permissions to the application. Select the application you just created on *App registration* view, then click *Required permissions*, then *Add->Select an API-> In search field type MicrosoftAzureQueryService* and press *Select*, then check the box *Have full access to the Azure Data Lake service* and save the changes. Repeat the same actions for *Windows Azure Active Directory* API (available on *Required permissions->Add->Select an API*) and the box *Sign in and read us [...]
+- Fill in ui form with the following parameters *Name* - put name of the new application, *Application type* - select
+ Native, *Sign-on URL* put any valid url as it will be updated later
+- Grant proper permissions to the application. Select the application you just created on *App registration* view, then
+ click *Required permissions*, then *Add->Select an API-> In search field type MicrosoftAzureQueryService* and press
+ *Select*, then check the box *Have full access to the Azure Data Lake service* and save the changes. Repeat the same
+ actions for *Windows Azure Active Directory* API (available on *Required permissions->Add->Select an API*) and the
+ box *Sign in and read user profile*
- Get *Application ID* from application properties it will be used as azure_application_id for deploy_dlap.py script
-2. Usage of Data Lake resource predicts shared folder where all users can write or read any data. To manage access to this folder please create ot use existing group in Active Directory. All users from this group will have RW access to the shared folder. Put ID(in Active Directory) of the group as *azure_ad_group_id* parameter to deploy_dlab.py script
-3. After execution of deploy_dlab.py script go to the application created in step 1 and change *Redirect URIs* value to the https://SSN_HOSTNAME/ where SSN_HOSTNAME - SSN node hostname
+2. Usage of Data Lake resource predicts shared folder where all users can write or read any data. To manage access to
+ this folder please create ot use existing group in Active Directory. All users from this group will have RW access to
+ the shared folder. Put ID(in Active Directory) of the group as *azure_ad_group_id* parameter to deploy_dlab.py script
+3. After execution of deploy_dlab.py script go to the application created in step 1 and change *Redirect URIs* value to
+ the https://SSN_HOSTNAME/ where SSN_HOSTNAME - SSN node hostname
After SSN node deployment following Azure resources will be created:
@@ -570,7 +614,6 @@ After SSN node deployment following Azure resources will be created:
- Network interface for SSN node
- Security Group for SSN node (if it was specified, script will attach the provided one)
- Virtual network and Subnet (if they have not been specified) for SSN and EDGE nodes
-- Storage account and blob container for necessary further dependencies and configuration files for Notebook nodes (such as .jar files, YARN configuration, etc.)
- Storage account and blob container for collaboration between Dlab users
- If support of Data Lake is enabled: Data Lake and shared directory will be created</details>
@@ -582,26 +625,27 @@ After SSN node deployment following Azure resources will be created:
List of parameters for SSN node deployment:
-| Parameter | Description/Value |
-|------------------------------|-----------------------------------------------------------------------------------------|
-| conf\_service\_base\_name | Any infrastructure value (should be unique if multiple SSN’s have been deployed before) |
-| gcp\_region | GCP region |
-| gcp\_zone | GCP zone |
-| conf\_os\_family | Name of the Linux distributive family, which is supported by DLab (Debian/RedHat) |
-| conf\_cloud\_provider | Name of the cloud provider, which is supported by DLab (GCP) |
-| gcp\_vpc\_name | Name of the Virtual Network (VN) (optional) |
-| gcp\_subnet\_name | Name of the GCP subnet (optional) |
-| gcp\_firewall\_name | One or more Name\`s of GCP Security Groups, which will be assigned to SSN node (optional) |
-| key\_path | Path to admin key (without key name) |
-| conf\_key\_name | Name of the uploaded SSH key file (without “.pem” extension) |
-| gcp\_service\_account\_path | Full path to auth json file |
-| gcp\_ssn\_instance\_size | Instance size of SSN instance in GCP |
-| gcp\_project\_id | ID of GCP project |
-| action | In case of SSN node creation, this parameter should be set to “create” |
+| Parameter | Description/Value |
+|------------------------------|---------------------------------------------------------------------------------------|
+| conf\_service\_base\_name | Any infrastructure value (should be unique if multiple SSN’s have been deployed before)|
+| gcp\_region | GCP region |
+| gcp\_zone | GCP zone |
+| conf\_os\_family | Name of the Linux distributive family, which is supported by DLab (Debian/RedHat) |
+| conf\_cloud\_provider | Name of the cloud provider, which is supported by DLab (GCP) |
+| gcp\_vpc\_name | Name of the Virtual Network (VN) (optional) |
+| gcp\_subnet\_name | Name of the GCP subnet (optional) |
+| gcp\_firewall\_name | One or more Name\`s of GCP Security Groups, which will be assigned to SSN node (optional)|
+| key\_path | Path to admin key (without key name) |
+| conf\_key\_name | Name of the uploaded SSH key file (without “.pem” extension) |
+| gcp\_service\_account\_path | Full path to auth json file |
+| gcp\_ssn\_instance\_size | Instance size of SSN instance in GCP |
+| gcp\_project\_id | ID of GCP project |
+| action | In case of SSN node creation, this parameter should be set to “create” |
| conf\_image\_enabled | Enable or Disable creating image at first time |
| billing\_dataset\_name | Name of GCP dataset (BigQuery service) |
-**Note:** If you gonna use Dataproc cluster, be aware that Dataproc has limited availability in GCP regions. [Cloud Dataproc availability by Region in GCP](https://cloud.google.com/about/locations/)
+**Note:** If you gonna use Dataproc cluster, be aware that Dataproc has limited availability in GCP regions.
+[Cloud Dataproc availability by Region in GCP](https://cloud.google.com/about/locations/)
After SSN node deployment following GCP resources will be created:
@@ -610,12 +654,13 @@ After SSN node deployment following GCP resources will be created:
- IAM role and Service account for SSN
- Security Groups for SSN node (if it was specified, script will attach the provided one)
- VPC, Subnet (if they have not been specified) for SSN and EDGE nodes
-- Bucket – its name will be \<service\_base\_name\>-ssn-bucket. This bucket will contain necessary dependencies and configuration files for Notebook nodes (such as .jar files, YARN configuration, etc.)
-- Bucket for for collaboration between Dlab users. Its name will be \<service\_base\_name\>-shared-bucket</details>
+- Bucket for for collaboration between Dlab users. Its name will be
+ \<service\_base\_name\>-\<endpoint\_name\>-shared-bucket</details>
### Terminating Self-Service Node
-Terminating SSN node will also remove all nodes and components related to it. Basically, terminating Self-service node will terminate all DLab’s infrastructure.
+Terminating SSN node will also remove all nodes and components related to it. Basically, terminating Self-service node
+will terminate all DLab’s infrastructure.
Example of command for terminating DLab environment:
<details><summary>In Amazon <i>(click to expand)</i></summary>
@@ -665,31 +710,39 @@ List of parameters for SSN node termination:
```
List of parameters for SSN node termination:
-| Parameter | Description/Value |
-|------------------------------|-----------------------------------------------------------------------------------------|
-| conf\_service\_base\_name | Any infrastructure value (should be unique if multiple SSN’s have been deployed before) |
-| gcp\_region | GCP region |
-| gcp\_zone | GCP zone |
-| conf\_os\_family | Name of the Linux distributive family, which is supported by DLab (Debian/RedHat) |
-| conf\_cloud\_provider | Name of the cloud provider, which is supported by DLab (GCP) |
-| gcp\_vpc\_name | Name of the Virtual Network (VN) (optional) |
-| gcp\_subnet\_name | Name of the GCP subnet (optional) |
-| key\_path | Path to admin key (without key name) |
-| conf\_key\_name | Name of the uploaded SSH key file (without “.pem” extension) |
-| gcp\_service\_account\_path | Full path to auth json file |
-| gcp\_project\_id | ID of GCP project |
-| action | In case of SSN node termination, this parameter should be set to “terminate” |
-
-Note: It is required to enter gcp_vpc_name and gcp_subnet_name parameters if Self-Service Node was deployed in pre-defined VPC and Subnet.
+| Parameter | Description/Value |
+|------------------------------|---------------------------------------------------------------------------------------|
+| conf\_service\_base\_name | Any infrastructure value (should be unique if multiple SSN’s have been deployed before)|
+| gcp\_region | GCP region |
+| gcp\_zone | GCP zone |
+| conf\_os\_family | Name of the Linux distributive family, which is supported by DLab (Debian/RedHat) |
+| conf\_cloud\_provider | Name of the cloud provider, which is supported by DLab (GCP) |
+| gcp\_vpc\_name | Name of the Virtual Network (VN) (optional) |
+| gcp\_subnet\_name | Name of the GCP subnet (optional) |
+| key\_path | Path to admin key (without key name) |
+| conf\_key\_name | Name of the uploaded SSH key file (without “.pem” extension) |
+| gcp\_service\_account\_path | Full path to auth json file |
+| gcp\_project\_id | ID of GCP project |
+| action | In case of SSN node termination, this parameter should be set to “terminate” |
+
+Note: It is required to enter gcp_vpc_name and gcp_subnet_name parameters if Self-Service Node was deployed in
+pre-defined VPC and Subnet.
</details>
## Edge Node <a name="Edge_Node"></a>
-Gateway node (or an Edge node) is an instance(virtual machine) provisioned in a public subnet. It serves as an entry point for accessing user’s personal analytical environment. It is created by an end-user, whose public key will be uploaded there. Only via Edge node, DLab user can access such application resources as notebook servers and dataengine clusters. Also, Edge Node is used to setup SOCKS proxy to access notebook servers via Web UI and SSH. Elastic(Static) IP address is assigned [...]
+Gateway node (or an Edge node) is an instance(virtual machine) provisioned in a public subnet. It serves as an entry
+point for accessing user’s personal analytical environment. It is created by an end-user, whose public key will be
+uploaded there. Only via Edge node, DLab user can access such application resources as notebook servers and dataengine
+clusters. Also, Edge Node is used to setup SOCKS proxy to access notebook servers via Web UI and SSH. Elastic(Static)
+IP address is assigned to an Edge Node.
### Create
-In order to create Edge node using DLab Web UI – login and, click on the button “Upload” (Depending on authorization provider that was chosen on deployment stage, user may be taken from [LDAP](#LDAP_Authentication) or from [Azure AD (Oauth2)](#Azure_OAuth2_Authentication)). Choose user’s SSH public key and after that click on the button “Create”. Edge node will be deployed and corresponding instance (virtual machine) will be started.
+In order to create Edge node using DLab Web UI – login and, click on the button “Upload” (Depending on authorization
+provider that was chosen on deployment stage, user may be taken from [LDAP](#LDAP_Authentication) or from
+[Azure AD (Oauth2)](#Azure_OAuth2_Authentication)). Choose user’s SSH public key and after that click on the button
+“Create”. Edge node will be deployed and corresponding instance (virtual machine) will be started.
<details><summary>In Amazon <i>(click to expand)</i></summary>
@@ -733,7 +786,8 @@ The following Azure resources will be created:
- Security Group for all further user's Notebook instances
- Security Groups for all further user's master nodes of data engine cluster
- Security Groups for all further user's slave nodes of data engine cluster
-- User's private subnet. All further nodes (Notebooks, data engine clusters) will be provisioned in different subnet than SSN.
+- User's private subnet. All further nodes (Notebooks, data engine clusters) will be provisioned in different subnet
+ than SSN.
- User's storage account and blob container
List of parameters for Edge node creation:
@@ -761,7 +815,8 @@ The following GCP resources will be created:
- Security Group for all further user's Notebook instances
- Security Groups for all further user's master nodes of data engine cluster
- Security Groups for all further user's slave nodes of data engine cluster
-- User's private subnet. All further nodes (Notebooks, data engine clusters) will be provisioned in different subnet than SSN.
+- User's private subnet. All further nodes (Notebooks, data engine clusters) will be provisioned in different subnet
+ than SSN.
- User's bucket
List of parameters for Edge node creation:
@@ -783,7 +838,8 @@ List of parameters for Edge node creation:
### Start/Stop <a name=""></a>
-To start/stop Edge node, click on the button which looks like a cycle on the top right corner, then click on the button which is located in “Action” field and in the drop-down menu click on the appropriate action.
+To start/stop Edge node, click on the button which looks like a cycle on the top right corner, then click on the button
+which is located in “Action” field and in the drop-down menu click on the appropriate action.
<details><summary>In Amazon <i>(click to expand)</i></summary>
@@ -837,72 +893,19 @@ List of parameters for Edge node starting/stopping:
| action | start/stop |
</details>
-### Recreate <a name=""></a>
-
-In case Edge node was damaged, or terminated manually, there is an option to re-create it.
-
-If Edge node was removed for some reason, to re-create it, click on the status button close to logged in users’s name (top right corner of the screen).Then click on gear icon in Actions column and choose “Recreate”.
-
-List of parameters for Edge node recreation:
-
-<details><summary>In Amazon <i>(click to expand)</i></summary>
-
-| Parameter | Description/Value |
-|----------------------------|-----------------------------------------------------------------------------------|
-| conf\_resource | edge |
-| conf\_os\_family | Name of the Linux distributive family, which is supported by DLAB (Debian/RedHat) |
-| conf\_service\_base\_name | Unique infrastructure value, specified during SSN deployment |
-| conf\_key\_name | Name of the uploaded SSH key file (without ".pem") |
-| edge\_user\_name | Name of the user |
-| aws\_vpc\_id | ID of AWS VPC where infrastructure is being deployed |
-| aws\_region | AWS region where infrastructure was deployed |
-| aws\_security\_groups\_ids | ID of the SSN instance's AWS security group |
-| aws\_subnet\_id | ID of the AWS public subnet where Edge was deployed |
-| edge\_elastic\_ip | AWS Elastic IP address which was associated to Edge node |
-| conf\_tag\_resource\_id | The name of tag for billing reports |
-| action | Create |
-</details>
-
-<details><summary>In Azure <i>(click to expand)</i></summary>
-
-| Parameter | Description/Value |
-|------------------------------|-----------------------------------------------------------------------------------|
-| conf\_resource | edge |
-| conf\_os\_family | Name of the Linux distributive family, which is supported by DLAB (Debian/RedHat) |
-| conf\_service\_base\_name | Unique infrastructure value, specified during SSN deployment |
-| conf\_key\_name | Name of the uploaded SSH key file (without ".pem") |
-| edge\_user\_name | Name of the user |
-| azure\_vpc\_name | NAme of Azure Virtual network where all infrastructure is being deployed |
-| azure\_region | Azure region where all infrastructure was deployed |
-| azure\_resource\_group\_name | Name of the resource group where all DLAb resources are being provisioned |
-| azure\_subnet\_name | Name of the Azure public subnet where Edge was deployed |
-| action | Create |
-</details>
-
-<details><summary>In Google cloud <i>(click to expand)</i></summary>
-
-| Parameter | Description/Value |
-|--------------------------------|-----------------------------------------------------------------------------------|
-| conf\_resource | edge |
-| conf\_os\_family | Name of the Linux distributive family, which is supported by DLAB (debian/redhat) |
-| conf\_service\_base\_name | Unique infrastructure value, specified during SSN deployment |
-| conf\_key\_name | Name of the uploaded SSH key file (without ".pem") |
-| edge\_user\_name | Name of the user |
-| gcp\_region | GCP region where infrastructure was deployed |
-| gcp\_zone | GCP zone where infrastructure was deployed |
-| gcp\_vpc\_name | Name of Azure Virtual network where all infrastructure is being deployed |
-| gcp\_subnet\_name | Name of the Azure public subnet where Edge will be deployed |
-| gcp\_project\_id | ID of GCP project |
-| action | create |
-</details>
-
## Notebook node <a name="Notebook_node"></a>
-Notebook node is an instance (virtual machine), with preinstalled analytical software, needed dependencies and with pre-configured kernels and interpreters. It is the main part of personal analytical environment, which is setup by a data scientist. It can be Created, Stopped and Terminated. To support variety of analytical needs - Notebook node can be provisioned on any of cloud supported instance shape for your particular region. From analytical software, which is already pre-installed [...]
+Notebook node is an instance (virtual machine), with preinstalled analytical software, needed dependencies and with
+pre-configured kernels and interpreters. It is the main part of personal analytical environment, which is setup by a
+data scientist. It can be Created, Stopped and Terminated. To support variety of analytical needs - Notebook node can
+be provisioned on any of cloud supported instance shape for your particular region. From analytical software, which is
+already pre-installed on a notebook node, end users can access (read/write) data stored on buckets/containers.
### Create
-To create Notebook node, click on the “Create new” button. Then, in drop-down menu choose template type (jupyter/rstudio/zeppelin/tensor), enter notebook name and choose instance shape. After clicking the button “Create”, notebook node will be deployed and started.
+To create Notebook node, click on the “Create new” button. Then, in drop-down menu choose template type
+(jupyter/rstudio/zeppelin/tensor/etc.), enter notebook name and choose instance shape. After clicking the button
+“Create”, notebook node will be deployed and started.
List of parameters for Notebook node creation:
@@ -965,7 +968,8 @@ List of parameters for Notebook node creation:
### Stop
-In order to stop Notebook node, click on the “gear” button in Actions column. From the drop-down menu click on “Stop” action.
+In order to stop Notebook node, click on the “gear” button in Actions column. From the drop-down menu click on “Stop”
+action.
List of parameters for Notebook node stopping:
@@ -1012,7 +1016,8 @@ List of parameters for Notebook node stopping:
### Start
-In order to start Notebook node, click on the button, which looks like gear in “Action” field. Then in drop-down menu choose “Start” action.
+In order to start Notebook node, click on the button, which looks like gear in “Action” field. Then in drop-down menu
+choose “Start” action.
List of parameters for Notebook node start:
@@ -1065,7 +1070,8 @@ List of parameters for Notebook node start:
### Terminate
-In order to terminate Notebook node, click on the button, which looks like gear in “Action” field. Then in drop-down menu choose “Terminate” action.
+In order to terminate Notebook node, click on the button, which looks like gear in “Action” field. Then in drop-down
+menu choose “Terminate” action.
List of parameters for Notebook node termination:
@@ -1113,7 +1119,8 @@ List of parameters for Notebook node termination:
### List/Install additional libraries
-In order to list available libraries (OS/Python2/Python3/R/Others) on Notebook node, click on the button, which looks like gear in “Action” field. Then in drop-down menu choose “Manage libraries” action.
+In order to list available libraries (OS/Python2/Python3/R/Others) on Notebook node, click on the button, which looks
+like gear in “Action” field. Then in drop-down menu choose “Manage libraries” action.
<details><summary>In Amazon <i>(click to expand)</i></summary>
@@ -1241,7 +1248,8 @@ List of parameters for Notebook node to **install** additional libraries:
### Manage git credentials
-In order to manage git credentials on Notebook node, click on the button “Git credentials”. Then in menu you can add or edit existing credentials.
+In order to manage git credentials on Notebook node, click on the button “Git credentials”. Then in menu you can add or
+edit existing credentials.
<details><summary>In Amazon <i>(click to expand)</i></summary>
@@ -1309,11 +1317,15 @@ List of parameters for Notebook node to **manage git credentials**:
## Dataengine-service cluster <a name="Dataengine-service_cluster"></a>
-Dataengine-service is a cluster provided by cloud as a service (EMR on AWS) can be created if more computational resources are needed for executing analytical algorithms and models, triggered from analytical tools. Jobs execution will be scaled to a cluster mode increasing the performance and decreasing execution time.
+Dataengine-service is a cluster provided by cloud as a service (EMR on AWS) can be created if more computational
+resources are needed for executing analytical algorithms and models, triggered from analytical tools. Jobs execution
+will be scaled to a cluster mode increasing the performance and decreasing execution time.
### Create
-To create dataengine-service cluster click on the “gear” button in Actions column, and click on “Add computational resources”. Specify dataengine-service version, fill in dataengine-service name, specify number of instances and instance shapes. Click on the “Create” button.
+To create dataengine-service cluster click on the “gear” button in Actions column, and click on “Add computational
+resources”. Specify dataengine-service version, fill in dataengine-service name, specify number of instances and
+instance shapes. Click on the “Create” button.
List of parameters for dataengine-service cluster creation:
@@ -1399,7 +1411,8 @@ List of parameters for dataengine-service cluster termination:
### List/Install additional libraries
-In order to list available libraries (OS/Python2/Python3/R/Others) on Dataengine-service, click on the button, which looks like gear in “Action” field. Then in drop-down menu choose “Manage libraries” action.
+In order to list available libraries (OS/Python2/Python3/R/Others) on Dataengine-service, click on the button, which
+looks like gear in “Action” field. Then in drop-down menu choose “Manage libraries” action.
<details><summary>In Amazon <i>(click to expand)</i></summary>
@@ -1492,11 +1505,14 @@ List of parameters for Dataengine-service node to **install** additional librari
## Dataengine cluster <a name="Dataengine_cluster"></a>
-Dataengine is cluster based on Standalone Spark framework can be created if more computational resources are needed for executing analytical algorithms, but without additional expenses for cloud provided service.
+Dataengine is cluster based on Standalone Spark framework can be created if more computational resources are needed for
+executing analytical algorithms, but without additional expenses for cloud provided service.
### Create
-To create Spark standalone cluster click on the “gear” button in Actions column, and click on “Add computational resources”. Specify dataengine version, fill in dataengine name, specify number of instances and instance shapes. Click on the “Create” button.
+To create Spark standalone cluster click on the “gear” button in Actions column, and click on “Add computational
+resources”. Specify dataengine version, fill in dataengine name, specify number of instances and instance shapes.
+Click on the “Create” button.
List of parameters for dataengine cluster creation:
@@ -1611,7 +1627,8 @@ List of parameters for dataengine cluster termination:
### List/Install additional libraries
-In order to list available libraries (OS/Python2/Python3/R/Others) on Dataengine, click on the button, which looks like gear in “Action” field. Then in drop-down menu choose “Manage libraries” action.
+In order to list available libraries (OS/Python2/Python3/R/Others) on Dataengine, click on the button, which looks like
+gear in “Action” field. Then in drop-down menu choose “Manage libraries” action.
<details><summary>In Amazon <i>(click to expand)</i></summary>
@@ -1757,12 +1774,15 @@ sudo supervisorctl {start | stop | status} [all | provserv | secserv | ui]
## DLab Web UI <a name="DLab Web UI"></a>
-DLab self service is listening to the secure 8443 port. This port is used for secure local communication with provisioning service.
+DLab self service is listening to the secure 8443 port. This port is used for secure local communication with
+provisioning service.
There is also Nginx proxy server running on Self-Service node, which proxies remote connection to local 8443 port.
-Nginx server is listening to both 80 and 443 ports by default. It means that you could access self-service Web UI using non-secure connections (80 port) or secure (443 port).
+Nginx server is listening to both 80 and 443 ports by default. It means that you could access self-service Web UI using
+non-secure connections (80 port) or secure (443 port).
-Establishing connection using 443 port you should take into account that DLab uses self-signed certificate from the box, however you are free to switch Nginx to use your own domain-verified certificate.
+Establishing connection using 443 port you should take into account that DLab uses self-signed certificate from the box,
+however you are free to switch Nginx to use your own domain-verified certificate.
To disable non-secure connection please do the following:
- uncomment at /etc/nginx/conf.d/nginx_proxy.conf file rule that rewrites all requests from 80 to 443 port;
@@ -1783,14 +1803,18 @@ Billing module is implemented as a separate jar file and can be running in the f
- separate system process;
- manual loading or use external scheduler;
-The billing module is running as part of the Self-Service (if billing was switched ON before SSN deployment). For details please refer to section [Self-Service Node](#Self_Service_Node). Otherwise, you should manually configure file billing.yml. See the descriptions how to do this in the configuration file. Please also note, that you should also add an entry in the Mongo database into collection:
+The billing module is running as part of the Self-Service (if billing was switched ON before SSN deployment). For
+details please refer to section [Self-Service Node](#Self_Service_Node). Otherwise, you should manually configure file
+billing.yml. See the descriptions how to do this in the configuration file. Please also note, that you should also add
+an entry in the Mongo database into collection:
```
{
"_id": "conf_tag_resource_id",
"Value": "<CONF_TAG_RESOURCE_ID>"
}
```
-After you have configured the billing, you can run it as a process of Self-Service. To do this, in the configuration file self-service.yml set the property **BillingSchedulerEnabled** to **true** and restart the Self-Service:
+After you have configured the billing, you can run it as a process of Self-Service. To do this, in the configuration
+file self-service.yml set the property **BillingSchedulerEnabled** to **true** and restart the Self-Service:
```
sudo supervisorctl stop ui
sudo supervisorctl start ui
@@ -1925,7 +1949,8 @@ docker build --build-arg OS=<os_family> --file general/files/<cloud_provider>/da
----------------
# Development <a name="Development"></a>
-DLab services could be ran in development mode. This mode emulates real work an does not create any resources on cloud provider environment.
+DLab services could be ran in development mode. This mode emulates real work an does not create any resources on cloud
+provider environment.
## Folder structure <a name="Folder_structure"></a>
@@ -1950,7 +1975,8 @@ DLab services could be ran in development mode. This mode emulates real work an
## Pre-requisites <a name="Pre-requisites"></a>
-In order to start development of Front-end Web UI part of DLab - Git repository should be cloned and the following packages should be installed:
+In order to start development of Front-end Web UI part of DLab - Git repository should be cloned and the following
+packages should be installed:
- Git 1.7 or higher
- Python 2.7 with library Fabric v1.14.0
@@ -1971,7 +1997,8 @@ Common is a module, which wraps set of reusable code over services. Commonly reu
#### Self-Service
-Self-Service provides REST based API’s. It tightly interacts with Provisioning Service and Security Service and actually delegates most of user\`s requests for execution.
+Self-Service provides REST based API’s. It tightly interacts with Provisioning Service and Security Service and actually
+delegates most of user\`s requests for execution.
| API class name | Supported actions | Description |
|---------------------------------|-------------------------------------------------------|------------------------|
@@ -1987,12 +2014,14 @@ Self-Service provides REST based API’s. It tightly interacts with Provisioning
| SecurityResource | Login<br>Authorize<br>Logout | User’s authentication. |
| UserSettingsResource | Get settings<br>Save settings | User’s preferences. |
-Some class names may have endings like Aws or Azure(e.g. ComputationalResourceAws, ComputationalResourceAzure, etc...). It means that it's cloud specific class with a proper API
+Some class names may have endings like Aws or Azure(e.g. ComputationalResourceAws, ComputationalResourceAzure, etc...).
+It means that it's cloud specific class with a proper API
#### Provisioning Service
-The Provisioning Service is key, REST based service for management of cloud specific or Docker based environment resources like computational, exploratory,
+The Provisioning Service is key, REST based service for management of cloud specific or Docker based environment
+resources like computational, exploratory,
edge, etc.
| API class name | Supported actions | Description |
@@ -2005,18 +2034,25 @@ edge, etc.
| InfrastructureResource | Status | Docker action for obtaining status of DLab infrastructure instances. |
| LibExploratoryResource | Lib list<br>Install lib | Docker actions to install libraries on netobboks |
-Some class names may have endings like Aws or Azure(e.g. ComputationalResourceAws, ComputationalResourceAzure, etc...). It means that it's cloud specific class with a proper API
+Some class names may have endings like Aws or Azure(e.g. ComputationalResourceAws, ComputationalResourceAzure, etc...).
+It means that it's cloud specific class with a proper API
#### Security service
-Security service is REST based service for user authentication against LDAP/LDAP + AWS/Azure OAuth2 depending on module configuration and cloud provider.
+Security service is REST based service for user authentication against LDAP/LDAP + AWS/Azure OAuth2 depending on module
+configuration and cloud provider.
LDAP only provides with authentication end point that allows to verify authenticity of users against LDAP instance.
-If you use AWS cloud provider LDAP + AWS authentication could be useful as it allows to combine LDAP authentication and verification if user has any role in AWS account
+If you use AWS cloud provider LDAP + AWS authentication could be useful as it allows to combine LDAP authentication and
+verification if user has any role in AWS account
-DLab provides OAuth2(client credentials and authorization code flow) security authorization mechanism for Azure users. This kind of authentication is required when you are going to use Data Lake. If Data Lake is not enabled you have two options LDAP or OAuth2
+DLab provides OAuth2(client credentials and authorization code flow) security authorization mechanism for Azure users.
+This kind of authentication is required when you are going to use Data Lake. If Data Lake is not enabled you have two
+options LDAP or OAuth2
If OAuth2 is in use security-service validates user's permissions to configured permission scope(resource in Azure).
-If Data Lake is enabled default permission scope(can be configured manually after deploy DLab) is Data Lake Store account so only if user has any role in scope of Data Lake Store Account resource he/she will be allowed to log in
-If Data Lake is disabled but Azure OAuth2 is in use default permission scope will be Resource Group where DLab is created and only users who have any roles in the resource group will be allowed to log in.
+If Data Lake is enabled default permission scope(can be configured manually after deploy DLab) is Data Lake Store
+account so only if user has any role in scope of Data Lake Store Account resource he/she will be allowed to log in
+If Data Lake is disabled but Azure OAuth2 is in use default permission scope will be Resource Group where DLab is
+created and only users who have any roles in the resource group will be allowed to log in.
## Front-end <a name="Front_end"></a>
@@ -2038,7 +2074,9 @@ Sources are located in dlab/services/self-service/src/main/resources/webapp
## How to setup local development environment <a name="setup_local_environment"></a>
-The development environment setup description is written with assumption that user already has installed Java8 (JDK), Maven3 and set environment variables (JAVA\_HOME, M2\_HOME). The description will cover Mongo installation, Mongo user creation, filling initial data into Mongo, Node.js installation
+The development environment setup description is written with assumption that user already has installed Java8 (JDK),
+Maven3 and set environment variables (JAVA\_HOME, M2\_HOME). The description will cover Mongo installation, Mongo
+user creation, filling initial data into Mongo, Node.js installation
### Install Mongo database
@@ -2080,7 +2118,8 @@ mongoimport -u admin -p <password> -d <database_name> --jsonArray -c roles mongo
### Setting up environment options
- * Set option CLOUD_TYPE to aws/azure, DEV\_MODE to **true**, mongo database name and password in configuration file dlab/infrastructure-provisioning/src/ssn/templates/ssn.yml
+ * Set option CLOUD_TYPE to aws/azure, DEV\_MODE to **true**, mongo database name and password in configuration file
+ dlab/infrastructure-provisioning/src/ssn/templates/ssn.yml
```
<#assign CLOUD_TYPE="aws">
@@ -2155,7 +2194,8 @@ To create a server certificate, follow these steps:
* Sign the certificate.
- * Import the certificate into a truststore: a repository of certificates used for verifying the certificates. A truststore typically contains more than one certificate.
+ * Import the certificate into a truststore: a repository of certificates used for verifying the certificates. A
+ truststore typically contains more than one certificate.
Please find below set of commands to create certificate, depending on OS.
@@ -2185,14 +2225,20 @@ Where the ```<DRIVE_LETTER>``` must be the drive letter where you run the DLab.
## How to run locally <a name="run_locally"></a>
-There is a possibility to run Self-Service and Provisioning Service locally. All requests from Provisioning Service to Docker are mocked and instance creation status will be persisted to Mongo (only without real impact on Docker and AWS). Security Service can\`t be running on local machine because of local LDAP mocking complexity.
+There is a possibility to run Self-Service and Provisioning Service locally. All requests from Provisioning Service to
+Docker are mocked and instance creation status will be persisted to Mongo (only without real impact on Docker and AWS).
+Security Service can\`t be running on local machine because of local LDAP mocking complexity.
-Both services, Self-Service and Provisioning Service are dependent on dlab/provisioning-infrastructure/ssn/templates/ssn.yml configuration file. Both services have main functions as entry point, SelfServiceApplication for Self-Service and ProvisioningServiceApplication for Provisioning Service. Services could be started by running main methods of these classes. Both main functions require two arguments:
+Both services, Self-Service and Provisioning Service are dependent on dlab/provisioning-infrastructure/ssn/templates/ssn.yml
+configuration file. Both services have main functions as entry point, SelfServiceApplication for Self-Service and ProvisioningServiceApplication for Provisioning Service. Services could be started by running main methods of these classes. Both main functions require two arguments:
* Run mode (“server”)
- * Configuration file name (“self-service.yml” or “provisioning.yml” depending on the service). Both files are located in root service directory. These configuration files contain service settings and are ready to use.
+ * Configuration file name (“self-service.yml” or “provisioning.yml” depending on the service). Both files are located
+ in root service directory. These configuration files contain service settings and are ready to use.
-The services start up order does matter. Since Self-Service depends on Provisioning Service, the last should be started first and Self-Service afterwards. Services could be started from local IDEA (Eclipse or Intellij Idea) “Run” functionality of toolbox.
+The services start up order does matter. Since Self-Service depends on Provisioning Service, the last should be started
+first and Self-Service afterwards. Services could be started from local IDEA (Eclipse or Intellij Idea) “Run”
+functionality of toolbox.
Run application flow is following:
@@ -2223,9 +2269,13 @@ The following list shows common structure of scripts for deploying DLab
├───edge
├───general
├───jupyter
+ ├───jupyterlab
+ ├───project
├───rstudio
├───ssn
+ ├───superset
├───tensor
+ ├───tensor-rstudio
└───zeppelin
Each directory except *general* contains Python scripts, Docker files, templates, files for appropriate Docker image.
@@ -2316,7 +2366,8 @@ Using this Docker file, all required scripts and files will be copied to Docker
docker run -i -v /root/KEYNAME.pem:/root/keys/KEYNAME.pem –v /web_app:/root/web_app -e "conf_os_family=debian" -e "conf_cloud_provider=aws" -e "conf_resource=ssn" -e "aws_ssn_instance_size=t2.medium" -e "aws_region=us-west-2" -e "aws_vpc_id=vpc-111111" -e "aws_subnet_id=subnet-111111" -e "aws_security_groups_ids=sg-11111,sg-22222,sg-33333" -e "conf_key_name=KEYNAME" -e "conf_service_base_name=dlab_test" -e "aws_access_key=Access_Key_ID" -e "aws_secret_access_key=Secret_Access_Key" -e "co [...]
```
-- Docker executes *entrypoint.py* script with action *create*. *Entrypoint.py* will set environment variables, which were provided from Docker and execute *general/api/create.py* script:
+- Docker executes *entrypoint.py* script with action *create*. *Entrypoint.py* will set environment variables,
+ which were provided from Docker and execute *general/api/create.py* script:
```
elif args.action == 'create':
with hide('running'):
@@ -2327,7 +2378,8 @@ docker run -i -v /root/KEYNAME.pem:/root/keys/KEYNAME.pem –v /web_app:/root/we
try:
local('cd /root; fab run')
```
-- Function *run()* in file *ssn/fabfile.py* will be executed. It will run two scripts *general/scripts/aws/ssn\_prepare.py* and *general/scripts/aws/ssn\_configure.py*:
+- Function *run()* in file *ssn/fabfile.py* will be executed. It will run two scripts *general/scripts/aws/ssn\_prepare.py*
+ and *general/scripts/aws/ssn\_configure.py*:
```
try:
local("~/scripts/{}.py".format('ssn_prepare'))
@@ -2343,7 +2395,8 @@ docker run -i -v /root/KEYNAME.pem:/root/keys/KEYNAME.pem –v /web_app:/root/we
append_result("Failed configuring SSN node. Exception: " + str(err))
sys.exit(1)
```
-- The scripts *general/scripts/<cloud_provider>/ssn\_prepare.py* an *general/scripts/<cloud_provider>/ssn\_configure.py* will execute other Python scripts/functions for:
+- The scripts *general/scripts/<cloud_provider>/ssn\_prepare.py* an *general/scripts/<cloud_provider>/ssn\_configure.py*
+ will execute other Python scripts/functions for:
1. *ssn\_prepate.py:*
1. Creating configuration file (for AWS)
2. Creating Cloud resources.
@@ -2414,9 +2467,12 @@ def run():
sys.exit(1)
```
-This function describes process of creating Jupyter node. It is divided into two parts – prepare and configure. Prepare part is common for all notebook templates and responsible for creating of necessary cloud resources, such as EC2 instances, etc. Configure part describes how the appropriate services will be installed.
+This function describes process of creating Jupyter node. It is divided into two parts – prepare and configure. Prepare
+part is common for all notebook templates and responsible for creating of necessary cloud resources, such as EC2
+instances, etc. Configure part describes how the appropriate services will be installed.
-To configure Jupyter node, the script *jupyter\_configure.py* is executed. This script describes steps for configuring Jupyter node. In each step, the appropriate Python script is executed.
+To configure Jupyter node, the script *jupyter\_configure.py* is executed. This script describes steps for configuring
+Jupyter node. In each step, the appropriate Python script is executed.
For example:
@@ -2457,7 +2513,8 @@ if __name__ == "__main__":
ensure_jre_jdk(args.os_user)
```
-This script call functions for configuring Jupyter node. If this function is OS dependent, it will be placed in *infrastructure-provisioning/src/general/lib/\<OS\_family\>/debian/notebook\_lib.py*
+This script call functions for configuring Jupyter node. If this function is OS dependent, it will be placed in
+*infrastructure-provisioning/src/general/lib/\<OS\_family\>/debian/notebook\_lib.py*
All functions in template directory (e.g. *infrastructure-provisioning/src/my-tool/*) should be OS and cloud independent.
@@ -2473,8 +2530,10 @@ Other scripts, responsible for configuring Jupyter node are placed in *infrastru
- scripts directory – contains all required configuration scripts.
-- *infrastructure-provisioning/src/general/files/<cloud_provider>/my-tool_Dockerfile* – used for building template Docker image and describes which files, scripts, templates are required and will be copied to template Docker image.
-- *infrastructure-provisioning/src/general/files/<cloud_provider>/my-tool_descriptsion.json* – JSON file for DLab Web UI. In this file you can specify:
+- *infrastructure-provisioning/src/general/files/<cloud_provider>/my-tool_Dockerfile* – used for building template
+ Docker image and describes which files, scripts, templates are required and will be copied to template Docker image.
+- *infrastructure-provisioning/src/general/files/<cloud_provider>/my-tool_descriptsion.json* – JSON file for DLab Web
+ UI. In this file you can specify:
* exploratory\_environment\_shapes – list of EC2 shapes
* exploratory\_environment\_versions – description of template
@@ -2516,8 +2575,8 @@ Additionally, following directories could be created:
- files – directory for files used by newly added templates only;
-All Docker images are being built while creating SSN node. To add newly created template, add it to the list of images in the
-following script:
+All Docker images are being built while creating SSN node. To add newly created template, add it to the list of images
+in the following script:
Path: *infrastructure-provisioning/src/general/scripts/aws/ssn\_configure.py*
```
@@ -2545,8 +2604,10 @@ For example:
### Unified logging and group management
-There are a few popular LDAP distributions on the market like Active Directory, Open LDap. That’s why some differences in configuration appear.
-Also depending on customization, there might be differences in attributes configuration. For example the DN(distinguished name) may contain different attributes:
+There are a few popular LDAP distributions on the market like Active Directory, Open LDap. That’s why some differences
+in configuration appear.
+Also depending on customization, there might be differences in attributes configuration. For example the
+DN(distinguished name) may contain different attributes:
- **DN=CN=Name Surname,OU=groups,OU=EPAM,DC=Company,DC=Cloud**
- **DN=UID=UID#53,OU=groups,OU=Company,DC=Company,DC=Cloud**
@@ -2555,13 +2616,15 @@ Also depending on customization, there might be differences in attributes config
The relation between users and groups also varies from vendor to vendor.
-For example, in Open LDAP the group object may contain set (from 0 to many) attributes **"memberuid"** with values equal to user`s attribute **“uid”**.
+For example, in Open LDAP the group object may contain set (from 0 to many) attributes **"memberuid"** with values
+equal to user`s attribute **“uid”**.
However, in Active Directory the mappings are done based on other attributes.
On a group size there is attribute **"member"** (from 0 to many values) and its value is user`s **DN** (distinguished name).
-To fit the unified way of LDAP usage, we introduced configuration file with set of properties and customized scripts (python and JavaScript based).
+To fit the unified way of LDAP usage, we introduced configuration file with set of properties and customized scripts
+(python and JavaScript based).
On backend side, all valuable attributes are further collected and passed to these scripts.
To apply some customization it is required to update a few properties in **security.yml** and customize the scripts.
@@ -2574,7 +2637,8 @@ There are just a few properties based in which the customization could be done:
- **ldapSearchAttribute: uid**
Where the:
-- **ldapBindTemplate** is a user`s DN template which should be filed with custom value. Here the template could be changed: uid=%s,ou=People,dc=example,dc=com -> cn=%s,ou=People,dc=example,dc=com.
+- **ldapBindTemplate** is a user`s DN template which should be filed with custom value. Here the template could be
+ changed: uid=%s,ou=People,dc=example,dc=com -> cn=%s,ou=People,dc=example,dc=com.
- **ldapBindAttribute** - this is a major attribute, on which the DN is based on. Usually it is any of: uid or cn, or email.
- **ldapSearchAttribute** - another attribute, based on which users will be looked up in LDAP.
@@ -2593,7 +2657,8 @@ There are 3 scripts in security.yml:
### Script structure
-The scripts above were created to flexibly manage user`s security configuration. They all are part of **security.yml** configuration. All scripts have following structure:
+The scripts above were created to flexibly manage user`s security configuration. They all are part of **security.yml**
+configuration. All scripts have following structure:
- **name**
- **cache**
- **expirationTimeMsec**
@@ -2609,7 +2674,8 @@ The scripts above were created to flexibly manage user`s security configuration.
Major properties are:
- **attributes** - list of attributes that will be retrieved from LDAP (-name, -cn, -uid, -member, etc);
- **filter** - the filter, based on which the object will be retrieved from LDAP;
-- **searchResultProcessor** - optional. If only LDAP object attributes retrieving is required, this property should be empty. For example, “userLookup” script only retrieves list of "attributes". Otherwise, code customization (like user enrichment, user to groups matching, etc.) should be added into sub-properties below:
+- **searchResultProcessor** - optional. If only LDAP object attributes retrieving is required, this property should
+ be empty. For example, “userLookup” script only retrieves list of "attributes". Otherwise, code customization (like user enrichment, user to groups matching, etc.) should be added into sub-properties below:
- **language** - the script language - "python" or "JavaScript"
- **code** - the script code.
@@ -2637,14 +2703,16 @@ Script code:
base: ou=users,ou=alxn,dc=alexion,dc=cloud
filter: "(&(objectCategory=person)(objectClass=user)(mail=%mail%))"
-In the example above, the user login passed from GUI is a mail (**ldapSearchAttribute: mail**) and based on the filer (**filter: "(&(objectCategory=person)(objectClass=user)(mail=%mail%))")** so, the service would search user by its **“mail”**.
+In the example above, the user login passed from GUI is a mail (**ldapSearchAttribute: mail**) and based on the filer
+(**filter: "(&(objectCategory=person)(objectClass=user)(mail=%mail%))")** so, the service would search user by its **“mail”**.
If corresponding users are found - the script will return additional user`s attributes:
- cn
- gidNumber
- mail
- memberOf
-User`s authentication into LDAP would be done for DN with following template **ldapBindTemplate: 'cn=%s,ou=users,ou=alxn,dc=alexion,dc=cloud'**, where CN is attribute retrieved by **“userLookUp”** script.
+User`s authentication into LDAP would be done for DN with following template **ldapBindTemplate: 'cn=%s,ou=users,ou=alxn,
+dc=alexion,dc=cloud'**, where CN is attribute retrieved by **“userLookUp”** script.
## Azure OAuth2 Authentication <a name="Azure_OAuth2_Authentication"></a>
DLab supports OAuth2 authentication that is configured automatically in Security Service and Self Service after DLab deployment.
@@ -2667,18 +2735,22 @@ DLab supports client credentials(username + password) and authorization code flo
maxSessionDurabilityMilliseconds: 288000000
where:
-- **useLdap** - defines if LDAP authentication is enabled(true/false). If false Azure OAuth2 takes place with configuration properties below
+- **useLdap** - defines if LDAP authentication is enabled(true/false). If false Azure OAuth2 takes place with
+ configuration properties below
- **tenant** - tenant id of your company
- **authority** - Microsoft login endpoint
- **clientId** - id of the application that users log in through
- **redirectUrl** - redirect URL to DLab application after try to login to Azure using OAuth2
- **responseMode** - defines how Azure sends authorization code or error information to DLab during log in procedure
- **prompt** - defines kind of prompt during Oauth2 login
-- **silent** - defines if DLab tries to log in user without interaction(true/false), if false DLab tries to login user with configured prompt
+- **silent** - defines if DLab tries to log in user without interaction(true/false), if false DLab tries to login user
+ with configured prompt
- **loginPage** - start page of DLab application
-- **maxSessionDurabilityMilliseconds** - max user session durability. user will be asked to login after this period of time and when he/she creates ot starts notebook/cluster. This operation is needed to update refresh_token that is used by notebooks to access Data Lake Store
+- **maxSessionDurabilityMilliseconds** - max user session durability. user will be asked to login after this period
+ of time and when he/she creates ot starts notebook/cluster. This operation is needed to update refresh_token that is used by notebooks to access Data Lake Store
-To get more info about *responseMode*, *prompt* parameters please visit [Authorize access to web applications using OAuth 2.0 and Azure Active Directory](https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-protocols-oauth-code)
+To get more info about *responseMode*, *prompt* parameters please visit
+[Authorize access to web applications using OAuth 2.0 and Azure Active Directory](https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-protocols-oauth-code)
### Azure OAuth2 Security Service configuration
@@ -2694,11 +2766,16 @@ To get more info about *responseMode*, *prompt* parameters please visit [Authori
managementApiAuthFile: /dlab/keys/azure_authentication.json
where:
-- **useLdap** - defines if LDAP authentication is enabled(true/false). If false Azure OAuth2 takes place with configuration properties below
+- **useLdap** - defines if LDAP authentication is enabled(true/false). If false Azure OAuth2 takes place with
+ configuration properties below
- **tenant** - tenant id of your company
- **authority** - Microsoft login endpoint
- **clientId** - id of the application that users log in through
- **redirectUrl** - redirect URL to DLab application after try to login to Azure using OAuth2
-- **validatePermissionScope** - defines(true/false) if user's permissions should be validated to resource that is provided in permissionScope parameter. User will be logged in onlu in case he/she has any role in resource IAM described with permissionScope parameter
-- **permissionScope** - describes Azure resource where user should have any role to pass authentication. If user has no role in resource IAM he/she will not be logged in
-- **managementApiAuthFile** - authentication file that is used to query Microsoft Graph API to check user roles in resource described in permissionScope
+- **validatePermissionScope** - defines(true/false) if user's permissions should be validated to resource that is
+ provided in permissionScope parameter. User will be logged in onlu in case he/she has any role in resource IAM
+ described with permissionScope parameter
+- **permissionScope** - describes Azure resource where user should have any role to pass authentication. If user has no
+ role in resource IAM he/she will not be logged in
+- **managementApiAuthFile** - authentication file that is used to query Microsoft Graph API to check user roles in
+ resource described in permissionScope
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@dlab.apache.org
For additional commands, e-mail: commits-help@dlab.apache.org