You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dlab.apache.org by lf...@apache.org on 2020/07/29 07:24:22 UTC
[incubator-dlab] branch DLAB-1966 updated (b1e2372 -> 6e72bbf)
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a change to branch DLAB-1966
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git.
from b1e2372 [DLAB-1966]: fixed R package installation for dataengine-service, yum installation, some other pip, R and apt improvements
add c4f72c4 [DLAB-1967]: Fixed additional packages are not shown (#832)
add 765eddb [DLAB-1968] - Jenkins installing suspended
add 4d58043 Merge pull request #833 from apache/DLAB-1968
add 159eb9d [DLAB-1870] Intigration with sonar
add 76a8249 Updated copyright
add f529b0f [DLAB-1870] Intigration with sonar
add dce5820 [DLAB-1870] Intigration with sonar
add 0309886 [DLAB-1870] Intigration with sonar
add 1b46e4c [DLAB-1870] Intigration with sonar
add fab7998 [DLAB-1870] Intigration with sonar
add 07b88e0 [DLAB-1870] Intigration with sonar
add 4e6dc6e [DLAB-1870] Intigration with sonar
add e9ea179 [DLAB-1870] Intigration with sonar
add c820746 Updated copyright
add cc4be26 Updated copyright
add 580a5e8 Updated copyright
add ca8d090 Updated pom
add 223f348 Merge remote-tracking branch 'origin/sonar' into sonar
add 3c743a7 Added sonar.py
add 727deb6 Added sonar.py
add 5335dc7 Updated sonar.py
add 82c35ed Updated sonar.py
add 6e404ae Updated sonar.py
add 4f8b5dd Updated sonar.py
add 95a6626 aaa
add 55931ff Updated sonar.py
add b62dd80 Updated sonar.py
add 78025b3 Updated sonar.py
add 5d8b9a5 Updated sonar.py
add db12a91 Updated sonar.py
add 840a045 Updated sonar.py
add 572981c Merge branch 'sonar' into develop
add 24f7781 [DLAB-1943]: Change style on grids (#834)
add c586248 Created test for AccessKeyServiceImpl
new c1d23dc [DLAB-1966]: fixed R package installation for dataengine-service, yum installation, some other pip, R and apt improvements
new 6e72bbf Merge branch 'DLAB-1966' of https://github.com/apache/incubator-dlab into DLAB-1966
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../scripts/jenkins/sonar.py | 33 +-
.../src/ssn/scripts/configure_ssn_node.py | 8 +-
.../aws/computational_resources/main/main.tf | 27 +-
.../aws/computational_resources/main/variables.tf | 21 ++
.../terraform/aws/endpoint/main/bucket.tf | 54 ++--
.../ssn-helm-charts/main/step-ca-chart/.helmignore | 21 ++
.../ssn-helm-charts/main/step-ca-chart/Chart.yaml | 23 +-
.../main/step-ca-chart/templates/_helpers.tpl | 21 ++
.../main/step-ca-chart/templates/bootstrap.yaml | 47 ++-
.../main/step-ca-chart/templates/ca.yaml | 203 ++++++------
.../main/step-ca-chart/templates/configmaps.yaml | 117 ++++---
.../main/step-ca-chart/templates/ingress.yaml | 63 ++--
.../main/step-ca-chart/templates/rbac.yaml | 23 +-
.../main/step-ca-chart/templates/secrets.yaml | 21 ++
.../main/step-ca-chart/templates/service.yaml | 35 +-
.../templates/tests/test-connection.yaml | 27 +-
.../ssn-helm-charts/main/step-ca-chart/values.yaml | 25 +-
.../terraform/bin/deploy/billing_app_aws.yml | 21 ++
.../terraform/bin/deploy/billing_azure.yml | 21 ++
.../terraform/bin/deploy/billing_gcp.yml | 21 ++
.../terraform/bin/deploy/endpoint_fab.py | 29 +-
infrastructure-provisioning/terraform/bin/dlab.py | 20 +-
.../terraform/bin/terraform-cli.py | 31 +-
.../terraform/gcp/endpoint/provisioning.py | 25 +-
.../terraform/gcp/main/main.tf | 29 +-
.../terraform/gcp/main/variables.tf | 21 ++
.../terraform/gcp/modules/common/iam.tf | 25 +-
.../terraform/gcp/modules/common/network.tf | 29 +-
.../terraform/gcp/modules/common/variables.tf | 21 ++
.../terraform/gcp/modules/data_engine/instance.tf | 28 +-
.../terraform/gcp/modules/data_engine/variables.tf | 21 ++
.../terraform/gcp/modules/dataproc/instance.tf | 83 +++--
.../terraform/gcp/modules/dataproc/variables.tf | 21 ++
.../terraform/gcp/modules/notebook/instance.tf | 23 +-
.../terraform/gcp/modules/notebook/variables.tf | 21 ++
.../helm_charts/files/get_configmap_values.sh | 21 ++
.../modules/helm_charts/step-ca-chart/.helmignore | 21 ++
.../modules/helm_charts/step-ca-chart/Chart.yaml | 23 +-
.../step-ca-chart/templates/_helpers.tpl | 21 ++
.../step-ca-chart/templates/bootstrap.yaml | 47 ++-
.../helm_charts/step-ca-chart/templates/ca.yaml | 203 ++++++------
.../step-ca-chart/templates/configmaps.yaml | 117 ++++---
.../step-ca-chart/templates/ingress.yaml | 62 ++--
.../helm_charts/step-ca-chart/templates/rbac.yaml | 23 +-
.../step-ca-chart/templates/secrets.yaml | 21 ++
.../step-ca-chart/templates/service.yaml | 35 +-
.../templates/tests/test-connection.yaml | 27 +-
.../modules/helm_charts/step-ca-chart/values.yaml | 25 +-
.../dlab/login/resources/css/login.css | 19 ++
.../keycloak-theme/dlab/login/theme.properties | 33 +-
integration-tests-cucumber/pom.xml | 19 ++
.../main/java/org/apache/dlab/dto/EndpointDTO.java | 19 ++
.../java/org/apache/dlab/mongo/MongoDBHelper.java | 19 ++
.../java/org/apache/dlab/util/JacksonMapper.java | 19 ++
.../java/org/apache/dlab/util/PropertyHelper.java | 19 ++
.../src/test/java/dlab/Constants.java | 19 ++
.../src/test/java/dlab/RunCucumberTest.java | 19 ++
.../src/test/java/dlab/endpoint/EndpointSteps.java | 19 ++
.../src/test/java/dlab/login/LoginSteps.java | 19 ++
.../src/test/resources/config.properties | 18 ++
.../src/test/resources/dlab/endpoint.feature | 18 ++
.../src/test/resources/dlab/login.feature | 18 ++
.../test_templates/rstudio/template_preparation.r | 17 +
.../rstudio/template_visualization.r | 17 +
pom.xml | 40 ++-
.../main/java/com/epam/dlab/BillingService.java | 1 +
.../billing-aws/src/main/resources/application.yml | 21 ++
.../src/main/resources/application.yml | 21 ++
services/billing-gcp/billing.yml | 21 ++
.../billing/gcp/dao/impl/BigQueryBillingDAO.java | 11 +-
.../dlab/billing/gcp/model/BillingHistory.java | 19 ++
.../gcp/repository/BillingHistoryRepository.java | 19 ++
.../billing-gcp/src/main/resources/application.yml | 21 ++
.../dlab/auth/conf/AzureLoginConfiguration.java | 39 ---
.../epam/dlab/auth/conf/GcpLoginConfiguration.java | 33 --
.../epam/dlab/dto/base/project/ProjectResult.java | 19 ++
.../epam/dlab/dto/project/ProjectActionDTO.java | 19 ++
.../epam/dlab/dto/project/ProjectCreateDTO.java | 19 ++
.../main/java/com/epam/dlab/model/StringList.java | 35 +-
.../main/java/com/epam/dlab/util/FileUtils.java | 2 +-
services/provisioning-service/pom.xml | 14 +
.../DropwizardBearerTokenFilterImpl.java | 19 ++
.../core/response/folderlistener/WatchItem.java | 20 +-
.../response/handlers/ProjectCallbackHandler.java | 23 +-
.../handlers/dao/FileSystemCallbackHandlerDao.java | 6 +-
.../dlab/backendapi/resources/ProjectResource.java | 19 ++
.../dlab/backendapi/service/ProjectService.java | 19 ++
.../service/impl/CheckInactivityServiceImpl.java | 2 +-
.../dlab/backendapi/service/impl/KeyService.java | 2 +-
.../service/impl/ProjectServiceImpl.java | 47 ++-
.../service/impl/aws/BucketServiceAwsImpl.java | 113 ++++---
.../service/impl/azure/BucketServiceAzureImpl.java | 8 +-
.../service/impl/gcp/BucketServiceGcpImpl.java | 10 +-
.../dlab/process/builder/ProcessInfoBuilder.java | 17 +-
services/self-service/entrypoint.sh | 21 ++
services/self-service/pom.xml | 11 +
.../epam/dlab/backendapi/annotation/Project.java | 19 ++
.../backendapi/auth/KeycloakAuthenticator.java | 23 +-
.../filters/DropwizardBearerTokenFilterImpl.java | 19 ++
.../dlab/backendapi/conf/CloudConfiguration.java | 21 +-
.../backendapi/conf/KeycloakConfiguration.java | 19 ++
.../com/epam/dlab/backendapi/dao/ProjectDAO.java | 19 ++
.../epam/dlab/backendapi/dao/ProjectDAOImpl.java | 19 ++
.../epam/dlab/backendapi/dao/UserRoleDaoImpl.java | 3 +
.../dlab/backendapi/domain/CreateProjectDTO.java | 19 ++
.../epam/dlab/backendapi/domain/ProjectDTO.java | 19 ++
.../dlab/backendapi/domain/ProjectEndpointDTO.java | 19 ++
.../backendapi/domain/UpdateProjectBudgetDTO.java | 19 ++
.../dlab/backendapi/domain/UpdateProjectDTO.java | 19 ++
.../dropwizard/bundles/DlabKeycloakBundle.java | 20 +-
.../dropwizard/listeners/MongoStartupListener.java | 23 +-
.../listeners/RestoreHandlerStartupListener.java | 2 +-
.../backendapi/healthcheck/MongoHealthCheck.java | 2 +-
.../interceptor/BudgetLimitInterceptor.java | 11 -
.../backendapi/resources/KeycloakResource.java | 24 +-
.../dlab/backendapi/resources/ProjectResource.java | 33 +-
.../resources/callback/ProjectCallback.java | 19 ++
.../dlab/backendapi/resources/dto/KeysDTO.java | 25 +-
.../resources/dto/ProjectActionFormDTO.java | 19 ++
.../schedulers/billing/BillingScheduler.java | 2 +-
.../endpoint/CheckEndpointStatusScheduler.java | 2 +-
.../dlab/backendapi/service/EndpointService.java | 19 ++
.../dlab/backendapi/service/GuacamoleService.java | 19 ++
.../dlab/backendapi/service/KeycloakService.java | 22 ++
.../backendapi/service/KeycloakServiceImpl.java | 23 +-
.../dlab/backendapi/service/ProjectService.java | 41 ++-
.../dlab/backendapi/service/SecurityService.java | 19 ++
.../backendapi/service/SecurityServiceImpl.java | 19 ++
.../epam/dlab/backendapi/service/TagService.java | 19 ++
.../dlab/backendapi/service/TagServiceImpl.java | 21 +-
.../service/impl/BillingServiceImpl.java | 38 +--
.../backendapi/service/impl/BucketServiceImpl.java | 10 +-
.../service/impl/ComputationalServiceImpl.java | 3 +-
.../service/impl/EndpointServiceImpl.java | 103 +++---
.../service/impl/ExploratoryServiceImpl.java | 4 +-
.../service/impl/GuacamoleServiceImpl.java | 25 +-
.../service/impl/ProjectServiceImpl.java | 134 ++++----
.../servlet/guacamole/GuacamoleServlet.java | 37 ++-
.../epam/dlab/backendapi/util/KeycloakUtil.java | 43 ++-
.../src/main/resources/webapp/browserslist | 23 +-
.../management-grid/management-grid.component.html | 4 +-
.../management-grid/management-grid.component.scss | 62 +++-
.../management/management.component.html | 2 +-
.../project-list/project-list.component.scss | 11 +-
.../administration/project/project.component.html | 2 +-
.../webapp/src/app/core/services/audit.service.ts | 19 ++
.../app/core/services/bucket-browser.service.ts | 19 ++
.../audit/audit-grid/audit-grid.component.html | 66 ++--
.../audit/audit-grid/audit-grid.component.scss | 77 +++--
.../audit/audit-grid/audit-grid.component.ts | 5 +-
.../audit-toolbar/audit-toolbar.component.html | 4 +-
.../audit-toolbar/audit-toolbar.component.scss | 76 +----
.../audit/audit-toolbar/audit-toolbar.component.ts | 34 +-
.../src/app/reports/audit/audit.component.ts | 8 +-
.../src/app/reports/audit/filter-audit.model.ts | 19 ++
.../reporting-grid/reporting-grid.component.html | 62 +++-
.../reporting-grid/reporting-grid.component.scss | 91 ++++--
.../reporting-grid/reporting-grid.component.ts | 4 +-
.../reporting/toolbar/toolbar.component.html | 2 +-
.../reporting/toolbar/toolbar.component.scss | 76 -----
.../reports/reporting/toolbar/toolbar.component.ts | 33 +-
.../bucket-browser/bucket-browser.component.html | 5 +-
.../bucket-browser/bucket-data.service.ts | 37 ++-
.../buckets-tree/bucket-tree.component.html | 21 +-
.../buckets-tree/bucket-tree.component.scss | 19 +-
.../buckets-tree/bucket-tree.component.ts | 19 ++
.../folder-tree/folder-tree.component.html | 21 +-
.../folder-tree/folder-tree.component.scss | 18 ++
.../folder-tree/folder-tree.component.ts | 19 ++
.../computational-resources-list.component.scss | 46 +--
.../cost-details-dialog.component.html | 3 +-
.../cost-details-dialog.component.scss | 3 +-
.../install-libraries.component.html | 47 ++-
.../install-libraries.component.scss | 113 ++++---
.../install-libraries.component.ts | 8 +-
.../resources-grid/resources-grid.component.html | 14 +-
.../resources-grid/resources-grid.component.scss | 358 +++++++++++----------
.../shared/form-controls/dropdowns.component.scss | 7 +-
.../multi-level-select-dropdown.component.scss | 5 +-
.../src/app/webterminal/webterminal.component.html | 2 +-
.../webapp/src/assets/styles/_dialogs.scss | 5 +
.../webapp/src/assets/styles/_general.scss | 9 +-
.../resources/webapp/src/assets/styles/_theme.scss | 153 ++++++++-
.../webapp/src/assets/styles/_variables.scss | 2 +
.../backendapi/resources/KeycloakResourceTest.java | 25 +-
.../backendapi/resources/ProjectResourceTest.java | 173 +++++-----
.../service/KeycloakServiceImplTest.java | 135 ++++----
.../service/impl/AccessKeyServiceImplTest.java | 57 ++++
188 files changed, 4365 insertions(+), 1583 deletions(-)
copy services/self-service/src/main/resources/webapp/.editorconfig => infrastructure-provisioning/scripts/jenkins/sonar.py (59%)
delete mode 100644 services/common/src/main/java/com/epam/dlab/auth/conf/AzureLoginConfiguration.java
delete mode 100644 services/common/src/main/java/com/epam/dlab/auth/conf/GcpLoginConfiguration.java
create mode 100644 services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/AccessKeyServiceImplTest.java
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@dlab.apache.org
For additional commands, e-mail: commits-help@dlab.apache.org
[incubator-dlab] 01/02: [DLAB-1966]: fixed R package installation
for dataengine-service, yum installation, some other pip,
R and apt improvements
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DLAB-1966
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git
commit c1d23dc763b0ca777fb6fd1add9770e67c497b60
Author: leonidfrolov <fr...@gmail.com>
AuthorDate: Tue Jul 28 18:14:27 2020 +0300
[DLAB-1966]: fixed R package installation for dataengine-service, yum installation, some other pip, R and apt improvements
---
.../src/general/lib/os/debian/notebook_lib.py | 6 ++-
.../src/general/lib/os/fab.py | 14 +++++--
.../src/general/lib/os/redhat/notebook_lib.py | 47 +++++++++++++++++++---
.../general/scripts/os/install_additional_libs.py | 3 ++
4 files changed, 58 insertions(+), 12 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py b/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
index d3bcefa..7a9a12b 100644
--- a/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
+++ b/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
@@ -380,7 +380,7 @@ def install_os_pkg(requisites):
sudo('DEBIAN_FRONTEND=noninteractive apt-get -y install --allow-downgrades {0} 2>&1 | tee /tmp/tee.tmp; if ! grep -w -E "({1})" /tmp/tee.tmp > '
'/tmp/os_install_{0}.log; then echo "" > /tmp/os_install_{0}.log;fi'.format(os_pkg, error_parser))
err = sudo('cat /tmp/os_install_{}.log'.format(os_pkg)).replace('"', "'")
- sudo('cat /tmp/tee.tmp | if ! grep -w -E -A 20 "({1})" /tmp/tee.tmp > '
+ sudo('cat /tmp/tee.tmp | if ! grep -w -E -A 30 "({1})" /tmp/tee.tmp > '
'/tmp/os_install_{0}.log; then echo "" > /tmp/os_install_{0}.log;fi'.format(os_pkg, new_pkgs_parser))
dep = sudo('cat /tmp/os_install_{}.log'.format(os_pkg))
if dep == '':
@@ -390,7 +390,9 @@ def install_os_pkg(requisites):
.replace('\n', '').replace(' ', ' ').replace(' {} '.format(os_pkg.split("=")[0]),
' ').strip().split(' ')
for n, i in enumerate(dep):
- dep[n] = sudo('apt show {} 2>&1 | grep Version:'.format(i)).replace('Version: ', '{} v.'.format(i))
+ sudo('apt show {0} 2>&1 | if ! grep Version: > '
+ '/tmp/os_install_{0}.log; then echo "" > /tmp/os_install_{0}.log;fi'.format(i))
+ dep[n] =sudo('cat /tmp/os_install_{}.log'.format(i)).replace('Version: ', '{} v.'.format(i))
dep = [i for i in dep if i]
versions = []
sudo('apt list --installed | if ! grep {0}/ > /tmp/os_install_{1}.list; then echo "" > /tmp/os_install_{1}.list;fi'.format(os_pkg.split("=")[0], os_pkg))
diff --git a/infrastructure-provisioning/src/general/lib/os/fab.py b/infrastructure-provisioning/src/general/lib/os/fab.py
index a277cf1..6b9d47c 100644
--- a/infrastructure-provisioning/src/general/lib/os/fab.py
+++ b/infrastructure-provisioning/src/general/lib/os/fab.py
@@ -54,7 +54,7 @@ def dataengine_dir_prepare(cluster_dir):
def install_pip_pkg(requisites, pip_version, lib_group):
status = list()
- error_parser = "Could not|No matching|ImportError:|failed|EnvironmentError:|requires"
+ error_parser = "Could not|No matching|ImportError:|failed|EnvironmentError:|requires|FileNotFoundError:|RuntimeError:|error:"
try:
if pip_version == 'pip3' and not exists('/bin/pip3'):
sudo('ln -s /bin/pip3.5 /bin/pip3')
@@ -111,7 +111,9 @@ def install_pip_pkg(requisites, pip_version, lib_group):
if i == pip_pkg.split("==")[0]:
dep[n] = ''
else:
- dep[n] = sudo('{} show {} 2>&1 | grep Version:'.format(pip_version, i)).replace('Version: ', '{} v.'.format(i))
+ sudo('{0} show {1} 2>&1 | if ! grep Version: /tmp/tee.tmp > '
+ '/tmp/{0}_install_{1}.log; then echo "" > /tmp/{0}_install_{1}.log;fi'.format(pip_version, i))
+ dep[n] = sudo('cat /tmp/{0}_install_{1}.log'.format(pip_version, i)).replace('Version: ', '{} v.'.format(i))
dep = [i for i in dep if i]
status.append({"group": lib_group, "name": pip_pkg.split("==")[0], "version": version, "status": status_msg,
"error_message": err, "available_versions": versions, "add_pkgs": dep})
@@ -439,7 +441,10 @@ def install_r_pkg(requisites):
else:
dep = dep.split(' ')
for n, i in enumerate(dep):
- dep[n] = '{} v.{}'.format(dep[n], dep_ver[n])
+ if i == name:
+ dep[n] = ''
+ else:
+ dep[n] = '{} v.{}'.format(dep[n], dep_ver[n])
dep = [i for i in dep if i]
err = sudo('cat /tmp/install_{0}.log'.format(name)).replace('"', "'")
sudo('R -e \'installed.packages()[,c(3:4)]\' | if ! grep -w {0} > /tmp/install_{0}.list; then echo "" > /tmp/install_{0}.list;fi'.format(name))
@@ -454,7 +459,8 @@ def install_r_pkg(requisites):
sudo('R -e \'install.packages("versions", repos="https://cloud.r-project.org", dep=TRUE)\'')
versions = sudo('R -e \'library(versions); available.versions("' + name + '")\' 2>&1 | grep -A 50 '
'\'date available\' | awk \'{print $2}\'').replace('\r\n', ' ')[5:].split(' ')
- status_msg = 'invalid_version'
+ if versions != '':
+ status_msg = 'invalid_version'
else:
versions = []
status.append({"group": "r_pkg", "name": name, "version": version, "status": status_msg, "error_message": err, "available_versions": versions, "add_pkgs": dep})
diff --git a/infrastructure-provisioning/src/general/lib/os/redhat/notebook_lib.py b/infrastructure-provisioning/src/general/lib/os/redhat/notebook_lib.py
index ce075c0..cf7d8d1 100644
--- a/infrastructure-provisioning/src/general/lib/os/redhat/notebook_lib.py
+++ b/infrastructure-provisioning/src/general/lib/os/redhat/notebook_lib.py
@@ -346,19 +346,54 @@ def install_nodejs(os_user):
def install_os_pkg(requisites):
status = list()
error_parser = "Could not|No matching|Error:|failed|Requires:|Errno"
+ new_pkgs_parser = "Dependency Installed:"
try:
print("Updating repositories and installing requested tools: {}".format(requisites))
manage_pkg('update-minimal --security -y --skip-broken', 'remote', '')
sudo('export LC_ALL=C')
for os_pkg in requisites:
- manage_pkg('-y install', 'remote', '{0} --nogpgcheck 2>&1 | if ! grep -w -E "({1})" > /tmp/os_install_{0}.log; then echo "" > /tmp/os_install_{0}.log;fi'.format(os_pkg, error_parser))
+ name, vers = os_pkg
+ if vers != '' and vers !='N/A':
+ version = vers
+ os_pkg = "{}-{}".format(name, vers)
+ else:
+ version = 'N/A'
+ os_pkg = name
+ manage_pkg('-y install', 'remote', '{0} --nogpgcheck 2>&1 | tee /tmp/tee.tmp; if ! grep -w -E "({1})" /tmp/tee.tmp > /tmp/os_install_{0}.log; then echo "" > /tmp/os_install_{0}.log;fi'.format(os_pkg, error_parser))
+ install_output = sudo('cat /tmp/tee.tmp')
err = sudo('cat /tmp/os_install_{}.log'.format(os_pkg)).replace('"', "'")
- try:
- res = sudo('python -c "import os,sys,yum; yb = yum.YumBase(); pl = yb.doPackageLists(); print [pkg.vr for pkg in pl.installed if pkg.name == \'{0}\'][0]"'.format(os_pkg))
+ sudo('cat /tmp/tee.tmp | if ! grep -w -E -A 30 "({1})" /tmp/tee.tmp > '
+ '/tmp/os_install_{0}.log; then echo "" > /tmp/os_install_{0}.log;fi'.format(os_pkg, new_pkgs_parser))
+ dep = sudo('cat /tmp/os_install_{}.log'.format(os_pkg))
+ if dep == '':
+ dep = []
+ else:
+ dep = dep[len(new_pkgs_parser): dep.find("Complete!") - 1].replace(' ', '').strip().split('\r\n')
+ for n, i in enumerate(dep):
+ i = i.split('.')[0]
+ sudo('yum info {0} 2>&1 | if ! grep Version > /tmp/os_install_{0}.log; then echo "" > /tmp/os_install_{0}.log;fi'.format(i))
+ dep[n] =sudo('cat /tmp/os_install_{}.log'.format(i)).replace('Version : ', '{} v.'.format(i))
+ dep = [i for i in dep if i]
+ versions = []
+ res = sudo(
+ 'python -c "import os,sys,yum; yb = yum.YumBase(); pl = yb.doPackageLists(); print [pkg.vr for pkg in pl.installed if pkg.name == \'{0}\'][0]"'.format(
+ name))
+ if err:
+ status_msg = 'installation_error'
+ elif res:
version = res.split('\r\n')[1].replace("'", "\"")
- status.append({"group": "os_pkg", "name": os_pkg, "version": version, "status": "installed"})
- except:
- status.append({"group": "os_pkg", "name": os_pkg, "status": "failed", "error_message": err})
+ status_msg = "installed"
+ if 'No package {} available'.format(os_pkg) in install_output:
+ versions = sudo ('yum --showduplicates list ' + name + ' | expand | grep -A 10 "Available Packages" | grep -v "Available Packages"| awk \'{print $2}\'').replace('\r\n', '').split(' ')
+ if versions != '':
+ status_msg = 'invalid_version'
+ for n, i in enumerate(versions):
+ if ':' in i:
+ versions[n] = i.split(':')[1].split('-')[0]
+ else:
+ versions[n] = i.split('-')[0]
+ status.append({"group": "os_pkg", "name": name, "version": version, "status": status_msg,
+ "error_message": err, "add_pkgs": dep, "available_versions": versions})
return status
except Exception as err:
for os_pkg in requisites:
diff --git a/infrastructure-provisioning/src/general/scripts/os/install_additional_libs.py b/infrastructure-provisioning/src/general/scripts/os/install_additional_libs.py
index ca81a97..075175c 100644
--- a/infrastructure-provisioning/src/general/scripts/os/install_additional_libs.py
+++ b/infrastructure-provisioning/src/general/scripts/os/install_additional_libs.py
@@ -115,6 +115,9 @@ if __name__ == "__main__":
or os.environ['application'] in ('rstudio', 'tensor-rstudio'):
try:
print('Installing R packages: {}'.format(pkgs['libraries']['r_pkg']))
+ if os.environ['conf_resource'] in ('dataengine-service'):
+ manage_pkg('-y install', 'remote', 'libcurl libcurl-devel')
+ sudo('R -e "install.packages(\'devtools\', repos = \'https://cloud.r-project.org\')"')
status = install_r_pkg(pkgs['libraries']['r_pkg'])
general_status = general_status + status
except KeyError:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@dlab.apache.org
For additional commands, e-mail: commits-help@dlab.apache.org
[incubator-dlab] 02/02: Merge branch 'DLAB-1966' of
https://github.com/apache/incubator-dlab into DLAB-1966
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DLAB-1966
in repository https://gitbox.apache.org/repos/asf/incubator-dlab.git
commit 6e72bbf3f772560f96e09c2b77013421e1da34db
Merge: c1d23dc b1e2372
Author: leonidfrolov <fr...@gmail.com>
AuthorDate: Wed Jul 29 10:23:56 2020 +0300
Merge branch 'DLAB-1966' of https://github.com/apache/incubator-dlab into DLAB-1966
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@dlab.apache.org
For additional commands, e-mail: commits-help@dlab.apache.org