You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datalab.apache.org by lf...@apache.org on 2022/02/14 17:23:04 UTC
[incubator-datalab] branch DATALAB-2698 updated (42e7adc -> a5d089e)
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a change to branch DATALAB-2698
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git.
from 42e7adc [DATALAB-2698]: fixed nvidia drivers for aws tensor
new 3e06945 [DATALAB-2698]: fixed install_venv_pip_pkg function usage
new bf6d18f [DATALAB-2698]: added venv libs install
new 5709754 [DATALAB-2698]: fixed function name
new a5d089e [DATALAB-2698]: added python3-opencv to packages installed by default
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../src/general/lib/os/debian/common_lib.py | 7 +++----
.../src/general/lib/os/debian/notebook_lib.py | 15 +++++++++++----
.../src/tensor/scripts/configure_tensor_node.py | 7 +++++--
3 files changed, 19 insertions(+), 10 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 04/04: [DATALAB-2698]: added python3-opencv to packages installed by default
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-2698
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit a5d089e8a6e62a0c52c4b2168de46aec35902348
Author: leonidfrolov <fr...@gmail.com>
AuthorDate: Mon Feb 14 18:46:52 2022 +0200
[DATALAB-2698]: added python3-opencv to packages installed by default
---
.../src/general/lib/os/debian/common_lib.py | 7 +++----
.../src/general/lib/os/debian/notebook_lib.py | 1 +
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/os/debian/common_lib.py b/infrastructure-provisioning/src/general/lib/os/debian/common_lib.py
index 648701f..27ce444 100644
--- a/infrastructure-provisioning/src/general/lib/os/debian/common_lib.py
+++ b/infrastructure-provisioning/src/general/lib/os/debian/common_lib.py
@@ -223,10 +223,9 @@ def manage_pkg(command, environment, requisites):
sys.exit(1)
-def ensure_pkg(os_user, requisites='linux-headers-$(uname -r) python3-pip python3-dev python3-virtualenv '
- 'groff gcc vim less git wget '
- 'libssl-dev unattended-upgrades nmap '
- 'libffi-dev unzip libxml2-dev haveged'):
+def ensure_pkg(os_user, requisites='linux-headers-$(uname -r) python3-pip python3-opencv python3-dev '
+ 'python3-virtualenv groff gcc vim less git wget libssl-dev unattended-upgrades '
+ 'nmap libffi-dev unzip libxml2-dev haveged'):
try:
if not exists(datalab.fab.conn,'/home/{}/.ensure_dir/pkg_upgraded'.format(os_user)):
count = 0
diff --git a/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py b/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
index e015d5b..3052753 100644
--- a/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
+++ b/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
@@ -400,6 +400,7 @@ def ensure_venv_libs(os_user, libs):
datalab.fab.conn.install_venv_pip_pkg(libs)
datalab.fab.conn.sudo('touch /home/' + os_user + '/.ensure_dir/venv_libs_ensured')
+
def ensure_pytorch(os_user, gpu=True):
if not exists(datalab.fab.conn, '/home/' + os_user + '/.ensure_dir/pytorch_ensured'):
if gpu:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 02/04: [DATALAB-2698]: added venv libs install
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-2698
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit bf6d18f0358aa7a490ab87becb28f325a50cedeb
Author: leonidfrolov <fr...@gmail.com>
AuthorDate: Mon Feb 14 18:41:50 2022 +0200
[DATALAB-2698]: added venv libs install
---
.../src/general/lib/os/debian/notebook_lib.py | 5 +++++
.../src/tensor/scripts/configure_tensor_node.py | 7 +++++--
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py b/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
index 7c094e1..4e9bdbd 100644
--- a/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
+++ b/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
@@ -395,6 +395,11 @@ def install_tensor(os_user, cuda_version, cuda_file_name,
sys.exit(1)
+def ensure_pytorch(os_user, libs):
+ if not exists(datalab.fab.conn, '/home/' + os_user + '/.ensure_dir/venv_libs_ensured'):
+ datalab.fab.conn.install_venv_pip_pkg(libs)
+ datalab.fab.conn.sudo('touch /home/' + os_user + '/.ensure_dir/venv_libs_ensured')
+
def ensure_pytorch(os_user, gpu=True):
if not exists(datalab.fab.conn, '/home/' + os_user + '/.ensure_dir/pytorch_ensured'):
if gpu:
diff --git a/infrastructure-provisioning/src/tensor/scripts/configure_tensor_node.py b/infrastructure-provisioning/src/tensor/scripts/configure_tensor_node.py
index 3566518..3686231 100644
--- a/infrastructure-provisioning/src/tensor/scripts/configure_tensor_node.py
+++ b/infrastructure-provisioning/src/tensor/scripts/configure_tensor_node.py
@@ -69,7 +69,7 @@ cuda_version = os.environ['notebook_cuda_version']
cuda_file_name = os.environ['notebook_cuda_file_name']
cudnn_version = os.environ['notebook_cudnn_version']
cudnn_file_name = os.environ['notebook_cudnn_file_name']
-
+venv_libs = 'numpy scipy matplotlib pandas scikit-learn opencv-python'
##############
# Run script #
@@ -147,7 +147,10 @@ if __name__ == "__main__":
ensure_pytorch(args.os_user)
print("Install Matplotlib")
ensure_matplot(args.os_user)
-
+
+ print("Install python venv required libs")
+ ensure_venv_libs(args.os_user, venv_libs)
+
#POST INSTALLATION PROCESS
print("Updating pyOpenSSL library")
update_pyopenssl_lib(args.os_user)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 01/04: [DATALAB-2698]: fixed install_venv_pip_pkg function usage
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-2698
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 3e06945cd320fead4c765574be32cc2b6e47c8a4
Author: leonidfrolov <fr...@gmail.com>
AuthorDate: Mon Feb 14 18:29:52 2022 +0200
[DATALAB-2698]: fixed install_venv_pip_pkg function usage
---
.../src/general/lib/os/debian/notebook_lib.py | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py b/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
index 768e6aa..7c094e1 100644
--- a/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
+++ b/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
@@ -398,11 +398,12 @@ def install_tensor(os_user, cuda_version, cuda_file_name,
def ensure_pytorch(os_user, gpu=True):
if not exists(datalab.fab.conn, '/home/' + os_user + '/.ensure_dir/pytorch_ensured'):
if gpu:
- install_venv_pip_pkg('torch==1.10.2+cu113 torchvision==0.11.3+cu113 torchaudio==0.10.2+cu113'
- ' -f https://download.pytorch.org/whl/cu113/torch_stable.html')
+ datalab.fab.conn.install_venv_pip_pkg('torch==1.10.2+cu113 torchvision==0.11.3+cu113 '
+ 'torchaudio==0.10.2+cu113 -f '
+ 'https://download.pytorch.org/whl/cu113/torch_stable.html')
else:
- datalab.fab.conn.sudo('pip3 install torch==1.10.2+cpu torchvision==0.11.3+cpu torchaudio==0.10.2+cpu -f '
- 'https://download.pytorch.org/whl/cpu/torch_stable.html --no-cache-dir')
+ datalab.fab.conn.install_venv_pip_pkg('torch==1.10.2+cpu torchvision==0.11.3+cpu torchaudio==0.10.2+cpu -f '
+ 'https://download.pytorch.org/whl/cpu/torch_stable.html')
datalab.fab.conn.sudo('touch /home/' + os_user + '/.ensure_dir/pytorch_ensured')
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 03/04: [DATALAB-2698]: fixed function name
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-2698
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 57097544fc6fd617331a592f8c17ad6c7631d400
Author: leonidfrolov <fr...@gmail.com>
AuthorDate: Mon Feb 14 18:43:23 2022 +0200
[DATALAB-2698]: fixed function name
---
infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py b/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
index 4e9bdbd..e015d5b 100644
--- a/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
+++ b/infrastructure-provisioning/src/general/lib/os/debian/notebook_lib.py
@@ -395,7 +395,7 @@ def install_tensor(os_user, cuda_version, cuda_file_name,
sys.exit(1)
-def ensure_pytorch(os_user, libs):
+def ensure_venv_libs(os_user, libs):
if not exists(datalab.fab.conn, '/home/' + os_user + '/.ensure_dir/venv_libs_ensured'):
datalab.fab.conn.install_venv_pip_pkg(libs)
datalab.fab.conn.sudo('touch /home/' + os_user + '/.ensure_dir/venv_libs_ensured')
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org