You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datalab.apache.org by lf...@apache.org on 2021/04/16 11:03:21 UTC
[incubator-datalab] branch DATALAB-2091 updated (58e7b1a -> abcf565)
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a change to branch DATALAB-2091
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git.
from 58e7b1a [DATALAB-2091]: fixed some errors during spark configuration
new 2b843e3 [DATALAB-2091]: fixed one of emr creation errors
new ce9fbb4 [DATALAB-2091]: fixed another of emr creation errors
new 1584a92 [DATALAB-2091]: added wrap in bash -c
new abcf565 [DATALAB-2091]: tmp use python2 instead of python3
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
infrastructure-provisioning/src/base/scripts/create_ssh_user.py | 2 +-
infrastructure-provisioning/src/general/lib/aws/meta_lib.py | 2 +-
.../src/general/scripts/aws/dataengine-service_create.py | 2 +-
.../src/general/scripts/aws/dataengine-service_prepare.py | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 02/04: [DATALAB-2091]: fixed another of emr
creation errors
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-2091
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit ce9fbb404e376e180be2ed58f5ed165201494534
Author: leonidfrolov <fr...@gmail.com>
AuthorDate: Thu Apr 15 15:48:20 2021 +0300
[DATALAB-2091]: fixed another of emr creation errors
---
.../src/general/scripts/aws/dataengine-service_prepare.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_prepare.py b/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_prepare.py
index b6def63..bf9da3b 100644
--- a/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_prepare.py
+++ b/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_prepare.py
@@ -308,7 +308,7 @@ if __name__ == "__main__":
emr_conf['additional_emr_sg_name'],
emr_conf['configurations'])
try:
- subprocess.run("~/scripts/{}.py {}".format('dataengine-service_create', params))
+ subprocess.run("~/scripts/{}.py {}".format('dataengine-service_create', params), shell=True, check=True)
except:
traceback.print_exc()
raise Exception
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 04/04: [DATALAB-2091]: tmp use python2 instead
of python3
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-2091
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit abcf565c8df8999970c0346e665ded8fee3dfc0e
Author: leonidfrolov <fr...@gmail.com>
AuthorDate: Fri Apr 16 14:02:58 2021 +0300
[DATALAB-2091]: tmp use python2 instead of python3
---
.../src/general/scripts/aws/dataengine-service_create.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_create.py b/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_create.py
index 093cbfd..6e9c8c6 100644
--- a/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_create.py
+++ b/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_create.py
@@ -121,7 +121,7 @@ cp_jars = "Name=CUSTOM_JAR, Args=aws " \
"Name=CUSTOM_JAR, Args=sudo " \
"/usr/bin/python3 /tmp/key_importer.py --user_name {4}, " \
"ActionOnFailure=TERMINATE_CLUSTER,Jar=command-runner.jar; " \
- "Name=CUSTOM_JAR, Args=/usr/bin/python3 /tmp/jars_parser.py " \
+ "Name=CUSTOM_JAR, Args=/usr/bin/python /tmp/jars_parser.py " \
"--bucket {0} --emr_version {3} --region {2} --user_name {4} " \
"--cluster_name {5}, " \
"ActionOnFailure=TERMINATE_CLUSTER,Jar=command-runner.jar".\
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 01/04: [DATALAB-2091]: fixed one of emr
creation errors
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-2091
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 2b843e32bc08b85a7bf3b90c078ca8b214f54423
Author: leonidfrolov <fr...@gmail.com>
AuthorDate: Thu Apr 15 15:35:26 2021 +0300
[DATALAB-2091]: fixed one of emr creation errors
---
infrastructure-provisioning/src/general/lib/aws/meta_lib.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/infrastructure-provisioning/src/general/lib/aws/meta_lib.py b/infrastructure-provisioning/src/general/lib/aws/meta_lib.py
index 692043b..07de8b0 100644
--- a/infrastructure-provisioning/src/general/lib/aws/meta_lib.py
+++ b/infrastructure-provisioning/src/general/lib/aws/meta_lib.py
@@ -797,7 +797,7 @@ def get_ec2_price(instance_shape, region):
# Price API require full name of region, for example: eu-west-1 -> 'EU (Ireland)'
# endpoints will be loaded from: botocore/botocore/data/endpoints.json
data = client._loader._cache.get(('load_data', 'endpoints'))
- standard_partition = filter(lambda x: 'AWS Standard' == x['partitionName'], data['partitions'])[0]
+ standard_partition = next(filter(lambda x: 'AWS Standard' == x['partitionName'], data['partitions']))
region_description = standard_partition['regions'][region]['description']
response = client.get_products(
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org
[incubator-datalab] 03/04: [DATALAB-2091]: added wrap in bash -c
Posted by lf...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lfrolov pushed a commit to branch DATALAB-2091
in repository https://gitbox.apache.org/repos/asf/incubator-datalab.git
commit 1584a92f166ac3a0222b3fa727c71b495ca7aa7b
Author: leonidfrolov <fr...@gmail.com>
AuthorDate: Fri Apr 16 14:01:41 2021 +0300
[DATALAB-2091]: added wrap in bash -c
---
infrastructure-provisioning/src/base/scripts/create_ssh_user.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/infrastructure-provisioning/src/base/scripts/create_ssh_user.py b/infrastructure-provisioning/src/base/scripts/create_ssh_user.py
index 7875056..7c99bd2 100644
--- a/infrastructure-provisioning/src/base/scripts/create_ssh_user.py
+++ b/infrastructure-provisioning/src/base/scripts/create_ssh_user.py
@@ -45,7 +45,7 @@ def ensure_ssh_user(initial_user, os_user, sudo_group):
conn.sudo('bash -c "echo \'{} ALL = NOPASSWD:ALL\' >> /etc/sudoers"'.format(os_user))
conn.sudo('mkdir /home/{}/.ssh'.format(os_user))
conn.sudo('chown -R {0}:{0} /home/{1}/.ssh/'.format(initial_user, os_user))
- conn.sudo('cat /home/{0}/.ssh/authorized_keys > /home/{1}/.ssh/authorized_keys'.format(initial_user, os_user))
+ conn.sudo('''bash -c 'cat /home/{0}/.ssh/authorized_keys > /home/{1}/.ssh/authorized_keys' '''.format(initial_user, os_user))
conn.sudo('chown -R {0}:{0} /home/{0}/.ssh/'.format(os_user))
conn.sudo('chmod 700 /home/{0}/.ssh'.format(os_user))
conn.sudo('chmod 600 /home/{0}/.ssh/authorized_keys'.format(os_user))
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datalab.apache.org
For additional commands, e-mail: commits-help@datalab.apache.org