You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@datalab.apache.org by "Leonid Frolov (Jira)" <ji...@apache.org> on 2021/12/14 13:00:00 UTC
[jira] [Commented] (DATALAB-2644) Investigate how image adding is implemented and possible future options
[ https://issues.apache.org/jira/browse/DATALAB-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459129#comment-17459129 ]
Leonid Frolov commented on DATALAB-2644:
----------------------------------------
*Answer these questions in DeepLearning content*
# *For now is only one image pulled from cloud (AWS/GCP/Azure)?*
Now we don`t pull images from cloud. We have image list specified in our code. Probably because we need not only image name but also it`s description so that user will understand what particular image contains.
On AWS there are two DeepLearning images: Base AMI only with some libraries and drivers and Conda AMI with many preinstalled libraries, frameworks, etc. We Conda AMI, but we use v.42.1 not the latest one and there is no way for it to changed by the user.
[https://aws.amazon.com/machine-learning/amis/]
[https://aws.amazon.com/marketplace/pp/prodview-x5nivojpquy6y?sr=0-1&ref_=beagle&applicationId=AWSMPContessa]
On Azure we use Deep Learning AMI from Azure docs (in docs there is only one on ubuntu)
[https://azure.microsoft.com/en-us/services/virtual-machines/data-science-virtual-machines/#product-overview]
https://azuremarketplace.microsoft.com/en-us/marketplace/apps/microsoft-dsvm.ubuntu-1804?tab=Overview
On GCP there are ~940 both GPU and CPU DeepLearning images divided by 12 frameworks
[https://cloud.google.com/deep-learning-vm/docs/images]
On Dlab UI we have images with 8 frameworks (except TensorFlow Enterprise 2.x,, TensorFlow Enterprise 1.x, R (only CPU), PyTorch XLA). 7 of 8 frameworks use latest GPU image and with base framework there are 3 images with different cuda versions to choose from (11.0, 10.0, 9.2)
[https://github.com/apache/incubator-datalab/blob/develop/infrastructure-provisioning/src/general/files/gcp/deeplearning_description.json|http://example.com]
# *Why we could not pull all available image list (templates) of related product from service cloud?*
On AWS we can probably add Base AMI but user will have to install all frameworks/packages/libraries by himself.
On Azure marketplace there are many AMIs with similar to DeepLearning sets of software but these images are placed not only by microsoft but also by other companies. All those images have different names and there is no easy way to distinguish them from another images (like base ubuntu, opensuse, jupyter, etc.).
On GCP list of all available DeepLearning images is too big.
# *Why is current version of image pulling implemented from cloud service? ( To add another we should add it manually)*
Probably because it was faster to implement. Our custom DeepLearning, that we had before, had many compatibility issues due to software updates and failed on creation. So Cloud AMIs, that were the most similar to our DeepLearning by frameworks/libraries were chosen.
> Investigate how image adding is implemented and possible future options
> -----------------------------------------------------------------------
>
> Key: DATALAB-2644
> URL: https://issues.apache.org/jira/browse/DATALAB-2644
> Project: Apache DataLab
> Issue Type: Task
> Security Level: Public(Regular Issues)
> Components: DataLab Main
> Reporter: Vira Vitanska
> Assignee: Leonid Frolov
> Priority: Major
> Labels: AWS, AZURE, DevOps, GCP
> Fix For: v.2.5.2
>
>
> *Answer these questions in DeepLearning content*
> # For now is only one image pulled from cloud (AWS/GCP/Azure)?
> # Why we could not pull all available image list (templates) of related product from service cloud?
> # Why is current version of image pulling implemented from cloud service? ( To add another we should add it manually)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datalab.apache.org
For additional commands, e-mail: dev-help@datalab.apache.org