[submarine] branch master updated: SUBMARINE-1311. Fix tensorflow dataset cannot be downloaded error

[pi...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

pingsutw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git


The following commit(s) were added to refs/heads/master by this push:
     new 01d6b21c SUBMARINE-1311. Fix tensorflow dataset cannot be downloaded error
01d6b21c is described below

commit 01d6b21cbca2b46ddda49955d48d318b135f2c78
Author: cdmikechen <cd...@hotmail.com>
AuthorDate: Sat Aug 20 11:17:09 2022 +0800

    SUBMARINE-1311. Fix tensorflow dataset cannot be downloaded error
    
    ### What is this PR for?
    When using submarine quickstart, the tensorflow dataset cannot be downloaded, the error is as follows:
    ```
    All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with "Not found: Could not locate the credentials file.". Retrieving token from GCE failed with "Failed precondition: Error executing an HTTP request: libcurl code 6 meaning 'Couldn't resolve host name', error details: Couldn't resolve host 'metadata'".
    ```
    
    ### What type of PR is it?
    Bug Fix
    
    ### Todos
    * [x] - Add version check to turn off GCS
    
    ### What is the Jira issue?
    https://issues.apache.org/jira/browse/SUBMARINE-1311
    
    ### How should this be tested?
    Test cases should be followed up with relevant image ([submarine-quickstart](https://hub.docker.com/r/apache/submarine/tags?page=1&name=quickstart)) test cases to facilitate regression testing.
    
    ### Screenshots (if appropriate)
    No
    
    ### Questions:
    * Do the license files need updating? No
    * Are there breaking changes for older versions? No
    * Does this need new documentation? No
    
    Author: cdmikechen <cd...@hotmail.com>
    
    Signed-off-by: Kevin <pi...@apache.org>
    
    Closes #990 from cdmikechen/SUBMARINE-1311 and squashes the following commits:
    
    c27c20c6 [cdmikechen] reformat codes
    3592810b [cdmikechen] Fix tensorflow dataset cannot be downloaded error
---
 dev-support/examples/quickstart/Dockerfile | 2 +-
 dev-support/examples/quickstart/train.py   | 5 +++++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/dev-support/examples/quickstart/Dockerfile b/dev-support/examples/quickstart/Dockerfile
index 5ce1045e..a5a35a00 100644
--- a/dev-support/examples/quickstart/Dockerfile
+++ b/dev-support/examples/quickstart/Dockerfile
@@ -19,6 +19,6 @@ MAINTAINER Apache Software Foundation <de...@submarine.apache.org>
 ADD ./tmp/submarine-sdk /opt/
 # install submarine-sdk locally
 RUN pip install /opt/pysubmarine/.[tf2]
-RUN pip install tensorflow_datasets
+RUN pip install tensorflow_datasets packaging
 
 ADD ./train.py /opt/
diff --git a/dev-support/examples/quickstart/train.py b/dev-support/examples/quickstart/train.py
index 2b4069a3..d0d1b041 100644
--- a/dev-support/examples/quickstart/train.py
+++ b/dev-support/examples/quickstart/train.py
@@ -18,6 +18,7 @@ https://github.com/kubeflow/tf-operator/blob/master/examples/v1/distribution_str
 """
 import tensorflow as tf
 import tensorflow_datasets as tfds
+from packaging.version import Version
 from tensorflow.keras import layers, models
 
 import submarine
@@ -32,6 +33,10 @@ def make_datasets_unbatched():
         image /= 255
         return image, label
 
+    # If we use tensorflow_datasets > 3.1.0, we need to disable GCS
+    # https://github.com/tensorflow/datasets/issues/2761#issuecomment-1187413141
+    if Version(tfds.__version__) > Version("3.1.0"):
+        tfds.core.utils.gcs_utils._is_gcs_disabled = True
     datasets, _ = tfds.load(name="mnist", with_info=True, as_supervised=True)
 
     return datasets["train"].map(scale).cache().shuffle(BUFFER_SIZE)


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org