You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/19 20:46:27 UTC

[GitHub] [airflow] vuppalli opened a new issue #9418: Deprecated AI Platform Operators and Runtimes in Example DAG

vuppalli opened a new issue #9418:
URL: https://github.com/apache/airflow/issues/9418


   <!--
   
   IMPORTANT!!!
   
   PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
   NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
   
   PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
   
   Please complete the next sections or the issue will be closed.
   This questions are the first thing we need to know to understand the context.
   
   -->
   
   **Apache Airflow version**: 1.10.10
   
   
   **Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
   
   **Environment**:
   
   - **Cloud provider or hardware configuration**: GCP
   - **OS** (e.g. from /etc/os-release):
   - **Kernel** (e.g. `uname -a`):
   - **Install tools**:
   - **Others**:
   
   **What happened**:
   
   The file, /airflow/providers/google/cloud/example_dags/example_mlengine.py, uses deprecated operators and a soon to be deprecated AI Platform runtime version.
   - MLEngineManageModelOperator is used to create and get a model. (https://airflow.readthedocs.io/en/latest/_api/airflow/providers/google/cloud/operators/mlengine/index.html#airflow.providers.google.cloud.operators.mlengine.MLEngineManageModelOperator)
   
   ```
   create_model = MLEngineManageModelOperator(
           task_id="create-model", 
           project_id=PROJECT_ID, 
           operation='create',
           model={
               "name": MODEL_NAME,
           },
   )
   ```
   
   ```
   get_model = MLEngineManageModelOperator(
           task_id="get-model",
           project_id=PROJECT_ID,
           operation="get",
           model={
               "name": MODEL_NAME,
           }
   )
   ```
   - Many operators use a runtime version of 1.14 which will be deprecated next month. (https://cloud.google.com/ai-platform/training/docs/runtime-version-list#1.14)
   
   ```
   create_version = MLEngineCreateVersionOperator(
           task_id="create-version", 
           project_id=PROJECT_ID, 
           model_name=MODEL_NAME,
           version={
               "name": "v1",
               "description": "First-version",
               "deployment_uri": '{}/keras_export/'.format(JOB_DIR),
               "runtime_version": "1.14",
               "machineType": "mls1-c1-m2",
               "framework": "TENSORFLOW",
               "pythonVersion": "3.5"
           }
   )
   ```
   **What you expected to happen**:
   
   The file, /airflow/providers/google/cloud/example_dags/example_mlengine.py, should be using new operators and a more recent AI Platform runtime version.
   - To create a model: MLEngineCreateModelOperator
   - To get a model: MLEngineGetModelOperator
   - All operators should use a runtime version of 1.15 or 2.1.
   
   **How to reproduce it**:
   <!---
   
   As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
   
   If you are using kubernetes, please attempt to recreate the issue using minikube or kind.
   
   ## Install minikube/kind
   
   - Minikube https://minikube.sigs.k8s.io/docs/start/
   - Kind https://kind.sigs.k8s.io/docs/user/quick-start/
   
   If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action
   
   You can include images using the .md sytle of
   ![alt text](http://url/to/img.png)
   
   To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.
   
   --->
   
   
   **Anything else we need to know**:
   I would be happy to take this issue!
   
   <!--
   
   How often does this problem occur? Once? Every time etc?
   
   Any relevant logs to include? Put them here in side a detail tag:
   <details><summary>x.log</summary> lots of stuff </details>
   
   -->
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] vuppalli commented on issue #9418: Deprecated AI Platform Operators and Runtimes in Example DAG

Posted by GitBox <gi...@apache.org>.
vuppalli commented on issue #9418:
URL: https://github.com/apache/airflow/issues/9418#issuecomment-648269805


   Thanks so much! I have a couple of quick questions: when running the DAG, I noticed that I do not have access to the GCP resources used in the file. Would it be possible to get added to the project so that I can run the DAG without making any changes successfully? Or, is there a place where I can access all of these resources for myself? Additionally, is there an official test file to confirm that the DAG works?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] vuppalli edited a comment on issue #9418: Deprecated AI Platform Operators and Runtimes in Example DAG

Posted by GitBox <gi...@apache.org>.
vuppalli edited a comment on issue #9418:
URL: https://github.com/apache/airflow/issues/9418#issuecomment-652672779


   Thank you for the information! I created a PR for this issue here: https://github.com/apache/airflow/pull/9727. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #9418: Deprecated AI Platform Operators and Runtimes in Example DAG

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #9418:
URL: https://github.com/apache/airflow/issues/9418#issuecomment-646875666


   I am very happy that you want to join this project. I assigned you to this ticket. I am the maintainer of this integration, so if you have questions, ask I look forward to your contribution.
   
   I also recommend reading our contribution guide.
   https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst
   
   I would like to pay particular attention to one fragment.
   
   > If you create pull-request, you don't have to create an issue first, but if you want, you can do it. Creating an issue will allow you to collect feedback or share plans with other people.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj closed issue #9418: Deprecated AI Platform Operators and Runtimes in Example DAG

Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #9418:
URL: https://github.com/apache/airflow/issues/9418


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9418: Deprecated AI Platform Operators and Runtimes in Example DAG

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #9418:
URL: https://github.com/apache/airflow/issues/9418#issuecomment-646854465


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] vuppalli commented on issue #9418: Deprecated AI Platform Operators and Runtimes in Example DAG

Posted by GitBox <gi...@apache.org>.
vuppalli commented on issue #9418:
URL: https://github.com/apache/airflow/issues/9418#issuecomment-652672779


   Thank you for the information! I created a PR for this issue here: https://github.com/apache/airflow/pull/9618. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #9418: Deprecated AI Platform Operators and Runtimes in Example DAG

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #9418:
URL: https://github.com/apache/airflow/issues/9418#issuecomment-648364194


   Unfortunately, the community has not developed procedures for storing files necessary for testing. We don't run these tests automatically on CI yet. Each team that works on integrations must provide these files on their own. This should not be difficult based on official guides for this service.
   https://cloud.google.com/ai-platform/training/docs/
   
   I have the following environment variables configured to run these tests. 
   ```
   GCP_PROJECT_ID=polidea-airflow
   
   GCP_MLENGINE_BUCKET_NAME_EPHEMERAL=polidea-airflow-tests-38
   GCP_MLENGINE_BUCKET_NAME_PERSISTENT=test-airflow-mlengine-persistent
   
   GCP_MLENGINE_MODEL_NAME=airflow_test_ci_model_name_19837
   
   GCP_MLENGINE_PREDICTION_INPUT=gs://test-airflow-mlengine-persistent/prediction_input.json
   GCP_MLENGINE_TRAINER_URI=gs://test-airflow-mlengine-persistent/trainer-0.0.0.tar.gz
   
   GCP_MLENGINE_DATAFLOW_STAGING=gs://polidea-airflow-tests-38/staging/
   GCP_MLENGINE_DATAFLOW_TMP=gs://polidea-airflow-tests-38/tmp/
   GCP_MLENGINE_JOB_DIR=gs://polidea-airflow-tests-38/job-dir
   GCP_MLENGINE_PREDICTION_OUTPUT=gs://polidea-airflow-tests-38/prediction_output/
   GCP_MLENGINE_SAVED_MODEL_PATH=gs://polidea-airflow-tests-38/job-dir/keras_export/
   ```
   I prepared copies of the necessary files for you and made them available in a public bucket.
   ```
   gs://airflow-polidea-googl-system-tests-resources-public
   ```
   I hope this information will help you.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj edited a comment on issue #9418: Deprecated AI Platform Operators and Runtimes in Example DAG

Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #9418:
URL: https://github.com/apache/airflow/issues/9418#issuecomment-648364194


   Unfortunately, the community has not developed procedures for storing files necessary for testing. We don't run these tests automatically on CI yet. Each team that works on integrations must provide these files on their own. This should not be difficult based on official guides for this service.
   https://cloud.google.com/ai-platform/training/docs/
   
   I have the following environment variables configured to run these tests. 
   ```
   GCP_PROJECT_ID=polidea-airflow
   
   GCP_MLENGINE_BUCKET_NAME_EPHEMERAL=polidea-airflow-tests-38
   GCP_MLENGINE_BUCKET_NAME_PERSISTENT=test-airflow-mlengine-persistent
   
   GCP_MLENGINE_MODEL_NAME=airflow_test_ci_model_name_19837
   
   GCP_MLENGINE_PREDICTION_INPUT=gs://test-airflow-mlengine-persistent/prediction_input.json
   GCP_MLENGINE_TRAINER_URI=gs://test-airflow-mlengine-persistent/trainer-0.0.0.tar.gz
   
   GCP_MLENGINE_DATAFLOW_STAGING=gs://polidea-airflow-tests-38/staging/
   GCP_MLENGINE_DATAFLOW_TMP=gs://polidea-airflow-tests-38/tmp/
   GCP_MLENGINE_JOB_DIR=gs://polidea-airflow-tests-38/job-dir
   GCP_MLENGINE_PREDICTION_OUTPUT=gs://polidea-airflow-tests-38/prediction_output/
   GCP_MLENGINE_SAVED_MODEL_PATH=gs://polidea-airflow-tests-38/job-dir/keras_export/
   ```
   I prepared copies of the necessary files for you and made them available in a public bucket. Please make a copy of this bucket if you want to work on this integration, because this bucket can be deleted at any time. 
   ```
   gs://airflow-polidea-googl-system-tests-resources-public
   ```
   I hope this information will help you.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org