You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/22 14:38:01 UTC
[GitHub] [airflow] potiuk opened a new issue #22447: Migrate Google example DAGs to new design
potiuk opened a new issue #22447:
URL: https://github.com/apache/airflow/issues/22447
There is a new design of system tests that was introduced by the [AIP-47](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-47+New+design+of+Airflow+System+Tests).
All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.
This is an aggregated issue for all example DAGs related to `Google` provider. It is created to track progress of their migration.
List of paths to example DAGs:
- [ ] airflow/providers/google/leveldb/example_dags/example_leveldb.py
- [ ] airflow/providers/google/firebase/example_dags/example_firestore.py
- [ ] airflow/providers/google/ads/example_dags/example_ads.py
- [ ] airflow/providers/google/suite/example_dags/example_sql_to_sheets.py
- [ ] airflow/providers/google/suite/example_dags/example_gcs_to_sheets.py
- [ ] airflow/providers/google/suite/example_dags/example_gcs_to_gdrive.py
- [ ] airflow/providers/google/suite/example_dags/example_sheets.py
- [ ] airflow/providers/google/marketing_platform/example_dags/example_analytics.py
- [ ] airflow/providers/google/marketing_platform/example_dags/example_display_video.py
- [ ] airflow/providers/google/marketing_platform/example_dags/example_campaign_manager.py
- [ ] airflow/providers/google/marketing_platform/example_dags/example_search_ads.py
- [ ] airflow/providers/google/cloud/example_dags/example_bigquery_queries.py
- [ ] airflow/providers/google/cloud/example_dags/example_gcs_to_sftp.py
- [ ] airflow/providers/google/cloud/example_dags/example_oracle_to_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_compute_igm.py
- [ ] airflow/providers/google/cloud/example_dags/example_automl_vision_classification.py
- [ ] airflow/providers/google/cloud/example_dags/example_datacatalog.py
- [ ] airflow/providers/google/cloud/example_dags/example_bigquery_to_mssql.py
- [ ] airflow/providers/google/cloud/example_dags/example_mssql_to_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_mysql_to_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_bigquery_to_bigquery.py
- [ ] airflow/providers/google/cloud/example_dags/example_natural_language.py
- [ ] airflow/providers/google/cloud/example_dags/example_life_sciences.py
- [ ] airflow/providers/google/cloud/example_dags/example_automl_nl_text_extraction.py
- [ ] airflow/providers/google/cloud/example_dags/example_automl_video_intelligence_classification.py
- [ ] airflow/providers/google/cloud/example_dags/example_trino_to_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_bigquery_transfer.py
- [ ] airflow/providers/google/cloud/example_dags/example_cloud_storage_transfer_service_gcp.py
- [ ] airflow/providers/google/cloud/example_dags/example_translate.py
- [ ] airflow/providers/google/cloud/example_dags/example_local_to_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_compute_ssh.py
- [ ] airflow/providers/google/cloud/example_dags/example_dataflow_sql.py
- [ ] airflow/providers/google/cloud/example_dags/example_text_to_speech.py
- [ ] airflow/providers/google/cloud/example_dags/example_presto_to_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_looker.py
- [ ] airflow/providers/google/cloud/example_dags/example_sftp_to_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_facebook_ads_to_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_gcs_to_bigquery.py
- [ ] airflow/providers/google/cloud/example_dags/example_postgres_to_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_spanner.py
- [ ] airflow/providers/google/cloud/example_dags/example_gdrive_to_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_dlp.py
- [ ] airflow/providers/google/cloud/example_dags/example_bigquery_dts.py
- [ ] airflow/providers/google/cloud/example_dags/example_datastore.py
- [ ] airflow/providers/google/cloud/example_dags/example_bigquery_operations.py
- [ ] airflow/providers/google/cloud/example_dags/example_automl_nl_text_classification.py
- [ ] airflow/providers/google/cloud/example_dags/example_bigquery_to_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_automl_vision_object_detection.py
- [ ] airflow/providers/google/cloud/example_dags/example_bigtable.py
- [ ] airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
- [ ] airflow/providers/google/cloud/example_dags/example_vertex_ai.py
- [ ] airflow/providers/google/cloud/example_dags/example_automl_video_intelligence_tracking.py
- [ ] airflow/providers/google/cloud/example_dags/example_gcs_timespan_file_transform.py
- [ ] airflow/providers/google/cloud/example_dags/example_dataproc_metastore.py
- [ ] airflow/providers/google/cloud/example_dags/example_salesforce_to_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_functions.py
- [ ] airflow/providers/google/cloud/example_dags/example_compute.py
- [ ] airflow/providers/google/cloud/example_dags/example_sheets_to_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_dataflow.py
- [ ] airflow/providers/google/cloud/example_dags/example_datafusion.py
- [ ] airflow/providers/google/cloud/example_dags/example_cloud_sql_query.py
- [ ] airflow/providers/google/cloud/example_dags/example_translate_speech.py
- [ ] airflow/providers/google/cloud/example_dags/example_automl_tables.py
- [ ] airflow/providers/google/cloud/example_dags/example_cloud_memorystore.py
- [ ] airflow/providers/google/cloud/example_dags/example_bigquery_sensors.py
- [ ] airflow/providers/google/cloud/example_dags/example_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_mlengine.py
- [ ] airflow/providers/google/cloud/example_dags/example_dataprep.py
- [ ] airflow/providers/google/cloud/example_dags/example_cloud_storage_transfer_service_aws.py
- [ ] airflow/providers/google/cloud/example_dags/example_pubsub.py
- [ ] airflow/providers/google/cloud/example_dags/example_gcs_to_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_s3_to_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_gdrive_to_local.py
- [ ] airflow/providers/google/cloud/example_dags/example_azure_fileshare_to_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_workflows.py
- [ ] airflow/providers/google/cloud/example_dags/example_kubernetes_engine.py
- [ ] airflow/providers/google/cloud/example_dags/example_dataproc.py
- [ ] airflow/providers/google/cloud/example_dags/example_speech_to_text.py
- [ ] airflow/providers/google/cloud/example_dags/example_cloud_composer.py
- [ ] airflow/providers/google/cloud/example_dags/example_automl_translation.py
- [ ] airflow/providers/google/cloud/example_dags/example_stackdriver.py
- [ ] airflow/providers/google/cloud/example_dags/example_tasks.py
- [ ] airflow/providers/google/cloud/example_dags/example_calendar_to_gcs.py
- [ ] airflow/providers/google/cloud/example_dags/example_video_intelligence.py
- [ ] airflow/providers/google/cloud/example_dags/example_cloud_build.py
- [ ] airflow/providers/google/cloud/example_dags/example_dataflow_flex_template.py
- [ ] airflow/providers/google/cloud/example_dags/example_automl_nl_text_sentiment.py
- [ ] airflow/providers/google/cloud/example_dags/example_cloud_sql.py
- [ ] airflow/providers/google/cloud/example_dags/example_vision.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mnojek commented on issue #22447: Migrate Google example DAGs to new design
Posted by GitBox <gi...@apache.org>.
mnojek commented on issue #22447:
URL: https://github.com/apache/airflow/issues/22447#issuecomment-1077688729
Hey, please update the checklist.
These example DAGs are being **removed by PR #22311**:
- airflow/providers/google/cloud/example_dags/example_bigquery_operations.py
- airflow/providers/google/cloud/example_dags/example_bigquery_queries.py
- airflow/providers/google/cloud/example_dags/example_bigquery_sensors.py
And these are **WIP** (Work In Progress) (not sure how we are going to mark them, but it would be good to notify others so that we will avoid work duplication):
- airflow/providers/google/cloud/example_dags/example_bigquery_to_bigquery.py
- airflow/providers/google/cloud/example_dags/example_dataflow.py
- airflow/providers/google/cloud/example_dags/example_datafusion.py
- airflow/providers/google/cloud/example_dags/example_dataproc.py
- airflow/providers/google/cloud/example_dags/example_dataproc_metastore.py
- airflow/providers/google/cloud/example_dags/example_datastore.py
- airflow/providers/google/cloud/example_dags/example_gcs.py
- airflow/providers/google/cloud/example_dags/example_gcs_timespan_file_transform.py
- airflow/providers/google/cloud/example_dags/example_gcs_to_gcs.py
- airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
- airflow/providers/google/cloud/example_dags/example_local_to_gcs.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #22447: Migrate Google example DAGs to new design
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #22447:
URL: https://github.com/apache/airflow/issues/22447#issuecomment-1080686217
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org