You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Kamil Bregula (JIRA)" <ji...@apache.org> on 2019/08/06 10:47:00 UTC
[jira] [Updated] (AIRFLOW-5121) Normalize *_conn_id parameters
[ https://issues.apache.org/jira/browse/AIRFLOW-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kamil Bregula updated AIRFLOW-5121:
-----------------------------------
Description:
Hello,
There are many ways to pass the connection ID to hooks and operators. We should introduce harmony in this matter.
List of all possible combinations.
{code:java}
217 gcp_conn_id=
40 aws_conn_id=
22 google_cloud_storage_conn_id=
21 bigquery_conn_id=
12 dingding_conn_id=
10 ssh_conn_id=
6 qubole_conn_id=
5 postgres_conn_id=
4 presto_conn_id=
4 http_conn_id=
3 jdbc_conn_id=
3 azure_data_lake_conn_id=
2 wasb_conn_id=
2 vertica_conn_id=
2 spark_conn_id=
2 snowflake_conn_id=
2 sftp_conn_id=
2 segment_conn_id=
2 redis_conn_id=
2 jira_conn_id=
2 imap_conn_id=
2 hive_cli_conn_id=
2 gcp_cloudsql_conn_id=
2 emr_conn_id=
2 druid_ingest_conn_id=
2 dest_gcs_conn_id=
2 datastore_conn_id=
2 databricks_conn_id=
2 azure_cosmos_conn_id=
1 sqlite_conn_id=
1 slack_conn_id=
1 samba_conn_id=
1 registry_conn_id=
1 pig_cli_conn_id=
1 oracle_conn_id=
1 opsgenie_conn_id=
1 mysql_conn_id=
1 mssql_conn_id=
1 metastore_conn_id=
1 grpc_conn_id=
1 druid_broker_conn_id=
1 docker_conn_id=
1 cloud_storage_conn_id=
1 ci_conn_id=
{code}
List generated by following command:
{code:java}
grep -o -R '[a-z_]\+_conn_id=' --include '*_operator.py' airflow/ | cut -d ":" -f 2 | sort | uniq -c |sort | tac
{code}
was:
Hello,
There are many ways to pass the connection ID to hooks and operators. We should introduce harmony in this matter.
List of all possible combinations.
{code:java}
217 gcp_conn_id=
40 aws_conn_id=
22 google_cloud_storage_conn_id=
21 bigquery_conn_id=
12 dingding_conn_id=
10 ssh_conn_id=
6 qubole_conn_id=
5 postgres_conn_id=
4 presto_conn_id=
4 http_conn_id=
3 jdbc_conn_id=
3 azure_data_lake_conn_id=
2 wasb_conn_id=
2 vertica_conn_id=
2 spark_conn_id=
2 snowflake_conn_id=
2 sftp_conn_id=
2 segment_conn_id=
2 redis_conn_id=
2 jira_conn_id=
2 imap_conn_id=
2 hive_cli_conn_id=
2 gcp_cloudsql_conn_id=
2 emr_conn_id=
2 druid_ingest_conn_id=
2 dest_gcs_conn_id=
2 datastore_conn_id=
2 databricks_conn_id=
2 azure_cosmos_conn_id=
1 sqlite_conn_id=
1 slack_conn_id=
1 samba_conn_id=
1 registry_conn_id=
1 pig_cli_conn_id=
1 oracle_conn_id=
1 opsgenie_conn_id=
1 mysql_conn_id=
1 mssql_conn_id=
1 metastore_conn_id=
1 grpc_conn_id=
1 druid_broker_conn_id=
1 docker_conn_id=
1 cloud_storage_conn_id=
1 ci_conn_id=
a
{code}
> Normalize *_conn_id parameters
> ------------------------------
>
> Key: AIRFLOW-5121
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5121
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks, operators
> Affects Versions: 1.10.3
> Reporter: Kamil Bregula
> Priority: Major
>
> Hello,
>
> There are many ways to pass the connection ID to hooks and operators. We should introduce harmony in this matter.
> List of all possible combinations.
> {code:java}
> 217 gcp_conn_id=
> 40 aws_conn_id=
> 22 google_cloud_storage_conn_id=
> 21 bigquery_conn_id=
> 12 dingding_conn_id=
> 10 ssh_conn_id=
> 6 qubole_conn_id=
> 5 postgres_conn_id=
> 4 presto_conn_id=
> 4 http_conn_id=
> 3 jdbc_conn_id=
> 3 azure_data_lake_conn_id=
> 2 wasb_conn_id=
> 2 vertica_conn_id=
> 2 spark_conn_id=
> 2 snowflake_conn_id=
> 2 sftp_conn_id=
> 2 segment_conn_id=
> 2 redis_conn_id=
> 2 jira_conn_id=
> 2 imap_conn_id=
> 2 hive_cli_conn_id=
> 2 gcp_cloudsql_conn_id=
> 2 emr_conn_id=
> 2 druid_ingest_conn_id=
> 2 dest_gcs_conn_id=
> 2 datastore_conn_id=
> 2 databricks_conn_id=
> 2 azure_cosmos_conn_id=
> 1 sqlite_conn_id=
> 1 slack_conn_id=
> 1 samba_conn_id=
> 1 registry_conn_id=
> 1 pig_cli_conn_id=
> 1 oracle_conn_id=
> 1 opsgenie_conn_id=
> 1 mysql_conn_id=
> 1 mssql_conn_id=
> 1 metastore_conn_id=
> 1 grpc_conn_id=
> 1 druid_broker_conn_id=
> 1 docker_conn_id=
> 1 cloud_storage_conn_id=
> 1 ci_conn_id=
> {code}
> List generated by following command:
> {code:java}
> grep -o -R '[a-z_]\+_conn_id=' --include '*_operator.py' airflow/ | cut -d ":" -f 2 | sort | uniq -c |sort | tac
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)