You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Juraj Šanko (Jira)" <ji...@apache.org> on 2020/03/16 13:11:00 UTC
[jira] [Updated] (AIRFLOW-7070) SimpleHttpOperator does not
consider extra attributes from Airflow connection
[ https://issues.apache.org/jira/browse/AIRFLOW-7070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Juraj Šanko updated AIRFLOW-7070:
---------------------------------
Description:
I have noticed this problem with SimpleHttpOperator in Airflow 1.10.9.
When we set it up with conn_id, it does not take into account 'extra' parameters from connection.
Example:
Airflow -> Admin -> Connection -> Extra:
{"proxies":"https":"http://xxxxxxxx:yyyyyyy@1.2.3.4:80"}}
My python skill is very bad, but I think there are 2 things happening:
*1.*
http_hook.py - function *get_conn* - line 78
Context:
{code:java}
if conn.extra:
try:
session.headers.update(conn.extra_dejson)
{code}
This is updating session header, however header can't accept type dictionary, only string. So first error with this is that the header is incorrectly formatted (type error).
To me, it makes no sense to use extra field as header source, but if this have to be kept for compatibility reason, I would suggest excluding python-request parameters from pushing to headers. Otherwise It may results also in insecure behavior.
{code:java}
session.proxies.update(extra.get("proxies"))
// This works for me if I disable header code above. (don't use it in solution){code}
*2.*
**http_hook.py - function *run_and_check* - ** lines 176-184
Extra options here are only taken into consideration from extra_option object, but they they should be taken from session object as well and overwritten by connection extra_option object afterwards.
{code:java}
proxies=session.proxies
// replaced this code, as brute force solution.(don't use it in solution){code}
PS: This prohibits us from having sensitive information like proxy login in secure place therefore creates security concern.
was:
I have noticed this problem with SimpleHttpOperator in Airflow 1.10.9.
When we set it up with conn_id, it does not take into account 'extra' parameters from connection.
Example:
Airflow -> Admin -> Connection -> Extra:
{"proxies":"https":"http://xxxxxxxx:yyyyyyy@1.2.3.4:80"}}
My python skill is very bad, but I think there are 2 things happening:
*1.*
http_hook.py - function *get_conn* - line 78
Context:
{code:java}
if conn.extra:
try:
session.headers.update(conn.extra_dejson)
{code}
This is updating session header, however header can't accept type dictionary, only string. So first error with this is that the header is incorrectly formatted (type error).
To me, it makes no sense to use extra field as header source, but if this have to be kept for compatibility reason, I would suggest excluding python-request parameters from pushing to headers. Otherwise It may results also in insecure behavior.
{code:java}
session.proxies.update(extra.get("proxies"))
// This works for me if I disable header code above. (don't use it in solution){code}
*2.*
**http_hook.py - function *run_and_check* - ** lines 176-184
Extra options here are only taken into consideration from extra_option object, but they they should be taken from session object as well and overwritten by connection extra_option object afterwards.
{code:java}
proxies=session.proxies
// replaced this code, as brute force solution.(don't use it in solution){code}
> SimpleHttpOperator does not consider extra attributes from Airflow connection
> -----------------------------------------------------------------------------
>
> Key: AIRFLOW-7070
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7070
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
> Affects Versions: 1.10.9
> Reporter: Juraj Šanko
> Priority: Minor
>
> I have noticed this problem with SimpleHttpOperator in Airflow 1.10.9.
> When we set it up with conn_id, it does not take into account 'extra' parameters from connection.
> Example:
> Airflow -> Admin -> Connection -> Extra:
> {"proxies":"https":"http://xxxxxxxx:yyyyyyy@1.2.3.4:80"}}
> My python skill is very bad, but I think there are 2 things happening:
> *1.*
> http_hook.py - function *get_conn* - line 78
> Context:
>
> {code:java}
> if conn.extra:
> try:
> session.headers.update(conn.extra_dejson)
> {code}
> This is updating session header, however header can't accept type dictionary, only string. So first error with this is that the header is incorrectly formatted (type error).
>
> To me, it makes no sense to use extra field as header source, but if this have to be kept for compatibility reason, I would suggest excluding python-request parameters from pushing to headers. Otherwise It may results also in insecure behavior.
>
> {code:java}
> session.proxies.update(extra.get("proxies"))
> // This works for me if I disable header code above. (don't use it in solution){code}
>
> *2.*
> **http_hook.py - function *run_and_check* - ** lines 176-184
> Extra options here are only taken into consideration from extra_option object, but they they should be taken from session object as well and overwritten by connection extra_option object afterwards.
> {code:java}
> proxies=session.proxies
> // replaced this code, as brute force solution.(don't use it in solution){code}
>
> PS: This prohibits us from having sensitive information like proxy login in secure place therefore creates security concern.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)