You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Abhishek Sinha <ab...@infoworks.io> on 2018/08/07 17:04:21 UTC

Using environment connection URL in operators

I am trying to use: MySqlToHiveTransfer Operator.


From the base_hook code, I see that there is a way to pass connection URL via environment variables. The variable needs to be pre-fixed with: AIRFLOW_CONN_ followed by the connection ID.

I have tried exporting this variable but the behaviour seems to be random. In some instances of the DAG Run, the variable is present in os.environ but in some of the runs it is not. Is this an issue?

We are on version 1.8.2 with Celery Executor.




Regards,

Abhishek

Re: Using environment connection URL in operators

Posted by Abhishek Sinha <ab...@infoworks.io>.
Deng,


I have only one node at present. 




Regards,

Abhishek 

> On 08-Aug-2018, at 5:30 AM, Deng Xiaodong <xd...@gmail.com> wrote:
> 
> Hi Abhishek,
> 
> Workers read connection info from its local environment variables. So if
> you have multiple Celery executor nodes (in distributed fashion), possibly
> check whether the same environement variable is exported on all of them?
> This may explain why the behaviour is random in your environment.
> 
> Folks please correct me if I’m wrong. Thanks.
> 
> 
> XD
> 
> On Wed, Aug 8, 2018 at 01:04 Abhishek Sinha <ab...@infoworks.io> wrote:
> 
>> I am trying to use: MySqlToHiveTransfer Operator.
>> 
>> 
>> From the base_hook code, I see that there is a way to pass connection URL
>> via environment variables. The variable needs to be pre-fixed with:
>> AIRFLOW_CONN_ followed by the connection ID.
>> 
>> I have tried exporting this variable but the behaviour seems to be random.
>> In some instances of the DAG Run, the variable is present in os.environ but
>> in some of the runs it is not. Is this an issue?
>> 
>> We are on version 1.8.2 with Celery Executor.
>> 
>> 
>> 
>> 
>> Regards,
>> 
>> Abhishek
>> 
>>

Re: Using environment connection URL in operators

Posted by Deng Xiaodong <xd...@gmail.com>.
Hi Abhishek,

Workers read connection info from its local environment variables. So if
you have multiple Celery executor nodes (in distributed fashion), possibly
check whether the same environement variable is exported on all of them?
This may explain why the behaviour is random in your environment.

Folks please correct me if I’m wrong. Thanks.


XD

On Wed, Aug 8, 2018 at 01:04 Abhishek Sinha <ab...@infoworks.io> wrote:

> I am trying to use: MySqlToHiveTransfer Operator.
>
>
> From the base_hook code, I see that there is a way to pass connection URL
> via environment variables. The variable needs to be pre-fixed with:
> AIRFLOW_CONN_ followed by the connection ID.
>
> I have tried exporting this variable but the behaviour seems to be random.
> In some instances of the DAG Run, the variable is present in os.environ but
> in some of the runs it is not. Is this an issue?
>
> We are on version 1.8.2 with Celery Executor.
>
>
>
>
> Regards,
>
> Abhishek
>
>