You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/05/03 09:03:09 UTC

[GitHub] [airflow] eladkal opened a new issue #8688: accessing Json variable jinja doesn't work when variable has hyphen 

eladkal opened a new issue #8688:
URL: https://github.com/apache/airflow/issues/8688


   
   **What happened**:
   
   Defined variable as:
   ![Screen Shot 2020-05-03 at 11 45 51](https://user-images.githubusercontent.com/45845474/80909798-b81d5500-8d33-11ea-93ee-2859997bf0ef.png)
   
   Trying to access the variable via:
   `{{ var.json.csba-catalog.sandbox }}`
   
   Gives:
   ```
   Traceback (most recent call last):
     File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py", line 1426, in _run_raw_task
       self.render_templates()
     File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py", line 1790, in render_templates
       rendered_content = rt(attr, content, jinja_context)
     File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py", line 2538, in render_template
       return self.render_template_from_field(attr, content, context, jinja_env)
     File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py", line 2514, in render_template_from_field
       result = jinja_env.from_string(content).render(**context)
     File "/usr/local/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
       return original_render(self, *args, **kwargs)
     File "/usr/local/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
       return self.environment.handle_exception(exc_info, True)
     File "/usr/local/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
       reraise(exc_type, exc_value, tb)
     File "/usr/local/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
       raise value.with_traceback(tb)
     File "<template>", line 1, in top-level template code
     File "/usr/local/lib/python3.6/site-packages/jinja2/environment.py", line 430, in getattr
       return getattr(obj, attribute)
     File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py", line 1728, in __getattr__
       self.var = Variable.get(item, deserialize_json=True)
     File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 73, in wrapper
       return func(*args, **kwargs)
     File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py", line 4368, in get
       raise KeyError('Variable {} does not exist'.format(key))
   KeyError: 'Variable csba does not exist'
   ```
   Trying to access with:
   `catalog_variable = Variable.get("csba-catalog", deserialize_json=True)` works.
   
   **How to reproduce it:**
   ```
   from airflow.operators.bash_operator import BashOperator
   catalog_variable = Variable.get("csba-catalog", deserialize_json=True)
   catalog= catalog_variable["sandbox"]
   
   default_args = {
       'owner': 'airflow',
       'depends_on_past': False,
       'start_date': datetime(2019, 6, 25),
   }
   
   with DAG(
       dag_id="reproduce_bug",
       default_args=default_args,
       schedule_interval=None,
       max_active_runs=1,
       catchup=False
   ) as dag:
   
       t1 = BashOperator(
           task_id='jinja',
           bash_command='echo {{ var.json.csba-catalog.sandbox }}',
           dag=dag,
       )
       t2 = BashOperator(
           task_id='variable_get',
           bash_command=f'echo {catalog}',
           dag=dag,
       )
   ```
   
   Gives:
   **task_id='jinja':**
   ![Screen Shot 2020-05-03 at 11 54 54](https://user-images.githubusercontent.com/45845474/80909969-1bf44d80-8d35-11ea-8f4f-b684923c4a99.png)
    **task_id='variable_get':**
   ![Screen Shot 2020-05-03 at 11 55 06](https://user-images.githubusercontent.com/45845474/80909974-21ea2e80-8d35-11ea-900a-fae6bcc2409f.png)
   
   
   **more info:**
   If I remove the hyphen:
   ![Screen Shot 2020-05-03 at 11 59 10](https://user-images.githubusercontent.com/45845474/80910050-a0df6700-8d35-11ea-9265-4af918979818.png)
   Both works.
   
   
   **Required fix**
   It should work in both cases. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] feluelle commented on issue #8688: accessing Json variable jinja doesn't work when variable has hyphen

Posted by GitBox <gi...@apache.org>.
feluelle commented on issue #8688:
URL: https://github.com/apache/airflow/issues/8688#issuecomment-659367130


   Also: https://stackoverflow.com/questions/17687534/jinja2-variables-naming-are-variables-naming-restrictions-the-same-as-for-pyth
   
   I could not find an officiial source regarding naming convention in jinja2. :/


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] feluelle closed issue #8688: accessing Json variable jinja doesn't work when variable has hyphen

Posted by GitBox <gi...@apache.org>.
feluelle closed issue #8688:
URL: https://github.com/apache/airflow/issues/8688


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] feluelle edited a comment on issue #8688: accessing Json variable jinja doesn't work when variable has hyphen

Posted by GitBox <gi...@apache.org>.
feluelle edited a comment on issue #8688:
URL: https://github.com/apache/airflow/issues/8688#issuecomment-659367130


   Also: https://stackoverflow.com/questions/17687534/jinja2-variables-naming-are-variables-naming-restrictions-the-same-as-for-pyth
   
   I could not find an official source regarding naming convention in jinja2. :/


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] feluelle commented on issue #8688: accessing Json variable jinja doesn't work when variable has hyphen

Posted by GitBox <gi...@apache.org>.
feluelle commented on issue #8688:
URL: https://github.com/apache/airflow/issues/8688#issuecomment-660872323


   Agree this isn't an Airflow Issue.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #8688: accessing Json variable jinja doesn't work when variable has hyphen

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #8688:
URL: https://github.com/apache/airflow/issues/8688#issuecomment-660838955


   > Can you try `{{ var.json.get("csba-catalog").sandbox }}` and see if that works.
   > 
   > Source: https://stackoverflow.com/questions/32744047/parsing-a-token-with-hyphen-in-jinja
   
   I worked around by not using hyphen changed it to underscore :)
   so this isn't really an Airflow issue. Shall I closed this?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] feluelle commented on issue #8688: accessing Json variable jinja doesn't work when variable has hyphen

Posted by GitBox <gi...@apache.org>.
feluelle commented on issue #8688:
URL: https://github.com/apache/airflow/issues/8688#issuecomment-659365603


   Can you try `{{ var.json.get("csba-catalog").sandbox }}` and see if that works.
   
   Source: https://stackoverflow.com/questions/32744047/parsing-a-token-with-hyphen-in-jinja


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8688: accessing Json variable jinja doesn't work when variable has hyphen

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #8688:
URL: https://github.com/apache/airflow/issues/8688#issuecomment-623078354


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org