You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "nolan-redox (via GitHub)" <gi...@apache.org> on 2024/09/27 00:01:21 UTC

[I] ObjectStorage jinja template is converting GCS to a different path type [airflow]

nolan-redox opened a new issue, #42528:
URL: https://github.com/apache/airflow/issues/42528

   ### Apache Airflow version
   
   2.10.1
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   When using the new jinja templating in ObjectStorage, it seems that it's misinterpreting the protocol of the string and prepending a forward slash "/" to my GCS bucket name. 
   
   
   
   ### What you think should happen instead?
   
   When using jinja templating for variable in ObjectStorage, I'm getting the following:
   /gs:/redox-n-airflow-workspace-us-central1 
   
   rather than the correct variable name:
   gs://redox-n-airflow-workspace-us-central1
   
   
   ### How to reproduce
   
   ```python
   import datetime
   import os
   import logging
   
   import pendulum
   
   from airflow.decorators import dag, task
   from airflow.io.path import ObjectStoragePath
   from airflow.models import Variable
   
   logger = logging.getLogger(__name__)
   
   @dag(
       dag_id='objectstorage_jinja_variable',
       schedule=None,
       start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
       catchup=False,
       tags=["testing"]
   )
   def dag_example():
       
       @task
       def print_variable(path: ObjectStoragePath):
           variable_path = Variable.get("gs_analytics_bucket")
           template_in_task_path = ObjectStoragePath("{{var.value.gs_analytics_bucket}}", conn_id="google_cloud_default")
           
           logger.debug(f"Jinja Path: {path}")
           logger.debug(f"Variable Path: {variable_path}")
           logger.debug(f"Template in Task Path: {template_in_task_path}")
           logger.debug(f"ENV VAR: {os.getenv('AIRFLOW_VAR_GS_ANALYTICS_BUCKET')}")
           
       
       
       base = ObjectStoragePath("{{var.value.gs_analytics_bucket}}", conn_id="google_cloud_default")
       print_variable(base)
   
   dag_example()
   ``` 
   
   Logging prints the following:
   ```shell
   DEBUG - Jinja Path: /gs:/redox-n-airflow-workspace-us-central1 
   DEBUG - Variable Path: gs://redox-n-airflow-workspace-us-central1
   DEBUG - Template in Task Path: /{{var.value.gs_analytics_bucket}}
   DEBUG - ENV VAR: gs://redox-n-airflow-workspace-us-central1
   ``` 
   
   
   ### Operating System
   
   linux amd64
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] ObjectStorage jinja template is converting GCS to a different path type [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #42528:
URL: https://github.com/apache/airflow/issues/42528#issuecomment-3449790543

   This issue has been closed because it has not received response from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] ObjectStorage jinja template is converting GCS to a different path type [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #42528: ObjectStorage jinja template is converting GCS to a different path type
URL: https://github.com/apache/airflow/issues/42528


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] ObjectStorage jinja template is converting GCS to a different path type [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #42528:
URL: https://github.com/apache/airflow/issues/42528#issuecomment-2378141531

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] ObjectStorage jinja template is converting GCS to a different path type [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #42528:
URL: https://github.com/apache/airflow/issues/42528#issuecomment-2378292544

   cc @bolkedebruin 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] ObjectStorage jinja template is converting GCS to a different path type [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #42528:
URL: https://github.com/apache/airflow/issues/42528#issuecomment-3341324443

   This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org