You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "nolan-redox (via GitHub)" <gi...@apache.org> on 2024/09/27 00:01:21 UTC
[I] ObjectStorage jinja template is converting GCS to a different path type [airflow]
nolan-redox opened a new issue, #42528:
URL: https://github.com/apache/airflow/issues/42528
### Apache Airflow version
2.10.1
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
When using the new jinja templating in ObjectStorage, it seems that it's misinterpreting the protocol of the string and prepending a forward slash "/" to my GCS bucket name.
### What you think should happen instead?
When using jinja templating for variable in ObjectStorage, I'm getting the following:
/gs:/redox-n-airflow-workspace-us-central1
rather than the correct variable name:
gs://redox-n-airflow-workspace-us-central1
### How to reproduce
```python
import datetime
import os
import logging
import pendulum
from airflow.decorators import dag, task
from airflow.io.path import ObjectStoragePath
from airflow.models import Variable
logger = logging.getLogger(__name__)
@dag(
dag_id='objectstorage_jinja_variable',
schedule=None,
start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
catchup=False,
tags=["testing"]
)
def dag_example():
@task
def print_variable(path: ObjectStoragePath):
variable_path = Variable.get("gs_analytics_bucket")
template_in_task_path = ObjectStoragePath("{{var.value.gs_analytics_bucket}}", conn_id="google_cloud_default")
logger.debug(f"Jinja Path: {path}")
logger.debug(f"Variable Path: {variable_path}")
logger.debug(f"Template in Task Path: {template_in_task_path}")
logger.debug(f"ENV VAR: {os.getenv('AIRFLOW_VAR_GS_ANALYTICS_BUCKET')}")
base = ObjectStoragePath("{{var.value.gs_analytics_bucket}}", conn_id="google_cloud_default")
print_variable(base)
dag_example()
```
Logging prints the following:
```shell
DEBUG - Jinja Path: /gs:/redox-n-airflow-workspace-us-central1
DEBUG - Variable Path: gs://redox-n-airflow-workspace-us-central1
DEBUG - Template in Task Path: /{{var.value.gs_analytics_bucket}}
DEBUG - ENV VAR: gs://redox-n-airflow-workspace-us-central1
```
### Operating System
linux amd64
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ObjectStorage jinja template is converting GCS to a different path type [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #42528:
URL: https://github.com/apache/airflow/issues/42528#issuecomment-3449790543
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ObjectStorage jinja template is converting GCS to a different path type [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #42528: ObjectStorage jinja template is converting GCS to a different path type
URL: https://github.com/apache/airflow/issues/42528
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ObjectStorage jinja template is converting GCS to a different path type [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #42528:
URL: https://github.com/apache/airflow/issues/42528#issuecomment-2378141531
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ObjectStorage jinja template is converting GCS to a different path type [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #42528:
URL: https://github.com/apache/airflow/issues/42528#issuecomment-2378292544
cc @bolkedebruin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ObjectStorage jinja template is converting GCS to a different path type [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #42528:
URL: https://github.com/apache/airflow/issues/42528#issuecomment-3341324443
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org