You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vatsrahul1001 (via GitHub)" <gi...@apache.org> on 2025/04/12 08:54:01 UTC
[I] Example tutorial_objectstorage DAG failing [airflow]
vatsrahul1001 opened a new issue, #49149:
URL: https://github.com/apache/airflow/issues/49149
### Apache Airflow version
3.0.0
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Example DAG failing error
DAG CODE: https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/tutorial_objectstorage.py
`KeyError: "Only a column name can be used for the key in a dtype mappings argument. 'fmisid' not found in columns."`
<img width="1728" alt="Image" src="https://github.com/user-attachments/assets/3cffd26d-fa6a-4d69-846e-4b8f36f736d2" />
### What you think should happen instead?
_No response_
### How to reproduce
1. Run `breeze start-airflow --executor CeleryExecutor --backend postgres --load-example-dag`
2. Run `tutorial_objectstorage`
### Operating System
Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Example tutorial_objectstorage DAG failing [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #49149:
URL: https://github.com/apache/airflow/issues/49149#issuecomment-2827616380
Is this still an issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Example tutorial_objectstorage DAG failing [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #49149:
URL: https://github.com/apache/airflow/issues/49149#issuecomment-2896716884
Closing fixed in [50828](https://github.com/apache/airflow/pull/50828)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Example tutorial_objectstorage DAG failing [airflow]
Posted by "Yonggege123 (via GitHub)" <gi...@apache.org>.
Yonggege123 commented on issue #49149:
URL: https://github.com/apache/airflow/issues/49149#issuecomment-2798780268
程序员必备！🚀 **code.mayoubang.cn** 免费 AI 代码工具，助你效率翻倍，告别加班！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Example tutorial_objectstorage DAG failing [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #49149:
URL: https://github.com/apache/airflow/issues/49149#issuecomment-2798754254
I see the issue here its failing at `conn.execute(f"CREATE OR REPLACE TABLE airquality_urban AS SELECT * FROM read_parquet('{path}')")` as path in below xcom have `aws_default` which does not exits in bucket
If I pass exact path `conn.execute(f"CREATE OR REPLACE TABLE airquality_urban AS SELECT * FROM read_parquet('
s3://airflow-tutorial-data/air_quality_20250412.parquet')")` it works.
Maybe we should not add connection details to the path. Also the way XCOM is being displayed could be better
`{'_data__': {'path': 's3://aws_default@airflow-tutorial-data/air_quality_20250412.parquet', 'kwargs': {}, 'conn_id': 'aws_default'}, '__version__': 1, '__classname__': 'airflow.sdk.io.path.ObjectStoragePath'}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Example tutorial_objectstorage DAG failing [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #49149:
URL: https://github.com/apache/airflow/issues/49149#issuecomment-3145302974
> left comment at [#39313 (comment)](https://github.com/apache/airflow/pull/39313#issuecomment-3145186252)
Actuailly @simi -> it would be better if you open a new issue for tha.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Example tutorial_objectstorage DAG failing [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #49149:
URL: https://github.com/apache/airflow/issues/49149#issuecomment-2801340551
Regarding this same behaviour is observed in 2.10.5 as well changing priority
> I see the issue here its failing at conn.execute(f"CREATE OR REPLACE TABLE airquality_urban AS SELECT * FROM read_parquet('{path}')") as path in below xcom have aws_default which does not exits in bucket`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Example tutorial_objectstorage DAG failing [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 closed issue #49149: Example tutorial_objectstorage DAG failing
URL: https://github.com/apache/airflow/issues/49149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Example tutorial_objectstorage DAG failing [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #49149:
URL: https://github.com/apache/airflow/issues/49149#issuecomment-2798744896
Looks like API(https://opendata.fmi.fi/timeseries) being used in Example DAG is returning 400. We might need to use something else.
<img width="767" alt="Image" src="https://github.com/user-attachments/assets/230d57d6-4ee2-4460-9cf3-5cef0c49ce4d" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Example tutorial_objectstorage DAG failing [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #49149:
URL: https://github.com/apache/airflow/issues/49149#issuecomment-2800685548
cc: @kaxil
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Example tutorial_objectstorage DAG failing [airflow]
Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on issue #49149:
URL: https://github.com/apache/airflow/issues/49149#issuecomment-3145186860
left comment at https://github.com/apache/airflow/pull/39313#issuecomment-3145186252
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Example tutorial_objectstorage DAG failing [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on issue #49149:
URL: https://github.com/apache/airflow/issues/49149#issuecomment-2802653825
@vatsrahul1001 an update, discussed with the team on this, i will try to take a stab at it tomorrow and try to squeeze it in for RC timeline. But im talking about the xcom issue. Could you create an issue for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Example tutorial_objectstorage DAG failing [airflow]
Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on issue #49149:
URL: https://github.com/apache/airflow/issues/49149#issuecomment-3145123939
The reported issue with `conn_id` being part of `ObjectStoragePath` in `__str__` form is still present @vatsrahul1001 and breaks DuckDB usage as explained at https://airflow.apache.org/docs/apache-airflow/stable/tutorial/objectstorage.html#analyzing-the-data-with-duckdb.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Example tutorial_objectstorage DAG failing [airflow]
Posted by "simi (via GitHub)" <gi...@apache.org>.
simi commented on issue #49149:
URL: https://github.com/apache/airflow/issues/49149#issuecomment-3145311972
> > left comment at [#39313 (comment)](https://github.com/apache/airflow/pull/39313#issuecomment-3145186252)
>
> Actuailly [@simi](https://github.com/simi) -> it would be better if you open a new issue for tha.
I think I have enough info, I'll open PR to fix with reasoning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Example tutorial_objectstorage DAG failing [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #49149: Example tutorial_objectstorage DAG failing
URL: https://github.com/apache/airflow/issues/49149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Example tutorial_objectstorage DAG failing [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #49149:
URL: https://github.com/apache/airflow/issues/49149#issuecomment-2798752019
Ok, so I have updated the DAG with a new API to test this out, and I see issue below.
1. Analyze task is failing with below error
```
[2025-04-12, 09:09:00] ERROR - Task failed with exception: source="task"
Error: ParamValidationError: Parameter validation failed:
Invalid bucket name "aws_default@airflow-tutorial-data": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$"
```
<img width="1625" alt="Image" src="https://github.com/user-attachments/assets/2caaf8e9-d1e0-4941-a6b7-dde60c3fc84d" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Example tutorial_objectstorage DAG failing [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #49149:
URL: https://github.com/apache/airflow/issues/49149#issuecomment-2849921351
I think yes. Let me chek on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org