You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jgoedeke (via GitHub)" <gi...@apache.org> on 2025/11/10 15:08:10 UTC

[I] Asset extra metadata not persisted: extra field always empty via API/DB for emitted asset alias events [airflow]

jgoedeke opened a new issue, #58146:
URL: https://github.com/apache/airflow/issues/58146

   ### Apache Airflow version
   
   3.1.2
   
   ### If "Other Airflow 2/3 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   When emitting an Asset with a non-empty `extra` dictionary (using the standard Asset/AssetAlias machinery in Airflow 3.1.2), the resulting asset has an empty `extra` field in the database and API, despite being created with extra metadata.
   
   ```python
   from datetime import UTC, datetime
   
   from airflow.providers.standard.operators.python import PythonOperator
   from airflow.sdk import Asset, AssetAlias, Metadata, dag, task
   
   first_alias = AssetAlias(
       name='a.first.data.alias',
   )
   first_asset = Asset(
       name='a.first.data.asset',
       uri='s3://bucket/mydata.csv',
       extra={'path': '/some/local/path.txt'},
   )
   
   
   @dag(
       schedule=None,
       tags=['test'],
   )
   def test_alias_producer_dag():
       def produce_asset():
           yield Metadata(
               first_asset,
               {'timestamp': datetime.now(tz=UTC).isoformat()},
               first_alias,
           )
   
       PythonOperator(
           task_id='produce_asset',
           python_callable=produce_asset,
           outlets=[first_alias],
       )
   
   
   @dag(
       schedule=[first_alias],
       tags=['test'],
   )
   def test_alias_consumer_dag():
       @task(inlets=[first_alias])
       def process_asset(triggering_asset_events: dict):
           for event in triggering_asset_events:
               print(f'Event: {event}')
   
       process_asset()
   
   
   test_alias_producer_dag()
   test_alias_consumer_dag()
   
   ```
   
   Task log:
   ```
   [2025-11-10 14:54:24] INFO - Event: Asset(name='a.first.data.asset', uri='s3://bucket/mydata.csv', group='asset', extra={}, watchers=[]) source=task.stdout
   [2025-11-10 14:54:24] INFO - Event: AssetAlias(name='a.first.data.alias', group='asset') source=task.stdout
   ```
   
   
   Example API response (assets endpoint):
   ```
   {
     "id": 116,
     "name": "a.first.data.asset",
     "uri": "s3://bucket/mydata.csv",
     "group": "",
     "extra": {},
     ...
   }
   ```
   
   ### What you think should happen instead?
   
   The `extra` dictionary provided at Asset creation should be saved and returned by the API and database. Assets should always include their extra metadata if set.
   
   ### How to reproduce
   
   See example
   
   ### Operating System
   
   docker
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Other Docker-based deployment
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   Can not verify Asset extra via the UI because of #57566
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Asset extra metadata not persisted: extra field always empty via API/DB for emitted asset alias events [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on issue #58146:
URL: https://github.com/apache/airflow/issues/58146#issuecomment-3521393042

   for https://github.com/apache/airflow/blob/3c823ecd20d5ca347f4120c74b2fc64f4d6d8773/airflow-core/src/airflow/models/taskinstance.py#L1428, yes, we'll need to do that
   
   for https://github.com/apache/airflow/blob/3c823ecd20d5ca347f4120c74b2fc64f4d6d8773/airflow-core/src/airflow/models/taskinstance.py#L1428, no, it's expected. We won't create an asset event if the asset is not active. (not consumed by some Dag)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Asset extra metadata not persisted: extra field always empty via API/DB for emitted asset alias events [airflow]

Posted by "mykola-shyshov (via GitHub)" <gi...@apache.org>.
mykola-shyshov commented on issue #58146:
URL: https://github.com/apache/airflow/issues/58146#issuecomment-3517663638

   @Lee-W good progress. check, please, should we persist in db asset [extras](https://github.com/apache/airflow/blob/3c823ecd20d5ca347f4120c74b2fc64f4d6d8773/airflow-core/src/airflow/models/taskinstance.py#L1428) extras as well.
   and what may confuse as well is [warning](https://github.com/apache/airflow/blob/9ba09f3e07206bfcef7c9b6fff80b7185999a61e/airflow-core/src/airflow/assets/manager.py#L138). if we are okay to persist "dynamically" than warning is weird. Actually as for me not good method invocations (checks on None). maybe better to use something like `ensureAssetModel`.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Asset extra metadata not persisted: extra field always empty via API/DB for emitted asset alias events [airflow]

Posted by "jgoedeke (via GitHub)" <gi...@apache.org>.
jgoedeke commented on issue #58146:
URL: https://github.com/apache/airflow/issues/58146#issuecomment-3515306426

   @jroachgolf84 I just started working with asset aware scheduling so I can´t tell for sure. I initially thought it did work under a previous version, but I may have mixed it up with directly emitted Assets.
   
   
   According to the docs it should work: https://github.com/apache/airflow/blob/main/airflow-core/docs/authoring-and-scheduling/assets.rst?plain=1#L93-L107
   
   
   Thanks for taking care of this @Lee-W.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Asset extra metadata not persisted: extra field always empty via API/DB for emitted asset alias events [airflow]

Posted by "mykola-shyshov (via GitHub)" <gi...@apache.org>.
mykola-shyshov commented on issue #58146:
URL: https://github.com/apache/airflow/issues/58146#issuecomment-3522619184

   you provided same links 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Asset extra metadata not persisted: extra field always empty via API/DB for emitted asset alias events [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on issue #58146:
URL: https://github.com/apache/airflow/issues/58146#issuecomment-3514532150

   I doubt it's a regression, but I've already started working on this one https://github.com/apache/airflow/pull/58038


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Asset extra metadata not persisted: extra field always empty via API/DB for emitted asset alias events [airflow]

Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #58146:
URL: https://github.com/apache/airflow/issues/58146#issuecomment-3513806704

   @jgoedeke - was this something that worked in a previous version?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Asset extra metadata not persisted: extra field always empty via API/DB for emitted asset alias events [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W closed issue #58146: Asset extra metadata not persisted: extra field always empty via API/DB for emitted asset alias events
URL: https://github.com/apache/airflow/issues/58146


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Asset extra metadata not persisted: extra field always empty via API/DB for emitted asset alias events [airflow]

Posted by "jgoedeke (via GitHub)" <gi...@apache.org>.
jgoedeke commented on issue #58146:
URL: https://github.com/apache/airflow/issues/58146#issuecomment-3512332922

   @Lee-W could this be a regression in 3.1.2 because of #57352 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Asset extra metadata not persisted: extra field always empty via API/DB for emitted asset alias events [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on issue #58146:
URL: https://github.com/apache/airflow/issues/58146#issuecomment-3524575631

   just updated🤦‍♂️


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Asset extra metadata not persisted: extra field always empty via API/DB for emitted asset alias events [airflow]

Posted by "mykola-shyshov (via GitHub)" <gi...@apache.org>.
mykola-shyshov commented on issue #58146:
URL: https://github.com/apache/airflow/issues/58146#issuecomment-3513127670

   I will investigate


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org