You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jedcunningham (via GitHub)" <gi...@apache.org> on 2025/01/14 17:04:44 UTC
[I] Add `bundle_name` and `latest_bundle_version` to `dags list` command [airflow]
jedcunningham opened a new issue, #45646:
URL: https://github.com/apache/airflow/issues/45646
### Body
We need to add `bundle_name` and `latest_bundle_version` to the `dags list` command
### Committer
- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `bundle_name` and `latest_bundle_version` to `dags list` command [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on issue #45646:
URL: https://github.com/apache/airflow/issues/45646#issuecomment-2617885733
Sounds like your db isn't migrated.
Here is an example. First you need to set up a connection:
```
AIRFLOW_CONN_AIRFLOW_GIT_CONN='{"host": "https://github.com/apache/airflow"}'
```
Then, add this to your config:
```
[dag_processor]
dag_bundle_config_list = [
{
"name": "airflow_example_dags",
"classpath": "airflow.dag_processing.bundles.git.GitDagBundle",
"kwargs": {"tracking_ref": "main", "git_conn_id": "airflow_git_conn", subdir="airflow/example_dags"}
}
]
```
However, something is broken right now and the processor is getting stuck on servicing the sockets. I'll have to dig into this tomorrow. In the meantime, you can safely go into the db and just put a string in the `dag.bundle_version` column. That will unblock you for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `bundle_name` and `latest_bundle_version` to `dags list` command [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham closed issue #45646: Add `bundle_name` and `latest_bundle_version` to `dags list` command
URL: https://github.com/apache/airflow/issues/45646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `bundle_name` and `latest_bundle_version` to `dags list` command [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on issue #45646:
URL: https://github.com/apache/airflow/issues/45646#issuecomment-2598715365
You set them in [config](https://github.com/apache/airflow/blob/051e617e0d7d0ebb995cb98063709350f279963c/airflow/config_templates/config.yml#L2673-L2706). The default and example there should get you going, but feel free to reach back out if you get stuck!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `bundle_name` and `latest_bundle_version` to `dags list` command [airflow]
Posted by "ambika-garg (via GitHub)" <gi...@apache.org>.
ambika-garg commented on issue #45646:
URL: https://github.com/apache/airflow/issues/45646#issuecomment-2599708062
I have created a draft PR (https://github.com/apache/airflow/pull/45779) that adds the bundle name as an argument to the `dags list` command. Could you please review it to ensure it's on the right track? Once confirmed, I will proceed to add the `bundle_version` argument as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `bundle_name` and `latest_bundle_version` to `dags list` command [airflow]
Posted by "ambika-garg (via GitHub)" <gi...@apache.org>.
ambika-garg commented on issue #45646:
URL: https://github.com/apache/airflow/issues/45646#issuecomment-2607354561
Once we are good with `list_dags` command, we can extend the output functionality to the other CLI commands as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `bundle_name` and `latest_bundle_version` to `dags list` command [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on issue #45646:
URL: https://github.com/apache/airflow/issues/45646#issuecomment-2614723538
> 1. I was able to display the bundle name and version in the output as shown in the image. Let me know if this aligns with your expectations. You can also review the changes in the [draft PR.](https://github.com/apache/airflow/pull/45779)
>
>
> <img alt="Image" width="714" src="https://private-user-images.githubusercontent.com/70703123/405475075-2772b1ff-ea7b-4ea0-902a-e3719dc01962.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mzc5NDI5NDEsIm5iZiI6MTczNzk0MjY0MSwicGF0aCI6Ii83MDcwMzEyMy80MDU0NzUwNzUtMjc3MmIxZmYtZWE3Yi00ZWEwLTkwMmEtZTM3MTlkYzAxOTYyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAxMjclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMTI3VDAxNTA0MVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWRlZDFlZGQxNWQ2ZmZlNzViYWMyYzc5OGFiYWM3NTNlOTRjM2M1OWM3NDEyYTljYzg4YTZkZWUxZjZkYjIyNDAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.jrq2T1IAEo6m8pu4hXklLkcZYXEQcfoGvk7TUVdT6m0">
Yep, exactly!
> 2. I tested this on LocalBundle, but since it doesn’t support versions, I wanted to test it on GitBundle. However, I encountered an error. I updated the config.yml with the following changes but ran into issues. Could you guide me on whether any specific configurations are required?
Without the error message, it's tough to say. There has been a bit of churn in GitHook and GitDagBundle recently. If you need a working config, let me know and I'll craft one to share.
> 3. As per your suggestion, I attempted to use Dagbag(read_dags_from_db=True). While exploring this approach, I tried using the collect_dags_from_db function from [dagbag](https://github.com/apache/airflow/blob/main/airflow/models/dagbag.py) file, It was unable to load any file from the db.
Hmm, in theory, you can just do:
```
>>> from airflow.models.dagbag import DagBag
>>> bag = DagBag(read_dags_from_db=True)
>>> bag.collect_dags_from_db()
[2025-01-26T19:00:50.726-0700] {dagbag.py:612} INFO - Filling up the DagBag from database
...
>>> bag.dag_ids
['hello', 'mapping_always_short', 'ke_image_override']
```
Did you get any errors? Or it was just empty?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `bundle_name` and `latest_bundle_version` to `dags list` command [airflow]
Posted by "ambika-garg (via GitHub)" <gi...@apache.org>.
ambika-garg commented on issue #45646:
URL: https://github.com/apache/airflow/issues/45646#issuecomment-2606282005
Hi @jedcunningham,
I have a few questions:
1. I was able to display the bundle name and version in the output as shown in the image. Let me know if this aligns with your expectations. You can also review the changes in the [draft PR.](https://github.com/apache/airflow/pull/45779)
<img width="714" alt="Image" src="https://github.com/user-attachments/assets/2772b1ff-ea7b-4ea0-902a-e3719dc01962" />
2. I tested this on LocalBundle, but since it doesn’t support versions, I wanted to test it on GitBundle. However, I encountered an error. I updated the config.yml with the following changes but ran into issues. Could you guide me on whether any specific configurations are required?
3. As per your suggestion, I attempted to use Dagbag(read_dags_from_db=True). While exploring this approach, I tried using the collect_dags_from_db function from [dagbag](https://github.com/apache/airflow/blob/main/airflow/models/dagbag.py) file, It was unable to load any file from the db.
Looking forward to your guidance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `bundle_name` and `latest_bundle_version` to `dags list` command [airflow]
Posted by "ambika-garg (via GitHub)" <gi...@apache.org>.
ambika-garg commented on issue #45646:
URL: https://github.com/apache/airflow/issues/45646#issuecomment-2594275967
Hey @jedcunningham, I can take this up. You can assign it to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `bundle_name` and `latest_bundle_version` to `dags list` command [airflow]
Posted by "ambika-garg (via GitHub)" <gi...@apache.org>.
ambika-garg commented on issue #45646:
URL: https://github.com/apache/airflow/issues/45646#issuecomment-2601031540
Thankyou @jedcunningham for reviewing it, I'll go with the approach enabling `DagBag(read_dags_from_db=True)` and using the added columns in the output of cli command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `bundle_name` and `latest_bundle_version` to `dags list` command [airflow]
Posted by "ambika-garg (via GitHub)" <gi...@apache.org>.
ambika-garg commented on issue #45646:
URL: https://github.com/apache/airflow/issues/45646#issuecomment-2617121312
> Hmm, in theory, you can just do:
>
> ```
> >>> from airflow.models.dagbag import DagBag
> >>> bag = DagBag(read_dags_from_db=True)
> >>> bag.collect_dags_from_db()
> [2025-01-26T19:00:50.726-0700] {dagbag.py:612} INFO - Filling up the DagBag from database
> ...
> >>> bag.dag_ids
> ['hello', 'mapping_always_short', 'ke_image_override']
> ```
>
> Did you get any errors? Or it was just empty?
Hey @jedcunningham , I get the following error upon trying it.
<img width="951" alt="Image" src="https://github.com/user-attachments/assets/d17d782f-6b4f-40b9-b23d-cba5a2e9aebb" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `bundle_name` and `latest_bundle_version` to `dags list` command [airflow]
Posted by "ambika-garg (via GitHub)" <gi...@apache.org>.
ambika-garg commented on issue #45646:
URL: https://github.com/apache/airflow/issues/45646#issuecomment-2617128824
Could you also please share a working configuration for the GithubDagBundle?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add `bundle_name` and `latest_bundle_version` to `dags list` command [airflow]
Posted by "ambika-garg (via GitHub)" <gi...@apache.org>.
ambika-garg commented on issue #45646:
URL: https://github.com/apache/airflow/issues/45646#issuecomment-2598573168
Hey @jedcunningham, I have a quick question about creating DAG Bundles. I noticed several classes under dag_processing/bundles. How can we use them to create bundles locally?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org