You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/03/04 16:28:45 UTC
[GitHub] [airflow] potiuk opened a new issue, #29918: Add Triggers to provider information
potiuk opened a new issue, #29918:
URL: https://github.com/apache/airflow/issues/29918
### Body
We do not have triggers information exposed in provider info but some Providerst already expose some (for example cncf.kubernetes).
What needs to be done:
- [ ] add triggers to provider.yaml schema
- [ ] add triggers to provider_info schema
- [ ] expose triggers via Providers Manager
- [ ] add CLI triggers command to `airflow providers`
- [ ] add automated exposure of triggers in the docs
- [ ] check trigge classes are present in provider.yaml in unit tests
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on issue #29918: Add Triggers to provider information
Posted by "Bowrna (via GitHub)" <gi...@apache.org>.
Bowrna commented on issue #29918:
URL: https://github.com/apache/airflow/issues/29918#issuecomment-1454956665
hi @potiuk I don't see trigger information as part of `providers.yaml` for `cncf.kubernetes` file too. Am i looking at the wrong place?
https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/provider.yaml#L76-L112
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on issue #29918: Add Triggers to provider information
Posted by "Bowrna (via GitHub)" <gi...@apache.org>.
Bowrna commented on issue #29918:
URL: https://github.com/apache/airflow/issues/29918#issuecomment-1457805699
I don't think I understood the changes in provider_info_schema
do you mean the below file? what changes has to go into this file to support triggers?
https://github.com/apache/airflow/blob/main/airflow/provider_info.schema.json#L1-L10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29918: Add Triggers to provider information
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29918:
URL: https://github.com/apache/airflow/issues/29918#issuecomment-1459607775
We expose the information there also for connections and other things:
```
"connection-types": {
"type": "array",
"description": "Map of connection types mapped to hook class names.",
"items": {
"type": "object",
"properties": {
"connection-type": {
"description": "Type of connection defined by the provider",
"type": "string"
},
"hook-class-name": {
"description": "Hook class name that implements the connection type",
"type": "string"
}
}
},
"required": ["connection-type", "hook-class-name"]
},
"extra-links": {
"type": "array",
"description": "Operator class names that provide extra link functionality",
"items": {
"type": "string"
}
},
...
```
This is what I call "core extensions" - i.e. the ways how providers are adding implementation of capabilities that Airflow exposes as public interface. It is nicely summarized here (this page is automatically generated based on the provider.yaml information, but also it is exposed for runtime query via CLI (`airflow providers` subcommands such as `hooks`, `auth` etc.)
This information is automatically discovered via ProvidersManager based on the information exposed via "get_provider_info" entrypoint (Providers Manager automatically queries the installed providers to get that information and it uses "provider_info.schema" to know how to query (simply get_provider_info returns a dict that conforms to the schema which is selected subset of information in provider.yaml.
https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/index.html
This is in order to be able to actively "query" the providers and figure out in automated way wha
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29918: Add Triggers to provider information
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29918:
URL: https://github.com/apache/airflow/issues/29918#issuecomment-1455013440
Yep. That's the point. We don't have them there and we need to add them :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29918: Add Triggers to provider information
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29918:
URL: https://github.com/apache/airflow/issues/29918#issuecomment-1454794555
cc: @eladkal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29918: Add Triggers to provider information
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29918:
URL: https://github.com/apache/airflow/issues/29918#issuecomment-1458979592
Those Schemas (JSON schemas) define what is the allowed content of the provider.yaml (provider.yaml.schema.json) and what kind of information can be exposed via provider_info entrypoint in the provider packages (provier_info.schema.json). Both need to be updated to add optional "triggerer" information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #29918: Add Triggers to provider information
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #29918:
URL: https://github.com/apache/airflow/issues/29918#issuecomment-1535924478
@Bowrna are you working on any of the other sub tasks listed in the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29918: Add Triggers to provider information
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29918:
URL: https://github.com/apache/airflow/issues/29918#issuecomment-1689602458
` check trigge classes are present in provider.yaml in unit tests` -> that is already there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on issue #29918: Add Triggers to provider information
Posted by "Bowrna (via GitHub)" <gi...@apache.org>.
Bowrna commented on issue #29918:
URL: https://github.com/apache/airflow/issues/29918#issuecomment-1536106429
I will pick one more task from this list and start working on it before today @eladkal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on issue #29918: Add Triggers to provider information
Posted by "Bowrna (via GitHub)" <gi...@apache.org>.
Bowrna commented on issue #29918:
URL: https://github.com/apache/airflow/issues/29918#issuecomment-1543573974
can expect delay at my side ( for a week) as i am relocating home. sorry @eladkal i thought i could find time in between and check this, but i am wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajastro commented on issue #29918: Add Triggers to provider information
Posted by "pankajastro (via GitHub)" <gi...@apache.org>.
pankajastro commented on issue #29918:
URL: https://github.com/apache/airflow/issues/29918#issuecomment-1547182021
I think we can mark the below task as done PR https://github.com/apache/airflow/pull/30822
> expose triggers via Providers Manager
add CLI triggers command to airflow providers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29918: Add Triggers to provider information
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29918:
URL: https://github.com/apache/airflow/issues/29918#issuecomment-1547424035
Marked :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on issue #29918: Add Triggers to provider information
Posted by "Bowrna (via GitHub)" <gi...@apache.org>.
Bowrna commented on issue #29918:
URL: https://github.com/apache/airflow/issues/29918#issuecomment-1459210017
the `provider_info.schema.json` file doesn't contain information about hooks, sensors, transfers, or operators as such. Does it have to contain `triggers` information alone?
could you explain to me why `triggers` alone have to be included? @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org