You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/19 15:41:34 UTC
[GitHub] [airflow] awildturtok opened a new issue #17085: Configuration validation for run-configs
awildturtok opened a new issue #17085:
URL: https://github.com/apache/airflow/issues/17085
**Description**
We make extensive use of Run-Configs in our project. They are a great feature, however misconfigurations happen which is why we're slowly establishing nodes that validate the input run-config before starting processing. This is somewhat problematic as a faulty run-config still produces a run which I'd like to avoid out of sanitation purposes. I'd love to be able to register a validation method on a dag that will validate the run config. And deny the dag-run, possibly giving the user feedback.
This would tighten the feedback loop for our users, who sometimes have to wade through multiple lines of logging to just find that they have a typo in a module name.
**Are you willing to submit a PR?**
I have the time/budget but not the knowledge of either Python-dev nor the Airflow Codebase in detail to create a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #17085:
URL: https://github.com/apache/airflow/issues/17085#issuecomment-944872558
closed by https://github.com/apache/airflow/pull/17100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on issue #17085:
URL: https://github.com/apache/airflow/issues/17085#issuecomment-882834756
cc @msumit -- Sumit has implemented something already for it :)
He will be opening a PR soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on issue #17085:
URL: https://github.com/apache/airflow/issues/17085#issuecomment-882834756
cc @msumit -- Sumit has implemented something already for it :)
He will be opening a PR soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #17085:
URL: https://github.com/apache/airflow/issues/17085#issuecomment-882834756
cc @msumit -- Sumit has implemented something already for it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] awildturtok commented on issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
awildturtok commented on issue #17085:
URL: https://github.com/apache/airflow/issues/17085#issuecomment-883155888
> What kind of validation do you have implemented? I can imagine you could check the structure as well as content of the configuration.
We mostly look for required keys and their contents. Some also derive from the content of normal variables (in that case validating the correct usage of datasource names and names of customers), this is pretty important as airflow is still pretty young in our team so we restructure/refactor it often to better align it with our application.
JsonSchema sounds like a neat Idea! the way I see it, I'd probably have to define some static validations and generate some more advanced at run time.
I hope you guys can keep up this pace from Issue to PR 🙃
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] awildturtok commented on issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
awildturtok commented on issue #17085:
URL: https://github.com/apache/airflow/issues/17085#issuecomment-883155888
> What kind of validation do you have implemented? I can imagine you could check the structure as well as content of the configuration.
We mostly look for required keys and their contents. Some also derive from the content of normal variables (in that case validating the correct usage of datasource names and names of customers), this is pretty important as airflow is still pretty young in our team so we restructure/refactor it often to better align it with our application.
JsonSchema sounds like a neat Idea! the way I see it, I'd probably have to define some static validations and generate some more advanced at run time.
I hope you guys can keep up this pace from Issue to PR 🙃
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] awildturtok commented on issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
awildturtok commented on issue #17085:
URL: https://github.com/apache/airflow/issues/17085#issuecomment-883155888
> What kind of validation do you have implemented? I can imagine you could check the structure as well as content of the configuration.
We mostly look for required keys and their contents. Some also derive from the content of normal variables (in that case validating the correct usage of datasource names and names of customers), this is pretty important as airflow is still pretty young in our team so we restructure/refactor it often to better align it with our application.
JsonSchema sounds like a neat Idea! the way I see it, I'd probably have to define some static validations and generate some more advanced at run time.
I hope you guys can keep up this pace from Issue to PR 🙃
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17085:
URL: https://github.com/apache/airflow/issues/17085#issuecomment-882767008
Happy to guide you with the implementation BTW.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on issue #17085:
URL: https://github.com/apache/airflow/issues/17085#issuecomment-882834756
cc @msumit -- Sumit has implemented something already for it :)
He will be opening a PR soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17085:
URL: https://github.com/apache/airflow/issues/17085#issuecomment-882765469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17085:
URL: https://github.com/apache/airflow/issues/17085#issuecomment-882765469
I think it's a good idea. What kind of validation do you have implemented? I can imagine you could check the structure as well as content of the configuration. I think both could be achieved by simply specifying a jsonschema that the run-config should validate with. We already use jsonschema for a few things in Airflow so we would not even have to add any dependency for that.
Also, if we have the schema we could add the monaco-editor (MIT licenced javascript editor which powers VS code)0 to our UI. It seems to support json-schema for json out-of-the-box, which could give not only validation as-you-type, but also autocompletion: See here https://microsoft.github.io/monaco-editor .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17085:
URL: https://github.com/apache/airflow/issues/17085#issuecomment-882765469
I think it's a good idea. What kind of validation do you have implemented? I can imagine you could check the structure as well as content of the configuration. I think both could be achieved by simply specifying a jsonschema that the run-config should validate with. We already use jsonschema for a few things in Airflow so we would not even have to add any dependency for that.
Also, if we have the schema we could add the monaco-editor (MIT licenced javascript editor which powers VS code)0 to our UI. It seems to support json-schema for json out-of-the-box, which could give not only validation as-you-type, but also autocompletion: See here https://microsoft.github.io/monaco-editor .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17085:
URL: https://github.com/apache/airflow/issues/17085#issuecomment-882765469
I think it's a good idea. What kind of validation do you have implemented? I can imagine you could check the structure as well as content of the configuration. I think both could be achieved by simply specifying a jsonschema that the run-config should validate with. We already use jsonschema for a few things in Airflow so we would not even have to add any dependency for that.
Also, if we have the schema we could add the monaco-editor (MIT licenced javascript editor which powers VS code)0 to our UI. It seems to support json-schema for json out-of-the-box, which could give not only validation as-you-type, but also autocompletion: See here https://microsoft.github.io/monaco-editor .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17085:
URL: https://github.com/apache/airflow/issues/17085#issuecomment-882765469
I think it's a good idea. What kind of validation do you have implemented? I can imagine you could check the structure as well as content of the configuration. I think both could be achieved by simply specifying a jsonschema that the run-config should validate with. We already use jsonschema for a few things in Airflow so we would not even have to add any dependency for that.
Also, if we have the schema we could add the monaco-editor (MIT licenced javascript editor which powers VS code in our UI). It seems to support json-schema for json out-of-the-box, which could give not only validation as-you-type, but also autocompletion: See here https://microsoft.github.io/monaco-editor .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #17085:
URL: https://github.com/apache/airflow/issues/17085#issuecomment-882834756
cc @msumit -- Sumit has implemented something already for it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17085:
URL: https://github.com/apache/airflow/issues/17085#issuecomment-882765469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #17085:
URL: https://github.com/apache/airflow/issues/17085#issuecomment-882834756
cc @msumit -- Sumit has implemented something already for it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #17085: Configuration validation for run-configs
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #17085:
URL: https://github.com/apache/airflow/issues/17085
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org