You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/18 19:03:18 UTC
[GitHub] [airflow] potiuk opened a new issue #10386: Add list of prerequisites that new provider package should fulfill
potiuk opened a new issue #10386:
URL: https://github.com/apache/airflow/issues/10386
**Description**
We need to have a folder template (ideally) and list of prerequisites that each provider should fulfill when it is created.
**Use case / motivation**
To make it easy to add new provider and to make new contributors aware what is needed in order to add provider. The following should be added:
- [ ] directory structure (possibly automated templates or script to create them)
- [ ] Information about what is required:
- [ ] hooks
- [ ] operators/sensors/transfers
- [ ] unit tests with mocks
- [ ] integration tests if applicable
- [ ] example dags for examples and their use (examples, howtos, system tests)
- [ ] howto guides
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10386: Add list of prerequisites that new provider package should fulfill
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10386:
URL: https://github.com/apache/airflow/issues/10386#issuecomment-749118178
Yeah. The docs in #12921 only describe prerequisites for "custom" providers - which can be structured in whatever way the 3rd-party provider wants them to be. But there are much "higher" requirements for the providers that are part of the Airflow repo (i.e. community-managed ones) - documentation, examples, proper structure and naming, provider.yaml file etc. We cannot expect 3rd-party provider to fulfill all that (and we have no way of enforcing it). But we have very good checks already and review process to require them from all the Airflow providers :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #10386: Add list of prerequisites that new provider package should fulfill
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #10386:
URL: https://github.com/apache/airflow/issues/10386
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10386: Add list of prerequisites that new provider package should fulfill
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10386:
URL: https://github.com/apache/airflow/issues/10386#issuecomment-971936393
Absolutely. Closing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pcandoalmeida commented on issue #10386: Add list of prerequisites that new provider package should fulfill
Posted by GitBox <gi...@apache.org>.
pcandoalmeida commented on issue #10386:
URL: https://github.com/apache/airflow/issues/10386#issuecomment-748114283
Hey @TobKed sorry for the delay on this! @potiuk shall we close this issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] TobKed commented on issue #10386: Add list of prerequisites that new provider package should fulfill
Posted by GitBox <gi...@apache.org>.
TobKed commented on issue #10386:
URL: https://github.com/apache/airflow/issues/10386#issuecomment-748592556
Hi @pcandoalmeida , it is only partly solved. I created draft PR https://github.com/apache/airflow/pull/13193 to push it forward
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #10386: Add list of prerequisites that new provider package should fulfill
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #10386:
URL: https://github.com/apache/airflow/issues/10386#issuecomment-971933239
@potiuk I think https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#developing-community-managed-provider-packages covers it quite nicely. Do you think we need something more?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10386: Add list of prerequisites that new provider package should fulfill
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10386:
URL: https://github.com/apache/airflow/issues/10386#issuecomment-676653158
Ah sorry - did not understand. I think it is mostly a 'CONtRIBUTING.rat' update or rather creating new 'how to create a new provider.rst' file + folder structure explanation( or anyone script creating a new provider with some scaffolding)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #10386: Add list of prerequisites that new provider package should fulfill
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #10386:
URL: https://github.com/apache/airflow/issues/10386#issuecomment-676653158
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] TobKed commented on issue #10386: Add list of prerequisites that new provider package should fulfill
Posted by GitBox <gi...@apache.org>.
TobKed commented on issue #10386:
URL: https://github.com/apache/airflow/issues/10386#issuecomment-741726075
Is it not solved by https://github.com/apache/airflow/pull/12921 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] TobKed commented on issue #10386: Add list of prerequisites that new provider package should fulfill
Posted by GitBox <gi...@apache.org>.
TobKed commented on issue #10386:
URL: https://github.com/apache/airflow/issues/10386#issuecomment-740561273
Any updates for this issue? @pcandoalmeida are you planning to come back working on this? I am happy to help with it.
I opened PR with new operator ([Apache Beam operators](https://github.com/apache/airflow/pull/12814)) and information about prerequisites for new providers will be very helpful.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10386: Add list of prerequisites that new provider package should fulfill
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10386:
URL: https://github.com/apache/airflow/issues/10386#issuecomment-676649940
It's rather about folder structure. We have monorepo - providers are in 'providers' packages/folders - both 'airflow' and 'tests' subfolders and they follow certain structure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pcandoalmeida commented on issue #10386: Add list of prerequisites that new provider package should fulfill
Posted by GitBox <gi...@apache.org>.
pcandoalmeida commented on issue #10386:
URL: https://github.com/apache/airflow/issues/10386#issuecomment-677896367
Thanks for the clarification @potiuk. I can certainly try working on the documentation to begin with if that suits.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] TobKed edited a comment on issue #10386: Add list of prerequisites that new provider package should fulfill
Posted by GitBox <gi...@apache.org>.
TobKed edited a comment on issue #10386:
URL: https://github.com/apache/airflow/issues/10386#issuecomment-748592556
Hi @pcandoalmeida , I think issue should not be closed yet, it is only partly solved. I created draft PR https://github.com/apache/airflow/pull/13193 to push it forward
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pcandoalmeida commented on issue #10386: Add list of prerequisites that new provider package should fulfill
Posted by GitBox <gi...@apache.org>.
pcandoalmeida commented on issue #10386:
URL: https://github.com/apache/airflow/issues/10386#issuecomment-676627091
Hi @potiuk 🙂 I was thinking about taking a look at this; would you mind if I asked to help clarify the issue a bit further? Would this require a Gitub repository PR template update and documentation or something completely different? Apologies if I am missing the obvious here!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org