You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/18 08:05:10 UTC
[GitHub] [airflow] vagetablechicken opened a new pull request, #25119: Add openmldb provider
vagetablechicken opened a new pull request, #25119:
URL: https://github.com/apache/airflow/pull/25119
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
closes: #25118
Add openmldb provider `apache-airflow-providers-openmldb`: hook,operators and tests. And an example dag in `tests/system/providers`.
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #25119: Add openmldb provider
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #25119:
URL: https://github.com/apache/airflow/pull/25119#discussion_r923175098
##########
airflow/providers/openmldb/provider.yaml:
##########
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+---
+package-name: apache-airflow-providers-openmldb
+name: OpenMLDB
+description: |
+ `OpenMLDB <https://openmldb.ai/>`__
+
+versions:
+ - 1.0.0
+
+dependencies:
+ - apache-airflow>=2.2.0
+ - apache-airflow-providers-http
Review Comment:
Also I thiink @vagetablechicken - you should consider releasing your provider on your own first.
We had a long discussion recently on that very subject and I would like to see a discussion whether you considered releasing it as your own provider. I am not sure if we want to take over the mainteance of OpenMLDB provider in the community.
Before we accept any new provider, there should be a discussion at the devlist where you will refer to the discussion you have and justify that it is better to have the provider in Airlfow rather than relase on your own.
See discussion here:
https://lists.apache.org/thread/6ngq79df7op541gfwntspdtsvzlv1cr6
There are certain limitiations and obligations of those who submit providers to be "community" providers rather than providers managed outside of Airlfow Community. If you release a provider on your own, you can fully control your release schedule and decide on how an when you release - so for product like open MLDB it seems more appropriate to release their provider on their own, unless there is a goo reason to get it in.
You can read about the release process and some of the approach we have for accepting the providers in https://github.com/apache/airflow#release-process-for-providers
I'd strongly encourage you to release the provider on your own first, and only approach contirbution to Airflow when it proves to be popular and useful.
You can add link to the provider of yours at https://airflow.apache.org/ecosystem/ and there are airlfow provider registries available that you can register it too.
If you really want to submit the provider to the community - please start discussion at the devlist, stating why you thnk is good, who are the stakeholders behind it, who from the stakeholder will be maintaing it and what kind of involvement they have and what kind of commitment the stakeholders can make for the new provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vagetablechicken commented on pull request #25119: Add openmldb provider
Posted by GitBox <gi...@apache.org>.
vagetablechicken commented on PR #25119:
URL: https://github.com/apache/airflow/pull/25119#issuecomment-1189733980
> Discussion at the devlist is needed explainig why this is better to be in community rather than outside and what is the commitment to maintain it from the stakeholders if it is submitted.
Thanks for your advice. I'll release openmldb provider on my own. And submit a pr to add link at https://airflow.apache.org/ecosystem/ later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #25119: Add openmldb provider
Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #25119:
URL: https://github.com/apache/airflow/pull/25119#discussion_r923119950
##########
airflow/providers/openmldb/provider.yaml:
##########
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+---
+package-name: apache-airflow-providers-openmldb
+name: OpenMLDB
+description: |
+ `OpenMLDB <https://openmldb.ai/>`__
+
+versions:
+ - 1.0.0
+
+dependencies:
+ - apache-airflow>=2.2.0
+ - apache-airflow-providers-http
Review Comment:
The project has Python SDK
https://openmldb.ai/docs/en/v0.4/quickstart/python_sdk.html
https://pypi.org/project/openmldb/
https://github.com/4paradigm/OpenMLDB
We prefer use Python SDK when existed unless there is a good reason to avoid it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #25119: Add openmldb provider
Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #25119:
URL: https://github.com/apache/airflow/pull/25119#discussion_r923119950
##########
airflow/providers/openmldb/provider.yaml:
##########
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+---
+package-name: apache-airflow-providers-openmldb
+name: OpenMLDB
+description: |
+ `OpenMLDB <https://openmldb.ai/>`__
+
+versions:
+ - 1.0.0
+
+dependencies:
+ - apache-airflow>=2.2.0
+ - apache-airflow-providers-http
Review Comment:
The project has Python SDK
https://openmldb.ai/docs/en/v0.4/quickstart/python_sdk.html
https://pypi.org/project/openmldb/
https://github.com/4paradigm/OpenMLDB
We prefer use official Python SDK when existed unless there is a good reason to avoid it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vagetablechicken commented on a diff in pull request #25119: Add openmldb provider
Posted by GitBox <gi...@apache.org>.
vagetablechicken commented on code in PR #25119:
URL: https://github.com/apache/airflow/pull/25119#discussion_r923215823
##########
airflow/providers/openmldb/provider.yaml:
##########
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+---
+package-name: apache-airflow-providers-openmldb
+name: OpenMLDB
+description: |
+ `OpenMLDB <https://openmldb.ai/>`__
+
+versions:
+ - 1.0.0
+
+dependencies:
+ - apache-airflow>=2.2.0
+ - apache-airflow-providers-http
Review Comment:
> The project has Python SDK https://openmldb.ai/docs/en/v0.4/quickstart/python_sdk.html https://pypi.org/project/openmldb/ https://github.com/4paradigm/OpenMLDB
>
> We prefer use official Python SDK when existed unless there is a good reason to avoid it?
The OpenMLDB python SDK is using the shared lib created by swig. It'll meet many import error when using OpenMLDB python sdk with some complicate systems, e.g. airflow, xgboost. So we prefer the lightweight way(restful api) to connect OpenMLDB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25119: Add openmldb provider
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25119:
URL: https://github.com/apache/airflow/pull/25119#issuecomment-1190105625
I think it's a better choice indeed :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #25119: Add openmldb provider
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on PR #25119:
URL: https://github.com/apache/airflow/pull/25119#issuecomment-1186891646
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vagetablechicken closed pull request #25119: Add openmldb provider
Posted by GitBox <gi...@apache.org>.
vagetablechicken closed pull request #25119: Add openmldb provider
URL: https://github.com/apache/airflow/pull/25119
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org