You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "RamanDamayeu (via GitHub)" <gi...@apache.org> on 2023/08/29 18:14:19 UTC
[GitHub] [airflow-site] RamanDamayeu opened a new pull request, #857: Adding information about Open Data Discovery (ODD) integration
RamanDamayeu opened a new pull request, #857:
URL: https://github.com/apache/airflow-site/pull/857
Adding information about Open Data Discovery (ODD) to the list of "Tools integrating with Airflow". It leverages Listeners capabilities of Airflow. Implemented with https://github.com/opendatadiscovery/odd-airflow-2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] RamanDamayeu commented on pull request #857: Adding information about Open Data Discovery (ODD) integration
Posted by "RamanDamayeu (via GitHub)" <gi...@apache.org>.
RamanDamayeu commented on PR #857:
URL: https://github.com/apache/airflow-site/pull/857#issuecomment-1698886905
Thanks, very valid point!
Of course, I left a link to the implementation of ODD integration with Airflow in a comment, but I agree that it will not be very easy for users to understand how exactly this platform is connected with Airflow.
Please advise how best to proceed here. The bottom line is that the ODD application implies several components, the main of which is the so-called platform itself. Here is a [link](https://github.com/opendatadiscovery/odd-platform) to a repository with it.
Also for various integrations with the platform (that is, in fact, the components that are mediators between external systems and the platform itself) are implemented as collectors (if we want to implement the pull approach, that is, the components themselves go to the systems to collect meta-information with some periodicity, for example, there are such collectors for a set of databases implemented [here](https://github.com/opendatadiscovery/odd-collector), or for collecting metadata from some of [AWS services]( https://github.com/opendatadiscovery/odd-collector-aws), there is own for [GCP](https://github.com/opendatadiscovery/odd-collector-gcp), for [Azure](https://github.com/opendatadiscovery/odd-collector-azure), etc. with their own repositories) or as adapters, if we want to expand the functionality of some external system and implement the push approach (here just like for the Airflow, this is a repository for versions up to [1.10.15](https://github.com/opendatadiscovery/odd
-airflow-adapter) and for versions >= 2.5.1 here is a new [rep](https://github.com/opendatadiscovery/odd-airflow-2) (uses Listeners for integration), the same approach is used to integrate [Great Expectations](https://github.com/opendatadiscovery/odd-great-expectations), [dbt](https://github.com/opendatadiscovery/odd-dbt), [Spark](https://github.com/opendatadiscovery/odd-spark-adapter)). The push/pull approach is described [here](https://github.com/opendatadiscovery/opendatadiscovery-specification/blob/main/specification/specification.md#push-and-pull-models).
An overview of the architecture we could find here: https://docs.opendatadiscovery.org/architecture
Having said all this, as you say, taking into account that there is a whole set of repositories - and for integration with Airflow we need to use at least two: the platform itself and the adapter - as we need to leave a link on the page should I change it to the adapter for [Airflow versions >= 2.5.1](https://github.com/opendatadiscovery/odd-airflow-2), to [a platform](https://github.com/opendatadiscovery/odd-platform) or maybe to be generally simple to [the git organization of ODD](https://github.com/opendatadiscovery)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #857: Adding information about Open Data Discovery (ODD) integration
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #857:
URL: https://github.com/apache/airflow-site/pull/857#issuecomment-1699083119
I think it's best to link to one of your GitHub repos and have a readme there explaining how to integrate airflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding information about Open Data Discovery (ODD) integration [airflow-site]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #857: Adding information about Open Data Discovery (ODD) integration
URL: https://github.com/apache/airflow-site/pull/857
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #857: Adding information about Open Data Discovery (ODD) integration
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #857:
URL: https://github.com/apache/airflow-site/pull/857#issuecomment-1698016593
Could you change it with the link to the Github project? the link you proposed does not refer to airflow in any way and it's unclear why it would be mentioned on Airflow's ecosystem page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org