You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "gyli (via GitHub)" <gi...@apache.org> on 2024/07/23 15:40:06 UTC
[I] Add Apache XTable provider [airflow]
gyli opened a new issue, #40962:
URL: https://github.com/apache/airflow/issues/40962
### Description
Apache XTable translates metadata among datalakes, allowing users to read from datalake with the tools don't have native support.
XTable can be executed with command like
```
java -jar xtable-utilities/target/xtable-utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml [--hadoopConfig hdfs-site.xml] [--convertersConfig converters.yaml] [--icebergCatalogConfig catalog.yaml]
```
An Airflow operator can be created to wrap this command and provide both file and dict input for those XTable config in YAML files.
### Use case/motivation
AWS provides an example [XTableOperator](https://aws.amazon.com/blogs/big-data/run-apache-xtable-on-amazon-mwaa-to-translate-open-table-formats/) for XTable. This blog has good explanation about the Open table formats XTable provides. While this [example operator](https://github.com/aws-samples/apache-xtable-on-aws-samples) is essentially an MVP version, and serves as an MWAA plugin. We can create Apache XTable provider making it available for more Airflow users, and providing more flexible user input.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add Apache XTable provider [airflow]
Posted by "gyli (via GitHub)" <gi...@apache.org>.
gyli commented on issue #40962:
URL: https://github.com/apache/airflow/issues/40962#issuecomment-2246413498
At this stage I'm only thinking about building a `XTableOperator`, similar to the example operator that AWS provides for MWAA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add Apache XTable provider [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on issue #40962:
URL: https://github.com/apache/airflow/issues/40962#issuecomment-2245713247
Apache Airflow is an orchestration tool not a compute engine or ETL tool
if you need to run custom code like this jar , trigger the run with a bashoperator or a KubernetesPodOperator
and if you want a "friendly" Xtable trigger operator
than extand the BashOperator or KubernetesPodOperator
example
```python
class EasyXTableOperator(BashOperator):
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add Apache XTable provider [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on issue #40962:
URL: https://github.com/apache/airflow/issues/40962#issuecomment-2246406663
so you want to add a XTable hook ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add Apache XTable provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #40962: Add Apache XTable provider
URL: https://github.com/apache/airflow/issues/40962
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add Apache XTable provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #40962:
URL: https://github.com/apache/airflow/issues/40962#issuecomment-2246617465
Read https://github.com/apache/airflow/blob/main/PROVIDERS.rst#accepting-new-community-providers about the process on how new providers are accepted here and feel free to follow it. Since this is not an issue or feature - converting that into discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add Apache XTable provider [airflow]
Posted by "gyli (via GitHub)" <gi...@apache.org>.
gyli commented on issue #40962:
URL: https://github.com/apache/airflow/issues/40962#issuecomment-2246388296
Hi @raphaelauv, I agree that such operator is a "friendly" XTable trigger and can be built on top of BashOperator. The example code you showed is exactly how I will build it, but my point here is it should be an Airflow community managed provider.
Just like Airflow also offers Iceberg hook and DatabricksSQLOperator in the corresponding providers, I believe XTable should also be added, as data engineering industry is embracing unified data format. For example, [Microsoft Fabric and OneLake is adopting XTable](https://clouddamcdnprodep.azureedge.net/gdc/gdc5iZCgN/original). I can understand that this Apache project is still in incubating stage, and Airflow might want to hold until it closer to an industry standard. While from a data engineer perspective, such unified datalake format is the way we are heading to, and I don't see a reason asking users to create custom operator for such work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org