You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/07 10:26:34 UTC

[GitHub] [airflow] dinowernli opened a new issue #21381: Databricks: sensor for Delta events

dinowernli opened a new issue #21381:
URL: https://github.com/apache/airflow/issues/21381


   ### Description
   
   Delta Lake has a native python library ([delta-io.github.io/delta-rs/python/](https://delta-io.github.io/delta-rs/python/)). This could be useful for use as a [sensor](https://airflow.apache.org/docs/apache-airflow/stable/concepts/sensors.html) or the newer [Deferrable Operators & Triggers](https://airflow.apache.org/docs/apache-airflow/stable/concepts/deferring.html) to wait until a table or partition is updated, similarly to [hive_partition_sensor](https://airflow.apache.org/docs/apache-airflow-providers-apache-hive/stable/_api/airflow/providers/apache/hive/sensors/hive_partition/index.html#module-airflow.providers.apache.hive.sensors.hive_partition) or [metastore_partition_sensor](https://airflow.apache.org/docs/apache-airflow-providers-apache-hive/stable/_api/airflow/providers/apache/hive/sensors/metastore_partition/index.html#module-airflow.providers.apache.hive.sensors.metastore_partition) or just for reading from Delta Lake directly.
   
   ### Use case/motivation
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] alexott commented on issue #21381: Databricks: sensor for Delta events

Posted by GitBox <gi...@apache.org>.
alexott commented on issue #21381:
URL: https://github.com/apache/airflow/issues/21381#issuecomment-1067762173


   @thcidale0808 what is the status of it? I have a customer that asks for it. If you haven't started, I can pickup it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] thcidale0808 commented on issue #21381: Databricks: sensor for Delta events

Posted by GitBox <gi...@apache.org>.
thcidale0808 commented on issue #21381:
URL: https://github.com/apache/airflow/issues/21381#issuecomment-1070817667


   @alexott , I've implemented and will raise a PR soon. Just finishing some tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] thcidale0808 commented on issue #21381: Databricks: sensor for Delta events

Posted by GitBox <gi...@apache.org>.
thcidale0808 commented on issue #21381:
URL: https://github.com/apache/airflow/issues/21381#issuecomment-1070817667


   @alexott , I've implemented and will raise a PR soon. Just finishing some tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] thcidale0808 commented on issue #21381: Databricks: sensor for Delta events

Posted by GitBox <gi...@apache.org>.
thcidale0808 commented on issue #21381:
URL: https://github.com/apache/airflow/issues/21381#issuecomment-1040884303


   I'm happy to contribute with this one. Can this be assigned to me?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org