You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "eladkal (via GitHub)" <gi...@apache.org> on 2023/05/20 12:06:31 UTC

[GitHub] [airflow] eladkal opened a new issue, #31438: Add GluePartitionSensor

eladkal opened a new issue, #31438:
URL: https://github.com/apache/airflow/issues/31438

   ### Body
   
   Sensor that waits for a specific partition(s) in a table.
   Can be constructed on top of boto3 [Glue.Client.get_partitions(**kwargs)](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue/client/get_partitions.html)
   
   Need to support the following use cases:
   1. Waiting for specific partition
   2. Given range of partitions wait for at least one partition present in range.
   
   
   ### Committer
   
   - [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] RachitSharma2001 commented on issue #31438: Add GluePartitionSensor

Posted by "RachitSharma2001 (via GitHub)" <gi...@apache.org>.
RachitSharma2001 commented on issue #31438:
URL: https://github.com/apache/airflow/issues/31438#issuecomment-1555959948

   I would love to work on this if that is ok.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] RachitSharma2001 commented on issue #31438: Add GluePartitionSensor

Posted by "RachitSharma2001 (via GitHub)" <gi...@apache.org>.
RachitSharma2001 commented on issue #31438:
URL: https://github.com/apache/airflow/issues/31438#issuecomment-1555973848

   Looking at the codebase, I see that there exists a `GlueCatalogPartitionSensor` already. This waits for a specific partition to show up in a Glue table. Would the `GluePartitionSensor` just be a more powerful version of this, as it would be able to handle a range of partitions as opposed to just one partition?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #31438: Add GluePartitionSensor

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #31438:
URL: https://github.com/apache/airflow/issues/31438#issuecomment-1556111295

   > Looking at the codebase, I see that there exists a GlueCatalogPartitionSensor already. This waits for a specific partition to show up in a Glue table. Would the GluePartitionSensor just be a more powerful version of this, as it would be able to handle a range of partitions as opposed to just one partition?
   
   ah yeah!
   I looked under `glue.py` rather than under `glue_catalog.py`
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal closed issue #31438: Add GluePartitionSensor

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #31438: Add GluePartitionSensor
URL: https://github.com/apache/airflow/issues/31438


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org