You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/16 22:11:09 UTC

[GitHub] [airflow-client-python] hhmahmood opened a new issue, #49: get_dags do not fetch more than 100 dags.

hhmahmood opened a new issue, #49:
URL: https://github.com/apache/airflow-client-python/issues/49

   Hi,
   
   The function does not return more than 100 dags even setting the limit to more than 100. So `get_dags(limit=500)` will only return max of 100 dags.
   
   I have to do the hack to mitigate this problem.
   ```
       def _get_dags(self, max_dags: int = 500):
           i = 0
           responses = []
           while i <= max_dags:
               response = self._api.get_dags(offset=i)
               responses += response['dags']
               i = i + 100
           return [dag['dag_id'] for dag in responses]
   ```
   
   Best,
   Hamid


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow-client-python] hhmahmood closed issue #49: get_dags does not fetch more than 100 dags.

Posted by GitBox <gi...@apache.org>.
hhmahmood closed issue #49: get_dags does not fetch more than 100 dags.  
URL: https://github.com/apache/airflow-client-python/issues/49


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow-client-python] potiuk commented on issue #49: get_dags does not fetch more than 100 dags.

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #49:
URL: https://github.com/apache/airflow-client-python/issues/49#issuecomment-1272915120

   airflow/config_templates/config.yaml - -> this is a source of truth and pre-commit will do all the automated updates of other places where it is generated


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow-client-python] potiuk commented on issue #49: get_dags does not fetch more than 100 dags.

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #49:
URL: https://github.com/apache/airflow-client-python/issues/49#issuecomment-1250912727

   The API  has fallback limit that kicks in automatically - you can increase it (but your queries might run much longer) https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#fallback-page-limit which 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow-client-python] hhmahmood commented on issue #49: get_dags does not fetch more than 100 dags.

Posted by GitBox <gi...@apache.org>.
hhmahmood commented on issue #49:
URL: https://github.com/apache/airflow-client-python/issues/49#issuecomment-1250916618

   Thanks @potiuk but it says that if no value is supplied from the api then it will be default 100 in my case I am supplying more than 100. For example `get_dags(limit=500)` will still gives 100 dags.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow-client-python] potiuk commented on issue #49: get_dags does not fetch more than 100 dags.

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #49:
URL: https://github.com/apache/airflow-client-python/issues/49#issuecomment-1250922748

   I think the description is a mistake. Try it (And then PR fixing the description would be most welcome).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow-client-python] hhmahmood commented on issue #49: get_dags does not fetch more than 100 dags.

Posted by GitBox <gi...@apache.org>.
hhmahmood commented on issue #49:
URL: https://github.com/apache/airflow-client-python/issues/49#issuecomment-1250926886

   Sure I will try it and let you know, thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org