You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/20 08:51:10 UTC
[GitHub] [airflow] ephraimbuddy opened a new pull request #9431: Move API page limit and offset parameters to views as kwargs
ephraimbuddy opened a new pull request #9431:
URL: https://github.com/apache/airflow/pull/9431
---
The limit and offset parameters are being parsed manually but it can be handled by connexion, giving a better error messages in the case of limit exceeding.
This change also adds `maximum` limit to spec and tests for default/maximum page limit requests
@mik-laj , you devised a way to allow user to set this maximum limit but I couldn't process it. Would appreciate if you could explain more.
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above provides context of the change
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Target Github ISSUE in description if exists
- [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#issuecomment-647118090
How do I add this change to docs? I can't find where to add it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#issuecomment-653185780
> This PR depends on #9503 ,
I don't see the common parts. Can you say something more?
Can you do a rebase? I would like to merge this change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy removed a comment on pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
ephraimbuddy removed a comment on pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#issuecomment-647118090
How do I add this change to docs? I can't find where to add it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r444260539
##########
File path: docs/index.rst
##########
@@ -108,5 +109,5 @@ Content
CLI <cli-ref>
Macros <macros-ref>
Python API <_api/index>
- REST API <rest-api-ref>
+ EXPERIMENTAL REST API <rest-api-ref>
Review comment:
```suggestion
Experimental REST API <rest-api-ref>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r443392444
##########
File path: tests/api_connexion/test_parameters.py
##########
@@ -50,6 +50,21 @@ def test_raises_400_for_invalid_arg(self):
format_datetime(invalid_datetime)
+class TestMaximumPagelimit(unittest.TestCase):
+
+ @mock.patch("airflow.api_connexion.parameters.conf.get")
Review comment:
Use conf_vars
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r444510988
##########
File path: airflow/api_connexion/parameters.py
##########
@@ -48,6 +46,21 @@ def format_datetime(value: str):
)
+def check_limit(value: int):
+ """
+ This checks the limit passed to view and raises BadRequest if
+ limit exceed user configured value
+ """
+ max_val = conf.getint("api", "maximum_page_limit")
+
+ if value > max_val:
Review comment:
Nice. I think It is nice. Let me work on it now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r443190851
##########
File path: airflow/api_connexion/parameters.py
##########
@@ -48,6 +45,27 @@ def format_datetime(value: str):
)
+def check_limit(value: int):
+ """
+ This checks the limit passed to view and raises BadRequest if
+ limit exceed user configured value
+ """
+ max_val = int(conf.get("api", "maximum_page_limit"))
Review comment:
```suggestion
max_val = conf.getint("api", "maximum_page_limit")
```
See: https://github.com/apache/airflow/pull/9436
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r444678716
##########
File path: airflow/api_connexion/parameters.py
##########
@@ -48,6 +49,22 @@ def format_datetime(value: str):
)
+def check_limit(value: int):
+ """
+ This checks the limit passed to view and raises BadRequest if
+ limit exceed user configured value
+ """
+ max_val = conf.getint("api", "maximum_page_limit") # user configured page limit
Review comment:
I tried having the default only on the config but it turned out that limit doesn't get parsed to views when no limit is on url.
It was giving some unexpected behaviours. But as it is now, it's giving us exactly as described.
I will also try again if I can have it solely on the config without the one on spec
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r444269065
##########
File path: docs/stable-rest-api/index.rst
##########
@@ -0,0 +1,44 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+REST API Documentation
+======================
+
+Airflow stable REST API exposes a lot of endpoints for working with Apache Airflow.
+
+Configuration
+-------------
+There are default configurations available for airflow REST API which can be changed in
+``airflow.cfg``.
+
+.. note::
+ For more information on setting the configuration, see :doc:`../howto/set-config`
+
+The default configurations for the REST API are as follows:
+
+.. list-table:: REST API CONFIGURATION DEFAULTS
+ :widths: 25, 25, 50
+ :header-rows: 1
+
+ * - Configuration
+ - Default
+ - Description
+
+ * - maximum_page_limit
+ - 100
+ - A positive integer, used to set the maximum page limit for API requests
Review comment:
```suggestion
Airflow has a REST API that allows third-party application to perform a wide wide range of operations.
Page size limit
---------------
To protect against requests that may lead to application instability, the API has a limit of items in response. The default is 100 items, but you can change it using `maximum_page_limit` option in `[api]` section in the `airflow.cfg` file.
.. note::
For more information on setting the configuration, see :doc:`../howto/set-config`
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r444615147
##########
File path: airflow/api_connexion/parameters.py
##########
@@ -48,6 +49,22 @@ def format_datetime(value: str):
)
+def check_limit(value: int):
+ """
+ This checks the limit passed to view and raises BadRequest if
+ limit exceed user configured value
+ """
+ max_val = conf.getint("api", "maximum_page_limit") # user configured page limit
Review comment:
Check how I implemented it https://github.com/apache/airflow/pull/9431/commits/49198927cba626957965c962cd098bb018e41df7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#issuecomment-653202146
> > This PR depends on #9503 ,
>
> I don't see the common parts. Can you say something more?
>
It is because I removed `format_parameters` tests in #9503. I was hoping to add it back with this PR. However, I will rebase now then add the tests on the other PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r443261155
##########
File path: docs/stable-rest-api-ref.rst
##########
@@ -0,0 +1,44 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+REST API Reference
Review comment:
I renamed it and placed it another point. Please take a look. https://github.com/apache/airflow/pull/9431/commits/ad5633fdba9b1e41b46df3f470250597fdb5ed03.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#issuecomment-649169418
Depends on #9503
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r443251019
##########
File path: docs/stable-rest-api-ref.rst
##########
@@ -0,0 +1,44 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+REST API Reference
Review comment:
We need a guide, not reference documentation. Reference docs will be generated in other tools:
Swaagger UI:
https://editor.swagger.io/?url=https://raw.githubusercontent.com/apache/airflow/master/airflow/api_connexion/openapi/v1.yaml
Redoc:
https://redocly.github.io/redoc/?url=https://raw.githubusercontent.com/apache/airflow/master/airflow/api_connexion/openapi/v1.yaml
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj merged pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
mik-laj merged pull request #9431:
URL: https://github.com/apache/airflow/pull/9431
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r444567648
##########
File path: airflow/api_connexion/parameters.py
##########
@@ -20,11 +20,11 @@
from pendulum.parsing import ParserError
from airflow.api_connexion.exceptions import BadRequest
+from airflow.configuration import conf
from airflow.utils import timezone
-# Page parameters
-page_offset = "offset"
-page_limit = "limit"
+MAXIMUM_PAGE_LIMIT = 1000
Review comment:
Why is there a hard limit in the code? I think we can trust the user that he will not configure the limit incorrectly. I am able to imagine situations that this limit may be insufficient if the user wants to migrate data from one instance to another.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r443212229
##########
File path: airflow/api_connexion/parameters.py
##########
@@ -48,6 +45,27 @@ def format_datetime(value: str):
)
+def check_limit(value: int):
+ """
+ This checks the limit passed to view and raises BadRequest if
+ limit exceed user configured value
+ """
+ max_val = int(conf.get("api", "maximum_page_limit"))
Review comment:
I added it to config.yml but can't find where to add it in docs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r443191107
##########
File path: tests/api_connexion/endpoints/test_connection_endpoint.py
##########
@@ -181,17 +182,42 @@ def test_handle_limit_offset(self, url, expected_conn_ids, session):
self.assertEqual(conn_ids, expected_conn_ids)
@provide_session
- def test_should_respect_page_size_limit(self, session):
+ def test_should_respect_page_size_limit_default(self, session):
connection_models = self._create_connections(200)
session.add_all(connection_models)
session.commit()
- response = self.client.get("/api/v1/connections?limit=150")
+ response = self.client.get("/api/v1/connections")
assert response.status_code == 200
self.assertEqual(response.json["total_entries"], 200)
self.assertEqual(len(response.json["connections"]), 100)
+ @provide_session
+ @mock.patch("airflow.api_connexion.parameters.conf.get")
Review comment:
Can you use conf_vars decorator here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r444678716
##########
File path: airflow/api_connexion/parameters.py
##########
@@ -48,6 +49,22 @@ def format_datetime(value: str):
)
+def check_limit(value: int):
+ """
+ This checks the limit passed to view and raises BadRequest if
+ limit exceed user configured value
+ """
+ max_val = conf.getint("api", "maximum_page_limit") # user configured page limit
Review comment:
I tried having the default only on the config but it turned out that limit doesn't get parsed to views. It was giving some unexpected behaviours. But as it is now, it's giving us exactly as described.
I will also try again if I can have it solely on the config without the one on spec
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r443212659
##########
File path: airflow/api_connexion/parameters.py
##########
@@ -48,6 +45,27 @@ def format_datetime(value: str):
)
+def check_limit(value: int):
+ """
+ This checks the limit passed to view and raises BadRequest if
+ limit exceed user configured value
+ """
+ max_val = int(conf.get("api", "maximum_page_limit"))
Review comment:
We don't have a specific page, so we have to create. My title suggestion: "Using the REST API". It will be similar to CLI. For CLI, we have [reference docs](https://airflow.readthedocs.io/en/latest/cli-ref.html) and [topic guide](https://airflow.readthedocs.io/en/latest/usage-cli.html).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r444272559
##########
File path: airflow/api_connexion/parameters.py
##########
@@ -48,6 +46,21 @@ def format_datetime(value: str):
)
+def check_limit(value: int):
+ """
+ This checks the limit passed to view and raises BadRequest if
+ limit exceed user configured value
+ """
+ max_val = conf.getint("api", "maximum_page_limit")
+
+ if value > max_val:
Review comment:
```
Request messages for collections should define an int32 page_size field, allowing users to specify the maximum number of results to return.
If the user does not specify page_size (or specifies 0), the API chooses an appropriate default, which the API should document.
If the user specifies page_size greater than the maximum permitted by the API, the API should coerce down to the maximum permitted page size.
If the user specifies a negative value for page_size, the API must send an INVALID_ARGUMENT error.
The API may return fewer results than the number requested (including zero results), even if not at the end of the collection.
```
https://google.aip.dev/158
I looked more closely at how it implements Google. What do you think? In my opinion this makes sense and it is behavior that is close to the previous one. Coerce down to the maximum permitted page size allows us to fetching the maximum number of items without checking the current application configuration.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r444567888
##########
File path: airflow/api_connexion/parameters.py
##########
@@ -48,6 +49,22 @@ def format_datetime(value: str):
)
+def check_limit(value: int):
+ """
+ This checks the limit passed to view and raises BadRequest if
+ limit exceed user configured value
+ """
+ max_val = conf.getint("api", "maximum_page_limit") # user configured page limit
Review comment:
```suggestion
# user configured page limit
max_val = conf.getint("api", "maximum_page_limit", fallback= PAGE_LIMIT_DEFAULT)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r443191107
##########
File path: tests/api_connexion/endpoints/test_connection_endpoint.py
##########
@@ -181,17 +182,42 @@ def test_handle_limit_offset(self, url, expected_conn_ids, session):
self.assertEqual(conn_ids, expected_conn_ids)
@provide_session
- def test_should_respect_page_size_limit(self, session):
+ def test_should_respect_page_size_limit_default(self, session):
connection_models = self._create_connections(200)
session.add_all(connection_models)
session.commit()
- response = self.client.get("/api/v1/connections?limit=150")
+ response = self.client.get("/api/v1/connections")
assert response.status_code == 200
self.assertEqual(response.json["total_entries"], 200)
self.assertEqual(len(response.json["connections"]), 100)
+ @provide_session
+ @mock.patch("airflow.api_connexion.parameters.conf.get")
Review comment:
Can you use conf_vars decorator here? We don't use mock for configuration because we have a more specialized decorator.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r443217145
##########
File path: airflow/api_connexion/parameters.py
##########
@@ -48,6 +45,27 @@ def format_datetime(value: str):
)
+def check_limit(value: int):
+ """
+ This checks the limit passed to view and raises BadRequest if
+ limit exceed user configured value
+ """
+ max_val = int(conf.get("api", "maximum_page_limit"))
Review comment:
Here is what I did. https://github.com/apache/airflow/pull/9431/commits/7e231ca7e48617c5c1af718ee4c134370aba211e
and this too https://github.com/apache/airflow/pull/9431/commits/861f33d5c86ec5777ea2084dab96e4919140d817
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r443190913
##########
File path: airflow/api_connexion/parameters.py
##########
@@ -48,6 +45,27 @@ def format_datetime(value: str):
)
+def check_limit(value: int):
+ """
+ This checks the limit passed to view and raises BadRequest if
+ limit exceed user configured value
+ """
+ max_val = int(conf.get("api", "maximum_page_limit"))
Review comment:
Can you add this option to reference docs(airflow/config_templates/config.yml file) and docs (/docs directory)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#discussion_r443191039
##########
File path: airflow/api_connexion/parameters.py
##########
@@ -48,6 +45,27 @@ def format_datetime(value: str):
)
+def check_limit(value: int):
+ """
+ This checks the limit passed to view and raises BadRequest if
+ limit exceed user configured value
+ """
+ max_val = int(conf.get("api", "maximum_page_limit"))
+ spec = {'default': 100, 'maximum': max_val,
+ 'minimum': 1, 'type': 'integer'}
+ if value > max_val:
+ # This message should not be formatted. If formatted please
+ # run tests
+ message = ("{value} is greater than the maximum of {max_val}\n"
+ "\n"
+ "Failed validating 'maximum' in schema:\n"
+ " {spec}\n"
+ "\nOn instance:\n {value}").format(value=value,
+ max_val=max_val, spec=spec)
Review comment:
```suggestion
message = (f"{value} is greater than the maximum of {max_val}")
```
This error is not due to schema validation. We have security on another layer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#issuecomment-649109375
Can you move datetime format related stuff to new PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org