You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/11 08:25:12 UTC
[GitHub] [airflow] ephraimbuddy opened a new pull request #9227: [WIP]add event log endpoints
ephraimbuddy opened a new pull request #9227:
URL: https://github.com/apache/airflow/pull/9227
---
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above provides context of the change
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Target Github ISSUE in description if exists
- [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #9227: Add event log endpoints
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#issuecomment-643795997
Thanks! @kaxil
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9227: Add event log endpoints
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r439781631
##########
File path: airflow/api_connexion/endpoints/event_log_endpoint.py
##########
@@ -15,19 +15,42 @@
# specific language governing permissions and limitations
# under the License.
-# TODO(mik-laj): We have to implement it.
-# Do you want to help? Please look at: https://github.com/apache/airflow/issues/8135
+from flask import request
+from airflow.api_connexion import parameters
+from airflow.api_connexion.exceptions import NotFound
+from airflow.api_connexion.schemas.event_log_schema import (
+ EventLogCollection, event_log_collection_schema, event_log_schema,
+)
+from airflow.models import Log
+from airflow.utils.session import provide_session
-def get_event_log():
+
+@provide_session
+def get_event_log(event_log_id, session):
"""
Get a log entry
"""
- raise NotImplementedError("Not implemented yet.")
+ query = session.query(Log)
+ query = query.filter(Log.id == event_log_id)
+ event_log = query.one_or_none()
+ if event_log is None:
+ raise NotFound("Event Log not found")
+ return event_log_schema.dump(event_log)
-def get_event_logs():
+@provide_session
+def get_event_logs(session):
"""
Get all log entries from event log
"""
- raise NotImplementedError("Not implemented yet.")
+ offset = request.args.get(parameters.page_offset, 0)
+ limit = min(int(request.args.get(parameters.page_limit, 100)), 100)
+
+ query = session.query(Log)
+ total_entries = query.count()
Review comment:
https://docs.sqlalchemy.org/en/13/orm/query.html?highlight=count#sqlalchemy.orm.query.Query.count
We should use:
```suggestion
total_entries = session.query(func.count(Log)).scalar()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9227: Add event log endpoints
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r439781657
##########
File path: airflow/api_connexion/endpoints/event_log_endpoint.py
##########
@@ -15,19 +15,42 @@
# specific language governing permissions and limitations
# under the License.
-# TODO(mik-laj): We have to implement it.
-# Do you want to help? Please look at: https://github.com/apache/airflow/issues/8135
+from flask import request
+from airflow.api_connexion import parameters
+from airflow.api_connexion.exceptions import NotFound
+from airflow.api_connexion.schemas.event_log_schema import (
+ EventLogCollection, event_log_collection_schema, event_log_schema,
+)
+from airflow.models import Log
+from airflow.utils.session import provide_session
-def get_event_log():
+
+@provide_session
+def get_event_log(event_log_id, session):
"""
Get a log entry
"""
- raise NotImplementedError("Not implemented yet.")
+ query = session.query(Log)
+ query = query.filter(Log.id == event_log_id)
+ event_log = query.one_or_none()
+ if event_log is None:
+ raise NotFound("Event Log not found")
+ return event_log_schema.dump(event_log)
-def get_event_logs():
+@provide_session
+def get_event_logs(session):
"""
Get all log entries from event log
"""
- raise NotImplementedError("Not implemented yet.")
+ offset = request.args.get(parameters.page_offset, 0)
+ limit = min(int(request.args.get(parameters.page_limit, 100)), 100)
+
+ query = session.query(Log)
+ total_entries = query.count()
Review comment:
Or just `len(event_logs)` after L54 below
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9227: Add event log endpoints
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r439445858
##########
File path: airflow/api_connexion/openapi/v1.yaml
##########
@@ -1315,6 +1315,7 @@ components:
extra:
type: string
readOnly: true
+ nullable: true
Review comment:
And if it is None then on serializing it, it would come out as None instead of a string. This would not conform to the spec
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9227: Add event log endpoints
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r439835162
##########
File path: tests/api_connexion/schemas/test_event_log_schema.py
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+from airflow import DAG
+from airflow.api_connexion.schemas.event_log_schema import (
+ EventLogCollection, event_log_collection_schema, event_log_schema,
+)
+from airflow.models import Log, TaskInstance
+from airflow.operators.dummy_operator import DummyOperator
+from airflow.utils import timezone
+from airflow.utils.session import create_session, provide_session
+
+
+class TestEventLogSchemaBase(unittest.TestCase):
+
+ def setUp(self) -> None:
+ with create_session() as session:
+ session.query(Log).delete()
+ self.default_time = "2020-06-09T13:00:00+00:00"
+ self.default_time2 = '2020-06-11T07:00:00+00:00'
+
+ def tearDown(self) -> None:
+ with create_session() as session:
+ session.query(Log).delete()
+
+ def _create_task_instance(self):
+ with DAG('TEST_DAG_ID', start_date=timezone.parse(self.default_time),
+ end_date=timezone.parse(self.default_time)) as dag:
+ DummyOperator(task_id="TEST_TASK_ID", owner="airflow")
+ return TaskInstance(task=op1, execution_date=timezone.parse(self.default_time))
Review comment:
Whoops my bad, static checks failed
```suggestion
with DAG('TEST_DAG_ID', start_date=timezone.parse(self.default_time),
end_date=timezone.parse(self.default_time)):
op1 = DummyOperator(task_id="TEST_TASK_ID", owner="airflow")
return TaskInstance(task=op1, execution_date=timezone.parse(self.default_time))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9227: [WIP]add event log endpoints
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r438905458
##########
File path: tests/api_connexion/endpoints/test_event_log_endpoint.py
##########
@@ -29,17 +34,177 @@ def setUpClass(cls) -> None:
def setUp(self) -> None:
self.client = self.app.test_client() # type:ignore
+ with create_session() as session:
+ session.query(Log).delete()
+ self.default_time = "2020-06-10T20:36:40.186116+00:00"
+ self.default_time = '2020-06-11T07:04:57.501393+00:00'
+
+ def tearDown(self) -> None:
+ with create_session() as session:
+ session.query(Log).delete()
+
+ def _create_task_instance(self):
+ dag = DAG('TEST_DAG_ID', start_date=timezone.parse(self.default_time),
+ end_date=timezone.parse(self.default_time))
+ op1 = DummyOperator(task_id="TEST_TASK_ID", owner="airflow",
+ )
+ dag.add_task(op1)
+ ti = TaskInstance(task=op1, execution_date=timezone.parse(self.default_time))
+ return ti
class TestGetEventLog(TestEventLogEndpoint):
- @pytest.mark.skip(reason="Not implemented yet")
- def test_should_response_200(self):
- response = self.client.get("/api/v1/eventLogs/1")
+
+ @provide_session
+ def test_should_response_200(self, session):
+ log_model = Log(
+ event='TEST_EVENT',
+ task_instance=self._create_task_instance(),
+ )
+ log_model.dttm = timezone.parse(self.default_time)
+ session.add(log_model)
+ session.commit()
+ event_log_id = log_model.id
+ response = self.client.get(f"/api/v1/eventLogs/{event_log_id}")
assert response.status_code == 200
+ self.assertEqual(
+ response.json,
+ {
+ "event_log_id": event_log_id,
+ "event": "TEST_EVENT",
+ "dag_id": "TEST_DAG_ID",
+ "task_id": "TEST_TASK_ID",
+ "execution_date": self.default_time,
+ "owner": 'airflow',
+ "when": self.default_time,
+ "extra": None
+ }
+ )
+
+ def test_should_response_404(self):
+ response = self.client.get("/api/v1/eventLogs/1")
+ assert response.status_code == 404
+ self.assertEqual(
+ {'detail': None, 'status': 404, 'title': 'Event Log not found', 'type': 'about:blank'},
+ response.json
+ )
class TestGetEventLogs(TestEventLogEndpoint):
- @pytest.mark.skip(reason="Not implemented yet")
- def test_should_response_200(self):
+
+ @provide_session
+ def test_should_response_200(self, session):
+ log_model_1 = Log(
+ event='TEST_EVENT_1',
+ task_instance=self._create_task_instance(),
+ )
+ log_model_2 = Log(
+ event='TEST_EVENT_2',
+ task_instance=self._create_task_instance(),
+ )
+ log_model_1.dttm = timezone.parse(self.default_time)
+ log_model_2.dttm = timezone.parse(self.default_time)
+ session.add_all([log_model_1, log_model_2])
+ session.commit()
response = self.client.get("/api/v1/eventLogs")
assert response.status_code == 200
+ self.assertEqual(
+ response.json,
+ {
+ "event_logs": [
+ {
+
+ "event_log_id": log_model_1.id,
+ "event": "TEST_EVENT_1",
+ "dag_id": "TEST_DAG_ID",
+ "task_id": "TEST_TASK_ID",
+ "execution_date": self.default_time,
+ "owner": 'airflow',
+ "when": self.default_time,
+ "extra": None
+
+ },
+ {
+ "event_log_id": log_model_2.id,
+ "event": "TEST_EVENT_2",
+ "dag_id": "TEST_DAG_ID",
+ "task_id": "TEST_TASK_ID",
+ "execution_date": self.default_time,
+ "owner": 'airflow',
+ "when": self.default_time,
Review comment:
```suggestion
"when": self.default_time_2,
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9227: [WIP]add event log endpoints
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r438904493
##########
File path: tests/api_connexion/endpoints/test_event_log_endpoint.py
##########
@@ -29,17 +34,177 @@ def setUpClass(cls) -> None:
def setUp(self) -> None:
self.client = self.app.test_client() # type:ignore
+ with create_session() as session:
+ session.query(Log).delete()
+ self.default_time = "2020-06-10T20:36:40.186116+00:00"
+ self.default_time = '2020-06-11T07:04:57.501393+00:00'
Review comment:
```suggestion
self.default_time_2 = '2020-06-11T07:04:57.501393+00:00'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9227: Add event log endpoints
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r440235998
##########
File path: airflow/api_connexion/openapi/v1.yaml
##########
@@ -1315,6 +1315,7 @@ components:
extra:
type: string
readOnly: true
+ nullable: true
Review comment:
Here is the PR https://github.com/apache/airflow/pull/9315
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9227: Add event log endpoints
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r439822479
##########
File path: airflow/api_connexion/endpoints/event_log_endpoint.py
##########
@@ -15,19 +15,42 @@
# specific language governing permissions and limitations
# under the License.
-# TODO(mik-laj): We have to implement it.
-# Do you want to help? Please look at: https://github.com/apache/airflow/issues/8135
+from flask import request
+from airflow.api_connexion import parameters
+from airflow.api_connexion.exceptions import NotFound
+from airflow.api_connexion.schemas.event_log_schema import (
+ EventLogCollection, event_log_collection_schema, event_log_schema,
+)
+from airflow.models import Log
+from airflow.utils.session import provide_session
-def get_event_log():
+
+@provide_session
+def get_event_log(event_log_id, session):
"""
Get a log entry
"""
- raise NotImplementedError("Not implemented yet.")
+ query = session.query(Log)
+ query = query.filter(Log.id == event_log_id)
+ event_log = query.one_or_none()
+ if event_log is None:
+ raise NotFound("Event Log not found")
+ return event_log_schema.dump(event_log)
-def get_event_logs():
+@provide_session
+def get_event_logs(session):
"""
Get all log entries from event log
"""
- raise NotImplementedError("Not implemented yet.")
+ offset = request.args.get(parameters.page_offset, 0)
+ limit = min(int(request.args.get(parameters.page_limit, 100)), 100)
+
+ query = session.query(Log)
+ total_entries = query.count()
Review comment:
I think using `len(event_logs)` would not conform to specification. I am taking the `total_entries` to mean the total of all `log entries` and not only what is returned after `limit` and `offset` have been applied. I had a discussion about it with @mik-laj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9227: Add event log endpoints
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r439782362
##########
File path: tests/api_connexion/schemas/test_event_log_schema.py
##########
@@ -0,0 +1,129 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+from airflow import DAG
+from airflow.api_connexion.schemas.event_log_schema import (
+ EventLogCollection, event_log_collection_schema, event_log_schema,
+)
+from airflow.models import Log, TaskInstance
+from airflow.operators.dummy_operator import DummyOperator
+from airflow.utils import timezone
+from airflow.utils.session import create_session, provide_session
+
+
+class TestEventLogSchemaBase(unittest.TestCase):
+
+ def setUp(self) -> None:
+ with create_session() as session:
+ session.query(Log).delete()
+ self.default_time = "2020-06-09T13:00:00+00:00"
+ self.default_time2 = '2020-06-11T07:00:00+00:00'
+
+ def tearDown(self) -> None:
+ with create_session() as session:
+ session.query(Log).delete()
+
+ def _create_task_instance(self):
+ dag = DAG('TEST_DAG_ID', start_date=timezone.parse(self.default_time),
+ end_date=timezone.parse(self.default_time))
+ op1 = DummyOperator(task_id="TEST_TASK_ID", owner="airflow",
+ )
+ dag.add_task(op1)
+ ti = TaskInstance(task=op1, execution_date=timezone.parse(self.default_time))
+ return ti
Review comment:
```suggestion
dag = DAG('TEST_DAG_ID', start_date=timezone.parse(self.default_time),
end_date=timezone.parse(self.default_time))
op1 = DummyOperator(task_id="TEST_TASK_ID", owner="airflow", dag=dag)
return TaskInstance(task=op1, execution_date=timezone.parse(self.default_time))
```
OR
```suggestion
with DAG('TEST_DAG_ID', start_date=timezone.parse(self.default_time),
end_date=timezone.parse(self.default_time)) as dag:
DummyOperator(task_id="TEST_TASK_ID", owner="airflow")
return TaskInstance(task=op1, execution_date=timezone.parse(self.default_time))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9227: Add event log endpoints
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r439878153
##########
File path: airflow/api_connexion/openapi/v1.yaml
##########
@@ -1315,6 +1315,7 @@ components:
extra:
type: string
readOnly: true
+ nullable: true
Review comment:
I agree with it. Could you do it in a separate PR? I would like to keep the specification change history clean. There is a good chance that someone will want to do [git blame](https://help.github.com/en/github/managing-files-in-a-repository/tracking-changes-in-a-file) on the specification.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9227: add event log endpoints
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r439368262
##########
File path: tests/api_connexion/endpoints/test_event_log_endpoint.py
##########
@@ -29,17 +34,177 @@ def setUpClass(cls) -> None:
def setUp(self) -> None:
self.client = self.app.test_client() # type:ignore
+ with create_session() as session:
+ session.query(Log).delete()
+ self.default_time = "2020-06-10T20:36:40.186116+00:00"
Review comment:
Can you reduce the precision on the dates given? Now it is difficult to compare them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9227: add event log endpoints
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r439367574
##########
File path: airflow/api_connexion/openapi/v1.yaml
##########
@@ -1315,6 +1315,7 @@ components:
extra:
type: string
readOnly: true
+ nullable: true
Review comment:
Can you tell me more about it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9227: Add event log endpoints
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r439878153
##########
File path: airflow/api_connexion/openapi/v1.yaml
##########
@@ -1315,6 +1315,7 @@ components:
extra:
type: string
readOnly: true
+ nullable: true
Review comment:
I agree with it. Could you do it in a separate PR? I would like to keep the specification change history clean. There is a good chance that someone will want to do git blame on the specification.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj merged pull request #9227: Add event log endpoints
Posted by GitBox <gi...@apache.org>.
mik-laj merged pull request #9227:
URL: https://github.com/apache/airflow/pull/9227
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9227: Add event log endpoints
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r439823752
##########
File path: airflow/api_connexion/endpoints/event_log_endpoint.py
##########
@@ -15,19 +15,42 @@
# specific language governing permissions and limitations
# under the License.
-# TODO(mik-laj): We have to implement it.
-# Do you want to help? Please look at: https://github.com/apache/airflow/issues/8135
+from flask import request
+from airflow.api_connexion import parameters
+from airflow.api_connexion.exceptions import NotFound
+from airflow.api_connexion.schemas.event_log_schema import (
+ EventLogCollection, event_log_collection_schema, event_log_schema,
+)
+from airflow.models import Log
+from airflow.utils.session import provide_session
-def get_event_log():
+
+@provide_session
+def get_event_log(event_log_id, session):
"""
Get a log entry
"""
- raise NotImplementedError("Not implemented yet.")
+ query = session.query(Log)
+ query = query.filter(Log.id == event_log_id)
+ event_log = query.one_or_none()
+ if event_log is None:
+ raise NotFound("Event Log not found")
+ return event_log_schema.dump(event_log)
-def get_event_logs():
+@provide_session
+def get_event_logs(session):
"""
Get all log entries from event log
"""
- raise NotImplementedError("Not implemented yet.")
+ offset = request.args.get(parameters.page_offset, 0)
+ limit = min(int(request.args.get(parameters.page_limit, 100)), 100)
+
+ query = session.query(Log)
+ total_entries = query.count()
Review comment:
In that case use 👍
```suggestion
total_entries = session.query(func.count(Log)).scalar()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #9227: Add event log endpoints
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#issuecomment-644172547
One thing is stopping me from merging this change.
https://github.com/apache/airflow/pull/9227/files#r439878153
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9227: [WIP]add event log endpoints
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r438905129
##########
File path: tests/api_connexion/endpoints/test_event_log_endpoint.py
##########
@@ -29,17 +34,177 @@ def setUpClass(cls) -> None:
def setUp(self) -> None:
self.client = self.app.test_client() # type:ignore
+ with create_session() as session:
+ session.query(Log).delete()
+ self.default_time = "2020-06-10T20:36:40.186116+00:00"
+ self.default_time = '2020-06-11T07:04:57.501393+00:00'
+
+ def tearDown(self) -> None:
+ with create_session() as session:
+ session.query(Log).delete()
+
+ def _create_task_instance(self):
+ dag = DAG('TEST_DAG_ID', start_date=timezone.parse(self.default_time),
+ end_date=timezone.parse(self.default_time))
+ op1 = DummyOperator(task_id="TEST_TASK_ID", owner="airflow",
+ )
+ dag.add_task(op1)
+ ti = TaskInstance(task=op1, execution_date=timezone.parse(self.default_time))
+ return ti
class TestGetEventLog(TestEventLogEndpoint):
- @pytest.mark.skip(reason="Not implemented yet")
- def test_should_response_200(self):
- response = self.client.get("/api/v1/eventLogs/1")
+
+ @provide_session
+ def test_should_response_200(self, session):
+ log_model = Log(
+ event='TEST_EVENT',
+ task_instance=self._create_task_instance(),
+ )
+ log_model.dttm = timezone.parse(self.default_time)
+ session.add(log_model)
+ session.commit()
+ event_log_id = log_model.id
+ response = self.client.get(f"/api/v1/eventLogs/{event_log_id}")
assert response.status_code == 200
+ self.assertEqual(
+ response.json,
+ {
+ "event_log_id": event_log_id,
+ "event": "TEST_EVENT",
+ "dag_id": "TEST_DAG_ID",
+ "task_id": "TEST_TASK_ID",
+ "execution_date": self.default_time,
+ "owner": 'airflow',
+ "when": self.default_time,
+ "extra": None
+ }
+ )
+
+ def test_should_response_404(self):
+ response = self.client.get("/api/v1/eventLogs/1")
+ assert response.status_code == 404
+ self.assertEqual(
+ {'detail': None, 'status': 404, 'title': 'Event Log not found', 'type': 'about:blank'},
+ response.json
+ )
class TestGetEventLogs(TestEventLogEndpoint):
- @pytest.mark.skip(reason="Not implemented yet")
- def test_should_response_200(self):
+
+ @provide_session
+ def test_should_response_200(self, session):
+ log_model_1 = Log(
+ event='TEST_EVENT_1',
+ task_instance=self._create_task_instance(),
+ )
+ log_model_2 = Log(
+ event='TEST_EVENT_2',
+ task_instance=self._create_task_instance(),
+ )
+ log_model_1.dttm = timezone.parse(self.default_time)
+ log_model_2.dttm = timezone.parse(self.default_time)
Review comment:
```suggestion
log_model_2.dttm = timezone.parse(self.default_time_2)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9227: Add event log endpoints
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r439835162
##########
File path: tests/api_connexion/schemas/test_event_log_schema.py
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+from airflow import DAG
+from airflow.api_connexion.schemas.event_log_schema import (
+ EventLogCollection, event_log_collection_schema, event_log_schema,
+)
+from airflow.models import Log, TaskInstance
+from airflow.operators.dummy_operator import DummyOperator
+from airflow.utils import timezone
+from airflow.utils.session import create_session, provide_session
+
+
+class TestEventLogSchemaBase(unittest.TestCase):
+
+ def setUp(self) -> None:
+ with create_session() as session:
+ session.query(Log).delete()
+ self.default_time = "2020-06-09T13:00:00+00:00"
+ self.default_time2 = '2020-06-11T07:00:00+00:00'
+
+ def tearDown(self) -> None:
+ with create_session() as session:
+ session.query(Log).delete()
+
+ def _create_task_instance(self):
+ with DAG('TEST_DAG_ID', start_date=timezone.parse(self.default_time),
+ end_date=timezone.parse(self.default_time)) as dag:
+ DummyOperator(task_id="TEST_TASK_ID", owner="airflow")
+ return TaskInstance(task=op1, execution_date=timezone.parse(self.default_time))
Review comment:
Whoops my bad, static checks failed
```suggestion
op1 = DummyOperator(task_id="TEST_TASK_ID", owner="airflow")
return TaskInstance(task=op1, execution_date=timezone.parse(self.default_time))
```
```suggestion
with DAG('TEST_DAG_ID', start_date=timezone.parse(self.default_time),
end_date=timezone.parse(self.default_time)):
DummyOperator(task_id="TEST_TASK_ID", owner="airflow")
return TaskInstance(task=op1, execution_date=timezone.parse(self.default_time))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9227: Add event log endpoints
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r440222477
##########
File path: airflow/api_connexion/openapi/v1.yaml
##########
@@ -1315,6 +1315,7 @@ components:
extra:
type: string
readOnly: true
+ nullable: true
Review comment:
Making the PR now, will be through in a few minutes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9227: Add event log endpoints
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r439438528
##########
File path: airflow/api_connexion/openapi/v1.yaml
##########
@@ -1315,6 +1315,7 @@ components:
extra:
type: string
readOnly: true
+ nullable: true
Review comment:
From the model init method, it looks to me that the` extra` parameter can be nullable.
https://github.com/apache/airflow/blob/dcf65765e58fb8beba206454075bbd6675b65721/airflow/models/log.py#L46-L67
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9227: Add event log endpoints
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #9227:
URL: https://github.com/apache/airflow/pull/9227#discussion_r439781507
##########
File path: airflow/api_connexion/endpoints/event_log_endpoint.py
##########
@@ -15,19 +15,42 @@
# specific language governing permissions and limitations
# under the License.
-# TODO(mik-laj): We have to implement it.
-# Do you want to help? Please look at: https://github.com/apache/airflow/issues/8135
+from flask import request
+from airflow.api_connexion import parameters
+from airflow.api_connexion.exceptions import NotFound
+from airflow.api_connexion.schemas.event_log_schema import (
+ EventLogCollection, event_log_collection_schema, event_log_schema,
+)
+from airflow.models import Log
+from airflow.utils.session import provide_session
-def get_event_log():
+
+@provide_session
+def get_event_log(event_log_id, session):
"""
Get a log entry
"""
- raise NotImplementedError("Not implemented yet.")
+ query = session.query(Log)
+ query = query.filter(Log.id == event_log_id)
+ event_log = query.one_or_none()
Review comment:
```suggestion
query = session.query(Log).filter(Log.id == event_log_id)
event_log = query.one_or_none()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org