You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/08/14 07:18:02 UTC
[GitHub] [airflow] potiuk opened a new issue, #33382: Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries
potiuk opened a new issue, #33382:
URL: https://github.com/apache/airflow/issues/33382
### Body
Seems that it has been starting happening affter https://github.com/apache/airflow/pull/31703
Worth looking at @aa3pankaj
https://github.com/apache/airflow/actions/runs/5852654575/job/15865548629?pr=33364
```
__________ TestDatabricksExecutionTrigger.test_sleep_between_retries ___________
self = <tests.providers.databricks.triggers.test_databricks.TestDatabricksExecutionTrigger object at 0x7f0c0e7baf50>
mock_get_run_state = <AsyncMock name='a_get_run_state' id='139689198176240'>
mock_sleep = <AsyncMock name='sleep' id='139689212547792'>
@pytest.mark.asyncio
@mock.patch("airflow.providers.databricks.triggers.databricks.asyncio.sleep")
@mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.a_get_run_state")
async def test_sleep_between_retries(self, mock_get_run_state, mock_sleep):
mock_get_run_state.side_effect = [
RunState(
life_cycle_state=LIFE_CYCLE_STATE_PENDING,
state_message="",
result_state="",
),
RunState(
life_cycle_state=LIFE_CYCLE_STATE_TERMINATED,
state_message="",
result_state="SUCCESS",
),
]
generator = self.trigger.run()
actual = await generator.asend(None)
assert actual == TriggerEvent(
{
"run_id": RUN_ID,
"run_state": RunState(
life_cycle_state=LIFE_CYCLE_STATE_TERMINATED, state_message="", result_state="SUCCESS"
).to_json(),
"run_page_url": RUN_PAGE_URL,
}
)
> mock_sleep.assert_called_once()
tests/providers/databricks/triggers/test_databricks.py:151:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <AsyncMock name='sleep' id='139689212547792'>
def assert_called_once(self):
"""assert that the mock was called only once.
"""
if not self.call_count == 1:
msg = ("Expected '%s' to have been called once. Called %s times.%s"
% (self._mock_name or 'mock',
self.call_count,
self._calls_repr()))
> raise AssertionError(msg)
E AssertionError: Expected 'sleep' to have been called once. Called 12 times.
E Calls: [call(30),
E call(3.483486096191374),
E call(0.813519164[8180](https://github.com/apache/airflow/actions/runs/5852654575/job/15865548629?pr=33364#step:6:8198)202),
E call(4.252464545552433),
E call(2.407743554553495),
E call(2.426392669267501),
E call(1.7065142153808315),
E call(2.0452043229765677),
E call(2.3704474685726793),
E call(3.9805234337374165),
E call(4.5757845484608195),
E call(3.865035447312122)].
/usr/local/lib/python3.10/unittest/mock.py:908: AssertionError
```
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #33382: Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries
URL: https://github.com/apache/airflow/issues/33382
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33382: Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33382:
URL: https://github.com/apache/airflow/issues/33382#issuecomment-1712745583
Hey @pankajastro . I think we are having some frequent flakiness in this test and likely it was introduced after this change: https://github.com/apache/airflow/pull/31703
It happens rather frequently (1/30-40 job runs as I see it) and it is currently the most frequent flaky test we have. Maybe you could take a look and see - because I have a feeling that this is also an example of "too many triggers generated when there. is not enough time to process the triggers firing".
Example recent failures:
* https://github.com/apache/airflow/actions/runs/6135701486/job/16649917145
* https://github.com/apache/airflow/actions/runs/6133271248/job/16645108748
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33382:
URL: https://github.com/apache/airflow/issues/33382#issuecomment-1821953271
https://github.com/apache/airflow/actions/runs/6951300980/job/18913526763 - another example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #33382: Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #33382: Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries
URL: https://github.com/apache/airflow/issues/33382
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajastro commented on issue #33382: Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries
Posted by "pankajastro (via GitHub)" <gi...@apache.org>.
pankajastro commented on issue #33382:
URL: https://github.com/apache/airflow/issues/33382#issuecomment-1714128480
hmm, interesting. I ran this test 1000 times but was not able to reproduce it on my machine.
But We reverted the Databricks related changes of https://github.com/apache/airflow/pull/31703 in PR https://github.com/apache/airflow/pull/31985 but looks like the tests we didn't revert. I'm changing tests in PR https://github.com/apache/airflow/pull/34272 like it was before https://github.com/apache/airflow/pull/31703 if it helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33382:
URL: https://github.com/apache/airflow/issues/33382#issuecomment-2510020296
Fixed already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33382: Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33382:
URL: https://github.com/apache/airflow/issues/33382#issuecomment-1715036320
And yes - this is a problem with Flaky tests, they are often not reproducible in isolation, this is the biggest problem with flakes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33382: Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33382:
URL: https://github.com/apache/airflow/issues/33382#issuecomment-1715034002
Closing it - let's see if it occurs again. Thatnks @pankajastro .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33382:
URL: https://github.com/apache/airflow/issues/33382#issuecomment-1821832497
Unfortunately we ssme that one re-appeared - it's very rare, but it happened again :)
* https://github.com/apache/airflow/actions/runs/6949888205/job/18909492387
```
=================================== FAILURES ===================================
__________ TestDatabricksExecutionTrigger.test_sleep_between_retries ___________
self = <tests.providers.databricks.triggers.test_databricks.TestDatabricksExecutionTrigger object at 0x7f67de189d50>
mock_get_run_state = <AsyncMock name='a_get_run_state' id='140084676774880'>
mock_sleep = <AsyncMock name='sleep' id='140084674550736'>
@pytest.mark.asyncio
@mock.patch("airflow.providers.databricks.triggers.databricks.asyncio.sleep")
@mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.a_get_run_state")
async def test_sleep_between_retries(self, mock_get_run_state, mock_sleep):
mock_get_run_state.side_effect = [
RunState(
life_cycle_state=LIFE_CYCLE_STATE_PENDING,
state_message="",
result_state="",
),
RunState(
life_cycle_state=LIFE_CYCLE_STATE_TERMINATED,
state_message="",
result_state="SUCCESS",
),
]
trigger_event = self.trigger.run()
async for event in trigger_event:
assert event == TriggerEvent(
{
"run_id": RUN_ID,
"run_state": RunState(
life_cycle_state=LIFE_CYCLE_STATE_TERMINATED, state_message="", result_state="SUCCESS"
).to_json(),
"run_page_url": RUN_PAGE_URL,
}
)
> mock_sleep.assert_called_once()
tests/providers/databricks/triggers/test_databricks.py:153:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <AsyncMock name='sleep' id='140084674550736'>
def assert_called_once(self):
"""assert that the mock was called only once.
"""
if not self.call_count == 1:
msg = ("Expected '%s' to have been called once. Called %s times.%s"
% (self._mock_name or 'mock',
self.call_count,
self._calls_repr()))
> raise AssertionError(msg)
E AssertionError: Expected 'sleep' to have been called once. Called 39 times.
E Calls: [call(30),
E call(1.5614954891818282),
E call(4.947393322584771),
E call(1.098632801403257),
E call(4.245258848656453),
E call(4.069514337592413),
E call(0.6677589362499334),
E call(1.3359160588431975),
E call(0.07929569295131796),
E call(5.404071101042226),
E call(5.743428527804218),
E call(2.7449642348197374),
E call(11.53270264980902),
E call(0.4576054206335382),
E call(10.185369522033069),
E call(0.3881695575480826),
E call(14.601938371047803),
E call(3.464100900781081),
E call(13.651888163702226),
E call(11.518445750880675),
E call(11.805540294945818),
E call(1.6654214641285763),
E call(12.116858954152775),
E call(0.4274309662069142),
E call(2.1419843921887987),
E call(15.108680099488625),
E call(14.200441665349484),
E call(17.461868373600186),
E call(9.176234144014751),
E call(7.823477464693354),
E call(9.709087497895718),
E call(12.996552399230477),
E call(17.364527077318467),
E call(12.750015645644263),
E call(9.405866521202778),
E call(5.319519737900769),
E call(2.652528572765807),
E call(6.5879276596641585),
E call(1.2949168686974422)].
/usr/local/lib/python3.10/unittest/mock.py:908: AssertionError
----------------------------- Captured stderr call -----------------------------
INFO [airflow.providers.databricks.triggers.databricks.DatabricksExecutionTrigger] run-id 1 in run state {'life_cycle_state': 'PENDING', 'result_state': '', 'state_message': ''}. sleeping for 30 seconds
------------------------------ Captured log call -------------------------------
INFO airflow.providers.databricks.triggers.databricks.DatabricksExecutionTrigger:databricks.py:94 run-id 1 in run state {'life_cycle_state': 'PENDING', 'result_state': '', 'state_message': ''}. sleeping for 30 seconds
- generated xml file: /files/test_result-providers_-amazon_google-postgres.xml -
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org