You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/08/14 07:18:02 UTC

[GitHub] [airflow] potiuk opened a new issue, #33382: Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries

potiuk opened a new issue, #33382:
URL: https://github.com/apache/airflow/issues/33382

   ### Body
   
   Seems that it has been starting happening affter https://github.com/apache/airflow/pull/31703
   
   Worth looking at @aa3pankaj 
   
   https://github.com/apache/airflow/actions/runs/5852654575/job/15865548629?pr=33364
   
   ```
   
   __________ TestDatabricksExecutionTrigger.test_sleep_between_retries ___________
   
   self = <tests.providers.databricks.triggers.test_databricks.TestDatabricksExecutionTrigger object at 0x7f0c0e7baf50>
   mock_get_run_state = <AsyncMock name='a_get_run_state' id='139689198176240'>
   mock_sleep = <AsyncMock name='sleep' id='139689212547792'>
   
       @pytest.mark.asyncio
       @mock.patch("airflow.providers.databricks.triggers.databricks.asyncio.sleep")
       @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.a_get_run_state")
       async def test_sleep_between_retries(self, mock_get_run_state, mock_sleep):
       
           mock_get_run_state.side_effect = [
               RunState(
                   life_cycle_state=LIFE_CYCLE_STATE_PENDING,
                   state_message="",
                   result_state="",
               ),
               RunState(
                   life_cycle_state=LIFE_CYCLE_STATE_TERMINATED,
                   state_message="",
                   result_state="SUCCESS",
               ),
           ]
       
           generator = self.trigger.run()
           actual = await generator.asend(None)
           assert actual == TriggerEvent(
               {
                   "run_id": RUN_ID,
                   "run_state": RunState(
                       life_cycle_state=LIFE_CYCLE_STATE_TERMINATED, state_message="", result_state="SUCCESS"
                   ).to_json(),
                   "run_page_url": RUN_PAGE_URL,
               }
           )
   >       mock_sleep.assert_called_once()
   
   tests/providers/databricks/triggers/test_databricks.py:151: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   
   self = <AsyncMock name='sleep' id='139689212547792'>
   
       def assert_called_once(self):
           """assert that the mock was called only once.
           """
           if not self.call_count == 1:
               msg = ("Expected '%s' to have been called once. Called %s times.%s"
                      % (self._mock_name or 'mock',
                         self.call_count,
                         self._calls_repr()))
   >           raise AssertionError(msg)
   E           AssertionError: Expected 'sleep' to have been called once. Called 12 times.
   E           Calls: [call(30),
   E            call(3.483486096191374),
   E            call(0.813519164[8180](https://github.com/apache/airflow/actions/runs/5852654575/job/15865548629?pr=33364#step:6:8198)202),
   E            call(4.252464545552433),
   E            call(2.407743554553495),
   E            call(2.426392669267501),
   E            call(1.7065142153808315),
   E            call(2.0452043229765677),
   E            call(2.3704474685726793),
   E            call(3.9805234337374165),
   E            call(4.5757845484608195),
   E            call(3.865035447312122)].
   
   /usr/local/lib/python3.10/unittest/mock.py:908: AssertionError
   ```
   
   ### Committer
   
   - [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #33382: Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries
URL: https://github.com/apache/airflow/issues/33382


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #33382: Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33382:
URL: https://github.com/apache/airflow/issues/33382#issuecomment-1712745583

   Hey @pankajastro . I think we are having some frequent flakiness in this test and  likely it was introduced after this change: https://github.com/apache/airflow/pull/31703
   
   It happens rather frequently (1/30-40 job runs as I see it) and it is currently the most frequent flaky test we have. Maybe you could take a look and see - because I have a feeling that this is also an example of "too many triggers generated when there. is not enough time to process the triggers firing". 
   
   Example recent failures:
   
   * https://github.com/apache/airflow/actions/runs/6135701486/job/16649917145
   * https://github.com/apache/airflow/actions/runs/6133271248/job/16645108748
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33382:
URL: https://github.com/apache/airflow/issues/33382#issuecomment-1821953271

   https://github.com/apache/airflow/actions/runs/6951300980/job/18913526763 - another example 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk closed issue #33382: Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #33382: Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries
URL: https://github.com/apache/airflow/issues/33382


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] pankajastro commented on issue #33382: Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries

Posted by "pankajastro (via GitHub)" <gi...@apache.org>.
pankajastro commented on issue #33382:
URL: https://github.com/apache/airflow/issues/33382#issuecomment-1714128480

   hmm, interesting. I ran this test 1000 times but was not able to reproduce it on my machine.
   But We reverted the Databricks related changes of https://github.com/apache/airflow/pull/31703 in PR https://github.com/apache/airflow/pull/31985 but looks like the tests we didn't revert. I'm changing tests in PR https://github.com/apache/airflow/pull/34272 like it was before https://github.com/apache/airflow/pull/31703 if it helps.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33382:
URL: https://github.com/apache/airflow/issues/33382#issuecomment-2510020296

   Fixed already.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #33382: Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33382:
URL: https://github.com/apache/airflow/issues/33382#issuecomment-1715036320

   And yes - this is a problem with Flaky tests, they are often not reproducible in isolation, this is the biggest problem with flakes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #33382: Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33382:
URL: https://github.com/apache/airflow/issues/33382#issuecomment-1715034002

   Closing it -  let's see if it occurs again. Thatnks @pankajastro .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Flaky TestDatabricksExecutionTrigger.test_sleep_between_retries [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33382:
URL: https://github.com/apache/airflow/issues/33382#issuecomment-1821832497

   Unfortunately we ssme that one re-appeared - it's very rare, but it happened again :) 
   
   * https://github.com/apache/airflow/actions/runs/6949888205/job/18909492387
   
   ```
   =================================== FAILURES ===================================
   __________ TestDatabricksExecutionTrigger.test_sleep_between_retries ___________
   
   self = <tests.providers.databricks.triggers.test_databricks.TestDatabricksExecutionTrigger object at 0x7f67de189d50>
   mock_get_run_state = <AsyncMock name='a_get_run_state' id='140084676774880'>
   mock_sleep = <AsyncMock name='sleep' id='140084674550736'>
   
       @pytest.mark.asyncio
       @mock.patch("airflow.providers.databricks.triggers.databricks.asyncio.sleep")
       @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.a_get_run_state")
       async def test_sleep_between_retries(self, mock_get_run_state, mock_sleep):
           mock_get_run_state.side_effect = [
               RunState(
                   life_cycle_state=LIFE_CYCLE_STATE_PENDING,
                   state_message="",
                   result_state="",
               ),
               RunState(
                   life_cycle_state=LIFE_CYCLE_STATE_TERMINATED,
                   state_message="",
                   result_state="SUCCESS",
               ),
           ]
       
           trigger_event = self.trigger.run()
           async for event in trigger_event:
               assert event == TriggerEvent(
                   {
                       "run_id": RUN_ID,
                       "run_state": RunState(
                           life_cycle_state=LIFE_CYCLE_STATE_TERMINATED, state_message="", result_state="SUCCESS"
                       ).to_json(),
                       "run_page_url": RUN_PAGE_URL,
                   }
               )
   >       mock_sleep.assert_called_once()
   
   tests/providers/databricks/triggers/test_databricks.py:153: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   
   self = <AsyncMock name='sleep' id='140084674550736'>
   
       def assert_called_once(self):
           """assert that the mock was called only once.
           """
           if not self.call_count == 1:
               msg = ("Expected '%s' to have been called once. Called %s times.%s"
                      % (self._mock_name or 'mock',
                         self.call_count,
                         self._calls_repr()))
   >           raise AssertionError(msg)
   E           AssertionError: Expected 'sleep' to have been called once. Called 39 times.
   E           Calls: [call(30),
   E            call(1.5614954891818282),
   E            call(4.947393322584771),
   E            call(1.098632801403257),
   E            call(4.245258848656453),
   E            call(4.069514337592413),
   E            call(0.6677589362499334),
   E            call(1.3359160588431975),
   E            call(0.07929569295131796),
   E            call(5.404071101042226),
   E            call(5.743428527804218),
   E            call(2.7449642348197374),
   E            call(11.53270264980902),
   E            call(0.4576054206335382),
   E            call(10.185369522033069),
   E            call(0.3881695575480826),
   E            call(14.601938371047803),
   E            call(3.464100900781081),
   E            call(13.651888163702226),
   E            call(11.518445750880675),
   E            call(11.805540294945818),
   E            call(1.6654214641285763),
   E            call(12.116858954152775),
   E            call(0.4274309662069142),
   E            call(2.1419843921887987),
   E            call(15.108680099488625),
   E            call(14.200441665349484),
   E            call(17.461868373600186),
   E            call(9.176234144014751),
   E            call(7.823477464693354),
   E            call(9.709087497895718),
   E            call(12.996552399230477),
   E            call(17.364527077318467),
   E            call(12.750015645644263),
   E            call(9.405866521202778),
   E            call(5.319519737900769),
   E            call(2.652528572765807),
   E            call(6.5879276596641585),
   E            call(1.2949168686974422)].
   
   /usr/local/lib/python3.10/unittest/mock.py:908: AssertionError
   ----------------------------- Captured stderr call -----------------------------
   INFO  [airflow.providers.databricks.triggers.databricks.DatabricksExecutionTrigger] run-id 1 in run state {'life_cycle_state': 'PENDING', 'result_state': '', 'state_message': ''}. sleeping for 30 seconds
   ------------------------------ Captured log call -------------------------------
   INFO     airflow.providers.databricks.triggers.databricks.DatabricksExecutionTrigger:databricks.py:94 run-id 1 in run state {'life_cycle_state': 'PENDING', 'result_state': '', 'state_message': ''}. sleeping for 30 seconds
   - generated xml file: /files/test_result-providers_-amazon_google-postgres.xml -
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org