You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "mobuchowski (via GitHub)" <gi...@apache.org> on 2024/08/30 12:33:25 UTC

[PR] move to dag_run.logical_date from execution date in OpenLineage provider [airflow]

mobuchowski opened a new pull request, #41889:
URL: https://github.com/apache/airflow/pull/41889

   Remove usage of deprecated `execution_date` in favor of `logical_date`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] move to dag_run.logical_date from execution date in OpenLineage provider [airflow]

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on PR #41889:
URL: https://github.com/apache/airflow/pull/41889#issuecomment-2324231418

   @potiuk `logical_date` property was added back in 2021
   
   <img width="579" alt="Screenshot 2024-09-02 at 11 14 00" src="https://github.com/user-attachments/assets/431b102b-277b-4d0c-8e83-af3fc2d2b206">
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] move to dag_run.logical_date from execution date in OpenLineage provider [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41889:
URL: https://github.com/apache/airflow/pull/41889#issuecomment-2324408518

   > logical_date teproperty was added back in 2021
   
   I know, just checking if it is ok to change it in the provider - which will be (for the next 6 months or so) only installed on Airflow 2 :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] move to dag_run.logical_date from execution date in OpenLineage provider [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41889:
URL: https://github.com/apache/airflow/pull/41889#issuecomment-2324411594

   Simply execution_date and logical_date have different semantics, so my question is, if that will impact observed open-lineage events. And if so - whether it should be somehow explained in the changelog so that users are not surprised.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] move to dag_run.logical_date from execution date in OpenLineage provider [airflow]

Posted by "Avihais12344 (via GitHub)" <gi...@apache.org>.
Avihais12344 commented on code in PR #41889:
URL: https://github.com/apache/airflow/pull/41889#discussion_r1740162410


##########
tests/providers/openlineage/plugins/test_listener.py:
##########
@@ -210,7 +210,7 @@ def mock_task_id(dag_id, task_id, try_number, execution_date):
     task_instance.state = State.RUNNING
     task_instance.start_date = dt.datetime(2023, 1, 1, 13, 1, 1)
     task_instance.end_date = dt.datetime(2023, 1, 3, 13, 1, 1)
-    task_instance.execution_date = "execution_date"
+    task_instance.execution_date = "logical_date"

Review Comment:
   It should be a certain date, and not just a string, I think. Maybe we can fix it here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] move to dag_run.logical_date from execution date in OpenLineage provider [airflow]

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski merged PR #41889:
URL: https://github.com/apache/airflow/pull/41889


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] move to dag_run.logical_date from execution date in OpenLineage provider [airflow]

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on PR #41889:
URL: https://github.com/apache/airflow/pull/41889#issuecomment-2324463504

   @potiuk does it really have different semantics?
   
   From what I can find, it's essentially a rename: 
   
   From AIP-39
   > We need a single date value that is the "canonical" date for the DagRun date, that can be used for sorting DagRuns for the UI view, so the current "execution_date" column on DagRun needs to be kept, but we should rename - and we've chosen "logical_date".
   
   Also [in the docstring](https://github.com/apache/airflow/pull/16352/files#diff-ffc28f0462bb68924dcaa55031ad885759562f6cb589744dd1e123dbbcd2ecf6R88):
   > This replaces ``execution_date`` in Airflow 2.1 and prior. The idea is essentially the same, just a different name.
   
   Basically, we want the same: the time when the DagRun is scheduled to be run, without consideration if it's actually going to run at that time, due to scheduler being backed up or other reason. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] move to dag_run.logical_date from execution date in OpenLineage provider [airflow]

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #41889:
URL: https://github.com/apache/airflow/pull/41889#discussion_r1740792145


##########
tests/providers/openlineage/plugins/test_listener.py:
##########
@@ -210,7 +210,7 @@ def mock_task_id(dag_id, task_id, try_number, execution_date):
     task_instance.state = State.RUNNING
     task_instance.start_date = dt.datetime(2023, 1, 1, 13, 1, 1)
     task_instance.end_date = dt.datetime(2023, 1, 3, 13, 1, 1)
-    task_instance.execution_date = "execution_date"
+    task_instance.execution_date = "logical_date"

Review Comment:
   Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] move to dag_run.logical_date from execution date in OpenLineage provider [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41889:
URL: https://github.com/apache/airflow/pull/41889#issuecomment-2324489881

   > Basically, we want the same: the time when the DagRun is scheduled to be run, without consideration if it's actually going to run at that time, due to scheduler being backed up or other reason.
   
   Yep. You are right. I had confused it with "start/end interval" and other macros available in context, but logical_date and execution_date are really aliases.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] move to dag_run.logical_date from execution date in OpenLineage provider [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41889:
URL: https://github.com/apache/airflow/pull/41889#issuecomment-2323494195

   Is it backwards-compatible with Airflow 2?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org