You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ka...@apache.org on 2025/04/23 18:22:25 UTC
(airflow) branch v3-0-test updated (04950b16441 -> 69debd3de60)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a change to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
from 04950b16441 [v3-0-test] Fix missing version substitution in docs (#49640) (#49642)
new e76879de5e8 Add `STRAIGHT_JOIN` prefix for MySQL query optimization in get_sorted_triggers (#46303)
new 9dae52a3d3b feature(ui): Make entire task box clickable to select the task in Air… (#49299)
new 88c498b2f46 Remove unused constant HANDLER_SUPPORTS_TRIGGERER (#49370)
new 63ec6f4cc7e Updating Airflow executor docs for AF3 (#49389)
new 020be9443a6 Vertically align task log header components in fullscreen mode. (#49569)
new 016350901d0 Ensure sqlalchemy[asyncio] extra is in core deps (#49452)
new 77e671514df Remove dag_code records with no serdag (#49478)
new 1a4ce7c4ecd Logout functionality should respect base_url in api server (#49545)
new 5b833c62110 Remove `gitpython` as a core dependency (#49537)
new 46e26489800 Remove sort indicators on XCom table (#49547)
new 416fdb6046a Null out the scheduler_interval field on downgrade (#49583)
new 69debd3de60 Update screenshots for Dags - Update GIFs (#49586)
The 12 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
airflow-core/docs/core-concepts/executor/index.rst | 40 ++++++++++++++++++---
airflow-core/docs/img/airflow_erd.sha256 | 2 +-
airflow-core/docs/img/ui-dark/task_group.gif | Bin 224817 -> 418819 bytes
airflow-core/docs/img/ui-light/task_group.gif | Bin 177836 -> 353262 bytes
.../docs/installation/upgrading_to_airflow3.rst | 5 +++
airflow-core/pyproject.toml | 3 +-
.../src/airflow/jobs/triggerer_job_runner.py | 8 -----
...3_0_0_rename_schedule_interval_to_timetable_.py | 1 +
.../versions/0047_3_0_0_add_dag_versioning.py | 3 ++
airflow-core/src/airflow/models/trigger.py | 1 +
.../airflow/ui/src/components/Graph/AssetNode.tsx | 6 ++--
.../airflow/ui/src/components/Graph/DagNode.tsx | 13 ++++---
.../airflow/ui/src/components/Graph/TaskLink.tsx | 27 +++++++-------
.../airflow/ui/src/components/Graph/TaskNode.tsx | 22 ++++++------
.../src/airflow/ui/src/layouts/Nav/LogoutModal.tsx | 6 +++-
.../ui/src/pages/TaskInstance/Logs/Logs.tsx | 24 +++++++------
.../src/airflow/ui/src/pages/XCom/XCom.tsx | 3 ++
17 files changed, 106 insertions(+), 58 deletions(-)
(airflow) 02/12: feature(ui): Make entire task box clickable to select the task in Air… (#49299)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 9dae52a3d3b588dd154d13146aef0a434463d0c2
Author: Jason <46...@users.noreply.github.com>
AuthorDate: Thu Apr 24 00:36:44 2025 +0700
feature(ui): Make entire task box clickable to select the task in Air… (#49299)
* feature(ui): Make entire task node clickable to select the task
* Add z-index to TogglePause for clickability
* Remove link component
* Remove unnecessary onClick handler from LinkOverlay
* Forward ref from LinkOverlay
(cherry picked from commit 20bdf5fc82415b6d67237794fd49174703af9230)
---
.../airflow/ui/src/components/Graph/AssetNode.tsx | 6 ++---
.../airflow/ui/src/components/Graph/DagNode.tsx | 13 +++++++----
.../airflow/ui/src/components/Graph/TaskLink.tsx | 27 +++++++++++-----------
.../airflow/ui/src/components/Graph/TaskNode.tsx | 22 ++++++++++--------
4 files changed, 37 insertions(+), 31 deletions(-)
diff --git a/airflow-core/src/airflow/ui/src/components/Graph/AssetNode.tsx b/airflow-core/src/airflow/ui/src/components/Graph/AssetNode.tsx
index 3d4539caf01..b95faf0fd86 100644
--- a/airflow-core/src/airflow/ui/src/components/Graph/AssetNode.tsx
+++ b/airflow-core/src/airflow/ui/src/components/Graph/AssetNode.tsx
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-import { Flex, Heading, HStack, Link, Text } from "@chakra-ui/react";
+import { Flex, Heading, HStack, LinkOverlay, Text } from "@chakra-ui/react";
import type { NodeProps, Node as NodeType } from "@xyflow/react";
import { FiDatabase } from "react-icons/fi";
import { useParams, Link as RouterLink } from "react-router-dom";
@@ -69,9 +69,9 @@ export const AssetNode = ({
<Heading ml={-2} size="sm">
<FiDatabase />
</Heading>
- <Link asChild color="fg.info">
+ <LinkOverlay asChild>
<RouterLink to={`/assets/${assetId}`}>{label}</RouterLink>
- </Link>
+ </LinkOverlay>
</HStack>
{assetEvent === undefined ? undefined : (
<>
diff --git a/airflow-core/src/airflow/ui/src/components/Graph/DagNode.tsx b/airflow-core/src/airflow/ui/src/components/Graph/DagNode.tsx
index 73dfaf537f4..83b3db2d4d0 100644
--- a/airflow-core/src/airflow/ui/src/components/Graph/DagNode.tsx
+++ b/airflow-core/src/airflow/ui/src/components/Graph/DagNode.tsx
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-import { Flex, HStack, Link } from "@chakra-ui/react";
+import { Flex, HStack, LinkOverlay } from "@chakra-ui/react";
import type { NodeProps, Node as NodeType } from "@xyflow/react";
import { Link as RouterLink } from "react-router-dom";
@@ -47,11 +47,16 @@ export const DagNode = ({
>
<HStack alignItems="center" justifyContent="space-between">
<DagIcon />
- <TogglePause dagId={dag?.dag_id ?? label} disabled={!Boolean(dag)} isPaused={dag?.is_paused} />
+ <TogglePause
+ dagId={dag?.dag_id ?? label}
+ disabled={!Boolean(dag)}
+ isPaused={dag?.is_paused}
+ style={{ zIndex: 2 }}
+ />
</HStack>
- <Link asChild color="fg.info" mb={2}>
+ <LinkOverlay asChild>
<RouterLink to={`/dags/${dag?.dag_id ?? label}`}>{dag?.dag_display_name ?? label}</RouterLink>
- </Link>
+ </LinkOverlay>
</Flex>
</NodeWrapper>
);
diff --git a/airflow-core/src/airflow/ui/src/components/Graph/TaskLink.tsx b/airflow-core/src/airflow/ui/src/components/Graph/TaskLink.tsx
index 685e79737ec..278c6af3092 100644
--- a/airflow-core/src/airflow/ui/src/components/Graph/TaskLink.tsx
+++ b/airflow-core/src/airflow/ui/src/components/Graph/TaskLink.tsx
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-import { Link } from "@chakra-ui/react";
+import { forwardRef } from "react";
import { useParams, useSearchParams, Link as RouterLink } from "react-router-dom";
import { TaskName, type TaskNameProps } from "src/components/TaskName";
@@ -25,7 +25,7 @@ type Props = {
readonly id: string;
} & TaskNameProps;
-export const TaskLink = ({ id, isGroup, isMapped, ...rest }: Props) => {
+export const TaskLink = forwardRef<HTMLAnchorElement, Props>(({ id, isGroup, isMapped, ...rest }, ref) => {
const { dagId = "", runId, taskId } = useParams();
const [searchParams] = useSearchParams();
@@ -35,16 +35,15 @@ export const TaskLink = ({ id, isGroup, isMapped, ...rest }: Props) => {
}
return (
- <Link asChild data-testid={id}>
- <RouterLink
- to={{
- // Do not include runId if there is no selected run, clicking a second time will deselect a task id
- pathname: `/dags/${dagId}/${runId === undefined ? "" : `runs/${runId}/`}${taskId === id ? "" : `tasks/${id}`}${isMapped && taskId !== id && runId !== undefined ? "/mapped" : ""}`,
- search: searchParams.toString(),
- }}
- >
- <TaskName isMapped={isMapped} {...rest} />
- </RouterLink>
- </Link>
+ <RouterLink
+ ref={ref}
+ to={{
+ // Do not include runId if there is no selected run, clicking a second time will deselect a task id
+ pathname: `/dags/${dagId}/${runId === undefined ? "" : `runs/${runId}/`}${taskId === id ? "" : `tasks/${id}`}${isMapped && taskId !== id && runId !== undefined ? "/mapped" : ""}`,
+ search: searchParams.toString(),
+ }}
+ >
+ <TaskName isMapped={isMapped} {...rest} />
+ </RouterLink>
);
-};
+});
diff --git a/airflow-core/src/airflow/ui/src/components/Graph/TaskNode.tsx b/airflow-core/src/airflow/ui/src/components/Graph/TaskNode.tsx
index 657b6d51cd1..6f513066a13 100644
--- a/airflow-core/src/airflow/ui/src/components/Graph/TaskNode.tsx
+++ b/airflow-core/src/airflow/ui/src/components/Graph/TaskNode.tsx
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-import { Box, Button, Flex, HStack, Text } from "@chakra-ui/react";
+import { Box, Button, Flex, HStack, LinkOverlay, Text } from "@chakra-ui/react";
import type { NodeProps, Node as NodeType } from "@xyflow/react";
import { CgRedo } from "react-icons/cg";
@@ -78,15 +78,17 @@ export const TaskNode = ({
width={`${width + (isSelected ? 4 : 0)}px`}
>
<Box>
- <TaskLink
- childCount={taskInstance?.task_count}
- id={id}
- isGroup={isGroup}
- isMapped={isMapped}
- isOpen={isOpen}
- label={label}
- setupTeardownType={setupTeardownType}
- />
+ <LinkOverlay asChild>
+ <TaskLink
+ childCount={taskInstance?.task_count}
+ id={id}
+ isGroup={isGroup}
+ isMapped={isMapped}
+ isOpen={isOpen}
+ label={label}
+ setupTeardownType={setupTeardownType}
+ />
+ </LinkOverlay>
<Text color="fg.muted" fontSize="sm" textTransform="capitalize">
{isGroup ? "Task Group" : operator}
</Text>
(airflow) 04/12: Updating Airflow executor docs for AF3 (#49389)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 63ec6f4cc7e1668cc0fdbb570f8a448fa6e620e9
Author: Amogh Desai <am...@gmail.com>
AuthorDate: Wed Apr 23 09:50:19 2025 +0530
Updating Airflow executor docs for AF3 (#49389)
(cherry picked from commit eca6a53110e7365bc013a181ea5a4746444ddc9e)
---
airflow-core/docs/core-concepts/executor/index.rst | 40 +++++++++++++++++++---
1 file changed, 36 insertions(+), 4 deletions(-)
diff --git a/airflow-core/docs/core-concepts/executor/index.rst b/airflow-core/docs/core-concepts/executor/index.rst
index c83f10cb050..33427ff2bcf 100644
--- a/airflow-core/docs/core-concepts/executor/index.rst
+++ b/airflow-core/docs/core-concepts/executor/index.rst
@@ -48,8 +48,6 @@ If you want to check which executor is currently set, you can use the ``airflow
$ airflow config get-value core executor
LocalExecutor
-
-
Executor Types
--------------
@@ -232,6 +230,40 @@ Some reasons you may want to write a custom executor include:
* You'd like to use an executor that leverages a compute service from your preferred cloud provider.
* You have a private tool/service for task execution that is only available to you or your organization.
+Workloads
+^^^^^^^^^
+
+A workload in context of an Executor is the fundamental unit of execution for an executor. It represents a discrete
+operation or job that the executor runs on a worker. For example, it can run user code encapsulated in an Airflow task
+on a worker.
+
+Example:
+
+.. code-block:: python
+
+ ExecuteTask(
+ token="mock",
+ ti=TaskInstance(
+ id=UUID("4d828a62-a417-4936-a7a6-2b3fabacecab"),
+ task_id="mock",
+ dag_id="mock",
+ run_id="mock",
+ try_number=1,
+ map_index=-1,
+ pool_slots=1,
+ queue="default",
+ priority_weight=1,
+ executor_config=None,
+ parent_context_carrier=None,
+ context_carrier=None,
+ queued_dttm=None,
+ ),
+ dag_rel_path=PurePosixPath("mock.py"),
+ bundle_info=BundleInfo(name="n/a", version="no matter"),
+ log_path="mock.log",
+ type="ExecuteTask",
+ )
+
Important BaseExecutor Methods
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -239,7 +271,7 @@ Important BaseExecutor Methods
These methods don't require overriding to implement your own executor, but are useful to be aware of:
* ``heartbeat``: The Airflow scheduler Job loop will periodically call heartbeat on the executor. This is one of the main points of interaction between the Airflow scheduler and the executor. This method updates some metrics, triggers newly queued tasks to execute and updates state of running/completed tasks.
-* ``queue_command``: The Airflow Executor will call this method of the BaseExecutor to provide tasks to be run by the executor. The BaseExecutor simply adds the TaskInstances to an internal list of queued tasks within the executor.
+* ``queue_workload``: The Airflow Executor will call this method of the BaseExecutor to provide tasks to be run by the executor. The BaseExecutor simply adds the *workloads* (check section above to understand) to an internal list of queued workloads to run within the executor. All executors present in the repository use this method.
* ``get_event_buffer``: The Airflow scheduler calls this method to retrieve the current state of the TaskInstances the executor is executing.
* ``has_task``: The scheduler uses this BaseExecutor method to determine if an executor already has a specific task instance queued or running.
* ``send_callback``: Sends any callbacks to the sink configured on the executor.
@@ -251,7 +283,7 @@ Mandatory Methods to Implement
The following methods must be overridden at minimum to have your executor supported by Airflow:
* ``sync``: Sync will get called periodically during executor heartbeats. Implement this method to update the state of the tasks which the executor knows about. Optionally, attempting to execute queued tasks that have been received from the scheduler.
-* ``execute_async``: Executes a command asynchronously. A command in this context is an Airflow CLI command to run an Airflow task. This method is called (after a few layers) during executor heartbeat which is run periodically by the scheduler. In practice, this method often just enqueues tasks into an internal or external queue of tasks to be run (e.g. ``KubernetesExecutor``). But can also execute the tasks directly as well (e.g. ``LocalExecutor``). This will depend on the executor.
+* ``execute_async``: Executes a *workload* asynchronously. This method is called (after a few layers) during executor heartbeat which is run periodically by the scheduler. In practice, this method often just enqueues tasks into an internal or external queue of tasks to be run (e.g. ``KubernetesExecutor``). But can also execute the tasks directly as well (e.g. ``LocalExecutor``). This will depend on the executor.
Optional Interface Methods to Implement
(airflow) 08/12: Logout functionality should respect base_url in api server (#49545)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1a4ce7c4ecd2be48fb9560ace0d754a8f9bf10a4
Author: Amogh Desai <am...@gmail.com>
AuthorDate: Wed Apr 23 09:49:03 2025 +0530
Logout functionality should respect base_url in api server (#49545)
(cherry picked from commit d17ca5c481beb984129737d2078834f95e24fecd)
---
airflow-core/src/airflow/ui/src/layouts/Nav/LogoutModal.tsx | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/airflow-core/src/airflow/ui/src/layouts/Nav/LogoutModal.tsx b/airflow-core/src/airflow/ui/src/layouts/Nav/LogoutModal.tsx
index ba6ae08eca0..97b5e67cb35 100644
--- a/airflow-core/src/airflow/ui/src/layouts/Nav/LogoutModal.tsx
+++ b/airflow-core/src/airflow/ui/src/layouts/Nav/LogoutModal.tsx
@@ -31,8 +31,12 @@ const LogoutModal: React.FC<LogoutModalProps> = ({ isOpen, onClose }) => (
<ConfirmationModal
header="Logout"
onConfirm={() => {
+ const baseHref = document.querySelector("head>base")?.getAttribute("href") ?? "";
+ const baseUrl = new URL(baseHref, globalThis.location.origin);
+ const logoutPath = new URL("api/v2/auth/logout", baseUrl).pathname;
+
localStorage.removeItem(TOKEN_STORAGE_KEY);
- globalThis.location.replace(`/api/v2/auth/logout`);
+ globalThis.location.replace(logoutPath);
}}
onOpenChange={onClose}
open={isOpen}
(airflow) 10/12: Remove sort indicators on XCom table (#49547)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 46e2648980084956039d0946ca659ce4483a2e31
Author: Pierre Jeambrun <pi...@gmail.com>
AuthorDate: Tue Apr 22 16:31:18 2025 +0200
Remove sort indicators on XCom table (#49547)
(cherry picked from commit c970317e18e4870f0b47b16f08293373e1e3a21e)
---
airflow-core/src/airflow/ui/src/pages/XCom/XCom.tsx | 3 +++
1 file changed, 3 insertions(+)
diff --git a/airflow-core/src/airflow/ui/src/pages/XCom/XCom.tsx b/airflow-core/src/airflow/ui/src/pages/XCom/XCom.tsx
index 39445c375ec..9614bf3d8d5 100644
--- a/airflow-core/src/airflow/ui/src/pages/XCom/XCom.tsx
+++ b/airflow-core/src/airflow/ui/src/pages/XCom/XCom.tsx
@@ -43,6 +43,7 @@ const columns: Array<ColumnDef<XComResponse>> = [
<RouterLink to={`/dags/${original.dag_id}`}>{original.dag_id}</RouterLink>
</Link>
),
+ enableSorting: false,
header: "Dag",
},
{
@@ -54,6 +55,7 @@ const columns: Array<ColumnDef<XComResponse>> = [
</RouterLink>
</Link>
),
+ enableSorting: false,
header: "Run Id",
},
{
@@ -77,6 +79,7 @@ const columns: Array<ColumnDef<XComResponse>> = [
},
{
accessorKey: "map_index",
+ enableSorting: false,
header: "Map Index",
},
{
(airflow) 12/12: Update screenshots for Dags - Update GIFs (#49586)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 69debd3de60e81a66cd3fc008f1d8dfc630f8e7b
Author: Jens Scheffler <95...@users.noreply.github.com>
AuthorDate: Tue Apr 22 23:56:15 2025 +0200
Update screenshots for Dags - Update GIFs (#49586)
(cherry picked from commit 3a6b821fbe380234e28f5415950e42799c6bf895)
---
airflow-core/docs/img/ui-dark/task_group.gif | Bin 224817 -> 418819 bytes
airflow-core/docs/img/ui-light/task_group.gif | Bin 177836 -> 353262 bytes
2 files changed, 0 insertions(+), 0 deletions(-)
diff --git a/airflow-core/docs/img/ui-dark/task_group.gif b/airflow-core/docs/img/ui-dark/task_group.gif
index 32ed29f8b2c..88e74965dde 100644
Binary files a/airflow-core/docs/img/ui-dark/task_group.gif and b/airflow-core/docs/img/ui-dark/task_group.gif differ
diff --git a/airflow-core/docs/img/ui-light/task_group.gif b/airflow-core/docs/img/ui-light/task_group.gif
index 39b7f94da12..743e86667d3 100644
Binary files a/airflow-core/docs/img/ui-light/task_group.gif and b/airflow-core/docs/img/ui-light/task_group.gif differ
(airflow) 06/12: Ensure sqlalchemy[asyncio] extra is in core deps (#49452)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 016350901d04779aac980ac0edbf56285d38989f
Author: Daniel Standish <15...@users.noreply.github.com>
AuthorDate: Tue Apr 22 13:27:45 2025 -0700
Ensure sqlalchemy[asyncio] extra is in core deps (#49452)
This is required for async sqlalchemy operations. This surfaces e.g. when we install with `--no-dev` and run the api-server, we'll get missing module `greenlet` error in api-server logs.
(cherry picked from commit 0c30c6e9885a48558a1aa87c73ab12097f758fda)
---
airflow-core/pyproject.toml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/airflow-core/pyproject.toml b/airflow-core/pyproject.toml
index d1b1e38fc9a..1cf72c33e5b 100644
--- a/airflow-core/pyproject.toml
+++ b/airflow-core/pyproject.toml
@@ -124,7 +124,7 @@ dependencies = [
# See https://sqlalche.me/e/b8d9 for details of deprecated features
# you can set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.
# The issue tracking it is https://github.com/apache/airflow/issues/28723
- "sqlalchemy>=1.4.49,<2.0",
+ "sqlalchemy[asyncio]>=1.4.49,<2.0",
"sqlalchemy-jsonfield>=1.0",
"sqlalchemy-utils>=0.41.2",
"svcs>=25.1.0",
(airflow) 03/12: Remove unused constant HANDLER_SUPPORTS_TRIGGERER (#49370)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 88c498b2f4696a9b2a5b12f38bd5e3358fc4f2c4
Author: Daniel Standish <15...@users.noreply.github.com>
AuthorDate: Tue Apr 22 13:59:59 2025 -0700
Remove unused constant HANDLER_SUPPORTS_TRIGGERER (#49370)
This was obsoleted by logging changes in task sdk, but its removal was missed, it seems.
(cherry picked from commit adffb476581b75e68f9a92fb487d902253355b8d)
---
airflow-core/src/airflow/jobs/triggerer_job_runner.py | 8 --------
1 file changed, 8 deletions(-)
diff --git a/airflow-core/src/airflow/jobs/triggerer_job_runner.py b/airflow-core/src/airflow/jobs/triggerer_job_runner.py
index 05ba3dd28ad..0752b59a01e 100644
--- a/airflow-core/src/airflow/jobs/triggerer_job_runner.py
+++ b/airflow-core/src/airflow/jobs/triggerer_job_runner.py
@@ -81,14 +81,6 @@ if TYPE_CHECKING:
from airflow.sdk.types import RuntimeTaskInstanceProtocol as RuntimeTI
from airflow.triggers.base import BaseTrigger
-HANDLER_SUPPORTS_TRIGGERER = False
-"""
-If this value is true, root handler is configured to log individual trigger messages
-visible in task logs.
-
-:meta private:
-"""
-
logger = logging.getLogger(__name__)
__all__ = [
(airflow) 09/12: Remove `gitpython` as a core dependency (#49537)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5b833c62110ceb8fe087e95bbb6487ae05c73de2
Author: Jed Cunningham <66...@users.noreply.github.com>
AuthorDate: Tue Apr 22 02:27:25 2025 -0600
Remove `gitpython` as a core dependency (#49537)
This was added when `GitDagBundle` lived in core, but it now lives in
the git provider. We no longer need gitpython as a core dependency.
(cherry picked from commit 6a970574cdf03dd4d646d8f46c4b0012719163a0)
---
airflow-core/pyproject.toml | 1 -
1 file changed, 1 deletion(-)
diff --git a/airflow-core/pyproject.toml b/airflow-core/pyproject.toml
index 1cf72c33e5b..cd475390eac 100644
--- a/airflow-core/pyproject.toml
+++ b/airflow-core/pyproject.toml
@@ -86,7 +86,6 @@ dependencies = [
"fastapi[standard]>=0.115.0,!=0.115.10",
# We could get rid of flask and gunicorn if we replace serve_logs with a starlette + unicorn
"flask>=2.1.1",
- "gitpython>=3.1.40",
# We could get rid of flask and gunicorn if we replace serve_logs with a starlette + unicorn
"gunicorn>=20.1.0",
"httpx>=0.25.0",
(airflow) 05/12: Vertically align task log header components in fullscreen mode. (#49569)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 020be9443a628938a4a6fbc163ad4a3619ac7e1c
Author: Karthikeyan Singaravelan <ti...@gmail.com>
AuthorDate: Wed Apr 23 00:45:27 2025 +0530
Vertically align task log header components in fullscreen mode. (#49569)
Co-authored-by: Shubham Raj <48...@users.noreply.github.com>
(cherry picked from commit c862e212e0ed6c6e1e4e1ee5176f1eeda98ce651)
---
.../ui/src/pages/TaskInstance/Logs/Logs.tsx | 24 ++++++++++++----------
1 file changed, 13 insertions(+), 11 deletions(-)
diff --git a/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx b/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
index e06c301da2c..931caf50d6e 100644
--- a/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
+++ b/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-import { Box, Heading } from "@chakra-ui/react";
+import { Box, Heading, VStack } from "@chakra-ui/react";
import { useState } from "react";
import { useParams, useSearchParams } from "react-router-dom";
@@ -104,16 +104,18 @@ export const Logs = () => {
<Dialog.Root onOpenChange={onOpenChange} open={fullscreen} scrollBehavior="inside" size="full">
<Dialog.Content backdrop>
<Dialog.Header>
- <Heading size="xl">{taskId}</Heading>
- <TaskLogHeader
- isFullscreen
- onSelectTryNumber={onSelectTryNumber}
- taskInstance={taskInstance}
- toggleFullscreen={toggleFullscreen}
- toggleWrap={toggleWrap}
- tryNumber={tryNumber}
- wrap={wrap}
- />
+ <VStack gap={2}>
+ <Heading size="xl">{taskId}</Heading>
+ <TaskLogHeader
+ isFullscreen
+ onSelectTryNumber={onSelectTryNumber}
+ taskInstance={taskInstance}
+ toggleFullscreen={toggleFullscreen}
+ toggleWrap={toggleWrap}
+ tryNumber={tryNumber}
+ wrap={wrap}
+ />
+ </VStack>
</Dialog.Header>
<Dialog.CloseTrigger />
(airflow) 01/12: Add `STRAIGHT_JOIN` prefix for MySQL query optimization in get_sorted_triggers (#46303)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e76879de5e8ec3ad64cd81ecdd504fc9fa10e52a
Author: oboki <ob...@kakao.com>
AuthorDate: Tue Apr 22 18:48:00 2025 +0900
Add `STRAIGHT_JOIN` prefix for MySQL query optimization in get_sorted_triggers (#46303)
This PR adds a STRAIGHT_JOIN hint to prevent an unintended full scan of TaskInstance table in get_sorted_triggers.
With MySQL backend, if there are many records in the trigger, it causes the task_instance to be scanned first as the driving table, leading to slow queries.
(cherry picked from commit 9f0ac9aeba7c64dc8a950470089c0a6868afc4c7)
---
airflow-core/src/airflow/models/trigger.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/airflow-core/src/airflow/models/trigger.py b/airflow-core/src/airflow/models/trigger.py
index 8bcff1d470e..94d8360edf9 100644
--- a/airflow-core/src/airflow/models/trigger.py
+++ b/airflow-core/src/airflow/models/trigger.py
@@ -350,6 +350,7 @@ class Trigger(Base):
"""
query = with_row_locks(
select(cls.id)
+ .prefix_with("STRAIGHT_JOIN", dialect="mysql")
.join(TaskInstance, cls.id == TaskInstance.trigger_id, isouter=False)
.where(or_(cls.triggerer_id.is_(None), cls.triggerer_id.not_in(alive_triggerer_ids)))
.order_by(coalesce(TaskInstance.priority_weight, 0).desc(), cls.created_date)
(airflow) 07/12: Remove dag_code records with no serdag (#49478)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 77e671514dfb5e83087bcd897824b8ec99a484b4
Author: Daniel Standish <15...@users.noreply.github.com>
AuthorDate: Mon Apr 21 11:38:14 2025 -0700
Remove dag_code records with no serdag (#49478)
It is assumed by the migration process that there will not be any dag_code records that don't have a serdag record.
It seems this only happens when there are two files with the same dag id.
---------
Co-authored-by: Tzu-ping Chung <ur...@gmail.com>
(cherry picked from commit 42024188b9553dd50072e717af754cb95c901972)
---
airflow-core/docs/img/airflow_erd.sha256 | 2 +-
airflow-core/docs/installation/upgrading_to_airflow3.rst | 5 +++++
.../src/airflow/migrations/versions/0047_3_0_0_add_dag_versioning.py | 3 +++
3 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/airflow-core/docs/img/airflow_erd.sha256 b/airflow-core/docs/img/airflow_erd.sha256
index 33fe09fd943..5ec0d8e84d1 100644
--- a/airflow-core/docs/img/airflow_erd.sha256
+++ b/airflow-core/docs/img/airflow_erd.sha256
@@ -1 +1 @@
-bc93e7288a7a8355b15dc721accaf80260f370f3afa0d478248f9fe4692a1f1d
\ No newline at end of file
+2a3ce828be7c3800cb61de4c081fcf46f1f94e31ac732f831e65a8a9581b5bb0
\ No newline at end of file
diff --git a/airflow-core/docs/installation/upgrading_to_airflow3.rst b/airflow-core/docs/installation/upgrading_to_airflow3.rst
index 04c9bda2fa0..d1e094bfb45 100644
--- a/airflow-core/docs/installation/upgrading_to_airflow3.rst
+++ b/airflow-core/docs/installation/upgrading_to_airflow3.rst
@@ -41,6 +41,11 @@ Step 2: Clean and back up your existing Airflow Instance
upgrade process. These schema changes can take a long time if the database is large. For a faster, safer migration, we recommend that you clean up your Airflow meta-database before the upgrade.
You can use the ``airflow db clean`` :ref:`Airflow CLI command<cli-db-clean>` to trim your Airflow database.
+- Ensure that there are no errors related to dag processing, such as ``AirflowDagDuplicatedIdException``. You should
+ be able to run ``airflow dags reserialize`` with no errors. If you have have to resolve errors from dag processing,
+ ensure you deploy your changes to your old instance prior to upgrade, and wait until your dags have all been reprocessed
+ (and all errors gone) before you proceed with upgrade.
+
Step 3: DAG Authors - Check your Airflow DAGs for compatibility
----------------------------------------------------------------
diff --git a/airflow-core/src/airflow/migrations/versions/0047_3_0_0_add_dag_versioning.py b/airflow-core/src/airflow/migrations/versions/0047_3_0_0_add_dag_versioning.py
index bd0a8ea725f..b6b08632f9f 100644
--- a/airflow-core/src/airflow/migrations/versions/0047_3_0_0_add_dag_versioning.py
+++ b/airflow-core/src/airflow/migrations/versions/0047_3_0_0_add_dag_versioning.py
@@ -66,6 +66,9 @@ def _airflow_2_fileloc_hash(fileloc):
def upgrade():
"""Apply add dag versioning."""
conn = op.get_bind()
+
+ op.execute("DELETE FROM dag_code WHERE fileloc_hash NOT IN (SELECT fileloc_hash FROM serialized_dag)")
+
op.create_table(
"dag_version",
sa.Column("id", UUIDType(binary=False), nullable=False),
(airflow) 11/12: Null out the scheduler_interval field on downgrade (#49583)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 416fdb6046af0f26493135ea74cc596359e189c4
Author: Daniel Standish <15...@users.noreply.github.com>
AuthorDate: Tue Apr 22 13:30:49 2025 -0700
Null out the scheduler_interval field on downgrade (#49583)
In 2.x, it was expected that this field was valid json -- e.g. `"0 0 * * *"`.
In 3.x, the column is renamed and repurposed and not json.
So if we downgrade it, we need to null it out. It will get updated when dag is reserialized.
If we *don't* do this, the scheduler crashes.
(cherry picked from commit 1bad247c2ad0e86a1816ac09a65bb944bde40085)
---
airflow-core/docs/img/airflow_erd.sha256 | 2 +-
.../versions/0030_3_0_0_rename_schedule_interval_to_timetable_.py | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/airflow-core/docs/img/airflow_erd.sha256 b/airflow-core/docs/img/airflow_erd.sha256
index 5ec0d8e84d1..fae49f1bfab 100644
--- a/airflow-core/docs/img/airflow_erd.sha256
+++ b/airflow-core/docs/img/airflow_erd.sha256
@@ -1 +1 @@
-2a3ce828be7c3800cb61de4c081fcf46f1f94e31ac732f831e65a8a9581b5bb0
\ No newline at end of file
+94023084fb9e30f248831a04f0bbabf5b7d90332b7edc013c7a5305340c4e1bf
\ No newline at end of file
diff --git a/airflow-core/src/airflow/migrations/versions/0030_3_0_0_rename_schedule_interval_to_timetable_.py b/airflow-core/src/airflow/migrations/versions/0030_3_0_0_rename_schedule_interval_to_timetable_.py
index b4434a65f3f..16f78293310 100644
--- a/airflow-core/src/airflow/migrations/versions/0030_3_0_0_rename_schedule_interval_to_timetable_.py
+++ b/airflow-core/src/airflow/migrations/versions/0030_3_0_0_rename_schedule_interval_to_timetable_.py
@@ -58,3 +58,4 @@ def downgrade():
type_=sa.Text,
nullable=True,
)
+ op.execute("UPDATE dag SET schedule_interval=NULL;")