You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jason810496 (via GitHub)" <gi...@apache.org> on 2026/08/05 04:45:35 UTC
[PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
jason810496 opened a new pull request, #71144:
URL: https://github.com/apache/airflow/pull/71144
- related: #69288
## Why
`registerTask({ dagId, taskId }, handler)` binds one handler to a Python stub Dag/task pair at a time and leaves no object that can carry native TypeScript Dag declaration later (#69288). This PR changes only the user-facing authoring interface so it can grow into native Dag support without another breaking change.
## How
- `new Dag(dagId, spec?)` takes a positional id plus an optional trailing spec object; `dag.task(taskId, handler, spec?)` returns an opaque `Task` handle. Both `DagSpec` and `TaskSpec` are empty extension points today, so future fields (schedule, tags, retries, dependency wiring) land without signature changes.
- A `Dag` instance retains its spec and each task's `(taskId, handler, spec)`, so a future `serialize()` can produce the serialized Dag JSON for `DagFileParsingResult.serialized_dags`. Producing that JSON is intentionally out of scope here; `handleParse` still returns `serialized_dags: []`.
- Variadic `registerDags(...dags)` records Dag instances in the default registry; lookups delegate live to each Dag's task map. `startCoordinator()`, the coordinator wire protocol, and the bundle manifest shape (`{supervisor_schema_version, dags: {<dagId>: {tasks: [...]}}}`) are unchanged.
- Remove`registerTask`, `listRegisteredTasks`, and `TaskRegistration`.
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes, with help of Claude Code Fable 5 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3771104555
##########
ts-sdk/src/cli/pack.ts:
##########
@@ -143,20 +143,52 @@ function readBundleManifest(bundlePath: string): BundleManifest {
throw new Error(`Bundle produced no ${AIRFLOW_METADATA_FLAG} output`);
}
- let manifest: BundleManifest;
+ let parsed: unknown;
try {
- manifest = JSON.parse(line.slice(AIRFLOW_METADATA_SENTINEL.length)) as BundleManifest;
+ parsed = JSON.parse(line.slice(AIRFLOW_METADATA_SENTINEL.length));
} catch (error) {
throw new Error(`Bundle produced invalid ${AIRFLOW_METADATA_FLAG} output: ${String(error)}`, {
cause: error,
});
}
- if (!manifest.supervisor_schema_version || !manifest.dags || typeof manifest.dags !== "object") {
+ if (!isBundleManifest(parsed)) {
throw new Error(`Bundle produced incomplete ${AIRFLOW_METADATA_FLAG} output`);
}
+ const manifest = parsed;
+ // The line is whatever the bundle printed and nothing downstream re-validates
+ // it, so check each Dag entry down to the task-id element.
+ for (const [dagId, dag] of Object.entries(manifest.dags)) {
+ if (dag == null || !isDagIdList(dag.tasks)) {
+ throw new Error(
+ `Bundle produced ${AIRFLOW_METADATA_FLAG} output with a malformed entry for Dag "${dagId}"`,
+ );
+ }
+ }
return manifest;
}
+// The document is checked before anything is read off it: JSON.parse also yields
+// null and primitives, and `null.supervisor_schema_version` would surface as a
+// raw TypeError rather than a report about the bundle.
+function isBundleManifest(value: unknown): value is BundleManifest {
+ if (typeof value !== "object" || value === null || Array.isArray(value)) return false;
+ const { supervisor_schema_version: version, dags } = value as Partial<BundleManifest>;
+ return (
+ // Rendered into the manifest verbatim, where the schema requires a non-empty
+ // string, so a truthy number or boolean would travel to Airflow as-is.
+ typeof version === "string" &&
+ version.length > 0 &&
Review Comment:
Does this need to check the version format? Or do you think it’s too restricted?
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,258 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import { brand, hasBrand } from "./brand.js";
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
Review Comment:
Same question from the other direction—do we need this check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3800606195
##########
ts-sdk/src/cli/pack.ts:
##########
Review Comment:
Addressed in https://github.com/apache/airflow/pull/71144/commits/07f2533f83dfc2bad368710a2162c3389f18ce11, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3810401102
##########
ts-sdk/src/cli/pack.ts:
##########
@@ -129,9 +129,15 @@ function readBundleManifest(bundlePath: string): BundleManifest {
maxBuffer: MANIFEST_MAX_BUFFER_BYTES,
});
} catch (error) {
- throw new Error(`Running the bundle with ${AIRFLOW_METADATA_FLAG} failed: ${String(error)}`, {
- cause: error,
- });
+ const stderr = (error as { stderr?: string }).stderr ?? "";
+ const reported = stderr
+ .split("\n")
+ .find((line) => /^\w*Error: /.test(line.trim()))
+ ?.trim();
Review Comment:
This takes the first match, but I think we should take the last instead.
Note that the process for stdout below uses a `reverse` to find the last instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #71144:
URL: https://github.com/apache/airflow/pull/71144#issuecomment-5209656022
What’s the plan to do the external vs pure TS dags after this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3754987644
##########
ts-sdk/src/coordinator/runtime.ts:
##########
@@ -51,13 +52,46 @@ import {
type RuntimeTaskState,
type StartupDetails,
} from "./protocol.js";
-import { getRegisteredTask, listRegisteredTasks } from "../sdk/registry.js";
+import { DagRegistry } from "../sdk/registry.js";
import type { TaskContext, TaskHandlerArgs } from "../sdk/task.js";
import type { JsonValue } from "../sdk/client-types.js";
export const ABORT_GRACE_PERIOD_MS = 30_000;
export const COORDINATOR_RESPONSE_TIMEOUT_MS = 30_000;
+let served = false;
Review Comment:
This is the only place where the global variable lives and `serveDags` should only be called once. From my perspective, it's fine to keep it as global since the lifecycle and the intention is clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3748386152
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,251 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
+
+/**
+ * Dag-level options. **Reserved: no fields yet.**
+ *
+ * Native TypeScript Dag declaration (schedule, tags, ...) will add optional
+ * fields here without changing the `Dag` constructor, generated from the
+ * serialized-Dag JSON schema the way `src/generated/supervisor.ts` is. Until
+ * then only `{}` is accepted, so a field that would be silently dropped —
+ * `new Dag("d", { schedule: "@daily" })` — is a compile error rather than a
+ * Dag that packs and runs without the schedule.
+ */
+export type DagSpec = Record<string, never>;
+
+/**
+ * Task-level options. **Reserved: no fields yet.**
+ *
+ * Future task fields (retries, ...) will land here without changing the
+ * `dag.task()` signature. As with {@link DagSpec}, only `{}` is accepted today.
+ */
+export type TaskSpec = Record<string, never>;
+
+/**
+ * Opaque handle to a task registered on a {@link Dag}, returned by
+ * `dag.task(...)`.
+ *
+ * Identity only — the handler is deliberately not exposed. Handles are what the
+ * reserved `inputs` option accepts, and what native TypeScript Dag declaration
+ * will use to wire dependencies.
+ */
+export interface TaskRef {
+ /** Identifier of the Dag this task belongs to. */
+ readonly dagId: string;
+ /** Airflow task ID, including any TaskGroup prefix. */
+ readonly taskId: string;
+}
+
+/**
+ * Upstream task handles keyed by input name. **Reserved: validated and
+ * retained, but inert today** — see {@link TaskOptions.inputs}.
+ *
+ * Values must be handles returned by `dag.task(...)`. Literal values are
+ * deliberately out of scope for now; the future native-Dag work decides how
+ * they are declared.
+ */
+export type TaskInputs = Readonly<Record<string, TaskRef>>;
+
+/**
+ * Named options for `dag.task()`.
+ *
+ * Keyword-only so neither field has to be positioned around the other, and so
+ * future fields can be added without a new parameter. Unknown keys are
+ * rejected, so a typo fails at import time rather than being ignored.
+ */
+export interface TaskOptions {
+ /**
+ * Upstream task handles this task consumes. **Reserved: validated and
+ * retained, but inert today.**
+ *
+ * Nothing reads it yet: a handler receives `{ctx, client}` only, and no
+ * dependency is declared from it — in today's Python-stub mode the stub Dag
+ * defines task order. To read an upstream task's return value, ask for it
+ * explicitly: `client.getXCom({ key: "return_value", taskId: "extract" })`.
+ * Omitting `taskId` there reads the *running* task's own XCom.
+ */
+ readonly inputs?: TaskInputs;
+ /** Task-level options. **Reserved: retained, but inert today.** */
+ readonly spec?: TaskSpec;
+}
+
+/** Per-task record a Dag retains: the handle, the handler, its spec, and the
+ * upstream handles feeding it. */
+export interface TaskRecord {
+ readonly task: TaskRef;
+ readonly handler: TaskHandler;
+ readonly spec: TaskSpec;
+ /** Upstream handles keyed by input name; empty when the task has no inputs. */
+ readonly inputs: TaskInputs;
+}
+
+// Assigned inside Dag's static block: gives package-internal code read access
+// to the #tasks private field without a public accessor on the Dag class.
+let taskRecordsOf: (dag: Dag) => ReadonlyMap<string, TaskRecord>;
+
+// Every Dag ever constructed, so the bundle manifest can report the ones that
+// were never passed to registerDags(...) and airflow-ts-pack can warn about them.
+const declaredDagIds = new Set<string>();
+
+/**
+ * A Dag declared in TypeScript.
+ *
+ * Today the Dag structure itself is still declared by a Python stub file; a
+ * `Dag` instance binds TypeScript handlers to that stub's Dag/task IDs. The
+ * instance retains its `spec` and every task's `(taskId, handler, spec)` so a
+ * future `serialize()` can produce the serialized Dag JSON for native
+ * TypeScript Dag declaration.
+ */
+export class Dag {
+ /** Identifier of this Dag. Must match the Python Dag's `dag_id`. */
+ readonly dagId: string;
+ /** Dag-level options this instance was constructed with, copied and frozen. */
+ readonly spec: DagSpec;
+ readonly #tasks = new Map<string, TaskRecord>();
+
+ static {
+ taskRecordsOf = (dag) => dag.#tasks;
+ }
+
+ constructor(dagId: string, spec: DagSpec = {}) {
+ validateKey("dagId", dagId);
+ this.dagId = dagId;
+ // Copied and frozen, as task specs and inputs are: nothing reads a spec
+ // until the bundle manifest is built, long after the user's module has run,
+ // so a later mutation of their object would silently change what is packed.
+ // Shallow — a nested value in a future generated spec stays mutable.
+ this.spec = Object.freeze({ ...spec });
+ declaredDagIds.add(dagId);
Review Comment:
Also not a big fan with using a global variable like this, especially in the constructor. Why not make the registry a public interface instead, similar to how the Java SDK has a Bundle class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3793731850
##########
ts-sdk/src/cli/pack.ts:
##########
@@ -143,20 +143,52 @@ function readBundleManifest(bundlePath: string): BundleManifest {
throw new Error(`Bundle produced no ${AIRFLOW_METADATA_FLAG} output`);
}
- let manifest: BundleManifest;
+ let parsed: unknown;
try {
- manifest = JSON.parse(line.slice(AIRFLOW_METADATA_SENTINEL.length)) as BundleManifest;
+ parsed = JSON.parse(line.slice(AIRFLOW_METADATA_SENTINEL.length));
} catch (error) {
throw new Error(`Bundle produced invalid ${AIRFLOW_METADATA_FLAG} output: ${String(error)}`, {
cause: error,
});
}
- if (!manifest.supervisor_schema_version || !manifest.dags || typeof manifest.dags !== "object") {
+ if (!isBundleManifest(parsed)) {
throw new Error(`Bundle produced incomplete ${AIRFLOW_METADATA_FLAG} output`);
}
+ const manifest = parsed;
+ // The line is whatever the bundle printed and nothing downstream re-validates
+ // it, so check each Dag entry down to the task-id element.
+ for (const [dagId, dag] of Object.entries(manifest.dags)) {
+ if (dag == null || !isDagIdList(dag.tasks)) {
+ throw new Error(
+ `Bundle produced ${AIRFLOW_METADATA_FLAG} output with a malformed entry for Dag "${dagId}"`,
+ );
+ }
+ }
return manifest;
}
+// The document is checked before anything is read off it: JSON.parse also yields
+// null and primitives, and `null.supervisor_schema_version` would surface as a
+// raw TypeError rather than a report about the bundle.
+function isBundleManifest(value: unknown): value is BundleManifest {
+ if (typeof value !== "object" || value === null || Array.isArray(value)) return false;
+ const { supervisor_schema_version: version, dags } = value as Partial<BundleManifest>;
+ return (
+ // Rendered into the manifest verbatim, where the schema requires a non-empty
+ // string, so a truthy number or boolean would travel to Airflow as-is.
+ typeof version === "string" &&
+ version.length > 0 &&
Review Comment:
But should this be stricter and check against a regex?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3754987644
##########
ts-sdk/src/coordinator/runtime.ts:
##########
@@ -51,13 +52,46 @@ import {
type RuntimeTaskState,
type StartupDetails,
} from "./protocol.js";
-import { getRegisteredTask, listRegisteredTasks } from "../sdk/registry.js";
+import { DagRegistry } from "../sdk/registry.js";
import type { TaskContext, TaskHandlerArgs } from "../sdk/task.js";
import type { JsonValue } from "../sdk/client-types.js";
export const ABORT_GRACE_PERIOD_MS = 30_000;
export const COORDINATOR_RESPONSE_TIMEOUT_MS = 30_000;
+let served = false;
Review Comment:
This is the only place where the global variable lives and `serveDags` should only be called once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3746910816
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,208 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
+
+/**
+ * Dag-level options.
+ *
+ * Empty today: native TypeScript Dag declaration (schedule, tags, ...) will add
+ * optional fields here without changing the `Dag` constructor signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future native-Dag fields
+export interface DagSpec {}
+
+/**
+ * Task-level options.
+ *
+ * Empty today: future task fields (retries, ...) will land here without
+ * changing the `dag.task()` signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future task fields
+export interface TaskSpec {}
+
+/**
+ * Opaque handle to a task registered on a {@link Dag}.
+ *
+ * Pass it as a downstream task's input to declare that the downstream task
Review Comment:
I went with your second option. `inputs`, `spec`, `DagSpec` and `TaskSpec` now lead with "Reserved: inert today" on every declaration, the `inputs` docstring names the exact trap (`client.getXCom` without `taskId` reads the running task's own XCom), and explicitly mentioned in`typescript.rst`.
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,208 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
+
+/**
+ * Dag-level options.
+ *
+ * Empty today: native TypeScript Dag declaration (schedule, tags, ...) will add
+ * optional fields here without changing the `Dag` constructor signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future native-Dag fields
+export interface DagSpec {}
+
+/**
+ * Task-level options.
+ *
+ * Empty today: future task fields (retries, ...) will land here without
+ * changing the `dag.task()` signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future task fields
+export interface TaskSpec {}
+
+/**
+ * Opaque handle to a task registered on a {@link Dag}.
+ *
+ * Pass it as a downstream task's input to declare that the downstream task
+ * consumes this task's return value:
+ *
+ * ```ts
+ * const extracted = dag.task("extract", extractFn);
+ * const transformed = dag.task("transform", transformFn, { inputs: { extracted } });
+ * dag.task("load", loadFn, { inputs: { transformed } });
+ * ```
+ *
+ * In today's Python-stub mode the stub Dag still defines task order; declared
+ * inputs are retained for the serialized Dag JSON that native TypeScript Dag
+ * declaration will emit. The handler is intentionally not exposed on the handle.
+ */
+export interface Task {
+ /** Identifier of the Dag this task belongs to. */
+ readonly dagId: string;
+ /** Airflow task ID, including any TaskGroup prefix. */
+ readonly taskId: string;
+}
+
+/**
+ * Upstream task handles a task consumes, keyed by input name.
+ *
+ * Values must be handles returned by `dag.task(...)`. Literal values are
+ * deliberately out of scope for now; the future native-Dag work decides how
+ * they are declared.
+ */
+export type TaskInputs = Readonly<Record<string, Task>>;
+
+/**
+ * Named options for `dag.task()`.
+ *
+ * Keyword-only so neither field has to be positioned around the other, and so
+ * future fields can be added without a new parameter.
+ */
+export interface TaskOptions {
+ /** Upstream task handles whose return values this task consumes. */
+ readonly inputs?: TaskInputs;
+ /** Task-level options. */
+ readonly spec?: TaskSpec;
+}
+
+/** Per-task record a Dag retains: the handle, the handler, its spec, and the
+ * upstream handles feeding it. */
+export interface TaskRecord {
+ readonly task: Task;
+ readonly handler: TaskHandler;
+ readonly spec: TaskSpec;
+ /** Upstream handles keyed by input name; empty when the task has no inputs. */
+ readonly inputs: TaskInputs;
+}
+
+// Assigned inside Dag's static block: gives package-internal code read access
+// to the #tasks private field without a public accessor on the Dag class.
+let taskRecordsOf: (dag: Dag) => ReadonlyMap<string, TaskRecord>;
+
+/**
+ * A Dag declared in TypeScript.
+ *
+ * Today the Dag structure itself is still declared by a Python stub file; a
+ * `Dag` instance binds TypeScript handlers to that stub's Dag/task IDs. The
+ * instance retains its `spec` and every task's `(taskId, handler, spec)` so a
+ * future `serialize()` can produce the serialized Dag JSON for native
+ * TypeScript Dag declaration.
+ */
+export class Dag {
+ /** Identifier of this Dag. Must match the Python Dag's `dag_id`. */
+ readonly dagId: string;
+ /** Dag-level options this instance was constructed with. */
+ readonly spec: DagSpec;
+ readonly #tasks = new Map<string, TaskRecord>();
+
+ static {
+ taskRecordsOf = (dag) => dag.#tasks;
+ }
+
+ constructor(dagId: string, spec: DagSpec = {}) {
+ validateKey("dagId", dagId);
+ this.dagId = dagId;
+ this.spec = spec;
+ }
+
+ /**
+ * Register a TypeScript handler for a task of this Dag.
+ *
+ * `taskId` must match the Dag-side operator's `task_id` exactly, including
+ * any TaskGroup prefix. `options.inputs` names the upstream task handles whose
+ * return values this task consumes. Returns this task's handle.
+ */
+ task<TReturn = unknown>(
+ taskId: string,
+ handler: TaskHandler<TReturn>,
+ options: TaskOptions = {},
+ ): Task {
+ const { inputs = {}, spec = {} } = options;
Review Comment:
Added `#validateOptions`: unknown keys throw `Unknown option "input" for Dag "d" task "t"`, and a non-object (including `null` and arrays) throws before the destructure, so no more opaque "Cannot destructure property".
Good catch on `tests/public-api.test.ts:178` — that comment was false for the `{ upstream }` line. It is now true rather than reworded, and a runtime test covers all three misuse shapes.
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,208 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
+
+/**
+ * Dag-level options.
+ *
+ * Empty today: native TypeScript Dag declaration (schedule, tags, ...) will add
+ * optional fields here without changing the `Dag` constructor signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future native-Dag fields
+export interface DagSpec {}
+
+/**
+ * Task-level options.
+ *
+ * Empty today: future task fields (retries, ...) will land here without
+ * changing the `dag.task()` signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future task fields
+export interface TaskSpec {}
+
+/**
+ * Opaque handle to a task registered on a {@link Dag}.
+ *
+ * Pass it as a downstream task's input to declare that the downstream task
+ * consumes this task's return value:
+ *
+ * ```ts
+ * const extracted = dag.task("extract", extractFn);
+ * const transformed = dag.task("transform", transformFn, { inputs: { extracted } });
+ * dag.task("load", loadFn, { inputs: { transformed } });
+ * ```
+ *
+ * In today's Python-stub mode the stub Dag still defines task order; declared
+ * inputs are retained for the serialized Dag JSON that native TypeScript Dag
+ * declaration will emit. The handler is intentionally not exposed on the handle.
+ */
+export interface Task {
+ /** Identifier of the Dag this task belongs to. */
+ readonly dagId: string;
+ /** Airflow task ID, including any TaskGroup prefix. */
+ readonly taskId: string;
+}
+
+/**
+ * Upstream task handles a task consumes, keyed by input name.
+ *
+ * Values must be handles returned by `dag.task(...)`. Literal values are
+ * deliberately out of scope for now; the future native-Dag work decides how
+ * they are declared.
+ */
+export type TaskInputs = Readonly<Record<string, Task>>;
+
+/**
+ * Named options for `dag.task()`.
+ *
+ * Keyword-only so neither field has to be positioned around the other, and so
+ * future fields can be added without a new parameter.
+ */
+export interface TaskOptions {
+ /** Upstream task handles whose return values this task consumes. */
+ readonly inputs?: TaskInputs;
+ /** Task-level options. */
+ readonly spec?: TaskSpec;
+}
+
+/** Per-task record a Dag retains: the handle, the handler, its spec, and the
+ * upstream handles feeding it. */
+export interface TaskRecord {
+ readonly task: Task;
+ readonly handler: TaskHandler;
+ readonly spec: TaskSpec;
+ /** Upstream handles keyed by input name; empty when the task has no inputs. */
+ readonly inputs: TaskInputs;
+}
+
+// Assigned inside Dag's static block: gives package-internal code read access
+// to the #tasks private field without a public accessor on the Dag class.
+let taskRecordsOf: (dag: Dag) => ReadonlyMap<string, TaskRecord>;
+
+/**
+ * A Dag declared in TypeScript.
+ *
+ * Today the Dag structure itself is still declared by a Python stub file; a
+ * `Dag` instance binds TypeScript handlers to that stub's Dag/task IDs. The
+ * instance retains its `spec` and every task's `(taskId, handler, spec)` so a
+ * future `serialize()` can produce the serialized Dag JSON for native
+ * TypeScript Dag declaration.
+ */
+export class Dag {
+ /** Identifier of this Dag. Must match the Python Dag's `dag_id`. */
+ readonly dagId: string;
+ /** Dag-level options this instance was constructed with. */
+ readonly spec: DagSpec;
+ readonly #tasks = new Map<string, TaskRecord>();
Review Comment:
Added exactly your suggestion
```ts
get taskIds(): readonly string[] { return [...this.#tasks.keys()]; }
```
`DagRegistry.listDags()` now goes through it instead of `getDagTaskRecords`.
##########
ts-sdk/src/cli/pack.ts:
##########
@@ -193,9 +202,19 @@ export async function runPack(argv: readonly string[]): Promise<void> {
});
const manifest = readBundleManifest(stagingPath);
- if (Object.keys(manifest.dags).length === 0) {
+ const dagEntries = Object.entries(manifest.dags);
+ if (dagEntries.length === 0) {
+ throw new Error(
+ `${args.entry} registered no Dags; register Dags with registerDags(...) before startCoordinator()`,
+ );
+ }
+ const emptyDags = dagEntries
+ .filter(([, dag]) => dag.tasks.length === 0)
+ .map(([dagId]) => dagId);
+ if (emptyDags.length > 0) {
Review Comment:
Matched all the handling behavior with Go side: `warning: dag %q has no tasks` and zero registered Dags is still a hard error.
##########
ts-sdk/src/cli/pack.ts:
##########
@@ -154,6 +154,15 @@ function readBundleManifest(bundlePath: string): BundleManifest {
if (!manifest.supervisor_schema_version || !manifest.dags || typeof manifest.dags !== "object") {
throw new Error(`Bundle produced incomplete ${AIRFLOW_METADATA_FLAG} output`);
}
+ // The line is whatever the bundle printed, so the per-Dag shape is not
+ // guaranteed by the type assertion above.
+ for (const [dagId, dag] of Object.entries(manifest.dags)) {
+ if (dag == null || !Array.isArray(dag.tasks)) {
Review Comment:
Both fixed. Task IDs are now checked as non-empty strings per `airflow-metadata.schema.json`, and the array case is rejected before `Object.entries` can turn it into Dags named `"0"`, `"1"`. Parametrized tests cover each malformed shape.
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,208 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
+
+/**
+ * Dag-level options.
+ *
+ * Empty today: native TypeScript Dag declaration (schedule, tags, ...) will add
+ * optional fields here without changing the `Dag` constructor signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future native-Dag fields
+export interface DagSpec {}
Review Comment:
Both are now `Record<string, never>`, and both `eslint-disable` lines are gone. I ran the `tsc` check you flagged as unverified:
| written today | `interface DagSpec {}` | `type DagSpec = object` | `Record<string, never>` |
|---|---|---|---|
| `new Dag("d", 42)` | compiles | TS2345 | TS2345 |
| `new Dag("d", { schedule: "@daily" })` | compiles, dropped | compiles, dropped | TS2322 |
`object` fixes only the primitive case, it still silently accepts the field a user would actually write. Your TS2559 point checks out too (but it can only bite call sites that pass a non-`{}` spec).
`Record<string, never>` makes unwritable, and `{}` stays assignable to an all-optional generated type.
##########
ts-sdk/src/sdk/registry.ts:
##########
@@ -17,85 +17,72 @@
* under the License.
*/
+import { Dag, getDagTaskRecords, type Task } from "./dag.js";
import type { TaskHandler } from "./task.js";
-// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
-const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
-const MAX_KEY_LENGTH = 250;
-
-function validateKey(name: string, value: string): void {
- if (typeof value !== "string" || !KEY_REGEX.test(value)) {
- throw new Error(
- `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
- );
- }
- if (value.length > MAX_KEY_LENGTH) {
- throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
- }
-}
-
-/** Identifies the Airflow task handled by a TypeScript function. */
-export interface TaskRegistration {
- /** Identifier of the Dag containing this task. */
+/** A registered Dag with its task IDs, empty Dags included — used for
+ * manifests, where a task-less Dag must stay visible. */
+export interface RegisteredDag {
+ /** Identifier of the registered Dag. */
readonly dagId: string;
- /** Airflow task ID, including any TaskGroup prefix. */
- readonly taskId: string;
+ /** Airflow task IDs, including any TaskGroup prefix. */
+ readonly tasks: string[];
}
-/** Registry of TypeScript task handlers keyed by Dag ID and task ID. */
-export class TaskRegistry {
- readonly #tasks = new Map<string, Map<string, TaskHandler>>();
+/**
+ * Registry of Dag instances keyed by Dag ID.
+ *
+ * Lookups delegate live to each Dag's task map, so tasks added to a Dag
+ * after registration are visible — the registry records Dag identity, not
+ * a snapshot of its tasks.
+ */
+export class DagRegistry {
+ readonly #dags = new Map<string, Dag>();
- /**
- * Register a TypeScript handler for an Airflow task.
- *
- * `dagId` must match the Python Dag's `dag_id`. `taskId` must match the
- * Dag-side operator's `task_id` exactly, including any TaskGroup prefix.
- */
- register<TReturn = unknown>(registration: TaskRegistration, handler: TaskHandler<TReturn>): void {
- const { dagId, taskId } = registration;
- validateKey("dagId", dagId);
- validateKey("taskId", taskId);
- if (typeof handler !== "function") {
- throw new Error(`handler for Dag "${dagId}" task "${taskId}" must be a function`);
+ /** Register Dags. Registering an already-registered `dagId` throws,
+ * and a call that throws registers none of its Dags. */
+ register(...dags: Dag[]): void {
+ const incoming = new Set<string>();
+ for (const dag of dags) {
+ if (!(dag instanceof Dag)) {
+ throw new Error("only Dag instances can be registered");
+ }
+ if (this.#dags.has(dag.dagId) || incoming.has(dag.dagId)) {
+ throw new Error(`Dag "${dag.dagId}" is already registered`);
+ }
+ incoming.add(dag.dagId);
}
- const dagTasks = this.#tasks.get(dagId) ?? new Map<string, TaskHandler>();
- if (dagTasks.has(taskId)) {
- throw new Error(`Task "${taskId}" is already registered for Dag "${dagId}"`);
+ for (const dag of dags) {
+ this.#dags.set(dag.dagId, dag);
}
- dagTasks.set(taskId, handler as TaskHandler);
- this.#tasks.set(dagId, dagTasks);
}
/** Look up a registered handler. Returns `undefined` when no handler exists. */
- get(dagId: string, taskId: string): TaskHandler | undefined {
- return this.#tasks.get(dagId)?.get(taskId);
+ getTaskHandler(dagId: string, taskId: string): TaskHandler | undefined {
+ const dag = this.#dags.get(dagId);
+ return dag ? getDagTaskRecords(dag).get(taskId)?.handler : undefined;
}
- /** List all registered tasks. */
- list(): TaskRegistration[] {
- return [...this.#tasks.entries()].flatMap(([dagId, tasks]) =>
- [...tasks.keys()].map((taskId) => ({ dagId, taskId })),
+ /** List the task handles across registered Dags. */
+ listTasks(): Task[] {
+ return [...this.#dags.values()].flatMap((dag) =>
+ [...getDagTaskRecords(dag).values()].map((record) => record.task),
);
}
-}
-const defaultRegistry = new TaskRegistry();
-
-/** Register a TypeScript handler in the default task registry. */
-export function registerTask<TReturn = unknown>(
- registration: TaskRegistration,
- handler: TaskHandler<TReturn>,
-): void {
- defaultRegistry.register(registration, handler);
+ /** List every registered Dag with its task IDs, empty Dags included. */
+ listDags(): RegisteredDag[] {
+ return [...this.#dags.values()].map((dag) => ({
+ dagId: dag.dagId,
+ tasks: [...getDagTaskRecords(dag).keys()],
+ }));
+ }
}
-/** Look up a registered handler. Returns `undefined` when no handler exists. */
-export function getRegisteredTask(dagId: string, taskId: string): TaskHandler | undefined {
- return defaultRegistry.get(dagId, taskId);
-}
+/** The registry `registerDags` writes to and the coordinator reads from. */
+export const defaultRegistry = new DagRegistry();
-/** List all registered tasks. */
-export function listRegisteredTasks(): TaskRegistration[] {
- return defaultRegistry.list();
+/** Record Dags in the default registry so the coordinator can run their tasks. */
+export function registerDags(...dags: Dag[]): void {
Review Comment:
`registerDags(...)` is now the single entrypoint, since a Dag author never needs to know Airflow's coordinator at all. A second `registerDags` call is rejected, which would otherwise start a second runtime.
For the Dag instance omission, I prefer to let `airflow-ts-pack` warns instead of having restrict error (same as Python side behavior, it's fine to "define the Dag but don't construct it".
I agreed to rename `Task` to `TaskRef` to match the convention of Go and Java and avoid having the `Task` too ambigious.
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,208 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
+
+/**
+ * Dag-level options.
+ *
+ * Empty today: native TypeScript Dag declaration (schedule, tags, ...) will add
+ * optional fields here without changing the `Dag` constructor signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future native-Dag fields
+export interface DagSpec {}
+
+/**
+ * Task-level options.
+ *
+ * Empty today: future task fields (retries, ...) will land here without
+ * changing the `dag.task()` signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future task fields
+export interface TaskSpec {}
+
+/**
+ * Opaque handle to a task registered on a {@link Dag}.
+ *
+ * Pass it as a downstream task's input to declare that the downstream task
+ * consumes this task's return value:
+ *
+ * ```ts
+ * const extracted = dag.task("extract", extractFn);
+ * const transformed = dag.task("transform", transformFn, { inputs: { extracted } });
+ * dag.task("load", loadFn, { inputs: { transformed } });
+ * ```
+ *
+ * In today's Python-stub mode the stub Dag still defines task order; declared
+ * inputs are retained for the serialized Dag JSON that native TypeScript Dag
+ * declaration will emit. The handler is intentionally not exposed on the handle.
+ */
+export interface Task {
+ /** Identifier of the Dag this task belongs to. */
+ readonly dagId: string;
+ /** Airflow task ID, including any TaskGroup prefix. */
+ readonly taskId: string;
+}
+
+/**
+ * Upstream task handles a task consumes, keyed by input name.
+ *
+ * Values must be handles returned by `dag.task(...)`. Literal values are
+ * deliberately out of scope for now; the future native-Dag work decides how
+ * they are declared.
+ */
+export type TaskInputs = Readonly<Record<string, Task>>;
+
+/**
+ * Named options for `dag.task()`.
+ *
+ * Keyword-only so neither field has to be positioned around the other, and so
+ * future fields can be added without a new parameter.
+ */
+export interface TaskOptions {
+ /** Upstream task handles whose return values this task consumes. */
+ readonly inputs?: TaskInputs;
+ /** Task-level options. */
+ readonly spec?: TaskSpec;
+}
+
+/** Per-task record a Dag retains: the handle, the handler, its spec, and the
+ * upstream handles feeding it. */
+export interface TaskRecord {
+ readonly task: Task;
+ readonly handler: TaskHandler;
+ readonly spec: TaskSpec;
+ /** Upstream handles keyed by input name; empty when the task has no inputs. */
+ readonly inputs: TaskInputs;
+}
+
+// Assigned inside Dag's static block: gives package-internal code read access
+// to the #tasks private field without a public accessor on the Dag class.
+let taskRecordsOf: (dag: Dag) => ReadonlyMap<string, TaskRecord>;
+
+/**
+ * A Dag declared in TypeScript.
+ *
+ * Today the Dag structure itself is still declared by a Python stub file; a
+ * `Dag` instance binds TypeScript handlers to that stub's Dag/task IDs. The
+ * instance retains its `spec` and every task's `(taskId, handler, spec)` so a
+ * future `serialize()` can produce the serialized Dag JSON for native
+ * TypeScript Dag declaration.
+ */
+export class Dag {
+ /** Identifier of this Dag. Must match the Python Dag's `dag_id`. */
+ readonly dagId: string;
+ /** Dag-level options this instance was constructed with. */
+ readonly spec: DagSpec;
+ readonly #tasks = new Map<string, TaskRecord>();
+
+ static {
+ taskRecordsOf = (dag) => dag.#tasks;
+ }
+
+ constructor(dagId: string, spec: DagSpec = {}) {
+ validateKey("dagId", dagId);
+ this.dagId = dagId;
+ this.spec = spec;
+ }
+
+ /**
+ * Register a TypeScript handler for a task of this Dag.
+ *
+ * `taskId` must match the Dag-side operator's `task_id` exactly, including
+ * any TaskGroup prefix. `options.inputs` names the upstream task handles whose
+ * return values this task consumes. Returns this task's handle.
+ */
+ task<TReturn = unknown>(
+ taskId: string,
+ handler: TaskHandler<TReturn>,
+ options: TaskOptions = {},
+ ): Task {
+ const { inputs = {}, spec = {} } = options;
+ validateKey("taskId", taskId);
+ if (typeof handler !== "function") {
+ throw new Error(`handler for Dag "${this.dagId}" task "${taskId}" must be a function`);
+ }
+ if (this.#tasks.has(taskId)) {
+ throw new Error(`Task "${taskId}" is already registered for Dag "${this.dagId}"`);
+ }
+ this.#validateInputs(taskId, inputs);
+ const task: Task = Object.freeze({ dagId: this.dagId, taskId });
+ this.#tasks.set(taskId, {
+ task,
+ handler: handler as TaskHandler,
+ spec,
+ inputs: Object.freeze({ ...inputs }),
Review Comment:
Both specs are now `Object.freeze({ ...spec })`, matching `inputs`, and the two `toBe` assertions relaxed to `toEqual`. I also added `Object.isFrozen` assertions and mutation-checked them and removed the `Object.freeze`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3748369878
##########
ts-sdk/example/src/main.ts:
##########
@@ -49,7 +49,7 @@ export async function readConnection({ client }: TaskHandlerArgs) {
};
}
-registerTask({ dagId: DAG_ID, taskId: "build_message" }, buildMessage);
-registerTask({ dagId: DAG_ID, taskId: "read_connection" }, readConnection);
+dag.task("build_message", buildMessage);
+dag.task("read_connection", readConnection);
-await startCoordinator();
+await registerDags(dag);
Review Comment:
I’m not sure how I feel with a function named `registerDags` handling sockets and a waiting server. Maybe use something more similar to the Java SDK such as `serveDags()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on PR #71144:
URL: https://github.com/apache/airflow/pull/71144#issuecomment-5215646281
> What’s the plan to do the external vs pure TS dags after this?
Along with https://github.com/apache/airflow/pull/71213, the user should set the `is_mixed_language_dag` for the mixed language Dag case, it will look somehow like `new Dag("my_ts_mixed_lang", spec: {is_mixed_language_dag: True, schedule: "..."})`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3755547419
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,252 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
+
+function validateEmptySpec(name: string, value: unknown): void {
+ if (
+ typeof value !== "object" ||
+ value === null ||
+ Array.isArray(value) ||
+ Reflect.ownKeys(value).length > 0
+ ) {
+ throw new Error(`${name} must be an empty object`);
+ }
+}
+
+/**
+ * Dag-level options. **Reserved: no fields yet.**
+ *
+ * Native TypeScript Dag declaration (schedule, tags, ...) will add optional
+ * fields here without changing the `Dag` constructor, generated from the
+ * serialized-Dag JSON schema the way `src/generated/supervisor.ts` is. Until
+ * then only `{}` is accepted, so a field that would be silently dropped —
+ * `new Dag("d", { schedule: "@daily" })` — is a compile error rather than a
+ * Dag that packs and runs without the schedule.
+ */
+export type DagSpec = Record<string, never>;
+
+/**
+ * Task-level options. **Reserved: no fields yet.**
+ *
+ * Future task fields (retries, ...) will land here without changing the
+ * `dag.task()` signature. As with {@link DagSpec}, only `{}` is accepted today.
+ */
+export type TaskSpec = Record<string, never>;
+
+/**
+ * Opaque handle to a task registered on a {@link Dag}, returned by
+ * `dag.task(...)`.
+ *
+ * Identity only — the handler is deliberately not exposed. Handles are what the
+ * reserved `inputs` option accepts, and what native TypeScript Dag declaration
+ * will use to wire dependencies.
+ */
+export interface TaskRef {
+ /** Identifier of the Dag this task belongs to. */
+ readonly dagId: string;
+ /** Airflow task ID, including any TaskGroup prefix. */
+ readonly taskId: string;
+}
+
+/**
+ * Upstream task handles keyed by input name. **Reserved: validated and
+ * retained, but inert today** — see {@link TaskOptions.inputs}.
+ *
+ * Values must be handles returned by `dag.task(...)`. Literal values are
+ * deliberately out of scope for now; the future native-Dag work decides how
+ * they are declared.
+ */
+export type TaskInputs = Readonly<Record<string, TaskRef>>;
+
+/**
+ * Named options for `dag.task()`.
+ *
+ * Keyword-only so neither field has to be positioned around the other, and so
+ * future fields can be added without a new parameter. Unknown keys are
+ * rejected, so a typo fails at import time rather than being ignored.
+ */
+export interface TaskOptions {
+ /**
+ * Upstream task handles this task consumes. **Reserved: validated and
+ * retained, but inert today.**
+ *
+ * Nothing reads it yet: a handler receives `{ctx, client}` only, and no
+ * dependency is declared from it — in today's Python-stub mode the stub Dag
+ * defines task order. To read an upstream task's return value, ask for it
+ * explicitly: `client.getXCom({ key: "return_value", taskId: "extract" })`.
+ * Omitting `taskId` there reads the *running* task's own XCom.
+ */
+ readonly inputs?: TaskInputs;
+ /** Task-level options. **Reserved: retained, but inert today.** */
+ readonly spec?: TaskSpec;
+}
+
+/** Per-task record a Dag retains: the handle, the handler, its spec, and the
+ * upstream handles feeding it. */
+export interface TaskRecord {
+ readonly task: TaskRef;
+ readonly handler: TaskHandler;
+ readonly spec: TaskSpec;
+ /** Upstream handles keyed by input name; empty when the task has no inputs. */
+ readonly inputs: TaskInputs;
+}
+
+// Assigned inside Dag's static block: gives package-internal code read access
+// to the #tasks private field without a public accessor on the Dag class.
+let taskRecordsOf: (dag: Dag) => ReadonlyMap<string, TaskRecord>;
+
+/**
+ * A Dag declared in TypeScript.
+ *
+ * Today the Dag structure itself is still declared by a Python stub file; a
+ * `Dag` instance binds TypeScript handlers to that stub's Dag/task IDs. The
+ * instance retains its `spec` and every task's `(taskId, handler, spec)` so a
+ * future `serialize()` can produce the serialized Dag JSON for native
+ * TypeScript Dag declaration.
+ *
+ * Constructing a Dag has no effect beyond the instance itself. Collect the ones
+ * a bundle should serve in a `DagRegistry` and pass it to `serveDags(...)`.
+ */
+export class Dag {
+ /** Identifier of this Dag. Must match the Python Dag's `dag_id`. */
+ readonly dagId: string;
+ /** Dag-level options this instance was constructed with, copied and frozen. */
+ readonly spec: DagSpec;
+ readonly #tasks = new Map<string, TaskRecord>();
+
+ static {
+ taskRecordsOf = (dag) => dag.#tasks;
+ }
+
+ constructor(dagId: string, spec: DagSpec = {}) {
+ validateKey("dagId", dagId);
+ validateEmptySpec(`spec for Dag "${dagId}"`, spec);
+ this.dagId = dagId;
+ // Copied and frozen, as task specs and inputs are: nothing reads a spec
+ // until the bundle manifest is built, long after the user's module has run,
+ // so a later mutation of their object would silently change what is packed.
+ // Shallow — a nested value in a future generated spec stays mutable.
+ this.spec = Object.freeze({ ...spec });
+ }
+
+ /** Task IDs attached to this Dag, in attachment order. */
+ get taskIds(): readonly string[] {
+ return [...this.#tasks.keys()];
+ }
+
+ /**
+ * Register a TypeScript handler for a task of this Dag.
+ *
+ * `taskId` must match the Dag-side operator's `task_id` exactly, including
+ * any TaskGroup prefix. Returns this task's handle.
+ */
+ task<TReturn = unknown>(
+ taskId: string,
+ handler: TaskHandler<TReturn>,
+ options: TaskOptions = {},
+ ): TaskRef {
Review Comment:
Good point! I would like to keep the interface as-is for now. Since they're no-op and user shouldn't set them at this moment (I added the `validateEmptySpec` to prevent it being set at all).
What the `TypeRef` and `TaskOutput` binding should be implemented in the follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3763989743
##########
ts-sdk/src/coordinator/runtime.ts:
##########
@@ -51,13 +52,46 @@ import {
type RuntimeTaskState,
type StartupDetails,
} from "./protocol.js";
-import { getRegisteredTask, listRegisteredTasks } from "../sdk/registry.js";
+import { DagRegistry } from "../sdk/registry.js";
import type { TaskContext, TaskHandlerArgs } from "../sdk/task.js";
import type { JsonValue } from "../sdk/client-types.js";
export const ABORT_GRACE_PERIOD_MS = 30_000;
export const COORDINATOR_RESPONSE_TIMEOUT_MS = 30_000;
+let served = false;
Review Comment:
Both fixed in 745461eb32:
1. the latch is released when a serve fails (so a retry reports the real cause instead of "already called"), it is keyed on the global symbol registry rather than module state
2. The `serveDags` is now `async` so validation and runtime errors arrive on one channel.
Thanks!
##########
ts-sdk/example/src/main.ts:
##########
@@ -49,7 +49,7 @@ export async function readConnection({ client }: TaskHandlerArgs) {
};
}
-registerTask({ dagId: DAG_ID, taskId: "build_message" }, buildMessage);
-registerTask({ dagId: DAG_ID, taskId: "read_connection" }, readConnection);
+dag.task("build_message", buildMessage);
+dag.task("read_connection", readConnection);
-await startCoordinator();
+await registerDags(dag);
Review Comment:
Renamed in 30eff07ec7. `serveDags(registry)` is the entrypoint now, `registerDags` is gone.
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,251 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
+
+/**
+ * Dag-level options. **Reserved: no fields yet.**
+ *
+ * Native TypeScript Dag declaration (schedule, tags, ...) will add optional
+ * fields here without changing the `Dag` constructor, generated from the
+ * serialized-Dag JSON schema the way `src/generated/supervisor.ts` is. Until
+ * then only `{}` is accepted, so a field that would be silently dropped —
+ * `new Dag("d", { schedule: "@daily" })` — is a compile error rather than a
+ * Dag that packs and runs without the schedule.
+ */
+export type DagSpec = Record<string, never>;
+
+/**
+ * Task-level options. **Reserved: no fields yet.**
+ *
+ * Future task fields (retries, ...) will land here without changing the
+ * `dag.task()` signature. As with {@link DagSpec}, only `{}` is accepted today.
+ */
+export type TaskSpec = Record<string, never>;
+
+/**
+ * Opaque handle to a task registered on a {@link Dag}, returned by
+ * `dag.task(...)`.
+ *
+ * Identity only — the handler is deliberately not exposed. Handles are what the
+ * reserved `inputs` option accepts, and what native TypeScript Dag declaration
+ * will use to wire dependencies.
+ */
+export interface TaskRef {
+ /** Identifier of the Dag this task belongs to. */
+ readonly dagId: string;
+ /** Airflow task ID, including any TaskGroup prefix. */
+ readonly taskId: string;
+}
+
+/**
+ * Upstream task handles keyed by input name. **Reserved: validated and
+ * retained, but inert today** — see {@link TaskOptions.inputs}.
+ *
+ * Values must be handles returned by `dag.task(...)`. Literal values are
+ * deliberately out of scope for now; the future native-Dag work decides how
+ * they are declared.
+ */
+export type TaskInputs = Readonly<Record<string, TaskRef>>;
+
+/**
+ * Named options for `dag.task()`.
+ *
+ * Keyword-only so neither field has to be positioned around the other, and so
+ * future fields can be added without a new parameter. Unknown keys are
+ * rejected, so a typo fails at import time rather than being ignored.
+ */
+export interface TaskOptions {
+ /**
+ * Upstream task handles this task consumes. **Reserved: validated and
+ * retained, but inert today.**
+ *
+ * Nothing reads it yet: a handler receives `{ctx, client}` only, and no
+ * dependency is declared from it — in today's Python-stub mode the stub Dag
+ * defines task order. To read an upstream task's return value, ask for it
+ * explicitly: `client.getXCom({ key: "return_value", taskId: "extract" })`.
+ * Omitting `taskId` there reads the *running* task's own XCom.
+ */
+ readonly inputs?: TaskInputs;
+ /** Task-level options. **Reserved: retained, but inert today.** */
+ readonly spec?: TaskSpec;
+}
+
+/** Per-task record a Dag retains: the handle, the handler, its spec, and the
+ * upstream handles feeding it. */
+export interface TaskRecord {
+ readonly task: TaskRef;
+ readonly handler: TaskHandler;
+ readonly spec: TaskSpec;
+ /** Upstream handles keyed by input name; empty when the task has no inputs. */
+ readonly inputs: TaskInputs;
+}
+
+// Assigned inside Dag's static block: gives package-internal code read access
+// to the #tasks private field without a public accessor on the Dag class.
+let taskRecordsOf: (dag: Dag) => ReadonlyMap<string, TaskRecord>;
+
+// Every Dag ever constructed, so the bundle manifest can report the ones that
+// were never passed to registerDags(...) and airflow-ts-pack can warn about them.
+const declaredDagIds = new Set<string>();
+
+/**
+ * A Dag declared in TypeScript.
+ *
+ * Today the Dag structure itself is still declared by a Python stub file; a
+ * `Dag` instance binds TypeScript handlers to that stub's Dag/task IDs. The
+ * instance retains its `spec` and every task's `(taskId, handler, spec)` so a
+ * future `serialize()` can produce the serialized Dag JSON for native
+ * TypeScript Dag declaration.
+ */
+export class Dag {
+ /** Identifier of this Dag. Must match the Python Dag's `dag_id`. */
+ readonly dagId: string;
+ /** Dag-level options this instance was constructed with, copied and frozen. */
+ readonly spec: DagSpec;
+ readonly #tasks = new Map<string, TaskRecord>();
+
+ static {
+ taskRecordsOf = (dag) => dag.#tasks;
+ }
+
+ constructor(dagId: string, spec: DagSpec = {}) {
+ validateKey("dagId", dagId);
+ this.dagId = dagId;
+ // Copied and frozen, as task specs and inputs are: nothing reads a spec
+ // until the bundle manifest is built, long after the user's module has run,
+ // so a later mutation of their object would silently change what is packed.
+ // Shallow — a nested value in a future generated spec stays mutable.
+ this.spec = Object.freeze({ ...spec });
+ declaredDagIds.add(dagId);
Review Comment:
Done in 30eff07ec7: the `Dag` constructor no longer writes to module state, and `DagRegistry` is the explicit object a bundle builds and hands to `serveDags`.
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,252 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
+
+function validateEmptySpec(name: string, value: unknown): void {
+ if (
+ typeof value !== "object" ||
+ value === null ||
+ Array.isArray(value) ||
+ Reflect.ownKeys(value).length > 0
+ ) {
+ throw new Error(`${name} must be an empty object`);
+ }
+}
+
+/**
+ * Dag-level options. **Reserved: no fields yet.**
+ *
+ * Native TypeScript Dag declaration (schedule, tags, ...) will add optional
+ * fields here without changing the `Dag` constructor, generated from the
+ * serialized-Dag JSON schema the way `src/generated/supervisor.ts` is. Until
+ * then only `{}` is accepted, so a field that would be silently dropped —
+ * `new Dag("d", { schedule: "@daily" })` — is a compile error rather than a
+ * Dag that packs and runs without the schedule.
+ */
+export type DagSpec = Record<string, never>;
+
+/**
+ * Task-level options. **Reserved: no fields yet.**
+ *
+ * Future task fields (retries, ...) will land here without changing the
+ * `dag.task()` signature. As with {@link DagSpec}, only `{}` is accepted today.
+ */
+export type TaskSpec = Record<string, never>;
+
+/**
+ * Opaque handle to a task registered on a {@link Dag}, returned by
+ * `dag.task(...)`.
+ *
+ * Identity only — the handler is deliberately not exposed. Handles are what the
+ * reserved `inputs` option accepts, and what native TypeScript Dag declaration
+ * will use to wire dependencies.
+ */
+export interface TaskRef {
+ /** Identifier of the Dag this task belongs to. */
+ readonly dagId: string;
+ /** Airflow task ID, including any TaskGroup prefix. */
+ readonly taskId: string;
+}
+
+/**
+ * Upstream task handles keyed by input name. **Reserved: validated and
+ * retained, but inert today** — see {@link TaskOptions.inputs}.
+ *
+ * Values must be handles returned by `dag.task(...)`. Literal values are
+ * deliberately out of scope for now; the future native-Dag work decides how
+ * they are declared.
+ */
Review Comment:
Agreed and resolved in 8d3a3882bc , thanks!
##########
ts-sdk/src/cli/pack.ts:
##########
Review Comment:
The `warnOnSuspiciousIDs ` issue will be address in https://github.com/apache/airflow/pull/70993 after we merge this one.
a21763416f resolved the validation for `supervisor_schema_version`
##########
ts-sdk/src/sdk/registry.ts:
##########
@@ -17,85 +17,84 @@
* under the License.
*/
+import { Dag, getDagTaskRecords, type TaskRef } from "./dag.js";
import type { TaskHandler } from "./task.js";
-// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
-const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
-const MAX_KEY_LENGTH = 250;
-
-function validateKey(name: string, value: string): void {
- if (typeof value !== "string" || !KEY_REGEX.test(value)) {
- throw new Error(
- `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
- );
- }
- if (value.length > MAX_KEY_LENGTH) {
- throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
- }
-}
-
-/** Identifies the Airflow task handled by a TypeScript function. */
-export interface TaskRegistration {
- /** Identifier of the Dag containing this task. */
+/** A registered Dag with its task IDs, returned by {@link DagRegistry.listDags}.
+ * A task-less Dag is included, so the bundle manifest keeps it visible. */
+export interface RegisteredDag {
+ /** Identifier of the registered Dag. */
readonly dagId: string;
- /** Airflow task ID, including any TaskGroup prefix. */
- readonly taskId: string;
+ /** Airflow task IDs, including any TaskGroup prefix. */
+ readonly tasks: string[];
}
-/** Registry of TypeScript task handlers keyed by Dag ID and task ID. */
-export class TaskRegistry {
- readonly #tasks = new Map<string, Map<string, TaskHandler>>();
+/**
+ * The Dags a bundle process can execute, keyed by Dag ID.
+ *
+ * This is what a bundle entry point builds and hands to `serveDags(registry)`:
+ *
+ * ```ts
+ * const dag = new Dag("my_dag");
+ * dag.task("extract", extractFn);
+ * await serveDags(new DagRegistry(dag));
+ * ```
+ *
+ * It holds no sockets and starts nothing, so a test can build one and invoke a
+ * handler through {@link getTaskHandler} without any runtime in scope.
+ *
+ * Lookups delegate live to each Dag's task map, so tasks added to a Dag
+ * after registration are visible — the registry records Dag identity, not
+ * a snapshot of its tasks.
+ */
+export class DagRegistry {
Review Comment:
Good point. I don't think `listTasks` and `listDags` are necessary as well, addressed in 6ae9254e1f
I kept `getTaskHandler` public deliberately: it is the documented way to unit-test a handler without a coordinator runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on PR #71144:
URL: https://github.com/apache/airflow/pull/71144#issuecomment-5249416146
Just curious about the longer-term direction. Do we expect tasks to become callable, similar to Python TaskFlow?
```
const result = load(transform(extract()));
```
These calls would build the Dag and return typed task outputs rather than execute the handlers.
Is something like this planned for future native TypeScript Dags, or is the current TaskRef plus inputs design intended to be the final authoring interface?
One option would be for dag.task() to return a callable TaskDefinition<TInputs, TOutput>. Calling that definition would record the input bindings and return a TaskOutput<TOutput>.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3754987644
##########
ts-sdk/src/coordinator/runtime.ts:
##########
@@ -51,13 +52,46 @@ import {
type RuntimeTaskState,
type StartupDetails,
} from "./protocol.js";
-import { getRegisteredTask, listRegisteredTasks } from "../sdk/registry.js";
+import { DagRegistry } from "../sdk/registry.js";
import type { TaskContext, TaskHandlerArgs } from "../sdk/task.js";
import type { JsonValue } from "../sdk/client-types.js";
export const ABORT_GRACE_PERIOD_MS = 30_000;
export const COORDINATOR_RESPONSE_TIMEOUT_MS = 30_000;
+let served = false;
Review Comment:
This is the only place where the global variable lives and `serveDags` should only be called once. It's fine to keep it as global since the lifecycle and the intention is clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3773763521
##########
ts-sdk/src/cli/pack.ts:
##########
@@ -143,20 +143,52 @@ function readBundleManifest(bundlePath: string): BundleManifest {
throw new Error(`Bundle produced no ${AIRFLOW_METADATA_FLAG} output`);
}
- let manifest: BundleManifest;
+ let parsed: unknown;
try {
- manifest = JSON.parse(line.slice(AIRFLOW_METADATA_SENTINEL.length)) as BundleManifest;
+ parsed = JSON.parse(line.slice(AIRFLOW_METADATA_SENTINEL.length));
} catch (error) {
throw new Error(`Bundle produced invalid ${AIRFLOW_METADATA_FLAG} output: ${String(error)}`, {
cause: error,
});
}
- if (!manifest.supervisor_schema_version || !manifest.dags || typeof manifest.dags !== "object") {
+ if (!isBundleManifest(parsed)) {
throw new Error(`Bundle produced incomplete ${AIRFLOW_METADATA_FLAG} output`);
}
+ const manifest = parsed;
+ // The line is whatever the bundle printed and nothing downstream re-validates
+ // it, so check each Dag entry down to the task-id element.
+ for (const [dagId, dag] of Object.entries(manifest.dags)) {
+ if (dag == null || !isDagIdList(dag.tasks)) {
+ throw new Error(
+ `Bundle produced ${AIRFLOW_METADATA_FLAG} output with a malformed entry for Dag "${dagId}"`,
+ );
+ }
+ }
return manifest;
}
+// The document is checked before anything is read off it: JSON.parse also yields
+// null and primitives, and `null.supervisor_schema_version` would surface as a
+// raw TypeError rather than a report about the bundle.
+function isBundleManifest(value: unknown): value is BundleManifest {
+ if (typeof value !== "object" || value === null || Array.isArray(value)) return false;
+ const { supervisor_schema_version: version, dags } = value as Partial<BundleManifest>;
+ return (
+ // Rendered into the manifest verbatim, where the schema requires a non-empty
+ // string, so a truthy number or boolean would travel to Airflow as-is.
+ typeof version === "string" &&
+ version.length > 0 &&
+ typeof dags === "object" &&
+ dags !== null &&
+ // An array would pass the typeof check and yield Dags named "0", "1", ...
+ !Array.isArray(dags)
+ );
+}
+
+function isDagIdList(value: unknown): value is string[] {
+ return Array.isArray(value) && value.every((item) => typeof item === "string" && item.length > 0);
+}
Review Comment:
`isDagIdList` is validating a list of *task IDs* (`dag.tasks`), not Dag IDs. Renaming it (and its call site in readBundleManifest) would make the intent clearer and avoid confusion when reading the schema validation logic.
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,241 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import { brand, hasBrand } from "./brand.js";
+import type { TaskHandler } from "./task.js";
+
+function validateEmptySpec(name: string, value: unknown): void {
+ if (
+ typeof value !== "object" ||
+ value === null ||
+ Array.isArray(value) ||
+ Reflect.ownKeys(value).length > 0
+ ) {
+ throw new Error(`${name} must be an empty object`);
+ }
+}
Review Comment:
validateEmptySpec currently treats any object with zero own keys as an “empty object”, so values like `new Date()` (or other non-plain objects) are accepted for Dag/task `spec` even though they are not `{}`. Since callers can bypass TypeScript via JS or casts, this should reject non-plain objects to avoid silently accepting invalid specs.
This issue also appears in the following locations of the same file:
- line 195
- line 206
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3793877748
##########
ts-sdk/src/cli/pack.ts:
##########
@@ -143,20 +143,52 @@ function readBundleManifest(bundlePath: string): BundleManifest {
throw new Error(`Bundle produced no ${AIRFLOW_METADATA_FLAG} output`);
}
- let manifest: BundleManifest;
+ let parsed: unknown;
try {
- manifest = JSON.parse(line.slice(AIRFLOW_METADATA_SENTINEL.length)) as BundleManifest;
+ parsed = JSON.parse(line.slice(AIRFLOW_METADATA_SENTINEL.length));
} catch (error) {
throw new Error(`Bundle produced invalid ${AIRFLOW_METADATA_FLAG} output: ${String(error)}`, {
cause: error,
});
}
- if (!manifest.supervisor_schema_version || !manifest.dags || typeof manifest.dags !== "object") {
+ if (!isBundleManifest(parsed)) {
throw new Error(`Bundle produced incomplete ${AIRFLOW_METADATA_FLAG} output`);
}
+ const manifest = parsed;
+ // The line is whatever the bundle printed and nothing downstream re-validates
+ // it, so check each Dag entry down to the task-id element.
+ for (const [dagId, dag] of Object.entries(manifest.dags)) {
+ if (dag == null || !isDagIdList(dag.tasks)) {
+ throw new Error(
+ `Bundle produced ${AIRFLOW_METADATA_FLAG} output with a malformed entry for Dag "${dagId}"`,
+ );
+ }
+ }
return manifest;
}
+// The document is checked before anything is read off it: JSON.parse also yields
+// null and primitives, and `null.supervisor_schema_version` would surface as a
+// raw TypeError rather than a report about the bundle.
+function isBundleManifest(value: unknown): value is BundleManifest {
+ if (typeof value !== "object" || value === null || Array.isArray(value)) return false;
+ const { supervisor_schema_version: version, dags } = value as Partial<BundleManifest>;
+ return (
+ // Rendered into the manifest verbatim, where the schema requires a non-empty
+ // string, so a truthy number or boolean would travel to Airflow as-is.
+ typeof version === "string" &&
+ version.length > 0 &&
Review Comment:
Yes, we will validate against regex in https://github.com/apache/airflow/pull/70993.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3755410378
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,252 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
+
+function validateEmptySpec(name: string, value: unknown): void {
+ if (
+ typeof value !== "object" ||
+ value === null ||
+ Array.isArray(value) ||
+ Reflect.ownKeys(value).length > 0
+ ) {
+ throw new Error(`${name} must be an empty object`);
+ }
+}
+
+/**
+ * Dag-level options. **Reserved: no fields yet.**
+ *
+ * Native TypeScript Dag declaration (schedule, tags, ...) will add optional
+ * fields here without changing the `Dag` constructor, generated from the
+ * serialized-Dag JSON schema the way `src/generated/supervisor.ts` is. Until
+ * then only `{}` is accepted, so a field that would be silently dropped —
+ * `new Dag("d", { schedule: "@daily" })` — is a compile error rather than a
+ * Dag that packs and runs without the schedule.
+ */
+export type DagSpec = Record<string, never>;
+
+/**
+ * Task-level options. **Reserved: no fields yet.**
+ *
+ * Future task fields (retries, ...) will land here without changing the
+ * `dag.task()` signature. As with {@link DagSpec}, only `{}` is accepted today.
+ */
+export type TaskSpec = Record<string, never>;
+
+/**
+ * Opaque handle to a task registered on a {@link Dag}, returned by
+ * `dag.task(...)`.
+ *
+ * Identity only — the handler is deliberately not exposed. Handles are what the
+ * reserved `inputs` option accepts, and what native TypeScript Dag declaration
+ * will use to wire dependencies.
+ */
+export interface TaskRef {
+ /** Identifier of the Dag this task belongs to. */
+ readonly dagId: string;
+ /** Airflow task ID, including any TaskGroup prefix. */
+ readonly taskId: string;
+}
+
+/**
+ * Upstream task handles keyed by input name. **Reserved: validated and
+ * retained, but inert today** — see {@link TaskOptions.inputs}.
+ *
+ * Values must be handles returned by `dag.task(...)`. Literal values are
+ * deliberately out of scope for now; the future native-Dag work decides how
+ * they are declared.
+ */
+export type TaskInputs = Readonly<Record<string, TaskRef>>;
+
+/**
+ * Named options for `dag.task()`.
+ *
+ * Keyword-only so neither field has to be positioned around the other, and so
+ * future fields can be added without a new parameter. Unknown keys are
+ * rejected, so a typo fails at import time rather than being ignored.
+ */
+export interface TaskOptions {
+ /**
+ * Upstream task handles this task consumes. **Reserved: validated and
+ * retained, but inert today.**
+ *
+ * Nothing reads it yet: a handler receives `{ctx, client}` only, and no
+ * dependency is declared from it — in today's Python-stub mode the stub Dag
+ * defines task order. To read an upstream task's return value, ask for it
+ * explicitly: `client.getXCom({ key: "return_value", taskId: "extract" })`.
+ * Omitting `taskId` there reads the *running* task's own XCom.
+ */
+ readonly inputs?: TaskInputs;
+ /** Task-level options. **Reserved: retained, but inert today.** */
+ readonly spec?: TaskSpec;
+}
+
+/** Per-task record a Dag retains: the handle, the handler, its spec, and the
+ * upstream handles feeding it. */
+export interface TaskRecord {
+ readonly task: TaskRef;
+ readonly handler: TaskHandler;
+ readonly spec: TaskSpec;
+ /** Upstream handles keyed by input name; empty when the task has no inputs. */
+ readonly inputs: TaskInputs;
+}
+
+// Assigned inside Dag's static block: gives package-internal code read access
+// to the #tasks private field without a public accessor on the Dag class.
+let taskRecordsOf: (dag: Dag) => ReadonlyMap<string, TaskRecord>;
+
+/**
+ * A Dag declared in TypeScript.
+ *
+ * Today the Dag structure itself is still declared by a Python stub file; a
+ * `Dag` instance binds TypeScript handlers to that stub's Dag/task IDs. The
+ * instance retains its `spec` and every task's `(taskId, handler, spec)` so a
+ * future `serialize()` can produce the serialized Dag JSON for native
+ * TypeScript Dag declaration.
+ *
+ * Constructing a Dag has no effect beyond the instance itself. Collect the ones
+ * a bundle should serve in a `DagRegistry` and pass it to `serveDags(...)`.
+ */
+export class Dag {
+ /** Identifier of this Dag. Must match the Python Dag's `dag_id`. */
+ readonly dagId: string;
+ /** Dag-level options this instance was constructed with, copied and frozen. */
+ readonly spec: DagSpec;
+ readonly #tasks = new Map<string, TaskRecord>();
+
+ static {
+ taskRecordsOf = (dag) => dag.#tasks;
+ }
+
+ constructor(dagId: string, spec: DagSpec = {}) {
+ validateKey("dagId", dagId);
+ validateEmptySpec(`spec for Dag "${dagId}"`, spec);
+ this.dagId = dagId;
+ // Copied and frozen, as task specs and inputs are: nothing reads a spec
+ // until the bundle manifest is built, long after the user's module has run,
+ // so a later mutation of their object would silently change what is packed.
+ // Shallow — a nested value in a future generated spec stays mutable.
+ this.spec = Object.freeze({ ...spec });
+ }
+
+ /** Task IDs attached to this Dag, in attachment order. */
+ get taskIds(): readonly string[] {
+ return [...this.#tasks.keys()];
+ }
+
+ /**
+ * Register a TypeScript handler for a task of this Dag.
+ *
+ * `taskId` must match the Dag-side operator's `task_id` exactly, including
+ * any TaskGroup prefix. Returns this task's handle.
+ */
+ task<TReturn = unknown>(
+ taskId: string,
+ handler: TaskHandler<TReturn>,
+ options: TaskOptions = {},
+ ): TaskRef {
Review Comment:
Since task references are intended for future dependency chaining, should the return type at least be preserved as TaskOutput<TReturn> or TaskRef<TReturn>?
```
const extracted = dag.task("extract", async () => ({ rows: ["a"] }));
// TaskOutput<{ rows: string[] }>
dag.task("transform", transformHandler, {
inputs: { extracted },
});
```
This would let TypeScript retain and check the value type passed from extract to transform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3755603430
##########
ts-sdk/src/sdk/registry.ts:
##########
@@ -17,85 +17,84 @@
* under the License.
*/
+import { Dag, getDagTaskRecords, type TaskRef } from "./dag.js";
import type { TaskHandler } from "./task.js";
-// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
-const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
-const MAX_KEY_LENGTH = 250;
-
-function validateKey(name: string, value: string): void {
- if (typeof value !== "string" || !KEY_REGEX.test(value)) {
- throw new Error(
- `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
- );
- }
- if (value.length > MAX_KEY_LENGTH) {
- throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
- }
-}
-
-/** Identifies the Airflow task handled by a TypeScript function. */
-export interface TaskRegistration {
- /** Identifier of the Dag containing this task. */
+/** A registered Dag with its task IDs, returned by {@link DagRegistry.listDags}.
+ * A task-less Dag is included, so the bundle manifest keeps it visible. */
+export interface RegisteredDag {
+ /** Identifier of the registered Dag. */
readonly dagId: string;
- /** Airflow task ID, including any TaskGroup prefix. */
- readonly taskId: string;
+ /** Airflow task IDs, including any TaskGroup prefix. */
+ readonly tasks: string[];
}
-/** Registry of TypeScript task handlers keyed by Dag ID and task ID. */
-export class TaskRegistry {
- readonly #tasks = new Map<string, Map<string, TaskHandler>>();
+/**
+ * The Dags a bundle process can execute, keyed by Dag ID.
+ *
+ * This is what a bundle entry point builds and hands to `serveDags(registry)`:
+ *
+ * ```ts
+ * const dag = new Dag("my_dag");
+ * dag.task("extract", extractFn);
+ * await serveDags(new DagRegistry(dag));
+ * ```
+ *
+ * It holds no sockets and starts nothing, so a test can build one and invoke a
+ * handler through {@link getTaskHandler} without any runtime in scope.
+ *
+ * Lookups delegate live to each Dag's task map, so tasks added to a Dag
+ * after registration are visible — the registry records Dag identity, not
+ * a snapshot of its tasks.
+ */
+export class DagRegistry {
Review Comment:
Do users need listTasks(), and listDags() directly? Since DagRegistry is exported from the package root, these become part of the public API. They are only required by the coordinator though. Could the public registry expose only registration, with package-internal helpers handling the rest?
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,252 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
+
+function validateEmptySpec(name: string, value: unknown): void {
+ if (
+ typeof value !== "object" ||
+ value === null ||
+ Array.isArray(value) ||
+ Reflect.ownKeys(value).length > 0
+ ) {
+ throw new Error(`${name} must be an empty object`);
+ }
+}
+
+/**
+ * Dag-level options. **Reserved: no fields yet.**
+ *
+ * Native TypeScript Dag declaration (schedule, tags, ...) will add optional
+ * fields here without changing the `Dag` constructor, generated from the
+ * serialized-Dag JSON schema the way `src/generated/supervisor.ts` is. Until
+ * then only `{}` is accepted, so a field that would be silently dropped —
+ * `new Dag("d", { schedule: "@daily" })` — is a compile error rather than a
+ * Dag that packs and runs without the schedule.
+ */
+export type DagSpec = Record<string, never>;
+
+/**
+ * Task-level options. **Reserved: no fields yet.**
+ *
+ * Future task fields (retries, ...) will land here without changing the
+ * `dag.task()` signature. As with {@link DagSpec}, only `{}` is accepted today.
+ */
+export type TaskSpec = Record<string, never>;
+
+/**
+ * Opaque handle to a task registered on a {@link Dag}, returned by
+ * `dag.task(...)`.
+ *
+ * Identity only — the handler is deliberately not exposed. Handles are what the
+ * reserved `inputs` option accepts, and what native TypeScript Dag declaration
+ * will use to wire dependencies.
+ */
+export interface TaskRef {
+ /** Identifier of the Dag this task belongs to. */
+ readonly dagId: string;
+ /** Airflow task ID, including any TaskGroup prefix. */
+ readonly taskId: string;
+}
+
+/**
+ * Upstream task handles keyed by input name. **Reserved: validated and
+ * retained, but inert today** — see {@link TaskOptions.inputs}.
+ *
+ * Values must be handles returned by `dag.task(...)`. Literal values are
+ * deliberately out of scope for now; the future native-Dag work decides how
+ * they are declared.
+ */
Review Comment:
Super nit: I found some of the comments a little hard to scan and seems like they are written in response to review comments rather than as docs for someone hovering in an editor. It might help to use a consistent order?
- What it represents today
- Current limitations
- Future purpose
I also prefer simpler language such as “not used yet” instead of “inert.” For example:
```
/**
* Task references keyed by input name.
*
* References are stored and validated, but they do not create dependencies or
* pass values to handlers yet. Each reference must identify an earlier task
* in the same Dag. Literal values are not currently supported.
*
* In the future, these references will be used to define dependencies in
* native TypeScript Dags.
*/
```
Not suggesting to drop the reasoning as it is useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3772638398
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,258 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import { brand, hasBrand } from "./brand.js";
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
Review Comment:
Therefore, I moved the validation into packing stage in https://github.com/apache/airflow/pull/71144/commits/f0edc1caea59087bb7bb86f6e052eed7f0df580c .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3793840334
##########
ts-sdk/src/cli/pack.ts:
##########
Review Comment:
Since key validation moved to buildBundleManifest, this catch is now the only place a bad Dag or task ID is reported. `String(error)` makes it hard to read. Packing `new Dag("bad id!")` currently gives:
```
Running the bundle with --airflow-metadata failed: Error: Command failed: /usr/bin/node /tmp/out/bundle.pack-staging.mjs --airflow-metadata
...
Error: Dag "bad id!" must be made of alphanumeric characters, dashes, dots, and underscores
at validateKey (file:///tmp/out/bundle.pack-staging.mjs:2527:11)
...
```
The one actionable sentence is buried, and every frame points into bundle.pack-staging.mjs, which the finally block deletes — so no line the trace names can be opened.
`execFileSync` puts the child's stderr on error.stderr, so the message is already available here:
```ts
} catch (error) {
const stderr = (error as { stderr?: string }).stderr ?? "";
const reported = stderr.split("\n").find((l) => /^\w*Error: /.test(l.trim()))?.trim();
throw new Error(
reported ?? `Running the bundle with ${AIRFLOW_METADATA_FLAG} failed: ${String(error)}`,
{ cause: error },
);
}
```
Worth considering to catch inside `--airflow-metadata` mode and emit the failure on the sentinel channel. That drops the stack entirely and lets pack distinguish "this Dag ID is invalid" from "the bundle threw on import", which are indistinguishable today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "guan404ming (via GitHub)" <gi...@apache.org>.
guan404ming commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3759192285
##########
ts-sdk/src/coordinator/runtime.ts:
##########
@@ -51,13 +52,46 @@ import {
type RuntimeTaskState,
type StartupDetails,
} from "./protocol.js";
-import { getRegisteredTask, listRegisteredTasks } from "../sdk/registry.js";
+import { DagRegistry } from "../sdk/registry.js";
import type { TaskContext, TaskHandlerArgs } from "../sdk/task.js";
import type { JsonValue } from "../sdk/client-types.js";
export const ABORT_GRACE_PERIOD_MS = 30_000;
export const COORDINATOR_RESPONSE_TIMEOUT_MS = 30_000;
+let served = false;
Review Comment:
Nice cleanup overall! I think there are two small leftovers worth a look:
1. served is set before `startCoordinator` runs and never cleared on failure, so a rejected serveDags (e.g. Missing --comm) blocks retries with a misleading "already called" -> public-api.test.ts actually relies on this sequence. Moving the state onto the registry would fix cleanup and test isolation.
2. The instanceof DagRegistry / instanceof Dag checks reject genuine objects when two SDK copies are installed (dual-package hazard). A structural check would avoid that.
Nit: `serveDags` throws sync for these but rejects for runtime errors; async would unify the channels.
##########
ts-sdk/src/cli/pack.ts:
##########
Review Comment:
One nit consistency thought: pack trusts the manifest more than the SDK layer trusts TypeScript, no suspicious-id warnings (go-pack has `warnOnSuspiciousIDs`), `supervisor_schema_version` accepts any truthy value, and top-level null crashes with a raw `TypeError`. Failing closed here for everything entering the manifest might be worth it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3740704527
##########
ts-sdk/src/sdk/registry.ts:
##########
@@ -17,85 +17,72 @@
* under the License.
*/
+import { Dag, getDagTaskRecords, type Task } from "./dag.js";
import type { TaskHandler } from "./task.js";
-// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
-const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
-const MAX_KEY_LENGTH = 250;
-
-function validateKey(name: string, value: string): void {
- if (typeof value !== "string" || !KEY_REGEX.test(value)) {
- throw new Error(
- `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
- );
- }
- if (value.length > MAX_KEY_LENGTH) {
- throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
- }
-}
-
-/** Identifies the Airflow task handled by a TypeScript function. */
-export interface TaskRegistration {
- /** Identifier of the Dag containing this task. */
+/** A registered Dag with its task IDs, empty Dags included — used for
+ * manifests, where a task-less Dag must stay visible. */
+export interface RegisteredDag {
+ /** Identifier of the registered Dag. */
readonly dagId: string;
- /** Airflow task ID, including any TaskGroup prefix. */
- readonly taskId: string;
+ /** Airflow task IDs, including any TaskGroup prefix. */
+ readonly tasks: string[];
}
-/** Registry of TypeScript task handlers keyed by Dag ID and task ID. */
-export class TaskRegistry {
- readonly #tasks = new Map<string, Map<string, TaskHandler>>();
+/**
+ * Registry of Dag instances keyed by Dag ID.
+ *
+ * Lookups delegate live to each Dag's task map, so tasks added to a Dag
+ * after registration are visible — the registry records Dag identity, not
+ * a snapshot of its tasks.
+ */
+export class DagRegistry {
+ readonly #dags = new Map<string, Dag>();
- /**
- * Register a TypeScript handler for an Airflow task.
- *
- * `dagId` must match the Python Dag's `dag_id`. `taskId` must match the
- * Dag-side operator's `task_id` exactly, including any TaskGroup prefix.
- */
- register<TReturn = unknown>(registration: TaskRegistration, handler: TaskHandler<TReturn>): void {
- const { dagId, taskId } = registration;
- validateKey("dagId", dagId);
- validateKey("taskId", taskId);
- if (typeof handler !== "function") {
- throw new Error(`handler for Dag "${dagId}" task "${taskId}" must be a function`);
+ /** Register Dags. Registering an already-registered `dagId` throws,
+ * and a call that throws registers none of its Dags. */
+ register(...dags: Dag[]): void {
+ const incoming = new Set<string>();
+ for (const dag of dags) {
+ if (!(dag instanceof Dag)) {
+ throw new Error("only Dag instances can be registered");
+ }
+ if (this.#dags.has(dag.dagId) || incoming.has(dag.dagId)) {
+ throw new Error(`Dag "${dag.dagId}" is already registered`);
+ }
+ incoming.add(dag.dagId);
}
- const dagTasks = this.#tasks.get(dagId) ?? new Map<string, TaskHandler>();
- if (dagTasks.has(taskId)) {
- throw new Error(`Task "${taskId}" is already registered for Dag "${dagId}"`);
+ for (const dag of dags) {
+ this.#dags.set(dag.dagId, dag);
}
- dagTasks.set(taskId, handler as TaskHandler);
- this.#tasks.set(dagId, dagTasks);
}
/** Look up a registered handler. Returns `undefined` when no handler exists. */
- get(dagId: string, taskId: string): TaskHandler | undefined {
- return this.#tasks.get(dagId)?.get(taskId);
+ getTaskHandler(dagId: string, taskId: string): TaskHandler | undefined {
+ const dag = this.#dags.get(dagId);
+ return dag ? getDagTaskRecords(dag).get(taskId)?.handler : undefined;
}
- /** List all registered tasks. */
- list(): TaskRegistration[] {
- return [...this.#tasks.entries()].flatMap(([dagId, tasks]) =>
- [...tasks.keys()].map((taskId) => ({ dagId, taskId })),
+ /** List the task handles across registered Dags. */
+ listTasks(): Task[] {
+ return [...this.#dags.values()].flatMap((dag) =>
+ [...getDagTaskRecords(dag).values()].map((record) => record.task),
);
}
-}
-const defaultRegistry = new TaskRegistry();
-
-/** Register a TypeScript handler in the default task registry. */
-export function registerTask<TReturn = unknown>(
- registration: TaskRegistration,
- handler: TaskHandler<TReturn>,
-): void {
- defaultRegistry.register(registration, handler);
+ /** List every registered Dag with its task IDs, empty Dags included. */
+ listDags(): RegisteredDag[] {
+ return [...this.#dags.values()].map((dag) => ({
+ dagId: dag.dagId,
+ tasks: [...getDagTaskRecords(dag).keys()],
+ }));
+ }
}
-/** Look up a registered handler. Returns `undefined` when no handler exists. */
-export function getRegisteredTask(dagId: string, taskId: string): TaskHandler | undefined {
- return defaultRegistry.get(dagId, taskId);
-}
+/** The registry `registerDags` writes to and the coordinator reads from. */
+export const defaultRegistry = new DagRegistry();
-/** List all registered tasks. */
-export function listRegisteredTasks(): TaskRegistration[] {
- return defaultRegistry.list();
+/** Record Dags in the default registry so the coordinator can run their tasks. */
+export function registerDags(...dags: Dag[]): void {
Review Comment:
Splitting declaration from registration makes this third step forgettable, and the chain fails quietly when it is missed.
`buildBundleManifest()` reads `defaultRegistry.listDags()`, so a Dag that was constructed and populated but never passed here is absent from the manifest. `runPack` catches only the all-or-nothing case (`registered no Dags`), so registering 3 of 4 Dags packs successfully. `NodeCoordinator._find_bundle` ignores the manifest's `dags` map when selecting a bundle (`task-sdk/src/airflow/sdk/coordinators/node/coordinator.py`), so the bundle is still launched for the missing Dag's tasks, `handleTask` finds no handler, and it returns `{type: "TaskState", state: "removed"}`. The user sees task instances quietly marked *removed*, which normally means the task is no longer in the Dag, with only a `warning` line in the task log.
Fair caveat: forgetting `import "./sales/tasks"` under the old side-effect pattern registered nothing either, so this is a new instance of an existing mistake rather than a regression in kind. What is new is that it is now cheaply detectable, because the Dag is an object the SDK can see.
Cheapest fix that keeps this shape: have the `Dag` constructor push `this` onto a module-level list, emit it in `--airflow-metadata` mode, and have pack fail with `Dag "billing" was declared but never passed to registerDags(...)`.
The structural version is worth a thought too. Java uses this same constructor shape (`java.rst`: `var dag = new Dag("my_dag"); dag.addTask("fetch", FetchTask.class); return List.of(dag);`) but returns the Dags from `getDags()`, so omission is unrepresentable; Go inverts ownership instead, with `Registry.AddDag(dagId) Dag`. A TypeScript analogue of either, `startCoordinator({ dags: [salesDag, billingDag] })` or an `addDag(dagId, spec?)` that registers eagerly and returns the handle, removes the forgettable step rather than detecting it.
On naming, while the package is still unpublished: `Task` here is an opaque `{dagId, taskId}` identity, whereas Go's `Task` is the executable thing the runtime calls and Java's is the interface a task class implements. `TaskRef` or `TaskHandle` would keep the vocabulary aligned across the three SDKs and leave `Task` free for the real task object native Dag declaration will need.
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,208 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
+
+/**
+ * Dag-level options.
+ *
+ * Empty today: native TypeScript Dag declaration (schedule, tags, ...) will add
+ * optional fields here without changing the `Dag` constructor signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future native-Dag fields
+export interface DagSpec {}
+
+/**
+ * Task-level options.
+ *
+ * Empty today: future task fields (retries, ...) will land here without
+ * changing the `dag.task()` signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future task fields
+export interface TaskSpec {}
+
+/**
+ * Opaque handle to a task registered on a {@link Dag}.
+ *
+ * Pass it as a downstream task's input to declare that the downstream task
+ * consumes this task's return value:
+ *
+ * ```ts
+ * const extracted = dag.task("extract", extractFn);
+ * const transformed = dag.task("transform", transformFn, { inputs: { extracted } });
+ * dag.task("load", loadFn, { inputs: { transformed } });
+ * ```
+ *
+ * In today's Python-stub mode the stub Dag still defines task order; declared
+ * inputs are retained for the serialized Dag JSON that native TypeScript Dag
+ * declaration will emit. The handler is intentionally not exposed on the handle.
+ */
+export interface Task {
+ /** Identifier of the Dag this task belongs to. */
+ readonly dagId: string;
+ /** Airflow task ID, including any TaskGroup prefix. */
+ readonly taskId: string;
+}
+
+/**
+ * Upstream task handles a task consumes, keyed by input name.
+ *
+ * Values must be handles returned by `dag.task(...)`. Literal values are
+ * deliberately out of scope for now; the future native-Dag work decides how
+ * they are declared.
+ */
+export type TaskInputs = Readonly<Record<string, Task>>;
+
+/**
+ * Named options for `dag.task()`.
+ *
+ * Keyword-only so neither field has to be positioned around the other, and so
+ * future fields can be added without a new parameter.
+ */
+export interface TaskOptions {
+ /** Upstream task handles whose return values this task consumes. */
+ readonly inputs?: TaskInputs;
+ /** Task-level options. */
+ readonly spec?: TaskSpec;
+}
+
+/** Per-task record a Dag retains: the handle, the handler, its spec, and the
+ * upstream handles feeding it. */
+export interface TaskRecord {
+ readonly task: Task;
+ readonly handler: TaskHandler;
+ readonly spec: TaskSpec;
+ /** Upstream handles keyed by input name; empty when the task has no inputs. */
+ readonly inputs: TaskInputs;
+}
+
+// Assigned inside Dag's static block: gives package-internal code read access
+// to the #tasks private field without a public accessor on the Dag class.
+let taskRecordsOf: (dag: Dag) => ReadonlyMap<string, TaskRecord>;
+
+/**
+ * A Dag declared in TypeScript.
+ *
+ * Today the Dag structure itself is still declared by a Python stub file; a
+ * `Dag` instance binds TypeScript handlers to that stub's Dag/task IDs. The
+ * instance retains its `spec` and every task's `(taskId, handler, spec)` so a
+ * future `serialize()` can produce the serialized Dag JSON for native
+ * TypeScript Dag declaration.
+ */
+export class Dag {
+ /** Identifier of this Dag. Must match the Python Dag's `dag_id`. */
+ readonly dagId: string;
+ /** Dag-level options this instance was constructed with. */
+ readonly spec: DagSpec;
+ readonly #tasks = new Map<string, TaskRecord>();
+
+ static {
+ taskRecordsOf = (dag) => dag.#tasks;
+ }
+
+ constructor(dagId: string, spec: DagSpec = {}) {
+ validateKey("dagId", dagId);
+ this.dagId = dagId;
+ this.spec = spec;
+ }
+
+ /**
+ * Register a TypeScript handler for a task of this Dag.
+ *
+ * `taskId` must match the Dag-side operator's `task_id` exactly, including
+ * any TaskGroup prefix. `options.inputs` names the upstream task handles whose
+ * return values this task consumes. Returns this task's handle.
+ */
+ task<TReturn = unknown>(
+ taskId: string,
+ handler: TaskHandler<TReturn>,
+ options: TaskOptions = {},
+ ): Task {
+ const { inputs = {}, spec = {} } = options;
+ validateKey("taskId", taskId);
+ if (typeof handler !== "function") {
+ throw new Error(`handler for Dag "${this.dagId}" task "${taskId}" must be a function`);
+ }
+ if (this.#tasks.has(taskId)) {
+ throw new Error(`Task "${taskId}" is already registered for Dag "${this.dagId}"`);
+ }
+ this.#validateInputs(taskId, inputs);
+ const task: Task = Object.freeze({ dagId: this.dagId, taskId });
+ this.#tasks.set(taskId, {
+ task,
+ handler: handler as TaskHandler,
+ spec,
+ inputs: Object.freeze({ ...inputs }),
Review Comment:
`inputs` is copied and frozen here, but `spec` on the line above and `this.spec` in the constructor are stored by reference, and `tests/sdk/dag.test.ts` locks that in with `toBe(dagSpec)` / `toBe(taskSpec)`. Once `DagSpec` carries real fields, a caller who mutates their spec object after construction silently mutates the Dag. `Object.freeze({ ...spec })` in both places, with those two assertions relaxed to `toEqual`, makes the two options behave alike.
Separately, the test named "records frozen inputs that later mutation of the caller's object cannot change" does not test the freeze. It adds a new *key* to the caller's object and asserts the record is unchanged, which passes on the spread copy alone; delete `Object.freeze` here and it stays green. One assertion covers it: `expect(Object.isFrozen(getDagTaskRecords(dag).get("transform")!.inputs)).toBe(true)`.
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,208 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
+
+/**
+ * Dag-level options.
+ *
+ * Empty today: native TypeScript Dag declaration (schedule, tags, ...) will add
+ * optional fields here without changing the `Dag` constructor signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future native-Dag fields
+export interface DagSpec {}
+
+/**
+ * Task-level options.
+ *
+ * Empty today: future task fields (retries, ...) will land here without
+ * changing the `dag.task()` signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future task fields
+export interface TaskSpec {}
+
+/**
+ * Opaque handle to a task registered on a {@link Dag}.
+ *
+ * Pass it as a downstream task's input to declare that the downstream task
+ * consumes this task's return value:
+ *
+ * ```ts
+ * const extracted = dag.task("extract", extractFn);
+ * const transformed = dag.task("transform", transformFn, { inputs: { extracted } });
+ * dag.task("load", loadFn, { inputs: { transformed } });
+ * ```
+ *
+ * In today's Python-stub mode the stub Dag still defines task order; declared
+ * inputs are retained for the serialized Dag JSON that native TypeScript Dag
+ * declaration will emit. The handler is intentionally not exposed on the handle.
+ */
+export interface Task {
+ /** Identifier of the Dag this task belongs to. */
+ readonly dagId: string;
+ /** Airflow task ID, including any TaskGroup prefix. */
+ readonly taskId: string;
+}
+
+/**
+ * Upstream task handles a task consumes, keyed by input name.
+ *
+ * Values must be handles returned by `dag.task(...)`. Literal values are
+ * deliberately out of scope for now; the future native-Dag work decides how
+ * they are declared.
+ */
+export type TaskInputs = Readonly<Record<string, Task>>;
+
+/**
+ * Named options for `dag.task()`.
+ *
+ * Keyword-only so neither field has to be positioned around the other, and so
+ * future fields can be added without a new parameter.
+ */
+export interface TaskOptions {
+ /** Upstream task handles whose return values this task consumes. */
+ readonly inputs?: TaskInputs;
+ /** Task-level options. */
+ readonly spec?: TaskSpec;
+}
+
+/** Per-task record a Dag retains: the handle, the handler, its spec, and the
+ * upstream handles feeding it. */
+export interface TaskRecord {
+ readonly task: Task;
+ readonly handler: TaskHandler;
+ readonly spec: TaskSpec;
+ /** Upstream handles keyed by input name; empty when the task has no inputs. */
+ readonly inputs: TaskInputs;
+}
+
+// Assigned inside Dag's static block: gives package-internal code read access
+// to the #tasks private field without a public accessor on the Dag class.
+let taskRecordsOf: (dag: Dag) => ReadonlyMap<string, TaskRecord>;
+
+/**
+ * A Dag declared in TypeScript.
+ *
+ * Today the Dag structure itself is still declared by a Python stub file; a
+ * `Dag` instance binds TypeScript handlers to that stub's Dag/task IDs. The
+ * instance retains its `spec` and every task's `(taskId, handler, spec)` so a
+ * future `serialize()` can produce the serialized Dag JSON for native
+ * TypeScript Dag declaration.
+ */
+export class Dag {
+ /** Identifier of this Dag. Must match the Python Dag's `dag_id`. */
+ readonly dagId: string;
+ /** Dag-level options this instance was constructed with. */
+ readonly spec: DagSpec;
+ readonly #tasks = new Map<string, TaskRecord>();
Review Comment:
With tasks in a `#`-private field and `listRegisteredTasks` removed, no public API is left that answers "which tasks did I wire up?". `defaultRegistry`, `DagRegistry.listTasks`, `listDags` and `getDagTaskRecords` are all module-exported but absent from `src/index.ts`, the two-entry `exports` map blocks deep imports, and `./coordinator` re-exports only `startCoordinator`, `StartCoordinatorOptions` and `SUPERVISOR_API_VERSION`.
That matters here specifically, because it removes the one test a user could write against the drift this stub-Dag model is exposed to: that their TypeScript handlers match the Python `@task.stub` names. It is also the test that would catch a forgotten `registerDags` at author time instead of as task instances marked *removed*. The Go SDK exposes this deliberately, through `EnumerableBundle.OrderedDags()`.
```ts
/** Task IDs attached to this Dag, in attachment order. */
get taskIds(): readonly string[] { return [...this.#tasks.keys()]; }
```
That also lets `registry.listDags()` stop going through `getDagTaskRecords`.
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,208 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
+
+/**
+ * Dag-level options.
+ *
+ * Empty today: native TypeScript Dag declaration (schedule, tags, ...) will add
+ * optional fields here without changing the `Dag` constructor signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future native-Dag fields
+export interface DagSpec {}
+
+/**
+ * Task-level options.
+ *
+ * Empty today: future task fields (retries, ...) will land here without
+ * changing the `dag.task()` signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future task fields
+export interface TaskSpec {}
+
+/**
+ * Opaque handle to a task registered on a {@link Dag}.
+ *
+ * Pass it as a downstream task's input to declare that the downstream task
Review Comment:
`inputs` does not declare anything today, but this headline says it does, and the gap can produce wrong data rather than an error.
Grepping `ts-sdk/src/`, nothing outside this file reads `TaskRecord.inputs`: `registry.ts` reads `.handler`, `.task` and `.keys()`, `manifest.ts` reads only `dagId`/`tasks`. There is also no channel to deliver an input to a handler, since `TaskHandlerArgs` is still `{ctx, client}` (`src/sdk/task.ts`).
The reachable failure: a user who reads this headline concludes the value is wired, so writes `await client.getXCom({ key: "return_value" })` with no `taskId`. Per `src/sdk/client-types.ts:32-33`, `taskId` defaults to the running task's context, so that reads the task's *own* XCom, returns `null`, and the task succeeds with the wrong value. No error, no warning, no failed task instance.
Neither `ts-sdk/README.md` nor `typescript.rst` mentions `inputs`, and the rst note next to the paragraph this PR edits still says dependencies are declared in the Python stub Dag, so autocomplete plus this JSDoc is the only discovery channel.
Either resolution works:
- Drop `inputs`/`TaskInputs`/`#validateInputs` for now and land `Dag` + `dag.task(taskId, handler)` + `registerDags`, which is the whole refactor the title promises. Adding an optional trailing `options` parameter later is not source-breaking in TypeScript, so waiting costs nothing.
- Keep it, but lead the headline with what it does today (nothing), and add a `dag.task` options table to the rst marking `inputs` reserved and inert.
##########
ts-sdk/src/cli/pack.ts:
##########
@@ -193,9 +202,19 @@ export async function runPack(argv: readonly string[]): Promise<void> {
});
const manifest = readBundleManifest(stagingPath);
- if (Object.keys(manifest.dags).length === 0) {
+ const dagEntries = Object.entries(manifest.dags);
+ if (dagEntries.length === 0) {
+ throw new Error(
+ `${args.entry} registered no Dags; register Dags with registerDags(...) before startCoordinator()`,
+ );
+ }
+ const emptyDags = dagEntries
+ .filter(([, dag]) => dag.tasks.length === 0)
+ .map(([dagId]) => dagId);
+ if (emptyDags.length > 0) {
Review Comment:
This is a policy divergence from the packer this file says it mirrors. For the same condition `airflow-go-pack` prints `warning: dag %q has no tasks` and continues (`go-sdk/cmd/airflow-go-pack/pack.go`), while this throws. The two agree on rejecting zero Dags, and the shared schema permits an empty list, since `dagEntry.tasks` has no `minItems`.
Failing closed is arguably the better call, and there is a forward-looking argument for it, since `executable/coordinator.py` claims bundle ownership from `set(dags.keys())` alone and Node is slated to grow the same routing. But it means one placeholder Dag, or conditional attachment like `if (process.env.FEATURE_X) dag.task(...)`, blocks the whole bundle build in TypeScript and not in Go. Worth picking one deliberately and applying it across both: match Go's warning, or hard-error in both and add `"minItems": 1` to the schema.
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,208 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
+
+/**
+ * Dag-level options.
+ *
+ * Empty today: native TypeScript Dag declaration (schedule, tags, ...) will add
+ * optional fields here without changing the `Dag` constructor signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future native-Dag fields
+export interface DagSpec {}
+
+/**
+ * Task-level options.
+ *
+ * Empty today: future task fields (retries, ...) will land here without
+ * changing the `dag.task()` signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future task fields
+export interface TaskSpec {}
+
+/**
+ * Opaque handle to a task registered on a {@link Dag}.
+ *
+ * Pass it as a downstream task's input to declare that the downstream task
+ * consumes this task's return value:
+ *
+ * ```ts
+ * const extracted = dag.task("extract", extractFn);
+ * const transformed = dag.task("transform", transformFn, { inputs: { extracted } });
+ * dag.task("load", loadFn, { inputs: { transformed } });
+ * ```
+ *
+ * In today's Python-stub mode the stub Dag still defines task order; declared
+ * inputs are retained for the serialized Dag JSON that native TypeScript Dag
+ * declaration will emit. The handler is intentionally not exposed on the handle.
+ */
+export interface Task {
+ /** Identifier of the Dag this task belongs to. */
+ readonly dagId: string;
+ /** Airflow task ID, including any TaskGroup prefix. */
+ readonly taskId: string;
+}
+
+/**
+ * Upstream task handles a task consumes, keyed by input name.
+ *
+ * Values must be handles returned by `dag.task(...)`. Literal values are
+ * deliberately out of scope for now; the future native-Dag work decides how
+ * they are declared.
+ */
+export type TaskInputs = Readonly<Record<string, Task>>;
+
+/**
+ * Named options for `dag.task()`.
+ *
+ * Keyword-only so neither field has to be positioned around the other, and so
+ * future fields can be added without a new parameter.
+ */
+export interface TaskOptions {
+ /** Upstream task handles whose return values this task consumes. */
+ readonly inputs?: TaskInputs;
+ /** Task-level options. */
+ readonly spec?: TaskSpec;
+}
+
+/** Per-task record a Dag retains: the handle, the handler, its spec, and the
+ * upstream handles feeding it. */
+export interface TaskRecord {
+ readonly task: Task;
+ readonly handler: TaskHandler;
+ readonly spec: TaskSpec;
+ /** Upstream handles keyed by input name; empty when the task has no inputs. */
+ readonly inputs: TaskInputs;
+}
+
+// Assigned inside Dag's static block: gives package-internal code read access
+// to the #tasks private field without a public accessor on the Dag class.
+let taskRecordsOf: (dag: Dag) => ReadonlyMap<string, TaskRecord>;
+
+/**
+ * A Dag declared in TypeScript.
+ *
+ * Today the Dag structure itself is still declared by a Python stub file; a
+ * `Dag` instance binds TypeScript handlers to that stub's Dag/task IDs. The
+ * instance retains its `spec` and every task's `(taskId, handler, spec)` so a
+ * future `serialize()` can produce the serialized Dag JSON for native
+ * TypeScript Dag declaration.
+ */
+export class Dag {
+ /** Identifier of this Dag. Must match the Python Dag's `dag_id`. */
+ readonly dagId: string;
+ /** Dag-level options this instance was constructed with. */
+ readonly spec: DagSpec;
+ readonly #tasks = new Map<string, TaskRecord>();
+
+ static {
+ taskRecordsOf = (dag) => dag.#tasks;
+ }
+
+ constructor(dagId: string, spec: DagSpec = {}) {
+ validateKey("dagId", dagId);
+ this.dagId = dagId;
+ this.spec = spec;
+ }
+
+ /**
+ * Register a TypeScript handler for a task of this Dag.
+ *
+ * `taskId` must match the Dag-side operator's `task_id` exactly, including
+ * any TaskGroup prefix. `options.inputs` names the upstream task handles whose
+ * return values this task consumes. Returns this task's handle.
+ */
+ task<TReturn = unknown>(
+ taskId: string,
+ handler: TaskHandler<TReturn>,
+ options: TaskOptions = {},
+ ): Task {
+ const { inputs = {}, spec = {} } = options;
Review Comment:
This destructure accepts any other shape silently, which is out of step with the three checks around it. `taskId`, `handler`, and every `inputs` value are runtime-validated precisely because TypeScript can be bypassed, but `dag.task("transform", fn, { input: { extracted } })` (singular typo) or `{ inpts: {...} }` registers a task with no declared upstream and says nothing. Reachable from plain JavaScript, since `airflow-ts-pack` bundles whatever esbuild accepts, and from an `as TaskOptions` cast. This package already fails closed on the same class of mistake: `parsePackArgs` throws `Unknown option ${arg}`.
Worth fixing in the same place: the new comment at `tests/public-api.test.ts:178` says "these constructor/method misuses also throw at runtime", and that is false for one of the five lines it covers. `dag.task("transform2", async () => undefined, { upstream })` does not throw; it registers with zero inputs and discards the declared upstream. The other four do throw. Because the arrow function is never invoked, nothing catches it.
```ts
for (const key of Object.keys(options)) {
if (key !== "inputs" && key !== "spec") {
throw new Error(`Unknown option "${key}" for Dag "${this.dagId}" task "${taskId}"`);
}
}
```
Also worth handling while here: `options === null` currently fails with an opaque "Cannot destructure property", because the default only applies to `undefined`.
##########
ts-sdk/src/cli/pack.ts:
##########
@@ -154,6 +154,15 @@ function readBundleManifest(bundlePath: string): BundleManifest {
if (!manifest.supervisor_schema_version || !manifest.dags || typeof manifest.dags !== "object") {
throw new Error(`Bundle produced incomplete ${AIRFLOW_METADATA_FLAG} output`);
}
+ // The line is whatever the bundle printed, so the per-Dag shape is not
+ // guaranteed by the type assertion above.
+ for (const [dagId, dag] of Object.entries(manifest.dags)) {
+ if (dag == null || !Array.isArray(dag.tasks)) {
Review Comment:
This stops one level short of the schema it protects. Element types are unchecked, so `{"dags":{"d":{"tasks":[null,{},"",123]}}}` passes here and flows through `renderMetadataYaml` into the base64 trailer, while `task-sdk/docs/airflow-metadata.schema.json` defines `dagEntry.tasks.items` as `{"type": "string", "minLength": 1}`. The Go reference packer cannot emit that, because it decodes into `Tasks []string`, and the Python reader does not close the gap either: `_bundle_metadata.py` validates only that the document is a mapping and that `sdk.supervisor_schema_version` is a non-empty string, so bad task ids propagate past pack.
```ts
if (dag == null || !Array.isArray(dag.tasks) ||
dag.tasks.some((t) => typeof t !== "string" || t.length === 0)) {
```
Cheap while you are here: the guard on the line above passes for an *array*, which `Object.entries` then turns into Dags named `"0"`, `"1"`. Pre-existing, but this loop is the natural place to close it.
##########
ts-sdk/src/sdk/dag.ts:
##########
@@ -0,0 +1,208 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The Dag authoring surface: `new Dag(dagId)` plus `dag.task(taskId, handler)`.
+
+import type { TaskHandler } from "./task.js";
+
+// Mirrors the Python task-SDK KEY_REGEX and validate_key in airflow.sdk.definitions._internal.node.
+const KEY_REGEX = /^[\p{L}\p{N}_.-]+$/u;
+const MAX_KEY_LENGTH = 250;
+
+function validateKey(name: string, value: string): void {
+ if (typeof value !== "string" || !KEY_REGEX.test(value)) {
+ throw new Error(
+ `${name} must be made of alphanumeric characters, dashes, dots, and underscores`,
+ );
+ }
+ if (value.length > MAX_KEY_LENGTH) {
+ throw new Error(`${name} must be less than ${MAX_KEY_LENGTH} characters, not ${value.length}`);
+ }
+}
+
+/**
+ * Dag-level options.
+ *
+ * Empty today: native TypeScript Dag declaration (schedule, tags, ...) will add
+ * optional fields here without changing the `Dag` constructor signature.
+ */
+// eslint-disable-next-line @typescript-eslint/no-empty-object-type -- extension point for future native-Dag fields
+export interface DagSpec {}
Review Comment:
`interface DagSpec {}` is the empty object type, so `spec` accepts every non-nullish value, and the rule suppressed here is the one that reports exactly that. Today `new Dag("d", 42)` and `dag.task("t", fn, { spec: { retries: 3 } })` both typecheck and store the value unvalidated. `TaskSpec`'s own docstring advertises `retries` as a future field, so that second call is a natural thing to write and have silently ignored.
The repo has already taken a position on the spelling: `airflow-core/src/airflow/ui/rules/typescript.js:668` sets `@typescript-eslint/no-empty-object-type` to `ERROR`, and its doc block lists `type FooType = {}` as incorrect while prescribing `type FooType = object`. That is the UI package's config rather than `ts-sdk/eslint.config.js`, so precedent rather than a rule this PR breaks, but the question has an in-repo answer.
The stated rationale also inverts. Once `DagSpec` gains its first field, necessarily optional, it becomes a weak type, and any call site passing an object with no overlapping property starts failing with TS2559. Every other extension the comment anticipates is already non-breaking: an optional trailing parameter, an optional property on `TaskOptions` (a type users pass and never implement), and `Task` becoming `Task<TReturn = unknown>` are all source-compatible. The only source-breaking change in that set is the one the placeholder introduces.
`export type DagSpec = object` rejects primitives and matches the spelling the repo prescribes; `Record<string, never>` also rejects `{retries: 3}`. Either one removes both `eslint-disable` lines.
One caveat on my side: I verified the lint policy and these declarations, but not the assignability behaviour with a `tsc` run, since `ts-sdk/` has no installed `node_modules` in my checkout. Both are one-line checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #71144:
URL: https://github.com/apache/airflow/pull/71144#issuecomment-5238626229
Maybe we can do something like this instead
```ts
const dag = new Dag(...);
// Add tasks...
const registry = new DagRegistry(dag);
registry.serve();
```
Or, even more similar to Java SDK, `new Server(registry).serve()` so we can separate dag registration logic and network logic into different types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3793877748
##########
ts-sdk/src/cli/pack.ts:
##########
@@ -143,20 +143,52 @@ function readBundleManifest(bundlePath: string): BundleManifest {
throw new Error(`Bundle produced no ${AIRFLOW_METADATA_FLAG} output`);
}
- let manifest: BundleManifest;
+ let parsed: unknown;
try {
- manifest = JSON.parse(line.slice(AIRFLOW_METADATA_SENTINEL.length)) as BundleManifest;
+ parsed = JSON.parse(line.slice(AIRFLOW_METADATA_SENTINEL.length));
} catch (error) {
throw new Error(`Bundle produced invalid ${AIRFLOW_METADATA_FLAG} output: ${String(error)}`, {
cause: error,
});
}
- if (!manifest.supervisor_schema_version || !manifest.dags || typeof manifest.dags !== "object") {
+ if (!isBundleManifest(parsed)) {
throw new Error(`Bundle produced incomplete ${AIRFLOW_METADATA_FLAG} output`);
}
+ const manifest = parsed;
+ // The line is whatever the bundle printed and nothing downstream re-validates
+ // it, so check each Dag entry down to the task-id element.
+ for (const [dagId, dag] of Object.entries(manifest.dags)) {
+ if (dag == null || !isDagIdList(dag.tasks)) {
+ throw new Error(
+ `Bundle produced ${AIRFLOW_METADATA_FLAG} output with a malformed entry for Dag "${dagId}"`,
+ );
+ }
+ }
return manifest;
}
+// The document is checked before anything is read off it: JSON.parse also yields
+// null and primitives, and `null.supervisor_schema_version` would surface as a
+// raw TypeError rather than a report about the bundle.
+function isBundleManifest(value: unknown): value is BundleManifest {
+ if (typeof value !== "object" || value === null || Array.isArray(value)) return false;
+ const { supervisor_schema_version: version, dags } = value as Partial<BundleManifest>;
+ return (
+ // Rendered into the manifest verbatim, where the schema requires a non-empty
+ // string, so a truthy number or boolean would travel to Airflow as-is.
+ typeof version === "string" &&
+ version.length > 0 &&
Review Comment:
Yes, we will validate against a regex in https://github.com/apache/airflow/pull/70993.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3772633778
##########
ts-sdk/src/cli/pack.ts:
##########
@@ -143,20 +143,52 @@ function readBundleManifest(bundlePath: string): BundleManifest {
throw new Error(`Bundle produced no ${AIRFLOW_METADATA_FLAG} output`);
}
- let manifest: BundleManifest;
+ let parsed: unknown;
try {
- manifest = JSON.parse(line.slice(AIRFLOW_METADATA_SENTINEL.length)) as BundleManifest;
+ parsed = JSON.parse(line.slice(AIRFLOW_METADATA_SENTINEL.length));
} catch (error) {
throw new Error(`Bundle produced invalid ${AIRFLOW_METADATA_FLAG} output: ${String(error)}`, {
cause: error,
});
}
- if (!manifest.supervisor_schema_version || !manifest.dags || typeof manifest.dags !== "object") {
+ if (!isBundleManifest(parsed)) {
throw new Error(`Bundle produced incomplete ${AIRFLOW_METADATA_FLAG} output`);
}
+ const manifest = parsed;
+ // The line is whatever the bundle printed and nothing downstream re-validates
+ // it, so check each Dag entry down to the task-id element.
+ for (const [dagId, dag] of Object.entries(manifest.dags)) {
+ if (dag == null || !isDagIdList(dag.tasks)) {
+ throw new Error(
+ `Bundle produced ${AIRFLOW_METADATA_FLAG} output with a malformed entry for Dag "${dagId}"`,
+ );
+ }
+ }
return manifest;
}
+// The document is checked before anything is read off it: JSON.parse also yields
+// null and primitives, and `null.supervisor_schema_version` would surface as a
+// raw TypeError rather than a report about the bundle.
+function isBundleManifest(value: unknown): value is BundleManifest {
+ if (typeof value !== "object" || value === null || Array.isArray(value)) return false;
+ const { supervisor_schema_version: version, dags } = value as Partial<BundleManifest>;
+ return (
+ // Rendered into the manifest verbatim, where the schema requires a non-empty
+ // string, so a truthy number or boolean would travel to Airflow as-is.
+ typeof version === "string" &&
+ version.length > 0 &&
Review Comment:
I think it's fine to keep it since the validation is in the packing stage instead of the runtime stage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on PR #71144:
URL: https://github.com/apache/airflow/pull/71144#issuecomment-5249545926
> Just curious about the longer-term direction. Do we expect tasks to become callable, similar to Python TaskFlow?
Yes, we will support the TaskFlow-style syntax to define the task dependencies as mentioned in https://github.com/apache/airflow/issues/69288#issuecomment-5093501655.
> One option would be for dag.task() to return a callable TaskDefinition<TInputs, TOutput>. Calling that definition would record the input bindings and return a TaskOutput.
This is what the `TaskRef` does today, though I haven't refine it into the "accurate" type like you mentioned, the `TaskRef` is more like a placeholder type right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71144:
URL: https://github.com/apache/airflow/pull/71144#discussion_r3810530494
##########
ts-sdk/src/cli/pack.ts:
##########
@@ -129,9 +129,15 @@ function readBundleManifest(bundlePath: string): BundleManifest {
maxBuffer: MANIFEST_MAX_BUFFER_BYTES,
});
} catch (error) {
- throw new Error(`Running the bundle with ${AIRFLOW_METADATA_FLAG} failed: ${String(error)}`, {
- cause: error,
- });
+ const stderr = (error as { stderr?: string }).stderr ?? "";
+ const reported = stderr
+ .split("\n")
+ .find((line) => /^\w*Error: /.test(line.trim()))
+ ?.trim();
Review Comment:
Addressed in https://github.com/apache/airflow/pull/71144/commits/5d853066b634663ba23d44585338df56bbf8e185, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] TS SDK: replace registerTask with Dag and registerDags [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 merged PR #71144:
URL: https://github.com/apache/airflow/pull/71144
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org