You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "shivaam (via GitHub)" <gi...@apache.org> on 2026/06/02 17:22:20 UTC
[PR] Add TypeScript Task SDK [airflow]
shivaam opened a new pull request, #67908:
URL: https://github.com/apache/airflow/pull/67908
Add \`ts-sdk/\` — a TypeScript SDK for authoring and running Airflow tasks in Node.js. Supports two execution modes:
**Coordinator mode** (AIP-108): spawned as a one-shot subprocess per task by \`TypescriptCoordinator\` (companion PR); communicates over TCP with the Airflow supervisor using the same 4-byte length-prefixed msgpack protocol as the Python and Java SDKs. Supports all 13 \`TaskClient\` APIs (Variables, XCom, Connections, Dag operations, secret masking) and DAG serialization.
**Edge Worker mode** (AIP-69/72): long-lived HTTPS worker using JWT authentication; picks up tasks from the edge executor queue.
Both modes share \`registerTask()\` and the same \`TaskClient\` interface — handlers are mode-agnostic.
## What's included
- \`ts-sdk/src/\` — full source (coordinator + edge + shared client interface)
- \`ts-sdk/tests/\` — 180 unit tests (vitest)
- \`ts-sdk/integration/\` — integration test workers and Python test scripts
- \`ts-sdk/schema/supervisor-schema.json\` — vendored supervisor wire schema (PR #67235, api_version \`2026-06-16\`)
- \`ts-sdk/scripts/generate-supervisor.mjs\` — regenerates types from schema
## What's deferred (follow-up PRs)
- CI workflow and pre-commit hooks for \`ts-sdk/\`
- \`TypescriptCoordinator\` Python-side wiring (companion PR)
- Structured log forwarding for edge mode (\`TODO(pr4)\` in \`worker.ts\`)
- Per-task token refresh and subprocess isolation for edge mode
## Testing
\`\`\`bash
cd ts-sdk
pnpm install
pnpm test # 180 tests, all pass
pnpm run typecheck
\`\`\`
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Sonnet 4.6)
Generated-by: Claude Code (Sonnet 4.6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
Drafted-by: Claude Code (Sonnet 4.6) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TypeScript task SDK public interface [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #67908:
URL: https://github.com/apache/airflow/pull/67908#discussion_r3414088225
##########
ts-sdk/src/client-types.ts:
##########
@@ -0,0 +1,68 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/** JSON-compatible value accepted by Airflow for XCom payloads. */
+export type JsonValue =
+ | string
+ | number
+ | boolean
+ | null
+ | JsonValue[]
+ | { [key: string]: JsonValue };
+
+/**
+ * Options for pulling an XCom value.
+ *
+ * `dagId`, `taskId`, and `runId` default to the running task's context.
+ * Pass them only when pulling an XCom value from another task or run.
+ */
+export interface GetXComOpts {
+ key: string;
+ dagId?: string;
+ runId?: string;
+ taskId?: string;
+ mapIndex?: number | null;
+ includePriorDates?: boolean;
+}
+
+/**
+ * Options for pushing an XCom value.
+ *
+ * `dagId`, `taskId`, and `runId` default to the running task's context.
+ */
+export interface SetXComOpts {
+ key: string;
+ value: JsonValue;
+ dagId?: string;
+ runId?: string;
+ taskId?: string;
+ mapIndex?: number | null;
+}
+
+/** Airflow Connection details returned by a task client. */
+export interface ConnectionResult {
+ connId: string;
+ connType: string;
Review Comment:
`connId`/`connType` diverge slightly from the sibling SDKs — Java's `Connection` uses `id`/`type`, Go's uses `ID`/`Type`. The camelCase-of-the-wire-name reads clearly on its own so I don't feel strongly, but since these public surfaces are meant to mirror each other, could we make it a conscious choice? Either align to `id`/`type`, or keep the `conn` prefix and note why it's preferred for TS.
##########
ts-sdk/src/registry.ts:
##########
@@ -0,0 +1,90 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import type { TaskHandler } from "./task.js";
+
+/** Identifies the Airflow task handled by a TypeScript function. */
+export interface TaskRegistration {
+ /** Identifier of the Dag containing this task. */
+ dagId: string;
+ /** Airflow task ID, including any TaskGroup prefix. */
+ taskId: string;
Review Comment:
Tiny nit: `TaskContext` and `TaskHandlerArgs` are `readonly`, but `TaskRegistration`'s fields aren't — could we make them `readonly` too for consistency?
```suggestion
/** Identifier of the Dag containing this task. */
readonly dagId: string;
/** Airflow task ID, including any TaskGroup prefix. */
readonly taskId: string;
```
##########
ts-sdk/src/task.ts:
##########
@@ -0,0 +1,63 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The task-handler call surface — types every user task handler sees.
+
+import type { TaskClient } from "./client.js";
+
+/** Runtime metadata for the current task invocation. */
+export interface TaskContext {
+ /** Identifier of the Dag containing this task. */
+ readonly dagId: string;
+ /** Task ID for this handler invocation, including any TaskGroup prefix. */
+ readonly taskId: string;
+ /** Dag run identifier for the current task attempt. */
+ readonly runId: string;
+ /** Airflow try number for the current task attempt. */
+ readonly tryNumber: number;
+ /** -1 for non-mapped tasks, 0..N-1 for mapped instances. */
+ readonly mapIndex: number;
+ /** Stable Airflow task instance identifier. */
+ readonly taskInstanceId: string;
Review Comment:
Small consistency check: neither the Go SDK's `TaskInstance` (`DagID`/`RunID`/`TaskID`/`MapIndex`/`TryNumber`) nor the Java SDK's `Context.TaskInstance` exposes a raw task-instance id — they locate a TI by the dag/run/task/map tuple. Is `taskInstanceId` needed on the public context here (for coordinator XCom addressing, maybe?), or could it drop to stay aligned with the other SDKs? Fine either way — just want it to be an intentional divergence rather than drift.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TypeScript task SDK public interface [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on PR #67908:
URL: https://github.com/apache/airflow/pull/67908#issuecomment-4818587463
@uranusjr can we merge this PR so I can build upon this and add the protocol.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TypeScript task SDK public interface [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on PR #67908:
URL: https://github.com/apache/airflow/pull/67908#issuecomment-4637501869
Addressed feedback from @ashb offline just on the user interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TypeScript task SDK public interface [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on code in PR #67908:
URL: https://github.com/apache/airflow/pull/67908#discussion_r3425397554
##########
ts-sdk/src/registry.ts:
##########
@@ -0,0 +1,90 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import type { TaskHandler } from "./task.js";
+
+/** Identifies the Airflow task handled by a TypeScript function. */
+export interface TaskRegistration {
+ /** Identifier of the Dag containing this task. */
+ dagId: string;
+ /** Airflow task ID, including any TaskGroup prefix. */
+ taskId: string;
Review Comment:
Yes, I have made them read only now. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TypeScript task SDK public interface [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #67908:
URL: https://github.com/apache/airflow/pull/67908#discussion_r3414088221
##########
ts-sdk/src/task.ts:
##########
@@ -0,0 +1,63 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The task-handler call surface — types every user task handler sees.
+
+import type { TaskClient } from "./client.js";
+
+/** Runtime metadata for the current task invocation. */
+export interface TaskContext {
+ /** Identifier of the Dag containing this task. */
+ readonly dagId: string;
+ /** Task ID for this handler invocation, including any TaskGroup prefix. */
+ readonly taskId: string;
+ /** Dag run identifier for the current task attempt. */
+ readonly runId: string;
+ /** Airflow try number for the current task attempt. */
+ readonly tryNumber: number;
+ /** -1 for non-mapped tasks, 0..N-1 for mapped instances. */
+ readonly mapIndex: number;
+ /** Stable Airflow task instance identifier. */
+ readonly taskInstanceId: string;
Review Comment:
Small consistency check: neither the Go SDK's `TaskInstance` (`DagID`/`RunID`/`TaskID`/`MapIndex`/`TryNumber`) nor the Java SDK's `Context.TaskInstance` exposes a raw task-instance id — they locate a TI by the dag/run/task/map tuple. Is `taskInstanceId` needed on the public context here (for coordinator XCom addressing, maybe?), or could it drop to stay aligned with the other SDKs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TypeScript task SDK public interface [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on code in PR #67908:
URL: https://github.com/apache/airflow/pull/67908#discussion_r3478400806
##########
ts-sdk/README.md:
##########
@@ -0,0 +1,137 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# Airflow TypeScript SDK
+
+Public TypeScript interfaces for writing Apache Airflow task handlers.
+
+**Status:** alpha · API will change · Node 22+ · ESM-only
+
+This package defines the user-facing task handler contract: task registration,
+runtime context types, and the `TaskClient` interface used for Airflow
+Variables, Connections, and XCom. Runtime transports implement this interface
+separately.
+
+## Install
+
+```bash
+pnpm add @apache-airflow/ts-sdk
+```
+
+## Task Handlers
+
+```ts
+import { registerTask } from "@apache-airflow/ts-sdk";
+
+registerTask({ dagId: "example_dag", taskId: "say_hello" }, async ({ ctx, client }) => {
Review Comment:
Yes, the current interface should be compatible with future Dag authoring in the TypeScript SDK. My thinking is that registerTask can be the minimal interface for the current Python stub Dag mode: TypeScript binds a handler to an existing Airflow Dag/task, but does not declare the Dag yet.
In the future, we can introduce a Dag-shaped API that uses the same registration primitive underneath. Users could then choose either the explicit registerTask form or the Dag-shaped API, and both would remain compatible.
```
const dag = registerDag("sales_pipeline");
dag.task("extract", extract);
```
##########
ts-sdk/src/sdk/client.ts:
##########
@@ -0,0 +1,88 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import type { ConnectionResult, GetXComOpts, SetXComOpts } from "./client-types.js";
Review Comment:
Good point. I moved the public implementation files under ts-sdk/src/sdk/ and kept ts-sdk/src/index.ts as the package-root re-export, so users still import from @apache-airflow/ts-sdk.
This keeps the user-facing SDK surface separate from future runtime internals. Later coordinator/runtime/bundle code can live under sibling folders such as
```
ts-sdk/src/index.ts # public package entrypoint
ts-sdk/src/sdk/ # public task-authoring API implementation
ts-sdk/src/coordinator/ # coordinator runtime internals
ts-sdk/src/bundle/ # bundle/build tooling internals
ts-sdk/src/generated/ # generated protocol/schema types
```
Let me know if you were thinking something else.
##########
.pre-commit-config.yaml:
##########
@@ -239,6 +239,21 @@ repos:
^pyproject\.toml$
pass_filenames: false
require_serial: true
+ - id: ts-sdk-lint
Review Comment:
Good point. I moved the TS SDK hook implementation under `ts-sdk/scripts/ci/prek/`.
The root `.pre-commit-config.yaml` still registers the `ts-sdk-lint` hook so root-level prek and CI can discover it, but the SDK-specific implementation now lives with the SDK code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TypeScript task SDK public interface [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #67908:
URL: https://github.com/apache/airflow/pull/67908#discussion_r3471504700
##########
.pre-commit-config.yaml:
##########
@@ -239,6 +239,21 @@ repos:
^pyproject\.toml$
pass_filenames: false
require_serial: true
+ - id: ts-sdk-lint
Review Comment:
Could we make this `ts-sdk` directory only instead of having at root level?
##########
ts-sdk/src/client.ts:
##########
@@ -0,0 +1,88 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import type { ConnectionResult, GetXComOpts, SetXComOpts } from "./client-types.js";
Review Comment:
Should we move all the modules under `ts-sdk/src` to `ts-sdk/src/sdk`?
Both Java and Go SDK follow this pattern, that the user-facing modules will be located under `sdk` prefix to not mix the runtime internal with the user interface.
https://github.com/apache/airflow/blob/c5059fe51e51c076b11dbd54be54a66b25c497bb/.agents/skills/airflow-java-sdk/SKILL.md#L33-L43
##########
ts-sdk/README.md:
##########
@@ -0,0 +1,137 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# Airflow TypeScript SDK
+
+Public TypeScript interfaces for writing Apache Airflow task handlers.
+
+**Status:** alpha · API will change · Node 22+ · ESM-only
+
+This package defines the user-facing task handler contract: task registration,
+runtime context types, and the `TaskClient` interface used for Airflow
+Variables, Connections, and XCom. Runtime transports implement this interface
+separately.
+
+## Install
+
+```bash
+pnpm add @apache-airflow/ts-sdk
+```
+
+## Task Handlers
+
+```ts
+import { registerTask } from "@apache-airflow/ts-sdk";
+
+registerTask({ dagId: "example_dag", taskId: "say_hello" }, async ({ ctx, client }) => {
Review Comment:
Having inline `registerTask` with DagId and TaskId makes sense for the 3.3 Multi-Language Dag scope.
However, from the further 3.4 going to the "defining whole Dag in Lang-SDK" feature. Then how will the user interface looks like for the latter one?
Will the `registerTask` still compatible? Or do we need to another new interface for it?
Since both Java and Go SDK went with two steps registration: 1. Define the callable (function / class representing a task, which will support task level args compatible ) 2. Register the Tasks with Dag relationship ( then we're able to support Dag level args at this point)
Would like to learn more about your thoughts on the compatible user interface to support both features. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TypeScript Task SDK [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on PR #67908:
URL: https://github.com/apache/airflow/pull/67908#issuecomment-4609844876
@jason810496 @uranusjr Appreciate your feedback
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TypeScript task SDK public interface [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on code in PR #67908:
URL: https://github.com/apache/airflow/pull/67908#discussion_r3425396619
##########
ts-sdk/src/client-types.ts:
##########
@@ -0,0 +1,68 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/** JSON-compatible value accepted by Airflow for XCom payloads. */
+export type JsonValue =
+ | string
+ | number
+ | boolean
+ | null
+ | JsonValue[]
+ | { [key: string]: JsonValue };
+
+/**
+ * Options for pulling an XCom value.
+ *
+ * `dagId`, `taskId`, and `runId` default to the running task's context.
+ * Pass them only when pulling an XCom value from another task or run.
+ */
+export interface GetXComOpts {
+ key: string;
+ dagId?: string;
+ runId?: string;
+ taskId?: string;
+ mapIndex?: number | null;
+ includePriorDates?: boolean;
+}
+
+/**
+ * Options for pushing an XCom value.
+ *
+ * `dagId`, `taskId`, and `runId` default to the running task's context.
+ */
+export interface SetXComOpts {
+ key: string;
+ value: JsonValue;
+ dagId?: string;
+ runId?: string;
+ taskId?: string;
+ mapIndex?: number | null;
+}
+
+/** Airflow Connection details returned by a task client. */
+export interface ConnectionResult {
+ connId: string;
+ connType: string;
Review Comment:
Done. Aligned with other SDKs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TypeScript task SDK public interface [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #67908:
URL: https://github.com/apache/airflow/pull/67908#discussion_r3478648883
##########
ts-sdk/src/sdk/client.ts:
##########
@@ -0,0 +1,88 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import type { ConnectionResult, GetXComOpts, SetXComOpts } from "./client-types.js";
Review Comment:
Personally I don’t think it’s a big deal either way, as long as the SDK is in its own directory (java-sdk, go-sdk, ts-sdk, etc.), the internal structure just needs to be whatever popular for the ecosystem. I think what you have right now is fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TypeScript task SDK public interface [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr merged PR #67908:
URL: https://github.com/apache/airflow/pull/67908
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TypeScript task SDK public interface [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on PR #67908:
URL: https://github.com/apache/airflow/pull/67908#issuecomment-4725693778
> I think in general this is good. We should add a pre-commit hook to lint ts files in here.
@uranusjr I have added a root prek hook for the TS SDK, following the existing Airflow TS/UI hooks. It runs the SDK’s local lint/format/typecheck scripts and auto-fixes lint/format issues where possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TypeScript task SDK public interface [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on code in PR #67908:
URL: https://github.com/apache/airflow/pull/67908#discussion_r3425395830
##########
ts-sdk/src/task.ts:
##########
@@ -0,0 +1,63 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The task-handler call surface — types every user task handler sees.
+
+import type { TaskClient } from "./client.js";
+
+/** Runtime metadata for the current task invocation. */
+export interface TaskContext {
+ /** Identifier of the Dag containing this task. */
+ readonly dagId: string;
+ /** Task ID for this handler invocation, including any TaskGroup prefix. */
+ readonly taskId: string;
+ /** Dag run identifier for the current task attempt. */
+ readonly runId: string;
+ /** Airflow try number for the current task attempt. */
+ readonly tryNumber: number;
+ /** -1 for non-mapped tasks, 0..N-1 for mapped instances. */
+ readonly mapIndex: number;
+ /** Stable Airflow task instance identifier. */
+ readonly taskInstanceId: string;
Review Comment:
Thanks for pointing this. I have made the changes and removed the taskInstanceId
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TypeScript task SDK public interface [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #67908:
URL: https://github.com/apache/airflow/pull/67908#issuecomment-4628500662
Can you provide an example in this PR? It would make review easier since we’d know what the end picture should be. It doesn’t need to work now, only illustrate the ideas.
Alternatively, ADR documents describing the intentions would also be helpful to understand your intention, and to have something to check against. A small writeup similar to the [_Writing a Non-Python Task_ section in the Java SDK ADR](https://github.com/apache/airflow/blob/main/java-sdk/adr/0001-java-sdk-airflow-integration.md#writing-a-non-python-task) would be enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TypeScript task SDK public interface [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #67908:
URL: https://github.com/apache/airflow/pull/67908#discussion_r3414088225
##########
ts-sdk/src/client-types.ts:
##########
@@ -0,0 +1,68 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/** JSON-compatible value accepted by Airflow for XCom payloads. */
+export type JsonValue =
+ | string
+ | number
+ | boolean
+ | null
+ | JsonValue[]
+ | { [key: string]: JsonValue };
+
+/**
+ * Options for pulling an XCom value.
+ *
+ * `dagId`, `taskId`, and `runId` default to the running task's context.
+ * Pass them only when pulling an XCom value from another task or run.
+ */
+export interface GetXComOpts {
+ key: string;
+ dagId?: string;
+ runId?: string;
+ taskId?: string;
+ mapIndex?: number | null;
+ includePriorDates?: boolean;
+}
+
+/**
+ * Options for pushing an XCom value.
+ *
+ * `dagId`, `taskId`, and `runId` default to the running task's context.
+ */
+export interface SetXComOpts {
+ key: string;
+ value: JsonValue;
+ dagId?: string;
+ runId?: string;
+ taskId?: string;
+ mapIndex?: number | null;
+}
+
+/** Airflow Connection details returned by a task client. */
+export interface ConnectionResult {
+ connId: string;
+ connType: string;
Review Comment:
`connId`/`connType` diverge slightly from the sibling SDKs — Java's `Connection` uses `id`/`type`, Go's uses `ID`/`Type`. The camelCase-of-the-wire-name reads clearly on its own so I don't feel strongly, but aligning to id/type is probably better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TypeScript task SDK public interface [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #67908:
URL: https://github.com/apache/airflow/pull/67908#issuecomment-4712791160
I think in general this is good. We should add a pre-commit hook to lint ts files in here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org