You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jason810496 (via GitHub)" <gi...@apache.org> on 2026/06/16 13:31:45 UTC
[PR] Generate data models from supervisor schema for Go-SDK [airflow]
jason810496 opened a new pull request, #68629:
URL: https://github.com/apache/airflow/pull/68629
- **Not finish yet, just create as draft in case I leave this git worktree somewhere and couldn't find it**
- closes: #68064
## Why
Same as Java-SDK implementation, we should auto generate the data models that will directly be sent to the Coordinator IPC channel from `task-sdk/src/airflow/sdk/execution_time/schema/schema.json`. Actually, all the Lang-SDK should follow the generating data models from supervisor schema pattern to avoid the schema drift.
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes, with help of Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate data models from supervisor schema for Go-SDK [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #68629:
URL: https://github.com/apache/airflow/pull/68629#issuecomment-4862578131
(Still from Claude)
`GetAssetEventByAsset.Ascending` and `GetAssetEventByAssetAlias.Ascending` are `bool` with `omitempty` and schema default `true`. The pointer widening in `pointerizedFields` only covers `builtinIntTypes` — `bool` is not in that set. So `Ascending: false` would be dropped on encode (Go zero for `bool`), and the supervisor would apply its default of `true`, silently reversing the caller's intent.
This is the same bug class as the original `int`+`omitempty` problem, just for a bool field. It's latent — `GetAssetEventByAsset` isn't called from `client.go` yet — but the `pointerizedFields` logic should be extended to cover `bool` fields with `default: true` for the same reason it covers integer fields with non-zero defaults. Or at minimum, the decode-side tests for `Ascending` should be paired with an encode-side test that confirms `false` is preserved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate data models from supervisor schema for Go-SDK [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #68629:
URL: https://github.com/apache/airflow/pull/68629#discussion_r3456960490
##########
go-sdk/pkg/execution/genmodels/models.gen.go:
##########
@@ -0,0 +1,1862 @@
+// Code generated by github.com/atombender/go-jsonschema, DO NOT EDIT.
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+package genmodels
+
+import "time"
+
+// Schema for AssetAliasModel used in AssetEventDagRunReference.
+type AssetAliasReferenceAssetEventDagRun struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+}
+
+// Schema for AssetEvent model used in DagRun.
+type AssetEventDagRunReference struct {
+ // Asset corresponds to the JSON schema field "asset".
+ Asset AssetReferenceAssetEventDagRun `msgpack:"asset"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra Extra `msgpack:"extra"`
+
+ // SourceAliases corresponds to the JSON schema field "source_aliases".
+ SourceAliases []AssetAliasReferenceAssetEventDagRun `msgpack:"source_aliases"`
+
+ // SourceDagID corresponds to the JSON schema field "source_dag_id".
+ SourceDagID interface{} `msgpack:"source_dag_id"`
+
+ // SourceMapIndex corresponds to the JSON schema field "source_map_index".
+ SourceMapIndex interface{} `msgpack:"source_map_index"`
+
+ // SourceRunID corresponds to the JSON schema field "source_run_id".
+ SourceRunID interface{} `msgpack:"source_run_id"`
+
+ // SourceTaskID corresponds to the JSON schema field "source_task_id".
+ SourceTaskID interface{} `msgpack:"source_task_id"`
+
+ // Timestamp corresponds to the JSON schema field "timestamp".
+ Timestamp time.Time `msgpack:"timestamp"`
+}
+
+// Asset event schema with fields that are needed for Runtime.
+type AssetEventResponse struct {
+ // Asset corresponds to the JSON schema field "asset".
+ Asset AssetResponse `msgpack:"asset"`
+
+ // CreatedDagruns corresponds to the JSON schema field "created_dagruns".
+ CreatedDagruns []DagRunAssetReference `msgpack:"created_dagruns"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // ID corresponds to the JSON schema field "id".
+ ID int `msgpack:"id"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // SourceDagID corresponds to the JSON schema field "source_dag_id".
+ SourceDagID interface{} `msgpack:"source_dag_id,omitempty"`
+
+ // SourceMapIndex corresponds to the JSON schema field "source_map_index".
+ SourceMapIndex interface{} `msgpack:"source_map_index,omitempty"`
+
+ // SourceRunID corresponds to the JSON schema field "source_run_id".
+ SourceRunID interface{} `msgpack:"source_run_id,omitempty"`
+
+ // SourceTaskID corresponds to the JSON schema field "source_task_id".
+ SourceTaskID interface{} `msgpack:"source_task_id,omitempty"`
+
+ // Timestamp corresponds to the JSON schema field "timestamp".
+ Timestamp time.Time `msgpack:"timestamp"`
+}
+
+// Response to GetAssetEvent request.
+type AssetEventsResult struct {
+ // AssetEvents corresponds to the JSON schema field "asset_events".
+ AssetEvents []AssetEventResponse `msgpack:"asset_events"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Profile of an asset-like object.
+//
+// Asset will have name, uri defined, with type set to 'Asset'.
+// AssetNameRef will have name defined, type set to 'AssetNameRef'.
+// AssetUriRef will have uri defined, type set to 'AssetUriRef'.
+// AssetAlias will have name defined, type set to 'AssetAlias'.
+//
+// Note that 'type' here is distinct from 'asset_type' the user declares on an
+// Asset (or subclass). This field is for distinguishing between different
+// asset-related types (Asset, AssetRef, or AssetAlias).
+type AssetProfile struct {
+ // Name corresponds to the JSON schema field "name".
+ Name interface{} `msgpack:"name,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI interface{} `msgpack:"uri,omitempty"`
+}
+
+// Schema for AssetModel used in AssetEventDagRunReference.
+type AssetReferenceAssetEventDagRun struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra Extra `msgpack:"extra"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Asset schema for responses with fields that are needed for Runtime.
+type AssetResponse struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // Group corresponds to the JSON schema field "group".
+ Group string `msgpack:"group"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Response to ReadXCom request.
+type AssetResult struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // Group corresponds to the JSON schema field "group".
+ Group string `msgpack:"group"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Response to GetAssetStateStore; wraps the generated API response for supervisor
+// to worker comms.
+type AssetStateStoreResult struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // Value corresponds to the JSON schema field "value".
+ Value JsonValue `msgpack:"value"`
+}
+
+// Response to GetAssetsByAlias; list of concrete assets resolved from an alias.
+type AssetsByAliasResult struct {
+ // Assets corresponds to the JSON schema field "assets".
+ Assets []AssetResult `msgpack:"assets"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type AssignedUsers []HITLUser
+
+// Park a task instance awaiting human input (Human-in-the-loop), without a
+// trigger.
+type AwaitInputTask struct {
+ // NextKwargs corresponds to the JSON schema field "next_kwargs".
+ NextKwargs *NextKwargs `msgpack:"next_kwargs,omitempty"`
+
+ // NextMethod corresponds to the JSON schema field "next_method".
+ NextMethod string `msgpack:"next_method"`
+
+ // RenderedMapIndex corresponds to the JSON schema field "rendered_map_index".
+ RenderedMapIndex interface{} `msgpack:"rendered_map_index,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // Timeout corresponds to the JSON schema field "timeout".
+ Timeout interface{} `msgpack:"timeout,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Schema for telling task which bundle to run with.
+type BundleInfo struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Version corresponds to the JSON schema field "version".
+ Version interface{} `msgpack:"version,omitempty"`
+}
+
+type ClearAssetStateStoreByName struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ClearAssetStateStoreByURI struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type ClearTaskStateStore struct {
+ // AllMapIndices corresponds to the JSON schema field "all_map_indices".
+ AllMapIndices bool `msgpack:"all_map_indices,omitempty"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type Conf map[string]interface{}
+
+// Connection schema for responses with fields that are needed for Runtime.
+type ConnectionResponse struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // ConnType corresponds to the JSON schema field "conn_type".
+ ConnType string `msgpack:"conn_type"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra interface{} `msgpack:"extra"`
+
+ // Host corresponds to the JSON schema field "host".
+ Host interface{} `msgpack:"host"`
+
+ // Login corresponds to the JSON schema field "login".
+ Login interface{} `msgpack:"login"`
+
+ // Password corresponds to the JSON schema field "password".
+ Password interface{} `msgpack:"password"`
+
+ // Port corresponds to the JSON schema field "port".
+ Port interface{} `msgpack:"port"`
+
+ // Schema corresponds to the JSON schema field "schema".
+ Schema interface{} `msgpack:"schema"`
+}
+
+type ConnectionResult struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // ConnType corresponds to the JSON schema field "conn_type".
+ ConnType string `msgpack:"conn_type"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra interface{} `msgpack:"extra,omitempty"`
+
+ // Host corresponds to the JSON schema field "host".
+ Host interface{} `msgpack:"host,omitempty"`
+
+ // Login corresponds to the JSON schema field "login".
+ Login interface{} `msgpack:"login,omitempty"`
+
+ // Password corresponds to the JSON schema field "password".
+ Password interface{} `msgpack:"password,omitempty"`
+
+ // Port corresponds to the JSON schema field "port".
+ Port interface{} `msgpack:"port,omitempty"`
+
+ // Schema corresponds to the JSON schema field "schema".
+ Schema interface{} `msgpack:"schema,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ContextCarrier map[string]interface{}
+
+// Add the input request part of a Human-in-the-loop response.
+type CreateHITLDetailPayload struct {
+ // AssignedUsers corresponds to the JSON schema field "assigned_users".
+ AssignedUsers *AssignedUsers `msgpack:"assigned_users,omitempty"`
+
+ // Body corresponds to the JSON schema field "body".
+ Body interface{} `msgpack:"body,omitempty"`
+
+ // Defaults corresponds to the JSON schema field "defaults".
+ Defaults *Defaults `msgpack:"defaults,omitempty"`
+
+ // Multiple corresponds to the JSON schema field "multiple".
+ Multiple interface{} `msgpack:"multiple,omitempty"`
+
+ // Options corresponds to the JSON schema field "options".
+ Options []string `msgpack:"options"`
+
+ // Params corresponds to the JSON schema field "params".
+ Params *Params `msgpack:"params,omitempty"`
+
+ // Subject corresponds to the JSON schema field "subject".
+ Subject string `msgpack:"subject"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Response containing count of Dag Runs matching certain filters.
+type DRCount struct {
+ // Count corresponds to the JSON schema field "count".
+ Count int `msgpack:"count"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// A Class with information about the success/failure DAG callback to be executed.
+type DagCallbackRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version"`
+
+ // ContextFromServer corresponds to the JSON schema field "context_from_server".
+ ContextFromServer *DagRunContext `msgpack:"context_from_server,omitempty"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Filepath corresponds to the JSON schema field "filepath".
+ Filepath string `msgpack:"filepath"`
+
+ // IsFailureCallback corresponds to the JSON schema field "is_failure_callback".
+ IsFailureCallback interface{} `msgpack:"is_failure_callback,omitempty"`
+
+ // Msg corresponds to the JSON schema field "msg".
+ Msg interface{} `msgpack:"msg,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Request for DAG File Parsing.
+//
+// This is the request that the manager will send to the DAG parser with the dag
+// file and
+// any other necessary metadata.
+type DagFileParseRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundlePath corresponds to the JSON schema field "bundle_path".
+ BundlePath string `msgpack:"bundle_path"`
+
+ // CallbackRequests corresponds to the JSON schema field "callback_requests".
+ CallbackRequests []interface{} `msgpack:"callback_requests,omitempty"`
+
+ // File corresponds to the JSON schema field "file".
+ File string `msgpack:"file"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Result of DAG File Parsing.
+//
+// This is the result of a successful DAG parse, in this class, we gather all
+// serialized DAGs,
+// import errors and warnings to send back to the scheduler to store in the DB.
+type DagFileParsingResult struct {
+ // Fileloc corresponds to the JSON schema field "fileloc".
+ Fileloc string `msgpack:"fileloc"`
+
+ // ImportErrors corresponds to the JSON schema field "import_errors".
+ ImportErrors *ImportErrors `msgpack:"import_errors,omitempty"`
+
+ // SerializedDags corresponds to the JSON schema field "serialized_dags".
+ SerializedDags []LazyDeserializedDAG `msgpack:"serialized_dags"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // Warnings corresponds to the JSON schema field "warnings".
+ Warnings *Warnings `msgpack:"warnings,omitempty"`
+}
+
+type DagResult struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName interface{} `msgpack:"bundle_name,omitempty"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version,omitempty"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // IsPaused corresponds to the JSON schema field "is_paused".
+ IsPaused bool `msgpack:"is_paused"`
+
+ // NextDagrun corresponds to the JSON schema field "next_dagrun".
+ NextDagrun interface{} `msgpack:"next_dagrun,omitempty"`
+
+ // Owners corresponds to the JSON schema field "owners".
+ Owners interface{} `msgpack:"owners,omitempty"`
+
+ // RelativeFileloc corresponds to the JSON schema field "relative_fileloc".
+ RelativeFileloc interface{} `msgpack:"relative_fileloc,omitempty"`
+
+ // Tags corresponds to the JSON schema field "tags".
+ Tags []string `msgpack:"tags"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Schema for DagRun model with minimal required fields needed for Runtime.
+type DagRun struct {
+ // ClearNumber corresponds to the JSON schema field "clear_number".
+ ClearNumber int `msgpack:"clear_number,omitempty"`
+
+ // Conf corresponds to the JSON schema field "conf".
+ Conf *Conf `msgpack:"conf,omitempty"`
+
+ // ConsumedAssetEvents corresponds to the JSON schema field
+ // "consumed_asset_events".
+ ConsumedAssetEvents []AssetEventDagRunReference `msgpack:"consumed_asset_events"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date"`
+
+ // Note corresponds to the JSON schema field "note".
+ Note interface{} `msgpack:"note,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key"`
+
+ // RunAfter corresponds to the JSON schema field "run_after".
+ RunAfter time.Time `msgpack:"run_after"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // RunType corresponds to the JSON schema field "run_type".
+ RunType DagRunType `msgpack:"run_type"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate interface{} `msgpack:"start_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // TeamName corresponds to the JSON schema field "team_name".
+ TeamName interface{} `msgpack:"team_name,omitempty"`
+
+ // TriggeringUserName corresponds to the JSON schema field "triggering_user_name".
+ TriggeringUserName interface{} `msgpack:"triggering_user_name,omitempty"`
+}
+
+// DagRun serializer for asset responses.
+type DagRunAssetReference struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end,omitempty"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start,omitempty"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date,omitempty"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate time.Time `msgpack:"start_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State string `msgpack:"state"`
+}
+
+// Class to pass context info from the server to build a Execution context object.
+type DagRunContext struct {
+ // DagRun corresponds to the JSON schema field "dag_run".
+ DagRun *DagRun `msgpack:"dag_run,omitempty"`
+
+ // LastTI corresponds to the JSON schema field "last_ti".
+ LastTI *TaskInstance `msgpack:"last_ti,omitempty"`
+}
+
+type DagRunResult struct {
+ // ClearNumber corresponds to the JSON schema field "clear_number".
+ ClearNumber interface{} `msgpack:"clear_number,omitempty"`
+
+ // Conf corresponds to the JSON schema field "conf".
+ Conf *Conf `msgpack:"conf,omitempty"`
+
+ // ConsumedAssetEvents corresponds to the JSON schema field
+ // "consumed_asset_events".
+ ConsumedAssetEvents []AssetEventDagRunReference `msgpack:"consumed_asset_events"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end,omitempty"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start,omitempty"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date,omitempty"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // Note corresponds to the JSON schema field "note".
+ Note interface{} `msgpack:"note,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // RunAfter corresponds to the JSON schema field "run_after".
+ RunAfter time.Time `msgpack:"run_after"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // RunType corresponds to the JSON schema field "run_type".
+ RunType DagRunType `msgpack:"run_type"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate interface{} `msgpack:"start_date,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // TeamName corresponds to the JSON schema field "team_name".
+ TeamName interface{} `msgpack:"team_name,omitempty"`
+
+ // TriggeringUserName corresponds to the JSON schema field "triggering_user_name".
+ TriggeringUserName interface{} `msgpack:"triggering_user_name,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DagRunState string
+
+const DagRunStateFailed DagRunState = "failed"
+const DagRunStateQueued DagRunState = "queued"
+
+type DagRunStateResult struct {
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+const DagRunStateRunning DagRunState = "running"
+const DagRunStateSuccess DagRunState = "success"
+
+type DagRunType string
+
+const DagRunTypeAssetMaterialization DagRunType = "asset_materialization"
+const DagRunTypeAssetTriggered DagRunType = "asset_triggered"
+const DagRunTypeBackfill DagRunType = "backfill"
+const DagRunTypeManual DagRunType = "manual"
+const DagRunTypeOperatorTriggered DagRunType = "operator_triggered"
+const DagRunTypeScheduled DagRunType = "scheduled"
+
+type Data map[string]interface{}
+
+type Defaults []string
+
+// Update a task instance state to deferred.
+type DeferTask struct {
+ // Classpath corresponds to the JSON schema field "classpath".
+ Classpath string `msgpack:"classpath"`
+
+ // NextKwargs corresponds to the JSON schema field "next_kwargs".
+ NextKwargs *NextKwargs `msgpack:"next_kwargs,omitempty"`
+
+ // NextMethod corresponds to the JSON schema field "next_method".
+ NextMethod string `msgpack:"next_method"`
+
+ // Queue corresponds to the JSON schema field "queue".
+ Queue interface{} `msgpack:"queue,omitempty"`
+
+ // RenderedMapIndex corresponds to the JSON schema field "rendered_map_index".
+ RenderedMapIndex interface{} `msgpack:"rendered_map_index,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // TriggerKwargs corresponds to the JSON schema field "trigger_kwargs".
+ TriggerKwargs *TriggerKwargs `msgpack:"trigger_kwargs,omitempty"`
+
+ // TriggerTimeout corresponds to the JSON schema field "trigger_timeout".
+ TriggerTimeout interface{} `msgpack:"trigger_timeout,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteAssetStateStoreByName struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteAssetStateStoreByURI struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type DeleteTaskStateStore struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteVariable struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteXCom struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // MapIndex corresponds to the JSON schema field "map_index".
+ MapIndex interface{} `msgpack:"map_index,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // TaskID corresponds to the JSON schema field "task_id".
+ TaskID string `msgpack:"task_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type Detail map[string]interface{}
+
+// Email notification request for task failures/retries.
+type EmailRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version"`
+
+ // ContextFromServer corresponds to the JSON schema field "context_from_server".
+ ContextFromServer TIRunContext `msgpack:"context_from_server"`
+
+ // EmailType corresponds to the JSON schema field "email_type".
+ EmailType EmailRequestEmailType `msgpack:"email_type,omitempty"`
+
+ // Filepath corresponds to the JSON schema field "filepath".
+ Filepath string `msgpack:"filepath"`
+
+ // Msg corresponds to the JSON schema field "msg".
+ Msg interface{} `msgpack:"msg,omitempty"`
+
+ // TI corresponds to the JSON schema field "ti".
+ TI TaskInstance `msgpack:"ti"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type EmailRequestEmailType string
+
+const EmailRequestEmailTypeFailure EmailRequestEmailType = "failure"
+const EmailRequestEmailTypeRetry EmailRequestEmailType = "retry"
+
+type ErrorResponse struct {
+ // Detail corresponds to the JSON schema field "detail".
+ Detail *Detail `msgpack:"detail,omitempty"`
+
+ // Error corresponds to the JSON schema field "error".
+ Error ErrorType `msgpack:"error,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ErrorType string
+
+const ErrorTypeAPISERVERERROR ErrorType = "API_SERVER_ERROR"
+const ErrorTypeASSETNOTFOUND ErrorType = "ASSET_NOT_FOUND"
+const ErrorTypeASSETSTORENOTFOUND ErrorType = "ASSET_STORE_NOT_FOUND"
+const ErrorTypeCONNECTIONNOTFOUND ErrorType = "CONNECTION_NOT_FOUND"
+const ErrorTypeDAGRUNALREADYEXISTS ErrorType = "DAGRUN_ALREADY_EXISTS"
+const ErrorTypeGENERICERROR ErrorType = "GENERIC_ERROR"
+const ErrorTypePERMISSIONDENIED ErrorType = "PERMISSION_DENIED"
+const ErrorTypeTASKSTORENOTFOUND ErrorType = "TASK_STORE_NOT_FOUND"
+const ErrorTypeVARIABLENOTFOUND ErrorType = "VARIABLE_NOT_FOUND"
+const ErrorTypeXCOMNOTFOUND ErrorType = "XCOM_NOT_FOUND"
+
+type Extra map[string]JsonValue
+
+type GetAssetByName struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetByURI struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type GetAssetEventByAsset struct {
+ // After corresponds to the JSON schema field "after".
+ After interface{} `msgpack:"after,omitempty"`
+
+ // Ascending corresponds to the JSON schema field "ascending".
+ Ascending bool `msgpack:"ascending,omitempty"`
+
+ // Before corresponds to the JSON schema field "before".
+ Before interface{} `msgpack:"before,omitempty"`
+
+ // Limit corresponds to the JSON schema field "limit".
+ Limit interface{} `msgpack:"limit,omitempty"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name interface{} `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI interface{} `msgpack:"uri"`
+}
+
+type GetAssetEventByAssetAlias struct {
+ // After corresponds to the JSON schema field "after".
+ After interface{} `msgpack:"after,omitempty"`
+
+ // AliasName corresponds to the JSON schema field "alias_name".
+ AliasName string `msgpack:"alias_name"`
+
+ // Ascending corresponds to the JSON schema field "ascending".
+ Ascending bool `msgpack:"ascending,omitempty"`
+
+ // Before corresponds to the JSON schema field "before".
+ Before interface{} `msgpack:"before,omitempty"`
+
+ // Limit corresponds to the JSON schema field "limit".
+ Limit interface{} `msgpack:"limit,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetStateStoreByName struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetStateStoreByURI struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type GetAssetsByAlias struct {
+ // AliasName corresponds to the JSON schema field "alias_name".
+ AliasName string `msgpack:"alias_name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetConnection struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDRCount struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDates corresponds to the JSON schema field "logical_dates".
+ LogicalDates *LogicalDates `msgpack:"logical_dates,omitempty"`
+
+ // RunIds corresponds to the JSON schema field "run_ids".
+ RunIds *RunIds `msgpack:"run_ids,omitempty"`
+
+ // States corresponds to the JSON schema field "states".
+ States *States `msgpack:"states,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDag struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDagRun struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDagRunState struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Get the response content part of a Human-in-the-loop response.
+type GetHITLDetailResponse struct {
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetPrevSuccessfulDagRun struct {
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetPreviousDagRun struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate time.Time `msgpack:"logical_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Request to get previous task instance.
+type GetPreviousTI struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // MapIndex corresponds to the JSON schema field "map_index".
+ MapIndex int `msgpack:"map_index,omitempty"`
Review Comment:
I think this is too brittle to my liking since you need to add a special case for each such field. If the supervisor adds a similar field declaration, it is too easy to miss adding an additional case to handle it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate data models from supervisor schema for Go-SDK [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #68629:
URL: https://github.com/apache/airflow/pull/68629#discussion_r3456847967
##########
go-sdk/Justfile:
##########
Review Comment:
How do you envision the hook should look like? I don’t think the hook should be tied to the in-tree supervisor schema since it would require whoever updates the schema also have the skill to resolve Go changes, which is a big ask. Schema upgrades in the SDK should be manual and separate from the supervisor side instead.
After we release 3.3, I want to publish the supervisor schema to somewhere on the internet with the version in the name (for example `https://gist.githubusercontent.com/.../2026-06-23.json`). When you want to upgrade the schema in the SDK, you bump the schema version number, and the pre-commit hook should keep this version bump (in message.go) and the generated source in sync.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate data models from supervisor schema for Go-SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on PR #68629:
URL: https://github.com/apache/airflow/pull/68629#issuecomment-4862608277
> GetAssetEventByAsset.Ascending and GetAssetEventByAssetAlias.Ascending are bool with omitempty and schema default true. The pointer widening in pointerizedFields only covers builtinIntTypes — bool is not in that set. So Ascending: false would be dropped on encode (Go zero for bool), and the supervisor would apply its default of true, silently reversing the caller's intent.
Yes, good point. I only considered the `int`, `null`, `-1` semantic issue in last round but didn't considered `bool`. I will verify all primitive type myself then mark as ready to review again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate data models from supervisor schema for Go-SDK [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #68629:
URL: https://github.com/apache/airflow/pull/68629#discussion_r3456816347
##########
go-sdk/Justfile:
##########
Review Comment:
There should be instructions somewhere to describe what needs to be done when a contributor wants to upgrade the schema version used by the Go SDK.
##########
go-sdk/pkg/execution/genmodels/models.gen.go:
##########
@@ -0,0 +1,1862 @@
+// Code generated by github.com/atombender/go-jsonschema, DO NOT EDIT.
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+package genmodels
+
+import "time"
+
+// Schema for AssetAliasModel used in AssetEventDagRunReference.
+type AssetAliasReferenceAssetEventDagRun struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+}
+
+// Schema for AssetEvent model used in DagRun.
+type AssetEventDagRunReference struct {
+ // Asset corresponds to the JSON schema field "asset".
+ Asset AssetReferenceAssetEventDagRun `msgpack:"asset"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra Extra `msgpack:"extra"`
+
+ // SourceAliases corresponds to the JSON schema field "source_aliases".
+ SourceAliases []AssetAliasReferenceAssetEventDagRun `msgpack:"source_aliases"`
+
+ // SourceDagID corresponds to the JSON schema field "source_dag_id".
+ SourceDagID interface{} `msgpack:"source_dag_id"`
+
+ // SourceMapIndex corresponds to the JSON schema field "source_map_index".
+ SourceMapIndex interface{} `msgpack:"source_map_index"`
+
+ // SourceRunID corresponds to the JSON schema field "source_run_id".
+ SourceRunID interface{} `msgpack:"source_run_id"`
+
+ // SourceTaskID corresponds to the JSON schema field "source_task_id".
+ SourceTaskID interface{} `msgpack:"source_task_id"`
+
+ // Timestamp corresponds to the JSON schema field "timestamp".
+ Timestamp time.Time `msgpack:"timestamp"`
+}
+
+// Asset event schema with fields that are needed for Runtime.
+type AssetEventResponse struct {
+ // Asset corresponds to the JSON schema field "asset".
+ Asset AssetResponse `msgpack:"asset"`
+
+ // CreatedDagruns corresponds to the JSON schema field "created_dagruns".
+ CreatedDagruns []DagRunAssetReference `msgpack:"created_dagruns"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // ID corresponds to the JSON schema field "id".
+ ID int `msgpack:"id"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // SourceDagID corresponds to the JSON schema field "source_dag_id".
+ SourceDagID interface{} `msgpack:"source_dag_id,omitempty"`
+
+ // SourceMapIndex corresponds to the JSON schema field "source_map_index".
+ SourceMapIndex interface{} `msgpack:"source_map_index,omitempty"`
+
+ // SourceRunID corresponds to the JSON schema field "source_run_id".
+ SourceRunID interface{} `msgpack:"source_run_id,omitempty"`
+
+ // SourceTaskID corresponds to the JSON schema field "source_task_id".
+ SourceTaskID interface{} `msgpack:"source_task_id,omitempty"`
+
+ // Timestamp corresponds to the JSON schema field "timestamp".
+ Timestamp time.Time `msgpack:"timestamp"`
+}
+
+// Response to GetAssetEvent request.
+type AssetEventsResult struct {
+ // AssetEvents corresponds to the JSON schema field "asset_events".
+ AssetEvents []AssetEventResponse `msgpack:"asset_events"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Profile of an asset-like object.
+//
+// Asset will have name, uri defined, with type set to 'Asset'.
+// AssetNameRef will have name defined, type set to 'AssetNameRef'.
+// AssetUriRef will have uri defined, type set to 'AssetUriRef'.
+// AssetAlias will have name defined, type set to 'AssetAlias'.
+//
+// Note that 'type' here is distinct from 'asset_type' the user declares on an
+// Asset (or subclass). This field is for distinguishing between different
+// asset-related types (Asset, AssetRef, or AssetAlias).
+type AssetProfile struct {
+ // Name corresponds to the JSON schema field "name".
+ Name interface{} `msgpack:"name,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI interface{} `msgpack:"uri,omitempty"`
+}
+
+// Schema for AssetModel used in AssetEventDagRunReference.
+type AssetReferenceAssetEventDagRun struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra Extra `msgpack:"extra"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Asset schema for responses with fields that are needed for Runtime.
+type AssetResponse struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // Group corresponds to the JSON schema field "group".
+ Group string `msgpack:"group"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Response to ReadXCom request.
+type AssetResult struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // Group corresponds to the JSON schema field "group".
+ Group string `msgpack:"group"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Response to GetAssetStateStore; wraps the generated API response for supervisor
+// to worker comms.
+type AssetStateStoreResult struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // Value corresponds to the JSON schema field "value".
+ Value JsonValue `msgpack:"value"`
+}
+
+// Response to GetAssetsByAlias; list of concrete assets resolved from an alias.
+type AssetsByAliasResult struct {
+ // Assets corresponds to the JSON schema field "assets".
+ Assets []AssetResult `msgpack:"assets"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type AssignedUsers []HITLUser
+
+// Park a task instance awaiting human input (Human-in-the-loop), without a
+// trigger.
+type AwaitInputTask struct {
+ // NextKwargs corresponds to the JSON schema field "next_kwargs".
+ NextKwargs *NextKwargs `msgpack:"next_kwargs,omitempty"`
+
+ // NextMethod corresponds to the JSON schema field "next_method".
+ NextMethod string `msgpack:"next_method"`
+
+ // RenderedMapIndex corresponds to the JSON schema field "rendered_map_index".
+ RenderedMapIndex interface{} `msgpack:"rendered_map_index,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // Timeout corresponds to the JSON schema field "timeout".
+ Timeout interface{} `msgpack:"timeout,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Schema for telling task which bundle to run with.
+type BundleInfo struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Version corresponds to the JSON schema field "version".
+ Version interface{} `msgpack:"version,omitempty"`
+}
+
+type ClearAssetStateStoreByName struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ClearAssetStateStoreByURI struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type ClearTaskStateStore struct {
+ // AllMapIndices corresponds to the JSON schema field "all_map_indices".
+ AllMapIndices bool `msgpack:"all_map_indices,omitempty"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type Conf map[string]interface{}
+
+// Connection schema for responses with fields that are needed for Runtime.
+type ConnectionResponse struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // ConnType corresponds to the JSON schema field "conn_type".
+ ConnType string `msgpack:"conn_type"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra interface{} `msgpack:"extra"`
+
+ // Host corresponds to the JSON schema field "host".
+ Host interface{} `msgpack:"host"`
+
+ // Login corresponds to the JSON schema field "login".
+ Login interface{} `msgpack:"login"`
+
+ // Password corresponds to the JSON schema field "password".
+ Password interface{} `msgpack:"password"`
+
+ // Port corresponds to the JSON schema field "port".
+ Port interface{} `msgpack:"port"`
+
+ // Schema corresponds to the JSON schema field "schema".
+ Schema interface{} `msgpack:"schema"`
+}
+
+type ConnectionResult struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // ConnType corresponds to the JSON schema field "conn_type".
+ ConnType string `msgpack:"conn_type"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra interface{} `msgpack:"extra,omitempty"`
+
+ // Host corresponds to the JSON schema field "host".
+ Host interface{} `msgpack:"host,omitempty"`
+
+ // Login corresponds to the JSON schema field "login".
+ Login interface{} `msgpack:"login,omitempty"`
+
+ // Password corresponds to the JSON schema field "password".
+ Password interface{} `msgpack:"password,omitempty"`
+
+ // Port corresponds to the JSON schema field "port".
+ Port interface{} `msgpack:"port,omitempty"`
+
+ // Schema corresponds to the JSON schema field "schema".
+ Schema interface{} `msgpack:"schema,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ContextCarrier map[string]interface{}
+
+// Add the input request part of a Human-in-the-loop response.
+type CreateHITLDetailPayload struct {
+ // AssignedUsers corresponds to the JSON schema field "assigned_users".
+ AssignedUsers *AssignedUsers `msgpack:"assigned_users,omitempty"`
+
+ // Body corresponds to the JSON schema field "body".
+ Body interface{} `msgpack:"body,omitempty"`
+
+ // Defaults corresponds to the JSON schema field "defaults".
+ Defaults *Defaults `msgpack:"defaults,omitempty"`
+
+ // Multiple corresponds to the JSON schema field "multiple".
+ Multiple interface{} `msgpack:"multiple,omitempty"`
+
+ // Options corresponds to the JSON schema field "options".
+ Options []string `msgpack:"options"`
+
+ // Params corresponds to the JSON schema field "params".
+ Params *Params `msgpack:"params,omitempty"`
+
+ // Subject corresponds to the JSON schema field "subject".
+ Subject string `msgpack:"subject"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Response containing count of Dag Runs matching certain filters.
+type DRCount struct {
+ // Count corresponds to the JSON schema field "count".
+ Count int `msgpack:"count"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// A Class with information about the success/failure DAG callback to be executed.
+type DagCallbackRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version"`
+
+ // ContextFromServer corresponds to the JSON schema field "context_from_server".
+ ContextFromServer *DagRunContext `msgpack:"context_from_server,omitempty"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Filepath corresponds to the JSON schema field "filepath".
+ Filepath string `msgpack:"filepath"`
+
+ // IsFailureCallback corresponds to the JSON schema field "is_failure_callback".
+ IsFailureCallback interface{} `msgpack:"is_failure_callback,omitempty"`
+
+ // Msg corresponds to the JSON schema field "msg".
+ Msg interface{} `msgpack:"msg,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Request for DAG File Parsing.
+//
+// This is the request that the manager will send to the DAG parser with the dag
+// file and
+// any other necessary metadata.
+type DagFileParseRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundlePath corresponds to the JSON schema field "bundle_path".
+ BundlePath string `msgpack:"bundle_path"`
+
+ // CallbackRequests corresponds to the JSON schema field "callback_requests".
+ CallbackRequests []interface{} `msgpack:"callback_requests,omitempty"`
+
+ // File corresponds to the JSON schema field "file".
+ File string `msgpack:"file"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Result of DAG File Parsing.
+//
+// This is the result of a successful DAG parse, in this class, we gather all
+// serialized DAGs,
+// import errors and warnings to send back to the scheduler to store in the DB.
+type DagFileParsingResult struct {
+ // Fileloc corresponds to the JSON schema field "fileloc".
+ Fileloc string `msgpack:"fileloc"`
+
+ // ImportErrors corresponds to the JSON schema field "import_errors".
+ ImportErrors *ImportErrors `msgpack:"import_errors,omitempty"`
+
+ // SerializedDags corresponds to the JSON schema field "serialized_dags".
+ SerializedDags []LazyDeserializedDAG `msgpack:"serialized_dags"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // Warnings corresponds to the JSON schema field "warnings".
+ Warnings *Warnings `msgpack:"warnings,omitempty"`
+}
+
+type DagResult struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName interface{} `msgpack:"bundle_name,omitempty"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version,omitempty"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // IsPaused corresponds to the JSON schema field "is_paused".
+ IsPaused bool `msgpack:"is_paused"`
+
+ // NextDagrun corresponds to the JSON schema field "next_dagrun".
+ NextDagrun interface{} `msgpack:"next_dagrun,omitempty"`
+
+ // Owners corresponds to the JSON schema field "owners".
+ Owners interface{} `msgpack:"owners,omitempty"`
+
+ // RelativeFileloc corresponds to the JSON schema field "relative_fileloc".
+ RelativeFileloc interface{} `msgpack:"relative_fileloc,omitempty"`
+
+ // Tags corresponds to the JSON schema field "tags".
+ Tags []string `msgpack:"tags"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Schema for DagRun model with minimal required fields needed for Runtime.
+type DagRun struct {
+ // ClearNumber corresponds to the JSON schema field "clear_number".
+ ClearNumber int `msgpack:"clear_number,omitempty"`
+
+ // Conf corresponds to the JSON schema field "conf".
+ Conf *Conf `msgpack:"conf,omitempty"`
+
+ // ConsumedAssetEvents corresponds to the JSON schema field
+ // "consumed_asset_events".
+ ConsumedAssetEvents []AssetEventDagRunReference `msgpack:"consumed_asset_events"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date"`
+
+ // Note corresponds to the JSON schema field "note".
+ Note interface{} `msgpack:"note,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key"`
+
+ // RunAfter corresponds to the JSON schema field "run_after".
+ RunAfter time.Time `msgpack:"run_after"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // RunType corresponds to the JSON schema field "run_type".
+ RunType DagRunType `msgpack:"run_type"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate interface{} `msgpack:"start_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // TeamName corresponds to the JSON schema field "team_name".
+ TeamName interface{} `msgpack:"team_name,omitempty"`
+
+ // TriggeringUserName corresponds to the JSON schema field "triggering_user_name".
+ TriggeringUserName interface{} `msgpack:"triggering_user_name,omitempty"`
+}
+
+// DagRun serializer for asset responses.
+type DagRunAssetReference struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end,omitempty"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start,omitempty"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date,omitempty"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate time.Time `msgpack:"start_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State string `msgpack:"state"`
+}
+
+// Class to pass context info from the server to build a Execution context object.
+type DagRunContext struct {
+ // DagRun corresponds to the JSON schema field "dag_run".
+ DagRun *DagRun `msgpack:"dag_run,omitempty"`
+
+ // LastTI corresponds to the JSON schema field "last_ti".
+ LastTI *TaskInstance `msgpack:"last_ti,omitempty"`
+}
+
+type DagRunResult struct {
+ // ClearNumber corresponds to the JSON schema field "clear_number".
+ ClearNumber interface{} `msgpack:"clear_number,omitempty"`
+
+ // Conf corresponds to the JSON schema field "conf".
+ Conf *Conf `msgpack:"conf,omitempty"`
+
+ // ConsumedAssetEvents corresponds to the JSON schema field
+ // "consumed_asset_events".
+ ConsumedAssetEvents []AssetEventDagRunReference `msgpack:"consumed_asset_events"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end,omitempty"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start,omitempty"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date,omitempty"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // Note corresponds to the JSON schema field "note".
+ Note interface{} `msgpack:"note,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // RunAfter corresponds to the JSON schema field "run_after".
+ RunAfter time.Time `msgpack:"run_after"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // RunType corresponds to the JSON schema field "run_type".
+ RunType DagRunType `msgpack:"run_type"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate interface{} `msgpack:"start_date,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // TeamName corresponds to the JSON schema field "team_name".
+ TeamName interface{} `msgpack:"team_name,omitempty"`
+
+ // TriggeringUserName corresponds to the JSON schema field "triggering_user_name".
+ TriggeringUserName interface{} `msgpack:"triggering_user_name,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DagRunState string
+
+const DagRunStateFailed DagRunState = "failed"
+const DagRunStateQueued DagRunState = "queued"
+
+type DagRunStateResult struct {
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+const DagRunStateRunning DagRunState = "running"
+const DagRunStateSuccess DagRunState = "success"
+
+type DagRunType string
+
+const DagRunTypeAssetMaterialization DagRunType = "asset_materialization"
+const DagRunTypeAssetTriggered DagRunType = "asset_triggered"
+const DagRunTypeBackfill DagRunType = "backfill"
+const DagRunTypeManual DagRunType = "manual"
+const DagRunTypeOperatorTriggered DagRunType = "operator_triggered"
+const DagRunTypeScheduled DagRunType = "scheduled"
+
+type Data map[string]interface{}
+
+type Defaults []string
+
+// Update a task instance state to deferred.
+type DeferTask struct {
+ // Classpath corresponds to the JSON schema field "classpath".
+ Classpath string `msgpack:"classpath"`
+
+ // NextKwargs corresponds to the JSON schema field "next_kwargs".
+ NextKwargs *NextKwargs `msgpack:"next_kwargs,omitempty"`
+
+ // NextMethod corresponds to the JSON schema field "next_method".
+ NextMethod string `msgpack:"next_method"`
+
+ // Queue corresponds to the JSON schema field "queue".
+ Queue interface{} `msgpack:"queue,omitempty"`
+
+ // RenderedMapIndex corresponds to the JSON schema field "rendered_map_index".
+ RenderedMapIndex interface{} `msgpack:"rendered_map_index,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // TriggerKwargs corresponds to the JSON schema field "trigger_kwargs".
+ TriggerKwargs *TriggerKwargs `msgpack:"trigger_kwargs,omitempty"`
+
+ // TriggerTimeout corresponds to the JSON schema field "trigger_timeout".
+ TriggerTimeout interface{} `msgpack:"trigger_timeout,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteAssetStateStoreByName struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteAssetStateStoreByURI struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type DeleteTaskStateStore struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteVariable struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteXCom struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // MapIndex corresponds to the JSON schema field "map_index".
+ MapIndex interface{} `msgpack:"map_index,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // TaskID corresponds to the JSON schema field "task_id".
+ TaskID string `msgpack:"task_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type Detail map[string]interface{}
+
+// Email notification request for task failures/retries.
+type EmailRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version"`
+
+ // ContextFromServer corresponds to the JSON schema field "context_from_server".
+ ContextFromServer TIRunContext `msgpack:"context_from_server"`
+
+ // EmailType corresponds to the JSON schema field "email_type".
+ EmailType EmailRequestEmailType `msgpack:"email_type,omitempty"`
+
+ // Filepath corresponds to the JSON schema field "filepath".
+ Filepath string `msgpack:"filepath"`
+
+ // Msg corresponds to the JSON schema field "msg".
+ Msg interface{} `msgpack:"msg,omitempty"`
+
+ // TI corresponds to the JSON schema field "ti".
+ TI TaskInstance `msgpack:"ti"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type EmailRequestEmailType string
+
+const EmailRequestEmailTypeFailure EmailRequestEmailType = "failure"
+const EmailRequestEmailTypeRetry EmailRequestEmailType = "retry"
+
+type ErrorResponse struct {
+ // Detail corresponds to the JSON schema field "detail".
+ Detail *Detail `msgpack:"detail,omitempty"`
+
+ // Error corresponds to the JSON schema field "error".
+ Error ErrorType `msgpack:"error,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ErrorType string
+
+const ErrorTypeAPISERVERERROR ErrorType = "API_SERVER_ERROR"
+const ErrorTypeASSETNOTFOUND ErrorType = "ASSET_NOT_FOUND"
+const ErrorTypeASSETSTORENOTFOUND ErrorType = "ASSET_STORE_NOT_FOUND"
+const ErrorTypeCONNECTIONNOTFOUND ErrorType = "CONNECTION_NOT_FOUND"
+const ErrorTypeDAGRUNALREADYEXISTS ErrorType = "DAGRUN_ALREADY_EXISTS"
+const ErrorTypeGENERICERROR ErrorType = "GENERIC_ERROR"
+const ErrorTypePERMISSIONDENIED ErrorType = "PERMISSION_DENIED"
+const ErrorTypeTASKSTORENOTFOUND ErrorType = "TASK_STORE_NOT_FOUND"
+const ErrorTypeVARIABLENOTFOUND ErrorType = "VARIABLE_NOT_FOUND"
+const ErrorTypeXCOMNOTFOUND ErrorType = "XCOM_NOT_FOUND"
+
+type Extra map[string]JsonValue
+
+type GetAssetByName struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetByURI struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type GetAssetEventByAsset struct {
+ // After corresponds to the JSON schema field "after".
+ After interface{} `msgpack:"after,omitempty"`
+
+ // Ascending corresponds to the JSON schema field "ascending".
+ Ascending bool `msgpack:"ascending,omitempty"`
+
+ // Before corresponds to the JSON schema field "before".
+ Before interface{} `msgpack:"before,omitempty"`
+
+ // Limit corresponds to the JSON schema field "limit".
+ Limit interface{} `msgpack:"limit,omitempty"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name interface{} `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI interface{} `msgpack:"uri"`
+}
+
+type GetAssetEventByAssetAlias struct {
+ // After corresponds to the JSON schema field "after".
+ After interface{} `msgpack:"after,omitempty"`
+
+ // AliasName corresponds to the JSON schema field "alias_name".
+ AliasName string `msgpack:"alias_name"`
+
+ // Ascending corresponds to the JSON schema field "ascending".
+ Ascending bool `msgpack:"ascending,omitempty"`
+
+ // Before corresponds to the JSON schema field "before".
+ Before interface{} `msgpack:"before,omitempty"`
+
+ // Limit corresponds to the JSON schema field "limit".
+ Limit interface{} `msgpack:"limit,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetStateStoreByName struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetStateStoreByURI struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type GetAssetsByAlias struct {
+ // AliasName corresponds to the JSON schema field "alias_name".
+ AliasName string `msgpack:"alias_name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetConnection struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDRCount struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDates corresponds to the JSON schema field "logical_dates".
+ LogicalDates *LogicalDates `msgpack:"logical_dates,omitempty"`
+
+ // RunIds corresponds to the JSON schema field "run_ids".
+ RunIds *RunIds `msgpack:"run_ids,omitempty"`
+
+ // States corresponds to the JSON schema field "states".
+ States *States `msgpack:"states,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDag struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDagRun struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDagRunState struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Get the response content part of a Human-in-the-loop response.
+type GetHITLDetailResponse struct {
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetPrevSuccessfulDagRun struct {
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetPreviousDagRun struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate time.Time `msgpack:"logical_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Request to get previous task instance.
+type GetPreviousTI struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // MapIndex corresponds to the JSON schema field "map_index".
+ MapIndex int `msgpack:"map_index,omitempty"`
Review Comment:
`GetPreviousTI.MapIndex` is generated as `int` with `omitempty` because the schema defines it as `{type: integer, default: -1}` — a non-nullable integer rather than `anyOf: [integer, null]`. This causes a silent bug: assigning `MapIndex = 0` produces a zero-value `int`, which `omitempty` drops from the payload, so the supervisor receives its default of `-1` and returns the unmapped task instance instead of index 0.
The other `map_index` fields (`GetXCom`, `SetXCom`, `GetTICount`, etc.) are nullable in the schema and so generate `interface{}` with `omitempty`, which avoids this because `omitempty` on `interface{}` only omits on nil — but `interface{}` is still weaker than it needs to be.
I'd suggest fixing this uniformly in `gen/main.go`: add a post-processing pass that rewrites any non-required integer field — whether it comes from a plain `{type: integer}` or an `anyOf: [integer, null]` — to `*int` with `omitempty`. The semantics are consistent in both cases: nil means "absent, let the server apply its default", non-nil means "send this exact value, including 0". This eliminates the `interface{}` fields for integers (and the `ifaceInt()` helper calls they require at usage sites), fixes the `GetPreviousTI` bug, and gives callers a type-safe representation that doesn't require knowing the server's sentinel value.
A test case for `map_index = 0` in `TestCoordinatorClientPushXComMapIndex` (and an equivalent for `GetXCom`) would also be worth adding to nail down the boundary that currently has no coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate data models from supervisor schema for Go-SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #68629:
URL: https://github.com/apache/airflow/pull/68629#discussion_r3457904439
##########
go-sdk/pkg/execution/genmodels/models.gen.go:
##########
@@ -0,0 +1,1862 @@
+// Code generated by github.com/atombender/go-jsonschema, DO NOT EDIT.
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+package genmodels
+
+import "time"
+
+// Schema for AssetAliasModel used in AssetEventDagRunReference.
+type AssetAliasReferenceAssetEventDagRun struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+}
+
+// Schema for AssetEvent model used in DagRun.
+type AssetEventDagRunReference struct {
+ // Asset corresponds to the JSON schema field "asset".
+ Asset AssetReferenceAssetEventDagRun `msgpack:"asset"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra Extra `msgpack:"extra"`
+
+ // SourceAliases corresponds to the JSON schema field "source_aliases".
+ SourceAliases []AssetAliasReferenceAssetEventDagRun `msgpack:"source_aliases"`
+
+ // SourceDagID corresponds to the JSON schema field "source_dag_id".
+ SourceDagID interface{} `msgpack:"source_dag_id"`
+
+ // SourceMapIndex corresponds to the JSON schema field "source_map_index".
+ SourceMapIndex interface{} `msgpack:"source_map_index"`
+
+ // SourceRunID corresponds to the JSON schema field "source_run_id".
+ SourceRunID interface{} `msgpack:"source_run_id"`
+
+ // SourceTaskID corresponds to the JSON schema field "source_task_id".
+ SourceTaskID interface{} `msgpack:"source_task_id"`
+
+ // Timestamp corresponds to the JSON schema field "timestamp".
+ Timestamp time.Time `msgpack:"timestamp"`
+}
+
+// Asset event schema with fields that are needed for Runtime.
+type AssetEventResponse struct {
+ // Asset corresponds to the JSON schema field "asset".
+ Asset AssetResponse `msgpack:"asset"`
+
+ // CreatedDagruns corresponds to the JSON schema field "created_dagruns".
+ CreatedDagruns []DagRunAssetReference `msgpack:"created_dagruns"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // ID corresponds to the JSON schema field "id".
+ ID int `msgpack:"id"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // SourceDagID corresponds to the JSON schema field "source_dag_id".
+ SourceDagID interface{} `msgpack:"source_dag_id,omitempty"`
+
+ // SourceMapIndex corresponds to the JSON schema field "source_map_index".
+ SourceMapIndex interface{} `msgpack:"source_map_index,omitempty"`
+
+ // SourceRunID corresponds to the JSON schema field "source_run_id".
+ SourceRunID interface{} `msgpack:"source_run_id,omitempty"`
+
+ // SourceTaskID corresponds to the JSON schema field "source_task_id".
+ SourceTaskID interface{} `msgpack:"source_task_id,omitempty"`
+
+ // Timestamp corresponds to the JSON schema field "timestamp".
+ Timestamp time.Time `msgpack:"timestamp"`
+}
+
+// Response to GetAssetEvent request.
+type AssetEventsResult struct {
+ // AssetEvents corresponds to the JSON schema field "asset_events".
+ AssetEvents []AssetEventResponse `msgpack:"asset_events"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Profile of an asset-like object.
+//
+// Asset will have name, uri defined, with type set to 'Asset'.
+// AssetNameRef will have name defined, type set to 'AssetNameRef'.
+// AssetUriRef will have uri defined, type set to 'AssetUriRef'.
+// AssetAlias will have name defined, type set to 'AssetAlias'.
+//
+// Note that 'type' here is distinct from 'asset_type' the user declares on an
+// Asset (or subclass). This field is for distinguishing between different
+// asset-related types (Asset, AssetRef, or AssetAlias).
+type AssetProfile struct {
+ // Name corresponds to the JSON schema field "name".
+ Name interface{} `msgpack:"name,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI interface{} `msgpack:"uri,omitempty"`
+}
+
+// Schema for AssetModel used in AssetEventDagRunReference.
+type AssetReferenceAssetEventDagRun struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra Extra `msgpack:"extra"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Asset schema for responses with fields that are needed for Runtime.
+type AssetResponse struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // Group corresponds to the JSON schema field "group".
+ Group string `msgpack:"group"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Response to ReadXCom request.
+type AssetResult struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // Group corresponds to the JSON schema field "group".
+ Group string `msgpack:"group"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Response to GetAssetStateStore; wraps the generated API response for supervisor
+// to worker comms.
+type AssetStateStoreResult struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // Value corresponds to the JSON schema field "value".
+ Value JsonValue `msgpack:"value"`
+}
+
+// Response to GetAssetsByAlias; list of concrete assets resolved from an alias.
+type AssetsByAliasResult struct {
+ // Assets corresponds to the JSON schema field "assets".
+ Assets []AssetResult `msgpack:"assets"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type AssignedUsers []HITLUser
+
+// Park a task instance awaiting human input (Human-in-the-loop), without a
+// trigger.
+type AwaitInputTask struct {
+ // NextKwargs corresponds to the JSON schema field "next_kwargs".
+ NextKwargs *NextKwargs `msgpack:"next_kwargs,omitempty"`
+
+ // NextMethod corresponds to the JSON schema field "next_method".
+ NextMethod string `msgpack:"next_method"`
+
+ // RenderedMapIndex corresponds to the JSON schema field "rendered_map_index".
+ RenderedMapIndex interface{} `msgpack:"rendered_map_index,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // Timeout corresponds to the JSON schema field "timeout".
+ Timeout interface{} `msgpack:"timeout,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Schema for telling task which bundle to run with.
+type BundleInfo struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Version corresponds to the JSON schema field "version".
+ Version interface{} `msgpack:"version,omitempty"`
+}
+
+type ClearAssetStateStoreByName struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ClearAssetStateStoreByURI struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type ClearTaskStateStore struct {
+ // AllMapIndices corresponds to the JSON schema field "all_map_indices".
+ AllMapIndices bool `msgpack:"all_map_indices,omitempty"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type Conf map[string]interface{}
+
+// Connection schema for responses with fields that are needed for Runtime.
+type ConnectionResponse struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // ConnType corresponds to the JSON schema field "conn_type".
+ ConnType string `msgpack:"conn_type"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra interface{} `msgpack:"extra"`
+
+ // Host corresponds to the JSON schema field "host".
+ Host interface{} `msgpack:"host"`
+
+ // Login corresponds to the JSON schema field "login".
+ Login interface{} `msgpack:"login"`
+
+ // Password corresponds to the JSON schema field "password".
+ Password interface{} `msgpack:"password"`
+
+ // Port corresponds to the JSON schema field "port".
+ Port interface{} `msgpack:"port"`
+
+ // Schema corresponds to the JSON schema field "schema".
+ Schema interface{} `msgpack:"schema"`
+}
+
+type ConnectionResult struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // ConnType corresponds to the JSON schema field "conn_type".
+ ConnType string `msgpack:"conn_type"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra interface{} `msgpack:"extra,omitempty"`
+
+ // Host corresponds to the JSON schema field "host".
+ Host interface{} `msgpack:"host,omitempty"`
+
+ // Login corresponds to the JSON schema field "login".
+ Login interface{} `msgpack:"login,omitempty"`
+
+ // Password corresponds to the JSON schema field "password".
+ Password interface{} `msgpack:"password,omitempty"`
+
+ // Port corresponds to the JSON schema field "port".
+ Port interface{} `msgpack:"port,omitempty"`
+
+ // Schema corresponds to the JSON schema field "schema".
+ Schema interface{} `msgpack:"schema,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ContextCarrier map[string]interface{}
+
+// Add the input request part of a Human-in-the-loop response.
+type CreateHITLDetailPayload struct {
+ // AssignedUsers corresponds to the JSON schema field "assigned_users".
+ AssignedUsers *AssignedUsers `msgpack:"assigned_users,omitempty"`
+
+ // Body corresponds to the JSON schema field "body".
+ Body interface{} `msgpack:"body,omitempty"`
+
+ // Defaults corresponds to the JSON schema field "defaults".
+ Defaults *Defaults `msgpack:"defaults,omitempty"`
+
+ // Multiple corresponds to the JSON schema field "multiple".
+ Multiple interface{} `msgpack:"multiple,omitempty"`
+
+ // Options corresponds to the JSON schema field "options".
+ Options []string `msgpack:"options"`
+
+ // Params corresponds to the JSON schema field "params".
+ Params *Params `msgpack:"params,omitempty"`
+
+ // Subject corresponds to the JSON schema field "subject".
+ Subject string `msgpack:"subject"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Response containing count of Dag Runs matching certain filters.
+type DRCount struct {
+ // Count corresponds to the JSON schema field "count".
+ Count int `msgpack:"count"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// A Class with information about the success/failure DAG callback to be executed.
+type DagCallbackRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version"`
+
+ // ContextFromServer corresponds to the JSON schema field "context_from_server".
+ ContextFromServer *DagRunContext `msgpack:"context_from_server,omitempty"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Filepath corresponds to the JSON schema field "filepath".
+ Filepath string `msgpack:"filepath"`
+
+ // IsFailureCallback corresponds to the JSON schema field "is_failure_callback".
+ IsFailureCallback interface{} `msgpack:"is_failure_callback,omitempty"`
+
+ // Msg corresponds to the JSON schema field "msg".
+ Msg interface{} `msgpack:"msg,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Request for DAG File Parsing.
+//
+// This is the request that the manager will send to the DAG parser with the dag
+// file and
+// any other necessary metadata.
+type DagFileParseRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundlePath corresponds to the JSON schema field "bundle_path".
+ BundlePath string `msgpack:"bundle_path"`
+
+ // CallbackRequests corresponds to the JSON schema field "callback_requests".
+ CallbackRequests []interface{} `msgpack:"callback_requests,omitempty"`
+
+ // File corresponds to the JSON schema field "file".
+ File string `msgpack:"file"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Result of DAG File Parsing.
+//
+// This is the result of a successful DAG parse, in this class, we gather all
+// serialized DAGs,
+// import errors and warnings to send back to the scheduler to store in the DB.
+type DagFileParsingResult struct {
+ // Fileloc corresponds to the JSON schema field "fileloc".
+ Fileloc string `msgpack:"fileloc"`
+
+ // ImportErrors corresponds to the JSON schema field "import_errors".
+ ImportErrors *ImportErrors `msgpack:"import_errors,omitempty"`
+
+ // SerializedDags corresponds to the JSON schema field "serialized_dags".
+ SerializedDags []LazyDeserializedDAG `msgpack:"serialized_dags"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // Warnings corresponds to the JSON schema field "warnings".
+ Warnings *Warnings `msgpack:"warnings,omitempty"`
+}
+
+type DagResult struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName interface{} `msgpack:"bundle_name,omitempty"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version,omitempty"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // IsPaused corresponds to the JSON schema field "is_paused".
+ IsPaused bool `msgpack:"is_paused"`
+
+ // NextDagrun corresponds to the JSON schema field "next_dagrun".
+ NextDagrun interface{} `msgpack:"next_dagrun,omitempty"`
+
+ // Owners corresponds to the JSON schema field "owners".
+ Owners interface{} `msgpack:"owners,omitempty"`
+
+ // RelativeFileloc corresponds to the JSON schema field "relative_fileloc".
+ RelativeFileloc interface{} `msgpack:"relative_fileloc,omitempty"`
+
+ // Tags corresponds to the JSON schema field "tags".
+ Tags []string `msgpack:"tags"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Schema for DagRun model with minimal required fields needed for Runtime.
+type DagRun struct {
+ // ClearNumber corresponds to the JSON schema field "clear_number".
+ ClearNumber int `msgpack:"clear_number,omitempty"`
+
+ // Conf corresponds to the JSON schema field "conf".
+ Conf *Conf `msgpack:"conf,omitempty"`
+
+ // ConsumedAssetEvents corresponds to the JSON schema field
+ // "consumed_asset_events".
+ ConsumedAssetEvents []AssetEventDagRunReference `msgpack:"consumed_asset_events"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date"`
+
+ // Note corresponds to the JSON schema field "note".
+ Note interface{} `msgpack:"note,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key"`
+
+ // RunAfter corresponds to the JSON schema field "run_after".
+ RunAfter time.Time `msgpack:"run_after"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // RunType corresponds to the JSON schema field "run_type".
+ RunType DagRunType `msgpack:"run_type"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate interface{} `msgpack:"start_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // TeamName corresponds to the JSON schema field "team_name".
+ TeamName interface{} `msgpack:"team_name,omitempty"`
+
+ // TriggeringUserName corresponds to the JSON schema field "triggering_user_name".
+ TriggeringUserName interface{} `msgpack:"triggering_user_name,omitempty"`
+}
+
+// DagRun serializer for asset responses.
+type DagRunAssetReference struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end,omitempty"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start,omitempty"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date,omitempty"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate time.Time `msgpack:"start_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State string `msgpack:"state"`
+}
+
+// Class to pass context info from the server to build a Execution context object.
+type DagRunContext struct {
+ // DagRun corresponds to the JSON schema field "dag_run".
+ DagRun *DagRun `msgpack:"dag_run,omitempty"`
+
+ // LastTI corresponds to the JSON schema field "last_ti".
+ LastTI *TaskInstance `msgpack:"last_ti,omitempty"`
+}
+
+type DagRunResult struct {
+ // ClearNumber corresponds to the JSON schema field "clear_number".
+ ClearNumber interface{} `msgpack:"clear_number,omitempty"`
+
+ // Conf corresponds to the JSON schema field "conf".
+ Conf *Conf `msgpack:"conf,omitempty"`
+
+ // ConsumedAssetEvents corresponds to the JSON schema field
+ // "consumed_asset_events".
+ ConsumedAssetEvents []AssetEventDagRunReference `msgpack:"consumed_asset_events"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end,omitempty"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start,omitempty"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date,omitempty"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // Note corresponds to the JSON schema field "note".
+ Note interface{} `msgpack:"note,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // RunAfter corresponds to the JSON schema field "run_after".
+ RunAfter time.Time `msgpack:"run_after"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // RunType corresponds to the JSON schema field "run_type".
+ RunType DagRunType `msgpack:"run_type"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate interface{} `msgpack:"start_date,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // TeamName corresponds to the JSON schema field "team_name".
+ TeamName interface{} `msgpack:"team_name,omitempty"`
+
+ // TriggeringUserName corresponds to the JSON schema field "triggering_user_name".
+ TriggeringUserName interface{} `msgpack:"triggering_user_name,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DagRunState string
+
+const DagRunStateFailed DagRunState = "failed"
+const DagRunStateQueued DagRunState = "queued"
+
+type DagRunStateResult struct {
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+const DagRunStateRunning DagRunState = "running"
+const DagRunStateSuccess DagRunState = "success"
+
+type DagRunType string
+
+const DagRunTypeAssetMaterialization DagRunType = "asset_materialization"
+const DagRunTypeAssetTriggered DagRunType = "asset_triggered"
+const DagRunTypeBackfill DagRunType = "backfill"
+const DagRunTypeManual DagRunType = "manual"
+const DagRunTypeOperatorTriggered DagRunType = "operator_triggered"
+const DagRunTypeScheduled DagRunType = "scheduled"
+
+type Data map[string]interface{}
+
+type Defaults []string
+
+// Update a task instance state to deferred.
+type DeferTask struct {
+ // Classpath corresponds to the JSON schema field "classpath".
+ Classpath string `msgpack:"classpath"`
+
+ // NextKwargs corresponds to the JSON schema field "next_kwargs".
+ NextKwargs *NextKwargs `msgpack:"next_kwargs,omitempty"`
+
+ // NextMethod corresponds to the JSON schema field "next_method".
+ NextMethod string `msgpack:"next_method"`
+
+ // Queue corresponds to the JSON schema field "queue".
+ Queue interface{} `msgpack:"queue,omitempty"`
+
+ // RenderedMapIndex corresponds to the JSON schema field "rendered_map_index".
+ RenderedMapIndex interface{} `msgpack:"rendered_map_index,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // TriggerKwargs corresponds to the JSON schema field "trigger_kwargs".
+ TriggerKwargs *TriggerKwargs `msgpack:"trigger_kwargs,omitempty"`
+
+ // TriggerTimeout corresponds to the JSON schema field "trigger_timeout".
+ TriggerTimeout interface{} `msgpack:"trigger_timeout,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteAssetStateStoreByName struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteAssetStateStoreByURI struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type DeleteTaskStateStore struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteVariable struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteXCom struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // MapIndex corresponds to the JSON schema field "map_index".
+ MapIndex interface{} `msgpack:"map_index,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // TaskID corresponds to the JSON schema field "task_id".
+ TaskID string `msgpack:"task_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type Detail map[string]interface{}
+
+// Email notification request for task failures/retries.
+type EmailRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version"`
+
+ // ContextFromServer corresponds to the JSON schema field "context_from_server".
+ ContextFromServer TIRunContext `msgpack:"context_from_server"`
+
+ // EmailType corresponds to the JSON schema field "email_type".
+ EmailType EmailRequestEmailType `msgpack:"email_type,omitempty"`
+
+ // Filepath corresponds to the JSON schema field "filepath".
+ Filepath string `msgpack:"filepath"`
+
+ // Msg corresponds to the JSON schema field "msg".
+ Msg interface{} `msgpack:"msg,omitempty"`
+
+ // TI corresponds to the JSON schema field "ti".
+ TI TaskInstance `msgpack:"ti"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type EmailRequestEmailType string
+
+const EmailRequestEmailTypeFailure EmailRequestEmailType = "failure"
+const EmailRequestEmailTypeRetry EmailRequestEmailType = "retry"
+
+type ErrorResponse struct {
+ // Detail corresponds to the JSON schema field "detail".
+ Detail *Detail `msgpack:"detail,omitempty"`
+
+ // Error corresponds to the JSON schema field "error".
+ Error ErrorType `msgpack:"error,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ErrorType string
+
+const ErrorTypeAPISERVERERROR ErrorType = "API_SERVER_ERROR"
+const ErrorTypeASSETNOTFOUND ErrorType = "ASSET_NOT_FOUND"
+const ErrorTypeASSETSTORENOTFOUND ErrorType = "ASSET_STORE_NOT_FOUND"
+const ErrorTypeCONNECTIONNOTFOUND ErrorType = "CONNECTION_NOT_FOUND"
+const ErrorTypeDAGRUNALREADYEXISTS ErrorType = "DAGRUN_ALREADY_EXISTS"
+const ErrorTypeGENERICERROR ErrorType = "GENERIC_ERROR"
+const ErrorTypePERMISSIONDENIED ErrorType = "PERMISSION_DENIED"
+const ErrorTypeTASKSTORENOTFOUND ErrorType = "TASK_STORE_NOT_FOUND"
+const ErrorTypeVARIABLENOTFOUND ErrorType = "VARIABLE_NOT_FOUND"
+const ErrorTypeXCOMNOTFOUND ErrorType = "XCOM_NOT_FOUND"
+
+type Extra map[string]JsonValue
+
+type GetAssetByName struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetByURI struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type GetAssetEventByAsset struct {
+ // After corresponds to the JSON schema field "after".
+ After interface{} `msgpack:"after,omitempty"`
+
+ // Ascending corresponds to the JSON schema field "ascending".
+ Ascending bool `msgpack:"ascending,omitempty"`
+
+ // Before corresponds to the JSON schema field "before".
+ Before interface{} `msgpack:"before,omitempty"`
+
+ // Limit corresponds to the JSON schema field "limit".
+ Limit interface{} `msgpack:"limit,omitempty"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name interface{} `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI interface{} `msgpack:"uri"`
+}
+
+type GetAssetEventByAssetAlias struct {
+ // After corresponds to the JSON schema field "after".
+ After interface{} `msgpack:"after,omitempty"`
+
+ // AliasName corresponds to the JSON schema field "alias_name".
+ AliasName string `msgpack:"alias_name"`
+
+ // Ascending corresponds to the JSON schema field "ascending".
+ Ascending bool `msgpack:"ascending,omitempty"`
+
+ // Before corresponds to the JSON schema field "before".
+ Before interface{} `msgpack:"before,omitempty"`
+
+ // Limit corresponds to the JSON schema field "limit".
+ Limit interface{} `msgpack:"limit,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetStateStoreByName struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetStateStoreByURI struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type GetAssetsByAlias struct {
+ // AliasName corresponds to the JSON schema field "alias_name".
+ AliasName string `msgpack:"alias_name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetConnection struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDRCount struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDates corresponds to the JSON schema field "logical_dates".
+ LogicalDates *LogicalDates `msgpack:"logical_dates,omitempty"`
+
+ // RunIds corresponds to the JSON schema field "run_ids".
+ RunIds *RunIds `msgpack:"run_ids,omitempty"`
+
+ // States corresponds to the JSON schema field "states".
+ States *States `msgpack:"states,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDag struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDagRun struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDagRunState struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Get the response content part of a Human-in-the-loop response.
+type GetHITLDetailResponse struct {
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetPrevSuccessfulDagRun struct {
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetPreviousDagRun struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate time.Time `msgpack:"logical_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Request to get previous task instance.
+type GetPreviousTI struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // MapIndex corresponds to the JSON schema field "map_index".
+ MapIndex int `msgpack:"map_index,omitempty"`
Review Comment:
Handled in https://github.com/apache/airflow/pull/68629/changes/df0858ec7231cf8c126f8936e22af8fdcd3fbc2c#diff-a5d9172b71274222737bbe5668a080c53ec5576bb9b500078c29aeae97f9c617 which produces `go-sdk/pkg/execution/genmodels/defaults.gen.go` that enable setting the default value at `msgpack.Unmarshal` stage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate data models from supervisor schema for Go-SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #68629:
URL: https://github.com/apache/airflow/pull/68629#discussion_r3459030855
##########
go-sdk/Justfile:
##########
Review Comment:
> it would require whoever updates the schema also have the skill to resolve Go changes, which is a big ask
Make sense, I went through the latter direction that you mentioned. Having ad-hoc dump instead of regenerating whenever supervisor schema change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate data models from supervisor schema for Go-SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #68629:
URL: https://github.com/apache/airflow/pull/68629#discussion_r3456842151
##########
go-sdk/pkg/execution/genmodels/models.gen.go:
##########
@@ -0,0 +1,1862 @@
+// Code generated by github.com/atombender/go-jsonschema, DO NOT EDIT.
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+package genmodels
+
+import "time"
+
+// Schema for AssetAliasModel used in AssetEventDagRunReference.
+type AssetAliasReferenceAssetEventDagRun struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+}
+
+// Schema for AssetEvent model used in DagRun.
+type AssetEventDagRunReference struct {
+ // Asset corresponds to the JSON schema field "asset".
+ Asset AssetReferenceAssetEventDagRun `msgpack:"asset"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra Extra `msgpack:"extra"`
+
+ // SourceAliases corresponds to the JSON schema field "source_aliases".
+ SourceAliases []AssetAliasReferenceAssetEventDagRun `msgpack:"source_aliases"`
+
+ // SourceDagID corresponds to the JSON schema field "source_dag_id".
+ SourceDagID interface{} `msgpack:"source_dag_id"`
+
+ // SourceMapIndex corresponds to the JSON schema field "source_map_index".
+ SourceMapIndex interface{} `msgpack:"source_map_index"`
+
+ // SourceRunID corresponds to the JSON schema field "source_run_id".
+ SourceRunID interface{} `msgpack:"source_run_id"`
+
+ // SourceTaskID corresponds to the JSON schema field "source_task_id".
+ SourceTaskID interface{} `msgpack:"source_task_id"`
+
+ // Timestamp corresponds to the JSON schema field "timestamp".
+ Timestamp time.Time `msgpack:"timestamp"`
+}
+
+// Asset event schema with fields that are needed for Runtime.
+type AssetEventResponse struct {
+ // Asset corresponds to the JSON schema field "asset".
+ Asset AssetResponse `msgpack:"asset"`
+
+ // CreatedDagruns corresponds to the JSON schema field "created_dagruns".
+ CreatedDagruns []DagRunAssetReference `msgpack:"created_dagruns"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // ID corresponds to the JSON schema field "id".
+ ID int `msgpack:"id"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // SourceDagID corresponds to the JSON schema field "source_dag_id".
+ SourceDagID interface{} `msgpack:"source_dag_id,omitempty"`
+
+ // SourceMapIndex corresponds to the JSON schema field "source_map_index".
+ SourceMapIndex interface{} `msgpack:"source_map_index,omitempty"`
+
+ // SourceRunID corresponds to the JSON schema field "source_run_id".
+ SourceRunID interface{} `msgpack:"source_run_id,omitempty"`
+
+ // SourceTaskID corresponds to the JSON schema field "source_task_id".
+ SourceTaskID interface{} `msgpack:"source_task_id,omitempty"`
+
+ // Timestamp corresponds to the JSON schema field "timestamp".
+ Timestamp time.Time `msgpack:"timestamp"`
+}
+
+// Response to GetAssetEvent request.
+type AssetEventsResult struct {
+ // AssetEvents corresponds to the JSON schema field "asset_events".
+ AssetEvents []AssetEventResponse `msgpack:"asset_events"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Profile of an asset-like object.
+//
+// Asset will have name, uri defined, with type set to 'Asset'.
+// AssetNameRef will have name defined, type set to 'AssetNameRef'.
+// AssetUriRef will have uri defined, type set to 'AssetUriRef'.
+// AssetAlias will have name defined, type set to 'AssetAlias'.
+//
+// Note that 'type' here is distinct from 'asset_type' the user declares on an
+// Asset (or subclass). This field is for distinguishing between different
+// asset-related types (Asset, AssetRef, or AssetAlias).
+type AssetProfile struct {
+ // Name corresponds to the JSON schema field "name".
+ Name interface{} `msgpack:"name,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI interface{} `msgpack:"uri,omitempty"`
+}
+
+// Schema for AssetModel used in AssetEventDagRunReference.
+type AssetReferenceAssetEventDagRun struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra Extra `msgpack:"extra"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Asset schema for responses with fields that are needed for Runtime.
+type AssetResponse struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // Group corresponds to the JSON schema field "group".
+ Group string `msgpack:"group"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Response to ReadXCom request.
+type AssetResult struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // Group corresponds to the JSON schema field "group".
+ Group string `msgpack:"group"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Response to GetAssetStateStore; wraps the generated API response for supervisor
+// to worker comms.
+type AssetStateStoreResult struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // Value corresponds to the JSON schema field "value".
+ Value JsonValue `msgpack:"value"`
+}
+
+// Response to GetAssetsByAlias; list of concrete assets resolved from an alias.
+type AssetsByAliasResult struct {
+ // Assets corresponds to the JSON schema field "assets".
+ Assets []AssetResult `msgpack:"assets"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type AssignedUsers []HITLUser
+
+// Park a task instance awaiting human input (Human-in-the-loop), without a
+// trigger.
+type AwaitInputTask struct {
+ // NextKwargs corresponds to the JSON schema field "next_kwargs".
+ NextKwargs *NextKwargs `msgpack:"next_kwargs,omitempty"`
+
+ // NextMethod corresponds to the JSON schema field "next_method".
+ NextMethod string `msgpack:"next_method"`
+
+ // RenderedMapIndex corresponds to the JSON schema field "rendered_map_index".
+ RenderedMapIndex interface{} `msgpack:"rendered_map_index,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // Timeout corresponds to the JSON schema field "timeout".
+ Timeout interface{} `msgpack:"timeout,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Schema for telling task which bundle to run with.
+type BundleInfo struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Version corresponds to the JSON schema field "version".
+ Version interface{} `msgpack:"version,omitempty"`
+}
+
+type ClearAssetStateStoreByName struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ClearAssetStateStoreByURI struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type ClearTaskStateStore struct {
+ // AllMapIndices corresponds to the JSON schema field "all_map_indices".
+ AllMapIndices bool `msgpack:"all_map_indices,omitempty"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type Conf map[string]interface{}
+
+// Connection schema for responses with fields that are needed for Runtime.
+type ConnectionResponse struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // ConnType corresponds to the JSON schema field "conn_type".
+ ConnType string `msgpack:"conn_type"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra interface{} `msgpack:"extra"`
+
+ // Host corresponds to the JSON schema field "host".
+ Host interface{} `msgpack:"host"`
+
+ // Login corresponds to the JSON schema field "login".
+ Login interface{} `msgpack:"login"`
+
+ // Password corresponds to the JSON schema field "password".
+ Password interface{} `msgpack:"password"`
+
+ // Port corresponds to the JSON schema field "port".
+ Port interface{} `msgpack:"port"`
+
+ // Schema corresponds to the JSON schema field "schema".
+ Schema interface{} `msgpack:"schema"`
+}
+
+type ConnectionResult struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // ConnType corresponds to the JSON schema field "conn_type".
+ ConnType string `msgpack:"conn_type"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra interface{} `msgpack:"extra,omitempty"`
+
+ // Host corresponds to the JSON schema field "host".
+ Host interface{} `msgpack:"host,omitempty"`
+
+ // Login corresponds to the JSON schema field "login".
+ Login interface{} `msgpack:"login,omitempty"`
+
+ // Password corresponds to the JSON schema field "password".
+ Password interface{} `msgpack:"password,omitempty"`
+
+ // Port corresponds to the JSON schema field "port".
+ Port interface{} `msgpack:"port,omitempty"`
+
+ // Schema corresponds to the JSON schema field "schema".
+ Schema interface{} `msgpack:"schema,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ContextCarrier map[string]interface{}
+
+// Add the input request part of a Human-in-the-loop response.
+type CreateHITLDetailPayload struct {
+ // AssignedUsers corresponds to the JSON schema field "assigned_users".
+ AssignedUsers *AssignedUsers `msgpack:"assigned_users,omitempty"`
+
+ // Body corresponds to the JSON schema field "body".
+ Body interface{} `msgpack:"body,omitempty"`
+
+ // Defaults corresponds to the JSON schema field "defaults".
+ Defaults *Defaults `msgpack:"defaults,omitempty"`
+
+ // Multiple corresponds to the JSON schema field "multiple".
+ Multiple interface{} `msgpack:"multiple,omitempty"`
+
+ // Options corresponds to the JSON schema field "options".
+ Options []string `msgpack:"options"`
+
+ // Params corresponds to the JSON schema field "params".
+ Params *Params `msgpack:"params,omitempty"`
+
+ // Subject corresponds to the JSON schema field "subject".
+ Subject string `msgpack:"subject"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Response containing count of Dag Runs matching certain filters.
+type DRCount struct {
+ // Count corresponds to the JSON schema field "count".
+ Count int `msgpack:"count"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// A Class with information about the success/failure DAG callback to be executed.
+type DagCallbackRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version"`
+
+ // ContextFromServer corresponds to the JSON schema field "context_from_server".
+ ContextFromServer *DagRunContext `msgpack:"context_from_server,omitempty"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Filepath corresponds to the JSON schema field "filepath".
+ Filepath string `msgpack:"filepath"`
+
+ // IsFailureCallback corresponds to the JSON schema field "is_failure_callback".
+ IsFailureCallback interface{} `msgpack:"is_failure_callback,omitempty"`
+
+ // Msg corresponds to the JSON schema field "msg".
+ Msg interface{} `msgpack:"msg,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Request for DAG File Parsing.
+//
+// This is the request that the manager will send to the DAG parser with the dag
+// file and
+// any other necessary metadata.
+type DagFileParseRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundlePath corresponds to the JSON schema field "bundle_path".
+ BundlePath string `msgpack:"bundle_path"`
+
+ // CallbackRequests corresponds to the JSON schema field "callback_requests".
+ CallbackRequests []interface{} `msgpack:"callback_requests,omitempty"`
+
+ // File corresponds to the JSON schema field "file".
+ File string `msgpack:"file"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Result of DAG File Parsing.
+//
+// This is the result of a successful DAG parse, in this class, we gather all
+// serialized DAGs,
+// import errors and warnings to send back to the scheduler to store in the DB.
+type DagFileParsingResult struct {
+ // Fileloc corresponds to the JSON schema field "fileloc".
+ Fileloc string `msgpack:"fileloc"`
+
+ // ImportErrors corresponds to the JSON schema field "import_errors".
+ ImportErrors *ImportErrors `msgpack:"import_errors,omitempty"`
+
+ // SerializedDags corresponds to the JSON schema field "serialized_dags".
+ SerializedDags []LazyDeserializedDAG `msgpack:"serialized_dags"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // Warnings corresponds to the JSON schema field "warnings".
+ Warnings *Warnings `msgpack:"warnings,omitempty"`
+}
+
+type DagResult struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName interface{} `msgpack:"bundle_name,omitempty"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version,omitempty"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // IsPaused corresponds to the JSON schema field "is_paused".
+ IsPaused bool `msgpack:"is_paused"`
+
+ // NextDagrun corresponds to the JSON schema field "next_dagrun".
+ NextDagrun interface{} `msgpack:"next_dagrun,omitempty"`
+
+ // Owners corresponds to the JSON schema field "owners".
+ Owners interface{} `msgpack:"owners,omitempty"`
+
+ // RelativeFileloc corresponds to the JSON schema field "relative_fileloc".
+ RelativeFileloc interface{} `msgpack:"relative_fileloc,omitempty"`
+
+ // Tags corresponds to the JSON schema field "tags".
+ Tags []string `msgpack:"tags"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Schema for DagRun model with minimal required fields needed for Runtime.
+type DagRun struct {
+ // ClearNumber corresponds to the JSON schema field "clear_number".
+ ClearNumber int `msgpack:"clear_number,omitempty"`
+
+ // Conf corresponds to the JSON schema field "conf".
+ Conf *Conf `msgpack:"conf,omitempty"`
+
+ // ConsumedAssetEvents corresponds to the JSON schema field
+ // "consumed_asset_events".
+ ConsumedAssetEvents []AssetEventDagRunReference `msgpack:"consumed_asset_events"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date"`
+
+ // Note corresponds to the JSON schema field "note".
+ Note interface{} `msgpack:"note,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key"`
+
+ // RunAfter corresponds to the JSON schema field "run_after".
+ RunAfter time.Time `msgpack:"run_after"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // RunType corresponds to the JSON schema field "run_type".
+ RunType DagRunType `msgpack:"run_type"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate interface{} `msgpack:"start_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // TeamName corresponds to the JSON schema field "team_name".
+ TeamName interface{} `msgpack:"team_name,omitempty"`
+
+ // TriggeringUserName corresponds to the JSON schema field "triggering_user_name".
+ TriggeringUserName interface{} `msgpack:"triggering_user_name,omitempty"`
+}
+
+// DagRun serializer for asset responses.
+type DagRunAssetReference struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end,omitempty"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start,omitempty"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date,omitempty"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate time.Time `msgpack:"start_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State string `msgpack:"state"`
+}
+
+// Class to pass context info from the server to build a Execution context object.
+type DagRunContext struct {
+ // DagRun corresponds to the JSON schema field "dag_run".
+ DagRun *DagRun `msgpack:"dag_run,omitempty"`
+
+ // LastTI corresponds to the JSON schema field "last_ti".
+ LastTI *TaskInstance `msgpack:"last_ti,omitempty"`
+}
+
+type DagRunResult struct {
+ // ClearNumber corresponds to the JSON schema field "clear_number".
+ ClearNumber interface{} `msgpack:"clear_number,omitempty"`
+
+ // Conf corresponds to the JSON schema field "conf".
+ Conf *Conf `msgpack:"conf,omitempty"`
+
+ // ConsumedAssetEvents corresponds to the JSON schema field
+ // "consumed_asset_events".
+ ConsumedAssetEvents []AssetEventDagRunReference `msgpack:"consumed_asset_events"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end,omitempty"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start,omitempty"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date,omitempty"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // Note corresponds to the JSON schema field "note".
+ Note interface{} `msgpack:"note,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // RunAfter corresponds to the JSON schema field "run_after".
+ RunAfter time.Time `msgpack:"run_after"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // RunType corresponds to the JSON schema field "run_type".
+ RunType DagRunType `msgpack:"run_type"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate interface{} `msgpack:"start_date,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // TeamName corresponds to the JSON schema field "team_name".
+ TeamName interface{} `msgpack:"team_name,omitempty"`
+
+ // TriggeringUserName corresponds to the JSON schema field "triggering_user_name".
+ TriggeringUserName interface{} `msgpack:"triggering_user_name,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DagRunState string
+
+const DagRunStateFailed DagRunState = "failed"
+const DagRunStateQueued DagRunState = "queued"
+
+type DagRunStateResult struct {
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+const DagRunStateRunning DagRunState = "running"
+const DagRunStateSuccess DagRunState = "success"
+
+type DagRunType string
+
+const DagRunTypeAssetMaterialization DagRunType = "asset_materialization"
+const DagRunTypeAssetTriggered DagRunType = "asset_triggered"
+const DagRunTypeBackfill DagRunType = "backfill"
+const DagRunTypeManual DagRunType = "manual"
+const DagRunTypeOperatorTriggered DagRunType = "operator_triggered"
+const DagRunTypeScheduled DagRunType = "scheduled"
+
+type Data map[string]interface{}
+
+type Defaults []string
+
+// Update a task instance state to deferred.
+type DeferTask struct {
+ // Classpath corresponds to the JSON schema field "classpath".
+ Classpath string `msgpack:"classpath"`
+
+ // NextKwargs corresponds to the JSON schema field "next_kwargs".
+ NextKwargs *NextKwargs `msgpack:"next_kwargs,omitempty"`
+
+ // NextMethod corresponds to the JSON schema field "next_method".
+ NextMethod string `msgpack:"next_method"`
+
+ // Queue corresponds to the JSON schema field "queue".
+ Queue interface{} `msgpack:"queue,omitempty"`
+
+ // RenderedMapIndex corresponds to the JSON schema field "rendered_map_index".
+ RenderedMapIndex interface{} `msgpack:"rendered_map_index,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // TriggerKwargs corresponds to the JSON schema field "trigger_kwargs".
+ TriggerKwargs *TriggerKwargs `msgpack:"trigger_kwargs,omitempty"`
+
+ // TriggerTimeout corresponds to the JSON schema field "trigger_timeout".
+ TriggerTimeout interface{} `msgpack:"trigger_timeout,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteAssetStateStoreByName struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteAssetStateStoreByURI struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type DeleteTaskStateStore struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteVariable struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteXCom struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // MapIndex corresponds to the JSON schema field "map_index".
+ MapIndex interface{} `msgpack:"map_index,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // TaskID corresponds to the JSON schema field "task_id".
+ TaskID string `msgpack:"task_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type Detail map[string]interface{}
+
+// Email notification request for task failures/retries.
+type EmailRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version"`
+
+ // ContextFromServer corresponds to the JSON schema field "context_from_server".
+ ContextFromServer TIRunContext `msgpack:"context_from_server"`
+
+ // EmailType corresponds to the JSON schema field "email_type".
+ EmailType EmailRequestEmailType `msgpack:"email_type,omitempty"`
+
+ // Filepath corresponds to the JSON schema field "filepath".
+ Filepath string `msgpack:"filepath"`
+
+ // Msg corresponds to the JSON schema field "msg".
+ Msg interface{} `msgpack:"msg,omitempty"`
+
+ // TI corresponds to the JSON schema field "ti".
+ TI TaskInstance `msgpack:"ti"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type EmailRequestEmailType string
+
+const EmailRequestEmailTypeFailure EmailRequestEmailType = "failure"
+const EmailRequestEmailTypeRetry EmailRequestEmailType = "retry"
+
+type ErrorResponse struct {
+ // Detail corresponds to the JSON schema field "detail".
+ Detail *Detail `msgpack:"detail,omitempty"`
+
+ // Error corresponds to the JSON schema field "error".
+ Error ErrorType `msgpack:"error,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ErrorType string
+
+const ErrorTypeAPISERVERERROR ErrorType = "API_SERVER_ERROR"
+const ErrorTypeASSETNOTFOUND ErrorType = "ASSET_NOT_FOUND"
+const ErrorTypeASSETSTORENOTFOUND ErrorType = "ASSET_STORE_NOT_FOUND"
+const ErrorTypeCONNECTIONNOTFOUND ErrorType = "CONNECTION_NOT_FOUND"
+const ErrorTypeDAGRUNALREADYEXISTS ErrorType = "DAGRUN_ALREADY_EXISTS"
+const ErrorTypeGENERICERROR ErrorType = "GENERIC_ERROR"
+const ErrorTypePERMISSIONDENIED ErrorType = "PERMISSION_DENIED"
+const ErrorTypeTASKSTORENOTFOUND ErrorType = "TASK_STORE_NOT_FOUND"
+const ErrorTypeVARIABLENOTFOUND ErrorType = "VARIABLE_NOT_FOUND"
+const ErrorTypeXCOMNOTFOUND ErrorType = "XCOM_NOT_FOUND"
+
+type Extra map[string]JsonValue
+
+type GetAssetByName struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetByURI struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type GetAssetEventByAsset struct {
+ // After corresponds to the JSON schema field "after".
+ After interface{} `msgpack:"after,omitempty"`
+
+ // Ascending corresponds to the JSON schema field "ascending".
+ Ascending bool `msgpack:"ascending,omitempty"`
+
+ // Before corresponds to the JSON schema field "before".
+ Before interface{} `msgpack:"before,omitempty"`
+
+ // Limit corresponds to the JSON schema field "limit".
+ Limit interface{} `msgpack:"limit,omitempty"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name interface{} `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI interface{} `msgpack:"uri"`
+}
+
+type GetAssetEventByAssetAlias struct {
+ // After corresponds to the JSON schema field "after".
+ After interface{} `msgpack:"after,omitempty"`
+
+ // AliasName corresponds to the JSON schema field "alias_name".
+ AliasName string `msgpack:"alias_name"`
+
+ // Ascending corresponds to the JSON schema field "ascending".
+ Ascending bool `msgpack:"ascending,omitempty"`
+
+ // Before corresponds to the JSON schema field "before".
+ Before interface{} `msgpack:"before,omitempty"`
+
+ // Limit corresponds to the JSON schema field "limit".
+ Limit interface{} `msgpack:"limit,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetStateStoreByName struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetStateStoreByURI struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type GetAssetsByAlias struct {
+ // AliasName corresponds to the JSON schema field "alias_name".
+ AliasName string `msgpack:"alias_name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetConnection struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDRCount struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDates corresponds to the JSON schema field "logical_dates".
+ LogicalDates *LogicalDates `msgpack:"logical_dates,omitempty"`
+
+ // RunIds corresponds to the JSON schema field "run_ids".
+ RunIds *RunIds `msgpack:"run_ids,omitempty"`
+
+ // States corresponds to the JSON schema field "states".
+ States *States `msgpack:"states,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDag struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDagRun struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDagRunState struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Get the response content part of a Human-in-the-loop response.
+type GetHITLDetailResponse struct {
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetPrevSuccessfulDagRun struct {
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetPreviousDagRun struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate time.Time `msgpack:"logical_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Request to get previous task instance.
+type GetPreviousTI struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // MapIndex corresponds to the JSON schema field "map_index".
+ MapIndex int `msgpack:"map_index,omitempty"`
Review Comment:
I aware of the issue that `map_index` default as `-1` from the Task-SDK supervisor schema side but omit on Go-SDK side.
However, I only handle at `decodeIncomingBody` in the first place.
https://github.com/apache/airflow/blob/a1e5a317f041e1cf24f5eaf7202bbd4df8af1b09/go-sdk/pkg/execution/messages.go#L63-L76
Nice catch, introducing another post-processing at `gen` level make sense to me and should be more robust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate data models from supervisor schema for Go-SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #68629:
URL: https://github.com/apache/airflow/pull/68629#discussion_r3459030855
##########
go-sdk/Justfile:
##########
Review Comment:
> it would require whoever updates the schema also have the skill to resolve Go changes, which is a big ask
Make sense, I went through the latter direction that you mentioned. Having ad-hoc dump instead of regenerating whenever supervisor schema change.
Added guide in https://github.com/apache/airflow/pull/68629/commits/c325977ffba3d5cc0a501f362952d5f84c532641.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate data models from supervisor schema for Go-SDK [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #68629:
URL: https://github.com/apache/airflow/pull/68629#issuecomment-4794955572
According to Claude:
The `defaults.gen.go` approach addresses inbound decoding only. The outbound encoding problem we discussed is still present: `GetPreviousTI.MapIndex` remains int with omitempty, so sending `GetPreviousTI{MapIndex: 0}` to the supervisor will silently drop the 0, and the supervisor will apply its default (-1), incorrectly treating a map_index=0 request as unmapped.
The minimal fix is to add a pass in `gen/main.go` that rewrites any struct field in `models.gen.go` where:
- the Go type is a concrete integer (`int`), **and**
- the schema property has a non-zero default (i.e., `defaultLiteral` would produce a seed for it)
...from `int` to `*int`, and strips `omitempty` from the tag (since with a pointer, nil already means "absent" and `omitempty` would be redundant and confusing).
With this type, the call site passes `&mapIndex` to get it encoded, or `nil` to omit it. The supervisor receives the explicit 0 and treats it as index 0, not unmapped.
`defaults.gen.go` would then also need to change: `GetPreviousTI.DecodeMsgpack` would use a `*int` pre-seed, which means `&-1` as the default pointer. The alias constructor becomes `alias{MapIndex: intPtr(-1)}` (with a small helper, or inline with `new(int)` and assignment). Or, since `*int` + absent key decodes to `nil`, the decode-side default for `GetPreviousTI.MapIndex` is just nil — and the call site treats nil the same as -1. That's actually the cleaner outcome: the Go SDK sends `nil` to mean "unmapped" (which the supervisor interprets as its -1 default), and sends `&0` to mean index 0. No decode-side seeding needed at all for that field.
The change to `gen/main.go` would sit inside `writeDefaults` (or a new pre-pass before it): scan `structs[structName]` for `int`-typed fields whose schema property has a non-zero default, emit an AST rewrite of `models.gen.go` that changes the field type to `*int` and strips `,omitempty` from the tag. That rewrite would be analogous to the dead-typedef stripping already done in `parseModels`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate data models from supervisor schema for Go-SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #68629:
URL: https://github.com/apache/airflow/pull/68629#discussion_r3456842151
##########
go-sdk/pkg/execution/genmodels/models.gen.go:
##########
@@ -0,0 +1,1862 @@
+// Code generated by github.com/atombender/go-jsonschema, DO NOT EDIT.
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+package genmodels
+
+import "time"
+
+// Schema for AssetAliasModel used in AssetEventDagRunReference.
+type AssetAliasReferenceAssetEventDagRun struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+}
+
+// Schema for AssetEvent model used in DagRun.
+type AssetEventDagRunReference struct {
+ // Asset corresponds to the JSON schema field "asset".
+ Asset AssetReferenceAssetEventDagRun `msgpack:"asset"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra Extra `msgpack:"extra"`
+
+ // SourceAliases corresponds to the JSON schema field "source_aliases".
+ SourceAliases []AssetAliasReferenceAssetEventDagRun `msgpack:"source_aliases"`
+
+ // SourceDagID corresponds to the JSON schema field "source_dag_id".
+ SourceDagID interface{} `msgpack:"source_dag_id"`
+
+ // SourceMapIndex corresponds to the JSON schema field "source_map_index".
+ SourceMapIndex interface{} `msgpack:"source_map_index"`
+
+ // SourceRunID corresponds to the JSON schema field "source_run_id".
+ SourceRunID interface{} `msgpack:"source_run_id"`
+
+ // SourceTaskID corresponds to the JSON schema field "source_task_id".
+ SourceTaskID interface{} `msgpack:"source_task_id"`
+
+ // Timestamp corresponds to the JSON schema field "timestamp".
+ Timestamp time.Time `msgpack:"timestamp"`
+}
+
+// Asset event schema with fields that are needed for Runtime.
+type AssetEventResponse struct {
+ // Asset corresponds to the JSON schema field "asset".
+ Asset AssetResponse `msgpack:"asset"`
+
+ // CreatedDagruns corresponds to the JSON schema field "created_dagruns".
+ CreatedDagruns []DagRunAssetReference `msgpack:"created_dagruns"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // ID corresponds to the JSON schema field "id".
+ ID int `msgpack:"id"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // SourceDagID corresponds to the JSON schema field "source_dag_id".
+ SourceDagID interface{} `msgpack:"source_dag_id,omitempty"`
+
+ // SourceMapIndex corresponds to the JSON schema field "source_map_index".
+ SourceMapIndex interface{} `msgpack:"source_map_index,omitempty"`
+
+ // SourceRunID corresponds to the JSON schema field "source_run_id".
+ SourceRunID interface{} `msgpack:"source_run_id,omitempty"`
+
+ // SourceTaskID corresponds to the JSON schema field "source_task_id".
+ SourceTaskID interface{} `msgpack:"source_task_id,omitempty"`
+
+ // Timestamp corresponds to the JSON schema field "timestamp".
+ Timestamp time.Time `msgpack:"timestamp"`
+}
+
+// Response to GetAssetEvent request.
+type AssetEventsResult struct {
+ // AssetEvents corresponds to the JSON schema field "asset_events".
+ AssetEvents []AssetEventResponse `msgpack:"asset_events"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Profile of an asset-like object.
+//
+// Asset will have name, uri defined, with type set to 'Asset'.
+// AssetNameRef will have name defined, type set to 'AssetNameRef'.
+// AssetUriRef will have uri defined, type set to 'AssetUriRef'.
+// AssetAlias will have name defined, type set to 'AssetAlias'.
+//
+// Note that 'type' here is distinct from 'asset_type' the user declares on an
+// Asset (or subclass). This field is for distinguishing between different
+// asset-related types (Asset, AssetRef, or AssetAlias).
+type AssetProfile struct {
+ // Name corresponds to the JSON schema field "name".
+ Name interface{} `msgpack:"name,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI interface{} `msgpack:"uri,omitempty"`
+}
+
+// Schema for AssetModel used in AssetEventDagRunReference.
+type AssetReferenceAssetEventDagRun struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra Extra `msgpack:"extra"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Asset schema for responses with fields that are needed for Runtime.
+type AssetResponse struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // Group corresponds to the JSON schema field "group".
+ Group string `msgpack:"group"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Response to ReadXCom request.
+type AssetResult struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // Group corresponds to the JSON schema field "group".
+ Group string `msgpack:"group"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Response to GetAssetStateStore; wraps the generated API response for supervisor
+// to worker comms.
+type AssetStateStoreResult struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // Value corresponds to the JSON schema field "value".
+ Value JsonValue `msgpack:"value"`
+}
+
+// Response to GetAssetsByAlias; list of concrete assets resolved from an alias.
+type AssetsByAliasResult struct {
+ // Assets corresponds to the JSON schema field "assets".
+ Assets []AssetResult `msgpack:"assets"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type AssignedUsers []HITLUser
+
+// Park a task instance awaiting human input (Human-in-the-loop), without a
+// trigger.
+type AwaitInputTask struct {
+ // NextKwargs corresponds to the JSON schema field "next_kwargs".
+ NextKwargs *NextKwargs `msgpack:"next_kwargs,omitempty"`
+
+ // NextMethod corresponds to the JSON schema field "next_method".
+ NextMethod string `msgpack:"next_method"`
+
+ // RenderedMapIndex corresponds to the JSON schema field "rendered_map_index".
+ RenderedMapIndex interface{} `msgpack:"rendered_map_index,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // Timeout corresponds to the JSON schema field "timeout".
+ Timeout interface{} `msgpack:"timeout,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Schema for telling task which bundle to run with.
+type BundleInfo struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Version corresponds to the JSON schema field "version".
+ Version interface{} `msgpack:"version,omitempty"`
+}
+
+type ClearAssetStateStoreByName struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ClearAssetStateStoreByURI struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type ClearTaskStateStore struct {
+ // AllMapIndices corresponds to the JSON schema field "all_map_indices".
+ AllMapIndices bool `msgpack:"all_map_indices,omitempty"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type Conf map[string]interface{}
+
+// Connection schema for responses with fields that are needed for Runtime.
+type ConnectionResponse struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // ConnType corresponds to the JSON schema field "conn_type".
+ ConnType string `msgpack:"conn_type"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra interface{} `msgpack:"extra"`
+
+ // Host corresponds to the JSON schema field "host".
+ Host interface{} `msgpack:"host"`
+
+ // Login corresponds to the JSON schema field "login".
+ Login interface{} `msgpack:"login"`
+
+ // Password corresponds to the JSON schema field "password".
+ Password interface{} `msgpack:"password"`
+
+ // Port corresponds to the JSON schema field "port".
+ Port interface{} `msgpack:"port"`
+
+ // Schema corresponds to the JSON schema field "schema".
+ Schema interface{} `msgpack:"schema"`
+}
+
+type ConnectionResult struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // ConnType corresponds to the JSON schema field "conn_type".
+ ConnType string `msgpack:"conn_type"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra interface{} `msgpack:"extra,omitempty"`
+
+ // Host corresponds to the JSON schema field "host".
+ Host interface{} `msgpack:"host,omitempty"`
+
+ // Login corresponds to the JSON schema field "login".
+ Login interface{} `msgpack:"login,omitempty"`
+
+ // Password corresponds to the JSON schema field "password".
+ Password interface{} `msgpack:"password,omitempty"`
+
+ // Port corresponds to the JSON schema field "port".
+ Port interface{} `msgpack:"port,omitempty"`
+
+ // Schema corresponds to the JSON schema field "schema".
+ Schema interface{} `msgpack:"schema,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ContextCarrier map[string]interface{}
+
+// Add the input request part of a Human-in-the-loop response.
+type CreateHITLDetailPayload struct {
+ // AssignedUsers corresponds to the JSON schema field "assigned_users".
+ AssignedUsers *AssignedUsers `msgpack:"assigned_users,omitempty"`
+
+ // Body corresponds to the JSON schema field "body".
+ Body interface{} `msgpack:"body,omitempty"`
+
+ // Defaults corresponds to the JSON schema field "defaults".
+ Defaults *Defaults `msgpack:"defaults,omitempty"`
+
+ // Multiple corresponds to the JSON schema field "multiple".
+ Multiple interface{} `msgpack:"multiple,omitempty"`
+
+ // Options corresponds to the JSON schema field "options".
+ Options []string `msgpack:"options"`
+
+ // Params corresponds to the JSON schema field "params".
+ Params *Params `msgpack:"params,omitempty"`
+
+ // Subject corresponds to the JSON schema field "subject".
+ Subject string `msgpack:"subject"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Response containing count of Dag Runs matching certain filters.
+type DRCount struct {
+ // Count corresponds to the JSON schema field "count".
+ Count int `msgpack:"count"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// A Class with information about the success/failure DAG callback to be executed.
+type DagCallbackRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version"`
+
+ // ContextFromServer corresponds to the JSON schema field "context_from_server".
+ ContextFromServer *DagRunContext `msgpack:"context_from_server,omitempty"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Filepath corresponds to the JSON schema field "filepath".
+ Filepath string `msgpack:"filepath"`
+
+ // IsFailureCallback corresponds to the JSON schema field "is_failure_callback".
+ IsFailureCallback interface{} `msgpack:"is_failure_callback,omitempty"`
+
+ // Msg corresponds to the JSON schema field "msg".
+ Msg interface{} `msgpack:"msg,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Request for DAG File Parsing.
+//
+// This is the request that the manager will send to the DAG parser with the dag
+// file and
+// any other necessary metadata.
+type DagFileParseRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundlePath corresponds to the JSON schema field "bundle_path".
+ BundlePath string `msgpack:"bundle_path"`
+
+ // CallbackRequests corresponds to the JSON schema field "callback_requests".
+ CallbackRequests []interface{} `msgpack:"callback_requests,omitempty"`
+
+ // File corresponds to the JSON schema field "file".
+ File string `msgpack:"file"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Result of DAG File Parsing.
+//
+// This is the result of a successful DAG parse, in this class, we gather all
+// serialized DAGs,
+// import errors and warnings to send back to the scheduler to store in the DB.
+type DagFileParsingResult struct {
+ // Fileloc corresponds to the JSON schema field "fileloc".
+ Fileloc string `msgpack:"fileloc"`
+
+ // ImportErrors corresponds to the JSON schema field "import_errors".
+ ImportErrors *ImportErrors `msgpack:"import_errors,omitempty"`
+
+ // SerializedDags corresponds to the JSON schema field "serialized_dags".
+ SerializedDags []LazyDeserializedDAG `msgpack:"serialized_dags"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // Warnings corresponds to the JSON schema field "warnings".
+ Warnings *Warnings `msgpack:"warnings,omitempty"`
+}
+
+type DagResult struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName interface{} `msgpack:"bundle_name,omitempty"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version,omitempty"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // IsPaused corresponds to the JSON schema field "is_paused".
+ IsPaused bool `msgpack:"is_paused"`
+
+ // NextDagrun corresponds to the JSON schema field "next_dagrun".
+ NextDagrun interface{} `msgpack:"next_dagrun,omitempty"`
+
+ // Owners corresponds to the JSON schema field "owners".
+ Owners interface{} `msgpack:"owners,omitempty"`
+
+ // RelativeFileloc corresponds to the JSON schema field "relative_fileloc".
+ RelativeFileloc interface{} `msgpack:"relative_fileloc,omitempty"`
+
+ // Tags corresponds to the JSON schema field "tags".
+ Tags []string `msgpack:"tags"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Schema for DagRun model with minimal required fields needed for Runtime.
+type DagRun struct {
+ // ClearNumber corresponds to the JSON schema field "clear_number".
+ ClearNumber int `msgpack:"clear_number,omitempty"`
+
+ // Conf corresponds to the JSON schema field "conf".
+ Conf *Conf `msgpack:"conf,omitempty"`
+
+ // ConsumedAssetEvents corresponds to the JSON schema field
+ // "consumed_asset_events".
+ ConsumedAssetEvents []AssetEventDagRunReference `msgpack:"consumed_asset_events"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date"`
+
+ // Note corresponds to the JSON schema field "note".
+ Note interface{} `msgpack:"note,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key"`
+
+ // RunAfter corresponds to the JSON schema field "run_after".
+ RunAfter time.Time `msgpack:"run_after"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // RunType corresponds to the JSON schema field "run_type".
+ RunType DagRunType `msgpack:"run_type"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate interface{} `msgpack:"start_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // TeamName corresponds to the JSON schema field "team_name".
+ TeamName interface{} `msgpack:"team_name,omitempty"`
+
+ // TriggeringUserName corresponds to the JSON schema field "triggering_user_name".
+ TriggeringUserName interface{} `msgpack:"triggering_user_name,omitempty"`
+}
+
+// DagRun serializer for asset responses.
+type DagRunAssetReference struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end,omitempty"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start,omitempty"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date,omitempty"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate time.Time `msgpack:"start_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State string `msgpack:"state"`
+}
+
+// Class to pass context info from the server to build a Execution context object.
+type DagRunContext struct {
+ // DagRun corresponds to the JSON schema field "dag_run".
+ DagRun *DagRun `msgpack:"dag_run,omitempty"`
+
+ // LastTI corresponds to the JSON schema field "last_ti".
+ LastTI *TaskInstance `msgpack:"last_ti,omitempty"`
+}
+
+type DagRunResult struct {
+ // ClearNumber corresponds to the JSON schema field "clear_number".
+ ClearNumber interface{} `msgpack:"clear_number,omitempty"`
+
+ // Conf corresponds to the JSON schema field "conf".
+ Conf *Conf `msgpack:"conf,omitempty"`
+
+ // ConsumedAssetEvents corresponds to the JSON schema field
+ // "consumed_asset_events".
+ ConsumedAssetEvents []AssetEventDagRunReference `msgpack:"consumed_asset_events"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end,omitempty"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start,omitempty"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date,omitempty"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // Note corresponds to the JSON schema field "note".
+ Note interface{} `msgpack:"note,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // RunAfter corresponds to the JSON schema field "run_after".
+ RunAfter time.Time `msgpack:"run_after"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // RunType corresponds to the JSON schema field "run_type".
+ RunType DagRunType `msgpack:"run_type"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate interface{} `msgpack:"start_date,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // TeamName corresponds to the JSON schema field "team_name".
+ TeamName interface{} `msgpack:"team_name,omitempty"`
+
+ // TriggeringUserName corresponds to the JSON schema field "triggering_user_name".
+ TriggeringUserName interface{} `msgpack:"triggering_user_name,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DagRunState string
+
+const DagRunStateFailed DagRunState = "failed"
+const DagRunStateQueued DagRunState = "queued"
+
+type DagRunStateResult struct {
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+const DagRunStateRunning DagRunState = "running"
+const DagRunStateSuccess DagRunState = "success"
+
+type DagRunType string
+
+const DagRunTypeAssetMaterialization DagRunType = "asset_materialization"
+const DagRunTypeAssetTriggered DagRunType = "asset_triggered"
+const DagRunTypeBackfill DagRunType = "backfill"
+const DagRunTypeManual DagRunType = "manual"
+const DagRunTypeOperatorTriggered DagRunType = "operator_triggered"
+const DagRunTypeScheduled DagRunType = "scheduled"
+
+type Data map[string]interface{}
+
+type Defaults []string
+
+// Update a task instance state to deferred.
+type DeferTask struct {
+ // Classpath corresponds to the JSON schema field "classpath".
+ Classpath string `msgpack:"classpath"`
+
+ // NextKwargs corresponds to the JSON schema field "next_kwargs".
+ NextKwargs *NextKwargs `msgpack:"next_kwargs,omitempty"`
+
+ // NextMethod corresponds to the JSON schema field "next_method".
+ NextMethod string `msgpack:"next_method"`
+
+ // Queue corresponds to the JSON schema field "queue".
+ Queue interface{} `msgpack:"queue,omitempty"`
+
+ // RenderedMapIndex corresponds to the JSON schema field "rendered_map_index".
+ RenderedMapIndex interface{} `msgpack:"rendered_map_index,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // TriggerKwargs corresponds to the JSON schema field "trigger_kwargs".
+ TriggerKwargs *TriggerKwargs `msgpack:"trigger_kwargs,omitempty"`
+
+ // TriggerTimeout corresponds to the JSON schema field "trigger_timeout".
+ TriggerTimeout interface{} `msgpack:"trigger_timeout,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteAssetStateStoreByName struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteAssetStateStoreByURI struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type DeleteTaskStateStore struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteVariable struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteXCom struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // MapIndex corresponds to the JSON schema field "map_index".
+ MapIndex interface{} `msgpack:"map_index,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // TaskID corresponds to the JSON schema field "task_id".
+ TaskID string `msgpack:"task_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type Detail map[string]interface{}
+
+// Email notification request for task failures/retries.
+type EmailRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version"`
+
+ // ContextFromServer corresponds to the JSON schema field "context_from_server".
+ ContextFromServer TIRunContext `msgpack:"context_from_server"`
+
+ // EmailType corresponds to the JSON schema field "email_type".
+ EmailType EmailRequestEmailType `msgpack:"email_type,omitempty"`
+
+ // Filepath corresponds to the JSON schema field "filepath".
+ Filepath string `msgpack:"filepath"`
+
+ // Msg corresponds to the JSON schema field "msg".
+ Msg interface{} `msgpack:"msg,omitempty"`
+
+ // TI corresponds to the JSON schema field "ti".
+ TI TaskInstance `msgpack:"ti"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type EmailRequestEmailType string
+
+const EmailRequestEmailTypeFailure EmailRequestEmailType = "failure"
+const EmailRequestEmailTypeRetry EmailRequestEmailType = "retry"
+
+type ErrorResponse struct {
+ // Detail corresponds to the JSON schema field "detail".
+ Detail *Detail `msgpack:"detail,omitempty"`
+
+ // Error corresponds to the JSON schema field "error".
+ Error ErrorType `msgpack:"error,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ErrorType string
+
+const ErrorTypeAPISERVERERROR ErrorType = "API_SERVER_ERROR"
+const ErrorTypeASSETNOTFOUND ErrorType = "ASSET_NOT_FOUND"
+const ErrorTypeASSETSTORENOTFOUND ErrorType = "ASSET_STORE_NOT_FOUND"
+const ErrorTypeCONNECTIONNOTFOUND ErrorType = "CONNECTION_NOT_FOUND"
+const ErrorTypeDAGRUNALREADYEXISTS ErrorType = "DAGRUN_ALREADY_EXISTS"
+const ErrorTypeGENERICERROR ErrorType = "GENERIC_ERROR"
+const ErrorTypePERMISSIONDENIED ErrorType = "PERMISSION_DENIED"
+const ErrorTypeTASKSTORENOTFOUND ErrorType = "TASK_STORE_NOT_FOUND"
+const ErrorTypeVARIABLENOTFOUND ErrorType = "VARIABLE_NOT_FOUND"
+const ErrorTypeXCOMNOTFOUND ErrorType = "XCOM_NOT_FOUND"
+
+type Extra map[string]JsonValue
+
+type GetAssetByName struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetByURI struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type GetAssetEventByAsset struct {
+ // After corresponds to the JSON schema field "after".
+ After interface{} `msgpack:"after,omitempty"`
+
+ // Ascending corresponds to the JSON schema field "ascending".
+ Ascending bool `msgpack:"ascending,omitempty"`
+
+ // Before corresponds to the JSON schema field "before".
+ Before interface{} `msgpack:"before,omitempty"`
+
+ // Limit corresponds to the JSON schema field "limit".
+ Limit interface{} `msgpack:"limit,omitempty"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name interface{} `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI interface{} `msgpack:"uri"`
+}
+
+type GetAssetEventByAssetAlias struct {
+ // After corresponds to the JSON schema field "after".
+ After interface{} `msgpack:"after,omitempty"`
+
+ // AliasName corresponds to the JSON schema field "alias_name".
+ AliasName string `msgpack:"alias_name"`
+
+ // Ascending corresponds to the JSON schema field "ascending".
+ Ascending bool `msgpack:"ascending,omitempty"`
+
+ // Before corresponds to the JSON schema field "before".
+ Before interface{} `msgpack:"before,omitempty"`
+
+ // Limit corresponds to the JSON schema field "limit".
+ Limit interface{} `msgpack:"limit,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetStateStoreByName struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetStateStoreByURI struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type GetAssetsByAlias struct {
+ // AliasName corresponds to the JSON schema field "alias_name".
+ AliasName string `msgpack:"alias_name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetConnection struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDRCount struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDates corresponds to the JSON schema field "logical_dates".
+ LogicalDates *LogicalDates `msgpack:"logical_dates,omitempty"`
+
+ // RunIds corresponds to the JSON schema field "run_ids".
+ RunIds *RunIds `msgpack:"run_ids,omitempty"`
+
+ // States corresponds to the JSON schema field "states".
+ States *States `msgpack:"states,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDag struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDagRun struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDagRunState struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Get the response content part of a Human-in-the-loop response.
+type GetHITLDetailResponse struct {
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetPrevSuccessfulDagRun struct {
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetPreviousDagRun struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate time.Time `msgpack:"logical_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Request to get previous task instance.
+type GetPreviousTI struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // MapIndex corresponds to the JSON schema field "map_index".
+ MapIndex int `msgpack:"map_index,omitempty"`
Review Comment:
I aware of the `map_index` default as `-1` from the Task-SDK supervisor schema side issue as well.
However, I only handle at `decodeIncomingBody` previously.
https://github.com/apache/airflow/blob/a1e5a317f041e1cf24f5eaf7202bbd4df8af1b09/go-sdk/pkg/execution/messages.go#L63-L76
Nice catch, introducing another post-processing at `gen` level make sense to me and should be more robust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate data models from supervisor schema for Go-SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #68629:
URL: https://github.com/apache/airflow/pull/68629#discussion_r3456829175
##########
go-sdk/Justfile:
##########
Review Comment:
I plan to introduce prek hook to auto generate the new schema in Go-SDK so that it will be done automatically without contributor involved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate data models from supervisor schema for Go-SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #68629:
URL: https://github.com/apache/airflow/pull/68629#discussion_r3456842151
##########
go-sdk/pkg/execution/genmodels/models.gen.go:
##########
@@ -0,0 +1,1862 @@
+// Code generated by github.com/atombender/go-jsonschema, DO NOT EDIT.
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+package genmodels
+
+import "time"
+
+// Schema for AssetAliasModel used in AssetEventDagRunReference.
+type AssetAliasReferenceAssetEventDagRun struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+}
+
+// Schema for AssetEvent model used in DagRun.
+type AssetEventDagRunReference struct {
+ // Asset corresponds to the JSON schema field "asset".
+ Asset AssetReferenceAssetEventDagRun `msgpack:"asset"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra Extra `msgpack:"extra"`
+
+ // SourceAliases corresponds to the JSON schema field "source_aliases".
+ SourceAliases []AssetAliasReferenceAssetEventDagRun `msgpack:"source_aliases"`
+
+ // SourceDagID corresponds to the JSON schema field "source_dag_id".
+ SourceDagID interface{} `msgpack:"source_dag_id"`
+
+ // SourceMapIndex corresponds to the JSON schema field "source_map_index".
+ SourceMapIndex interface{} `msgpack:"source_map_index"`
+
+ // SourceRunID corresponds to the JSON schema field "source_run_id".
+ SourceRunID interface{} `msgpack:"source_run_id"`
+
+ // SourceTaskID corresponds to the JSON schema field "source_task_id".
+ SourceTaskID interface{} `msgpack:"source_task_id"`
+
+ // Timestamp corresponds to the JSON schema field "timestamp".
+ Timestamp time.Time `msgpack:"timestamp"`
+}
+
+// Asset event schema with fields that are needed for Runtime.
+type AssetEventResponse struct {
+ // Asset corresponds to the JSON schema field "asset".
+ Asset AssetResponse `msgpack:"asset"`
+
+ // CreatedDagruns corresponds to the JSON schema field "created_dagruns".
+ CreatedDagruns []DagRunAssetReference `msgpack:"created_dagruns"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // ID corresponds to the JSON schema field "id".
+ ID int `msgpack:"id"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // SourceDagID corresponds to the JSON schema field "source_dag_id".
+ SourceDagID interface{} `msgpack:"source_dag_id,omitempty"`
+
+ // SourceMapIndex corresponds to the JSON schema field "source_map_index".
+ SourceMapIndex interface{} `msgpack:"source_map_index,omitempty"`
+
+ // SourceRunID corresponds to the JSON schema field "source_run_id".
+ SourceRunID interface{} `msgpack:"source_run_id,omitempty"`
+
+ // SourceTaskID corresponds to the JSON schema field "source_task_id".
+ SourceTaskID interface{} `msgpack:"source_task_id,omitempty"`
+
+ // Timestamp corresponds to the JSON schema field "timestamp".
+ Timestamp time.Time `msgpack:"timestamp"`
+}
+
+// Response to GetAssetEvent request.
+type AssetEventsResult struct {
+ // AssetEvents corresponds to the JSON schema field "asset_events".
+ AssetEvents []AssetEventResponse `msgpack:"asset_events"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Profile of an asset-like object.
+//
+// Asset will have name, uri defined, with type set to 'Asset'.
+// AssetNameRef will have name defined, type set to 'AssetNameRef'.
+// AssetUriRef will have uri defined, type set to 'AssetUriRef'.
+// AssetAlias will have name defined, type set to 'AssetAlias'.
+//
+// Note that 'type' here is distinct from 'asset_type' the user declares on an
+// Asset (or subclass). This field is for distinguishing between different
+// asset-related types (Asset, AssetRef, or AssetAlias).
+type AssetProfile struct {
+ // Name corresponds to the JSON schema field "name".
+ Name interface{} `msgpack:"name,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI interface{} `msgpack:"uri,omitempty"`
+}
+
+// Schema for AssetModel used in AssetEventDagRunReference.
+type AssetReferenceAssetEventDagRun struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra Extra `msgpack:"extra"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Asset schema for responses with fields that are needed for Runtime.
+type AssetResponse struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // Group corresponds to the JSON schema field "group".
+ Group string `msgpack:"group"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Response to ReadXCom request.
+type AssetResult struct {
+ // Extra corresponds to the JSON schema field "extra".
+ Extra *Extra `msgpack:"extra,omitempty"`
+
+ // Group corresponds to the JSON schema field "group".
+ Group string `msgpack:"group"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+// Response to GetAssetStateStore; wraps the generated API response for supervisor
+// to worker comms.
+type AssetStateStoreResult struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // Value corresponds to the JSON schema field "value".
+ Value JsonValue `msgpack:"value"`
+}
+
+// Response to GetAssetsByAlias; list of concrete assets resolved from an alias.
+type AssetsByAliasResult struct {
+ // Assets corresponds to the JSON schema field "assets".
+ Assets []AssetResult `msgpack:"assets"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type AssignedUsers []HITLUser
+
+// Park a task instance awaiting human input (Human-in-the-loop), without a
+// trigger.
+type AwaitInputTask struct {
+ // NextKwargs corresponds to the JSON schema field "next_kwargs".
+ NextKwargs *NextKwargs `msgpack:"next_kwargs,omitempty"`
+
+ // NextMethod corresponds to the JSON schema field "next_method".
+ NextMethod string `msgpack:"next_method"`
+
+ // RenderedMapIndex corresponds to the JSON schema field "rendered_map_index".
+ RenderedMapIndex interface{} `msgpack:"rendered_map_index,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // Timeout corresponds to the JSON schema field "timeout".
+ Timeout interface{} `msgpack:"timeout,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Schema for telling task which bundle to run with.
+type BundleInfo struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Version corresponds to the JSON schema field "version".
+ Version interface{} `msgpack:"version,omitempty"`
+}
+
+type ClearAssetStateStoreByName struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ClearAssetStateStoreByURI struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type ClearTaskStateStore struct {
+ // AllMapIndices corresponds to the JSON schema field "all_map_indices".
+ AllMapIndices bool `msgpack:"all_map_indices,omitempty"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type Conf map[string]interface{}
+
+// Connection schema for responses with fields that are needed for Runtime.
+type ConnectionResponse struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // ConnType corresponds to the JSON schema field "conn_type".
+ ConnType string `msgpack:"conn_type"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra interface{} `msgpack:"extra"`
+
+ // Host corresponds to the JSON schema field "host".
+ Host interface{} `msgpack:"host"`
+
+ // Login corresponds to the JSON schema field "login".
+ Login interface{} `msgpack:"login"`
+
+ // Password corresponds to the JSON schema field "password".
+ Password interface{} `msgpack:"password"`
+
+ // Port corresponds to the JSON schema field "port".
+ Port interface{} `msgpack:"port"`
+
+ // Schema corresponds to the JSON schema field "schema".
+ Schema interface{} `msgpack:"schema"`
+}
+
+type ConnectionResult struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // ConnType corresponds to the JSON schema field "conn_type".
+ ConnType string `msgpack:"conn_type"`
+
+ // Extra corresponds to the JSON schema field "extra".
+ Extra interface{} `msgpack:"extra,omitempty"`
+
+ // Host corresponds to the JSON schema field "host".
+ Host interface{} `msgpack:"host,omitempty"`
+
+ // Login corresponds to the JSON schema field "login".
+ Login interface{} `msgpack:"login,omitempty"`
+
+ // Password corresponds to the JSON schema field "password".
+ Password interface{} `msgpack:"password,omitempty"`
+
+ // Port corresponds to the JSON schema field "port".
+ Port interface{} `msgpack:"port,omitempty"`
+
+ // Schema corresponds to the JSON schema field "schema".
+ Schema interface{} `msgpack:"schema,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ContextCarrier map[string]interface{}
+
+// Add the input request part of a Human-in-the-loop response.
+type CreateHITLDetailPayload struct {
+ // AssignedUsers corresponds to the JSON schema field "assigned_users".
+ AssignedUsers *AssignedUsers `msgpack:"assigned_users,omitempty"`
+
+ // Body corresponds to the JSON schema field "body".
+ Body interface{} `msgpack:"body,omitempty"`
+
+ // Defaults corresponds to the JSON schema field "defaults".
+ Defaults *Defaults `msgpack:"defaults,omitempty"`
+
+ // Multiple corresponds to the JSON schema field "multiple".
+ Multiple interface{} `msgpack:"multiple,omitempty"`
+
+ // Options corresponds to the JSON schema field "options".
+ Options []string `msgpack:"options"`
+
+ // Params corresponds to the JSON schema field "params".
+ Params *Params `msgpack:"params,omitempty"`
+
+ // Subject corresponds to the JSON schema field "subject".
+ Subject string `msgpack:"subject"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Response containing count of Dag Runs matching certain filters.
+type DRCount struct {
+ // Count corresponds to the JSON schema field "count".
+ Count int `msgpack:"count"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// A Class with information about the success/failure DAG callback to be executed.
+type DagCallbackRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version"`
+
+ // ContextFromServer corresponds to the JSON schema field "context_from_server".
+ ContextFromServer *DagRunContext `msgpack:"context_from_server,omitempty"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Filepath corresponds to the JSON schema field "filepath".
+ Filepath string `msgpack:"filepath"`
+
+ // IsFailureCallback corresponds to the JSON schema field "is_failure_callback".
+ IsFailureCallback interface{} `msgpack:"is_failure_callback,omitempty"`
+
+ // Msg corresponds to the JSON schema field "msg".
+ Msg interface{} `msgpack:"msg,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Request for DAG File Parsing.
+//
+// This is the request that the manager will send to the DAG parser with the dag
+// file and
+// any other necessary metadata.
+type DagFileParseRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundlePath corresponds to the JSON schema field "bundle_path".
+ BundlePath string `msgpack:"bundle_path"`
+
+ // CallbackRequests corresponds to the JSON schema field "callback_requests".
+ CallbackRequests []interface{} `msgpack:"callback_requests,omitempty"`
+
+ // File corresponds to the JSON schema field "file".
+ File string `msgpack:"file"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Result of DAG File Parsing.
+//
+// This is the result of a successful DAG parse, in this class, we gather all
+// serialized DAGs,
+// import errors and warnings to send back to the scheduler to store in the DB.
+type DagFileParsingResult struct {
+ // Fileloc corresponds to the JSON schema field "fileloc".
+ Fileloc string `msgpack:"fileloc"`
+
+ // ImportErrors corresponds to the JSON schema field "import_errors".
+ ImportErrors *ImportErrors `msgpack:"import_errors,omitempty"`
+
+ // SerializedDags corresponds to the JSON schema field "serialized_dags".
+ SerializedDags []LazyDeserializedDAG `msgpack:"serialized_dags"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // Warnings corresponds to the JSON schema field "warnings".
+ Warnings *Warnings `msgpack:"warnings,omitempty"`
+}
+
+type DagResult struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName interface{} `msgpack:"bundle_name,omitempty"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version,omitempty"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // IsPaused corresponds to the JSON schema field "is_paused".
+ IsPaused bool `msgpack:"is_paused"`
+
+ // NextDagrun corresponds to the JSON schema field "next_dagrun".
+ NextDagrun interface{} `msgpack:"next_dagrun,omitempty"`
+
+ // Owners corresponds to the JSON schema field "owners".
+ Owners interface{} `msgpack:"owners,omitempty"`
+
+ // RelativeFileloc corresponds to the JSON schema field "relative_fileloc".
+ RelativeFileloc interface{} `msgpack:"relative_fileloc,omitempty"`
+
+ // Tags corresponds to the JSON schema field "tags".
+ Tags []string `msgpack:"tags"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Schema for DagRun model with minimal required fields needed for Runtime.
+type DagRun struct {
+ // ClearNumber corresponds to the JSON schema field "clear_number".
+ ClearNumber int `msgpack:"clear_number,omitempty"`
+
+ // Conf corresponds to the JSON schema field "conf".
+ Conf *Conf `msgpack:"conf,omitempty"`
+
+ // ConsumedAssetEvents corresponds to the JSON schema field
+ // "consumed_asset_events".
+ ConsumedAssetEvents []AssetEventDagRunReference `msgpack:"consumed_asset_events"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date"`
+
+ // Note corresponds to the JSON schema field "note".
+ Note interface{} `msgpack:"note,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key"`
+
+ // RunAfter corresponds to the JSON schema field "run_after".
+ RunAfter time.Time `msgpack:"run_after"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // RunType corresponds to the JSON schema field "run_type".
+ RunType DagRunType `msgpack:"run_type"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate interface{} `msgpack:"start_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // TeamName corresponds to the JSON schema field "team_name".
+ TeamName interface{} `msgpack:"team_name,omitempty"`
+
+ // TriggeringUserName corresponds to the JSON schema field "triggering_user_name".
+ TriggeringUserName interface{} `msgpack:"triggering_user_name,omitempty"`
+}
+
+// DagRun serializer for asset responses.
+type DagRunAssetReference struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end,omitempty"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start,omitempty"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date,omitempty"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate time.Time `msgpack:"start_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State string `msgpack:"state"`
+}
+
+// Class to pass context info from the server to build a Execution context object.
+type DagRunContext struct {
+ // DagRun corresponds to the JSON schema field "dag_run".
+ DagRun *DagRun `msgpack:"dag_run,omitempty"`
+
+ // LastTI corresponds to the JSON schema field "last_ti".
+ LastTI *TaskInstance `msgpack:"last_ti,omitempty"`
+}
+
+type DagRunResult struct {
+ // ClearNumber corresponds to the JSON schema field "clear_number".
+ ClearNumber interface{} `msgpack:"clear_number,omitempty"`
+
+ // Conf corresponds to the JSON schema field "conf".
+ Conf *Conf `msgpack:"conf,omitempty"`
+
+ // ConsumedAssetEvents corresponds to the JSON schema field
+ // "consumed_asset_events".
+ ConsumedAssetEvents []AssetEventDagRunReference `msgpack:"consumed_asset_events"`
+
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // DataIntervalEnd corresponds to the JSON schema field "data_interval_end".
+ DataIntervalEnd interface{} `msgpack:"data_interval_end,omitempty"`
+
+ // DataIntervalStart corresponds to the JSON schema field "data_interval_start".
+ DataIntervalStart interface{} `msgpack:"data_interval_start,omitempty"`
+
+ // EndDate corresponds to the JSON schema field "end_date".
+ EndDate interface{} `msgpack:"end_date,omitempty"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // Note corresponds to the JSON schema field "note".
+ Note interface{} `msgpack:"note,omitempty"`
+
+ // PartitionKey corresponds to the JSON schema field "partition_key".
+ PartitionKey interface{} `msgpack:"partition_key,omitempty"`
+
+ // RunAfter corresponds to the JSON schema field "run_after".
+ RunAfter time.Time `msgpack:"run_after"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // RunType corresponds to the JSON schema field "run_type".
+ RunType DagRunType `msgpack:"run_type"`
+
+ // StartDate corresponds to the JSON schema field "start_date".
+ StartDate interface{} `msgpack:"start_date,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // TeamName corresponds to the JSON schema field "team_name".
+ TeamName interface{} `msgpack:"team_name,omitempty"`
+
+ // TriggeringUserName corresponds to the JSON schema field "triggering_user_name".
+ TriggeringUserName interface{} `msgpack:"triggering_user_name,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DagRunState string
+
+const DagRunStateFailed DagRunState = "failed"
+const DagRunStateQueued DagRunState = "queued"
+
+type DagRunStateResult struct {
+ // State corresponds to the JSON schema field "state".
+ State DagRunState `msgpack:"state"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+const DagRunStateRunning DagRunState = "running"
+const DagRunStateSuccess DagRunState = "success"
+
+type DagRunType string
+
+const DagRunTypeAssetMaterialization DagRunType = "asset_materialization"
+const DagRunTypeAssetTriggered DagRunType = "asset_triggered"
+const DagRunTypeBackfill DagRunType = "backfill"
+const DagRunTypeManual DagRunType = "manual"
+const DagRunTypeOperatorTriggered DagRunType = "operator_triggered"
+const DagRunTypeScheduled DagRunType = "scheduled"
+
+type Data map[string]interface{}
+
+type Defaults []string
+
+// Update a task instance state to deferred.
+type DeferTask struct {
+ // Classpath corresponds to the JSON schema field "classpath".
+ Classpath string `msgpack:"classpath"`
+
+ // NextKwargs corresponds to the JSON schema field "next_kwargs".
+ NextKwargs *NextKwargs `msgpack:"next_kwargs,omitempty"`
+
+ // NextMethod corresponds to the JSON schema field "next_method".
+ NextMethod string `msgpack:"next_method"`
+
+ // Queue corresponds to the JSON schema field "queue".
+ Queue interface{} `msgpack:"queue,omitempty"`
+
+ // RenderedMapIndex corresponds to the JSON schema field "rendered_map_index".
+ RenderedMapIndex interface{} `msgpack:"rendered_map_index,omitempty"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // TriggerKwargs corresponds to the JSON schema field "trigger_kwargs".
+ TriggerKwargs *TriggerKwargs `msgpack:"trigger_kwargs,omitempty"`
+
+ // TriggerTimeout corresponds to the JSON schema field "trigger_timeout".
+ TriggerTimeout interface{} `msgpack:"trigger_timeout,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteAssetStateStoreByName struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteAssetStateStoreByURI struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type DeleteTaskStateStore struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteVariable struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type DeleteXCom struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // MapIndex corresponds to the JSON schema field "map_index".
+ MapIndex interface{} `msgpack:"map_index,omitempty"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // TaskID corresponds to the JSON schema field "task_id".
+ TaskID string `msgpack:"task_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type Detail map[string]interface{}
+
+// Email notification request for task failures/retries.
+type EmailRequest struct {
+ // BundleName corresponds to the JSON schema field "bundle_name".
+ BundleName string `msgpack:"bundle_name"`
+
+ // BundleVersion corresponds to the JSON schema field "bundle_version".
+ BundleVersion interface{} `msgpack:"bundle_version"`
+
+ // ContextFromServer corresponds to the JSON schema field "context_from_server".
+ ContextFromServer TIRunContext `msgpack:"context_from_server"`
+
+ // EmailType corresponds to the JSON schema field "email_type".
+ EmailType EmailRequestEmailType `msgpack:"email_type,omitempty"`
+
+ // Filepath corresponds to the JSON schema field "filepath".
+ Filepath string `msgpack:"filepath"`
+
+ // Msg corresponds to the JSON schema field "msg".
+ Msg interface{} `msgpack:"msg,omitempty"`
+
+ // TI corresponds to the JSON schema field "ti".
+ TI TaskInstance `msgpack:"ti"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type EmailRequestEmailType string
+
+const EmailRequestEmailTypeFailure EmailRequestEmailType = "failure"
+const EmailRequestEmailTypeRetry EmailRequestEmailType = "retry"
+
+type ErrorResponse struct {
+ // Detail corresponds to the JSON schema field "detail".
+ Detail *Detail `msgpack:"detail,omitempty"`
+
+ // Error corresponds to the JSON schema field "error".
+ Error ErrorType `msgpack:"error,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type ErrorType string
+
+const ErrorTypeAPISERVERERROR ErrorType = "API_SERVER_ERROR"
+const ErrorTypeASSETNOTFOUND ErrorType = "ASSET_NOT_FOUND"
+const ErrorTypeASSETSTORENOTFOUND ErrorType = "ASSET_STORE_NOT_FOUND"
+const ErrorTypeCONNECTIONNOTFOUND ErrorType = "CONNECTION_NOT_FOUND"
+const ErrorTypeDAGRUNALREADYEXISTS ErrorType = "DAGRUN_ALREADY_EXISTS"
+const ErrorTypeGENERICERROR ErrorType = "GENERIC_ERROR"
+const ErrorTypePERMISSIONDENIED ErrorType = "PERMISSION_DENIED"
+const ErrorTypeTASKSTORENOTFOUND ErrorType = "TASK_STORE_NOT_FOUND"
+const ErrorTypeVARIABLENOTFOUND ErrorType = "VARIABLE_NOT_FOUND"
+const ErrorTypeXCOMNOTFOUND ErrorType = "XCOM_NOT_FOUND"
+
+type Extra map[string]JsonValue
+
+type GetAssetByName struct {
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetByURI struct {
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type GetAssetEventByAsset struct {
+ // After corresponds to the JSON schema field "after".
+ After interface{} `msgpack:"after,omitempty"`
+
+ // Ascending corresponds to the JSON schema field "ascending".
+ Ascending bool `msgpack:"ascending,omitempty"`
+
+ // Before corresponds to the JSON schema field "before".
+ Before interface{} `msgpack:"before,omitempty"`
+
+ // Limit corresponds to the JSON schema field "limit".
+ Limit interface{} `msgpack:"limit,omitempty"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name interface{} `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI interface{} `msgpack:"uri"`
+}
+
+type GetAssetEventByAssetAlias struct {
+ // After corresponds to the JSON schema field "after".
+ After interface{} `msgpack:"after,omitempty"`
+
+ // AliasName corresponds to the JSON schema field "alias_name".
+ AliasName string `msgpack:"alias_name"`
+
+ // Ascending corresponds to the JSON schema field "ascending".
+ Ascending bool `msgpack:"ascending,omitempty"`
+
+ // Before corresponds to the JSON schema field "before".
+ Before interface{} `msgpack:"before,omitempty"`
+
+ // Limit corresponds to the JSON schema field "limit".
+ Limit interface{} `msgpack:"limit,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetStateStoreByName struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Name corresponds to the JSON schema field "name".
+ Name string `msgpack:"name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetAssetStateStoreByURI struct {
+ // Key corresponds to the JSON schema field "key".
+ Key string `msgpack:"key"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+
+ // URI corresponds to the JSON schema field "uri".
+ URI string `msgpack:"uri"`
+}
+
+type GetAssetsByAlias struct {
+ // AliasName corresponds to the JSON schema field "alias_name".
+ AliasName string `msgpack:"alias_name"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetConnection struct {
+ // ConnID corresponds to the JSON schema field "conn_id".
+ ConnID string `msgpack:"conn_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDRCount struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDates corresponds to the JSON schema field "logical_dates".
+ LogicalDates *LogicalDates `msgpack:"logical_dates,omitempty"`
+
+ // RunIds corresponds to the JSON schema field "run_ids".
+ RunIds *RunIds `msgpack:"run_ids,omitempty"`
+
+ // States corresponds to the JSON schema field "states".
+ States *States `msgpack:"states,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDag struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDagRun struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetDagRunState struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // RunID corresponds to the JSON schema field "run_id".
+ RunID string `msgpack:"run_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Get the response content part of a Human-in-the-loop response.
+type GetHITLDetailResponse struct {
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetPrevSuccessfulDagRun struct {
+ // TIID corresponds to the JSON schema field "ti_id".
+ TIID string `msgpack:"ti_id"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+type GetPreviousDagRun struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate time.Time `msgpack:"logical_date"`
+
+ // State corresponds to the JSON schema field "state".
+ State interface{} `msgpack:"state,omitempty"`
+
+ // Type corresponds to the JSON schema field "type".
+ Type string `msgpack:"type,omitempty"`
+}
+
+// Request to get previous task instance.
+type GetPreviousTI struct {
+ // DagID corresponds to the JSON schema field "dag_id".
+ DagID string `msgpack:"dag_id"`
+
+ // LogicalDate corresponds to the JSON schema field "logical_date".
+ LogicalDate interface{} `msgpack:"logical_date,omitempty"`
+
+ // MapIndex corresponds to the JSON schema field "map_index".
+ MapIndex int `msgpack:"map_index,omitempty"`
Review Comment:
I aware of the issue that `map_index` default as `-1` from the Task-SDK supervisor schema side but omit on Go-SDK side.
However, I only handle at `decodeIncomingBody` in the first place (which only handle the `StartupDetails` and miss the `GetPreviousTI` you mentioned).
https://github.com/apache/airflow/blob/a1e5a317f041e1cf24f5eaf7202bbd4df8af1b09/go-sdk/pkg/execution/messages.go#L63-L76
Nice catch, introducing another post-processing at `gen` level make sense to me and should be more robust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org