You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jobs@airflow.apache.org by GitBox <gi...@apache.org> on 2025/08/30 01:40:56 UTC
[GH] (airflow/v3-1-test): Workflow run "Tests ARM" failed!
The GitHub Actions job "Tests ARM" on airflow.git/v3-1-test has failed.
Run started by GitHub user kaxil (triggered by kaxil).
Head commit for run:
d9969be6c9b44e9c3bbccf7dc5e9e0c9d06db734 / Kaxil Naik <ka...@gmail.com>
Decouple serialization and deserialization code for tasks (#54569)
Remove Task SDK dependencies from airflow-core deserialization by establishing
a schema-based contract between client and server components. This
change enables independent deployment and upgrades while laying the foundation
for multi-language SDK support.
Key Decoupling Achievements:
- Replace dynamic get_serialized_fields() calls with hardcoded class methods
- Add schema-driven default resolution with get_operator_defaults_from_schema()
- Remove OPERATOR_DEFAULTS import dependency from airflow-core
- Implement SerializedBaseOperator class attributes for all operator defaults
- Update _is_excluded() logic to use schema defaults for efficient serialization
Serialization Optimizations:
- Unified partial_kwargs optimization supporting both encoded/non-encoded formats
- Intelligent default exclusion reducing storage redundancy
- MappedOperator.operator_class memory optimization (~90-95% reduction)
- Comprehensive client_defaults system with hierarchical resolution
Compatibility & Performance:
- Significant size reduction for typical DAGs with mapped operators
- Minimal overhead for client_defaults section (excellent efficiency)
- All existing serialized DAGs continue to work unchanged
Technical Implementation:
- Add generate_client_defaults() with LRU caching for optimal performance
- Implement _deserialize_partial_kwargs() supporting dual formats
- Centralized field deserialization eliminating code duplication
- Consolidated preprocessing logic in _preprocess_encoded_operator()
- Callback field preprocessing for backward compatibility
Testing & Validation:
- Added TestMappedOperatorSerializationAndClientDefaults with 9 comprehensive tests
- Parameterized tests for multiple serialization formats
- End-to-end validation of serialization/deserialization workflows
- Backward compatibility validation for callback field migration
This decoupling enables independent deployment/upgrades and provides the
foundation for multi-language SDK ecosystem alongside the Task Execution API.
Part of https://github.com/apache/airflow/issues/45428
Report URL: https://github.com/apache/airflow/actions/runs/17337104193
With regards,
GitHub Actions via GitBox
---------------------------------------------------------------------
To unsubscribe, e-mail: jobs-unsubscribe@airflow.apache.org
For additional commands, e-mail: jobs-help@airflow.apache.org