You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/07 19:19:24 UTC
[GitHub] [airflow] ashb commented on a change in pull request #21332: Add option to compress dag data
ashb commented on a change in pull request #21332:
URL: https://github.com/apache/airflow/pull/21332#discussion_r800972981
##########
File path: airflow/migrations/versions/a3bcd0914482_add_data_compressed_to_serialized_dag.py
##########
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""add data_compressed to serialized_dag
+
+Revision ID: a3bcd0914482
+Revises: e655c0453f75
+Create Date: 2022-02-03 22:40:59.841119
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = 'a3bcd0914482'
+down_revision = 'e655c0453f75'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+ op.add_column('serialized_dag', sa.Column('data_compressed', sa.LargeBinary, nullable=True))
+ with op.batch_alter_table('serialized_dag') as batch_op:
+ batch_op.alter_column('data', existing_type=sa.JSON, nullable=True)
Review comment:
It probably doesn't make much difference but:
```suggestion
with op.batch_alter_table('serialized_dag') as batch_op:
batch_op.alter_column('data', existing_type=sa.JSON, nullable=True)
batch_op.add_column(sa.Column('data_compressed', sa.LargeBinary, nullable=True))
```
##########
File path: airflow/config_templates/config.yml
##########
@@ -368,6 +368,13 @@
type: string
example: ~
default: "30"
+ - name: compress_serialized_dags
+ description: |
+ If True, serialized DAGs are compressed before writing to DB
+ version_added: ~
Review comment:
```suggestion
version_added: "2.3.0"
```
##########
File path: airflow/models/serialized_dag.py
##########
@@ -20,11 +20,12 @@
import hashlib
import logging
+import zlib
Review comment:
Without meaning to start a compression format war: Is zlib really an appropriate choice?
The python stdlib includes `bz2` which can produce blobs as much as 50% smaller -- though obviously that depends upon the data https://starbeamrainbowlabs.com/blog/article.php?article=posts%2F275-Compression-Comparison.html
##########
File path: airflow/config_templates/config.yml
##########
@@ -368,6 +368,13 @@
type: string
example: ~
default: "30"
+ - name: compress_serialized_dags
+ description: |
+ If True, serialized DAGs are compressed before writing to DB
Review comment:
We should note that turning this on would (currently) break the DAG dependencies view.
##########
File path: airflow/migrations/versions/a3bcd0914482_add_data_compressed_to_serialized_dag.py
##########
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""add data_compressed to serialized_dag
+
+Revision ID: a3bcd0914482
+Revises: e655c0453f75
+Create Date: 2022-02-03 22:40:59.841119
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = 'a3bcd0914482'
+down_revision = 'e655c0453f75'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+ op.add_column('serialized_dag', sa.Column('data_compressed', sa.LargeBinary, nullable=True))
+ with op.batch_alter_table('serialized_dag') as batch_op:
+ batch_op.alter_column('data', existing_type=sa.JSON, nullable=True)
+
+
+def downgrade():
+ with op.batch_alter_table('serialized_dag') as batch_op:
+ batch_op.alter_column('data', existing_type=sa.JSON, nullable=False)
+ op.drop_column('serialized_dag', 'data_compressed')
Review comment:
```suggestion
batch_op.drop_column('data_compressed')
```
##########
File path: tests/models/test_serialized_dag.py
##########
@@ -63,60 +72,68 @@ def _write_example_dags(self):
return example_dags
def test_write_dag(self):
- """DAGs can be written into database."""
- example_dags = self._write_example_dags()
+ """DAGs can be written into database when no comprehension"""
+ with mock.patch(
+ 'airflow.models.serialized_dag.COMPRESS_SERIALIZED_DAGS', self.compress_serialized_dags
+ ):
Review comment:
Rather than having to do this and touch most of the lines of the files you can do this:
```python
def setUp(self):
self.patcher = mock.patch(
'airflow.models.serialized_dag.COMPRESS_SERIALIZED_DAGS', self.compress_serialized_dags
)
self.patcher.start()
def tearDown(sefl):
self.patcher.stop()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org