You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/04 18:26:33 UTC
[GitHub] [airflow] pingzh opened a new pull request #21332: Add option to compress dag data
pingzh opened a new pull request #21332:
URL: https://github.com/apache/airflow/pull/21332
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
The uncompressed dag data size can be very large for large DAGs. In our prod db, the dag size can be up to `514MB`.
Adding this optional feature to compress the dag data. It reduces the size from `514MB` to `44MB`.
By default, `compress_serialized_dags` is `False`.
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pingzh edited a comment on pull request #21332: Add option to compress dag data
Posted by GitBox <gi...@apache.org>.
pingzh edited a comment on pull request #21332:
URL: https://github.com/apache/airflow/pull/21332#issuecomment-1031811212
@potiuk @ashb @kaxil @XD-DENG could you please take a look at this PR?
---
The CI failure isn't related to this PR.
https://github.com/apache/airflow/runs/5098104683?check_suite_focus=true#step:11:281
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pingzh commented on pull request #21332: Add option to compress dag data
Posted by GitBox <gi...@apache.org>.
pingzh commented on pull request #21332:
URL: https://github.com/apache/airflow/pull/21332#issuecomment-1039557117
@kaxil addressed your feedback. also the the static CI check isn't related to this PR. can we merge this PR. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #21332: Add option to compress dag data
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #21332:
URL: https://github.com/apache/airflow/pull/21332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pingzh commented on a change in pull request #21332: Add option to compress dag data
Posted by GitBox <gi...@apache.org>.
pingzh commented on a change in pull request #21332:
URL: https://github.com/apache/airflow/pull/21332#discussion_r801092172
##########
File path: airflow/models/serialized_dag.py
##########
@@ -20,11 +20,12 @@
import hashlib
import logging
+import zlib
Review comment:
although `bz2` can achieve smaller size of compress data, but we noticed that the `zlib` is faster than `bz2` when compressing and decompressing data with the same level of compression (zlib with default level 6, also set compresslevel as 6 for bz2).
test with python3.7.9, `raw json data size`: `514MB`, `bz2` compressed: `14M` and `zlib` compressed: `44M`.
```
~$ python test_bz2.py
bz2 took 89.23221111297607
bz2 decompress took 15.989518880844116
~$ python test_zlib.py
zlib compress took 8.829093933105469
zlib decompress took 1.1581130027770996
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pingzh commented on pull request #21332: Add option to compress dag data
Posted by GitBox <gi...@apache.org>.
pingzh commented on pull request #21332:
URL: https://github.com/apache/airflow/pull/21332#issuecomment-1034087831
@ashb nice, after your rebase, the CI passed. hmm, very strange that my pushes did not pass the CI 🤦
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pingzh commented on a change in pull request #21332: Add option to compress dag data
Posted by GitBox <gi...@apache.org>.
pingzh commented on a change in pull request #21332:
URL: https://github.com/apache/airflow/pull/21332#discussion_r801008951
##########
File path: tests/models/test_serialized_dag.py
##########
@@ -63,60 +72,68 @@ def _write_example_dags(self):
return example_dags
def test_write_dag(self):
- """DAGs can be written into database."""
- example_dags = self._write_example_dags()
+ """DAGs can be written into database when no comprehension"""
+ with mock.patch(
+ 'airflow.models.serialized_dag.COMPRESS_SERIALIZED_DAGS', self.compress_serialized_dags
+ ):
Review comment:
Good call. thanks
##########
File path: airflow/config_templates/config.yml
##########
@@ -368,6 +368,13 @@
type: string
example: ~
default: "30"
+ - name: compress_serialized_dags
+ description: |
+ If True, serialized DAGs are compressed before writing to DB
Review comment:
Good call👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #21332: Add option to compress dag data
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #21332:
URL: https://github.com/apache/airflow/pull/21332#discussion_r801437260
##########
File path: airflow/models/serialized_dag.py
##########
@@ -20,11 +20,12 @@
import hashlib
import logging
+import zlib
Review comment:
Oh yeah, 15s to decompress clearly makes bz2 unsuitable!
(I knew bz2 was slow to compress, but I mistakenly thought it was fast to decompress too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pingzh commented on pull request #21332: Add option to compress dag data
Posted by GitBox <gi...@apache.org>.
pingzh commented on pull request #21332:
URL: https://github.com/apache/airflow/pull/21332#issuecomment-1032991821
hi @ashb @potiuk , do you think we can merge this PR? the CI failure isn't related to this PR. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pingzh commented on pull request #21332: Add option to compress dag data
Posted by GitBox <gi...@apache.org>.
pingzh commented on pull request #21332:
URL: https://github.com/apache/airflow/pull/21332#issuecomment-1039557117
@kaxil addressed your feedback. also the the static CI check isn't related to this PR. can we merge this PR. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #21332: Add option to compress dag data
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #21332:
URL: https://github.com/apache/airflow/pull/21332#issuecomment-1034090682
That's cos we fixed the broken main :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pingzh commented on pull request #21332: Add option to compress dag data
Posted by GitBox <gi...@apache.org>.
pingzh commented on pull request #21332:
URL: https://github.com/apache/airflow/pull/21332#issuecomment-1031811212
@potiuk @ashb @kaxil @XD-DENG could you please take a look at this PR?
The CI failure isn't related to this PR.
https://github.com/apache/airflow/runs/5098104683?check_suite_focus=true#step:11:281
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #21332: Add option to compress dag data
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #21332:
URL: https://github.com/apache/airflow/pull/21332#issuecomment-1032409172
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #21332: Add option to compress dag data
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #21332:
URL: https://github.com/apache/airflow/pull/21332#discussion_r805979165
##########
File path: docs/apache-airflow/dag-serialization.rst
##########
@@ -84,6 +85,8 @@ Add the following settings in ``airflow.cfg``:
load on the DB, but at the expense of displaying a possibly stale cached version of the DAG.
* ``max_num_rendered_ti_fields_per_task``: This option controls the maximum number of Rendered Task Instance
Fields (Template Fields) per task to store in the Database.
+* ``compress_serialized_dags``: This option controls whether compressing the dag data to the Database. It is
+ useful when there are very large DAGs in your cluster. When True, this will disable the DAG dependencies view.
Review comment:
```suggestion
* ``compress_serialized_dags``: This option controls whether to compress the Serialized dag to the Database. It is
useful when there are very large DAGs in your cluster. When ``True``, this will disable the DAG dependencies view.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pingzh commented on a change in pull request #21332: Add option to compress dag data
Posted by GitBox <gi...@apache.org>.
pingzh commented on a change in pull request #21332:
URL: https://github.com/apache/airflow/pull/21332#discussion_r801092172
##########
File path: airflow/models/serialized_dag.py
##########
@@ -20,11 +20,12 @@
import hashlib
import logging
+import zlib
Review comment:
although `bz2` can achieve smaller size of compress data, but we noticed that the `zlib` is faster than `bz2` when compressing and decompressing data with the same level of compression (zlib with default level 6, also set compresslevel as 6 for bz2).
test with python3.7.9, `raw json data size`: `514MB`, `bz2` compressed: `14M` and `zlib` compressed: `44M`.
```
ping_zhang@i-001e445e8c0ed963f:~$ python test_bz2.py
bz2 took 89.23221111297607
bz2 decompress took 15.989518880844116
ping_zhang@i-001e445e8c0ed963f:~$ python test_zlib.py
zlib compress took 8.829093933105469
zlib decompress took 1.1581130027770996
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #21332: Add option to compress dag data
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #21332:
URL: https://github.com/apache/airflow/pull/21332#discussion_r800972981
##########
File path: airflow/migrations/versions/a3bcd0914482_add_data_compressed_to_serialized_dag.py
##########
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""add data_compressed to serialized_dag
+
+Revision ID: a3bcd0914482
+Revises: e655c0453f75
+Create Date: 2022-02-03 22:40:59.841119
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = 'a3bcd0914482'
+down_revision = 'e655c0453f75'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+ op.add_column('serialized_dag', sa.Column('data_compressed', sa.LargeBinary, nullable=True))
+ with op.batch_alter_table('serialized_dag') as batch_op:
+ batch_op.alter_column('data', existing_type=sa.JSON, nullable=True)
Review comment:
It probably doesn't make much difference but:
```suggestion
with op.batch_alter_table('serialized_dag') as batch_op:
batch_op.alter_column('data', existing_type=sa.JSON, nullable=True)
batch_op.add_column(sa.Column('data_compressed', sa.LargeBinary, nullable=True))
```
##########
File path: airflow/config_templates/config.yml
##########
@@ -368,6 +368,13 @@
type: string
example: ~
default: "30"
+ - name: compress_serialized_dags
+ description: |
+ If True, serialized DAGs are compressed before writing to DB
+ version_added: ~
Review comment:
```suggestion
version_added: "2.3.0"
```
##########
File path: airflow/models/serialized_dag.py
##########
@@ -20,11 +20,12 @@
import hashlib
import logging
+import zlib
Review comment:
Without meaning to start a compression format war: Is zlib really an appropriate choice?
The python stdlib includes `bz2` which can produce blobs as much as 50% smaller -- though obviously that depends upon the data https://starbeamrainbowlabs.com/blog/article.php?article=posts%2F275-Compression-Comparison.html
##########
File path: airflow/config_templates/config.yml
##########
@@ -368,6 +368,13 @@
type: string
example: ~
default: "30"
+ - name: compress_serialized_dags
+ description: |
+ If True, serialized DAGs are compressed before writing to DB
Review comment:
We should note that turning this on would (currently) break the DAG dependencies view.
##########
File path: airflow/migrations/versions/a3bcd0914482_add_data_compressed_to_serialized_dag.py
##########
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""add data_compressed to serialized_dag
+
+Revision ID: a3bcd0914482
+Revises: e655c0453f75
+Create Date: 2022-02-03 22:40:59.841119
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = 'a3bcd0914482'
+down_revision = 'e655c0453f75'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+ op.add_column('serialized_dag', sa.Column('data_compressed', sa.LargeBinary, nullable=True))
+ with op.batch_alter_table('serialized_dag') as batch_op:
+ batch_op.alter_column('data', existing_type=sa.JSON, nullable=True)
+
+
+def downgrade():
+ with op.batch_alter_table('serialized_dag') as batch_op:
+ batch_op.alter_column('data', existing_type=sa.JSON, nullable=False)
+ op.drop_column('serialized_dag', 'data_compressed')
Review comment:
```suggestion
batch_op.drop_column('data_compressed')
```
##########
File path: tests/models/test_serialized_dag.py
##########
@@ -63,60 +72,68 @@ def _write_example_dags(self):
return example_dags
def test_write_dag(self):
- """DAGs can be written into database."""
- example_dags = self._write_example_dags()
+ """DAGs can be written into database when no comprehension"""
+ with mock.patch(
+ 'airflow.models.serialized_dag.COMPRESS_SERIALIZED_DAGS', self.compress_serialized_dags
+ ):
Review comment:
Rather than having to do this and touch most of the lines of the files you can do this:
```python
def setUp(self):
self.patcher = mock.patch(
'airflow.models.serialized_dag.COMPRESS_SERIALIZED_DAGS', self.compress_serialized_dags
)
self.patcher.start()
def tearDown(sefl):
self.patcher.stop()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #21332: Add option to compress dag data
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #21332:
URL: https://github.com/apache/airflow/pull/21332#discussion_r805979165
##########
File path: docs/apache-airflow/dag-serialization.rst
##########
@@ -84,6 +85,8 @@ Add the following settings in ``airflow.cfg``:
load on the DB, but at the expense of displaying a possibly stale cached version of the DAG.
* ``max_num_rendered_ti_fields_per_task``: This option controls the maximum number of Rendered Task Instance
Fields (Template Fields) per task to store in the Database.
+* ``compress_serialized_dags``: This option controls whether compressing the dag data to the Database. It is
+ useful when there are very large DAGs in your cluster. When True, this will disable the DAG dependencies view.
Review comment:
```suggestion
* ``compress_serialized_dags``: This option controls whether to compress the Serialized dag to the Database. It is
useful when there are very large DAGs in your cluster. When ``True``, this will disable the DAG dependencies view.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #21332: Add option to compress dag data
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #21332:
URL: https://github.com/apache/airflow/pull/21332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org