You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "claudevdm (via GitHub)" <gi...@apache.org> on 2025/10/09 14:20:54 UTC
[PR] Add flag for disabling dill check in coders. [beam]
claudevdm opened a new pull request, #36453:
URL: https://github.com/apache/beam/pull/36453
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add flag for disabling dill check in coders. [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #36453:
URL: https://github.com/apache/beam/pull/36453#discussion_r2418018720
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -874,6 +874,18 @@ def _add_argparse_args(cls, parser):
'their condition met. Some operations, such as GroupByKey, disallow '
'this. This exists for cases where such loss is acceptable and for '
'backwards compatibility. See BEAM-9487.')
+ parser.add_argument(
+ '--force_cloudpickle_deterministic_coders',
Review Comment:
But to answer the original question: I made this a flag so that if someone knows they arent really triggering the code path where the type is pickled using coders and are doing a pipeline update they don't need dill installed.
This can still be done but it is a bit more complicated and probably not worth it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add flag for disabling dill check in coders. [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #36453:
URL: https://github.com/apache/beam/pull/36453#discussion_r2418018720
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -874,6 +874,18 @@ def _add_argparse_args(cls, parser):
'their condition met. Some operations, such as GroupByKey, disallow '
'this. This exists for cases where such loss is acceptable and for '
'backwards compatibility. See BEAM-9487.')
+ parser.add_argument(
+ '--force_cloudpickle_deterministic_coders',
Review Comment:
But to answer the original question: I made this a flag so that if someone knows they arent really triggering the code path where the type is pickled using coders and are doing a pipeline update they don't need dill 0.3.1.1 installed.
This can still be done but it is a bit more complicated and probably not worth it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add flag for disabling dill check in coders. [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #36453:
URL: https://github.com/apache/beam/pull/36453#discussion_r2426399500
##########
.github/workflows/beam_PreCommit_Python_Dill.yml:
##########
@@ -106,9 +106,9 @@ jobs:
arguments: |
-Pposargs="${{
contains(matrix.os, 'self-hosted') &&
- 'apache_beam/internal/ apache_beam/ml/ apache_beam/transforms/ apache_beam/typehints/ apache_beam/runners/portability/ -m (uses_dill and not require_docker_in_docker)' ||
- 'apache_beam/internal/ apache_beam/ml/ apache_beam/transforms/ apache_beam/typehints/ apache_beam/runners/portability/ -m (uses_dill and require_docker_in_docker)'
- }}" \
+ 'apache_beam/internal/pickler_test.py apache_beam/io/gcp/bigquery_file_loads_test.py apache_beam/options/pipeline_options_validator_test.py apache_beam/transforms/combinefn_lifecycle_test.py apache_beam/transforms/util_test.py apache_beam/transforms/ptransform_test.py apache_beam/typehints/schemas_test.py apache_beam/runners/portability/stager_test.py -m (uses_dill and not require_docker_in_docker)' ||
+ 'apache_beam/internal/pickler_test.py apache_beam/io/gcp/bigquery_file_loads_test.py apache_beam/options/pipeline_options_validator_test.py apache_beam/transforms/combinefn_lifecycle_test.py apache_beam/transforms/util_test.py apache_beam/transforms/ptransform_test.py apache_beam/typehints/schemas_test.py apache_beam/runners/portability/stager_test.py -m (uses_dill and require_docker_in_docker)'
Review Comment:
I think we should avoid doing this in CI - the downside here is that now adding a test in a different file also requires updating the github config (which you can't even do in the same PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add flag for disabling dill check in coders. [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #36453:
URL: https://github.com/apache/beam/pull/36453#discussion_r2418008474
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -874,6 +874,18 @@ def _add_argparse_args(cls, parser):
'their condition met. Some operations, such as GroupByKey, disallow '
'this. This exists for cases where such loss is acceptable and for '
'backwards compatibility. See BEAM-9487.')
+ parser.add_argument(
+ '--force_cloudpickle_deterministic_coders',
Review Comment:
Actually as I was typing out the explanation I realized this flag wont work in practice (because this change will result in a new coder name and therefore failing update compat check in dataflow.
I added it so that tests that use update compat flags (like reshuffle tests) that for some reason have a DeterministicFastPrimitivesCoder (but dont actually trigger the pickle code path) fail because dill is not installed. These test have to be run in the dill test suite even though they dont really need dill.
Maybe it should not be a flag/pipeline option, and just a property that we hack on in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add flag for disabling dill check in coders. [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #36453:
URL: https://github.com/apache/beam/pull/36453#discussion_r2418105478
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -874,6 +874,18 @@ def _add_argparse_args(cls, parser):
'their condition met. Some operations, such as GroupByKey, disallow '
'this. This exists for cases where such loss is acceptable and for '
'backwards compatibility. See BEAM-9487.')
+ parser.add_argument(
+ '--force_cloudpickle_deterministic_coders',
Review Comment:
I removed it as a pipeline option and just monkey patch it in tests, let me know what you think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add flag for disabling dill check in coders. [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #36453:
URL: https://github.com/apache/beam/pull/36453
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add flag for disabling dill check in coders. [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #36453:
URL: https://github.com/apache/beam/pull/36453#discussion_r2418008474
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -874,6 +874,18 @@ def _add_argparse_args(cls, parser):
'their condition met. Some operations, such as GroupByKey, disallow '
'this. This exists for cases where such loss is acceptable and for '
'backwards compatibility. See BEAM-9487.')
+ parser.add_argument(
+ '--force_cloudpickle_deterministic_coders',
Review Comment:
Actually as I was typing out the explanation I realized this flag wont work in practice (because this change will result in a new coder name and therefore failing update compat check in dataflow.
I added it so that tests that use update compat flags (like reshuffle tests) that for some reason have a DeterministicFastPrimitivesCoder (but dont actually trigger the pickle code path) fail because dill is not installed can be run in non-dill test suites. These test have to be run in the dill test suite even though they dont really need dill.
Maybe it should not be a flag/pipeline option, and just a property that we hack on in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add flag for disabling dill check in coders. [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on PR #36453:
URL: https://github.com/apache/beam/pull/36453#issuecomment-3387137050
R: @damccorm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add flag for disabling dill check in coders. [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #36453:
URL: https://github.com/apache/beam/pull/36453#discussion_r2417911376
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -874,6 +874,18 @@ def _add_argparse_args(cls, parser):
'their condition met. Some operations, such as GroupByKey, disallow '
'this. This exists for cases where such loss is acceptable and for '
'backwards compatibility. See BEAM-9487.')
+ parser.add_argument(
+ '--force_cloudpickle_deterministic_coders',
Review Comment:
I don't quite follow the motivation - why would a user ever want to do this? Won't this always lead to correctness problems?
##########
.github/workflows/beam_PreCommit_Python_Dill.yml:
##########
@@ -106,9 +106,9 @@ jobs:
arguments: |
-Pposargs="${{
contains(matrix.os, 'self-hosted') &&
- 'apache_beam/internal/ apache_beam/ml/ apache_beam/transforms/ apache_beam/typehints/ apache_beam/runners/portability/ -m (uses_dill and not require_docker_in_docker)' ||
- 'apache_beam/internal/ apache_beam/ml/ apache_beam/transforms/ apache_beam/typehints/ apache_beam/runners/portability/ -m (uses_dill and require_docker_in_docker)'
- }}" \
+ 'apache_beam/internal/pickler_test.py apache_beam/io/gcp/bigquery_file_loads_test.py apache_beam/options/pipeline_options_validator_test.py apache_beam/transforms/combinefn_lifecycle_test.py apache_beam/transforms/util_test.py apache_beam/transforms/ptransform_test.py apache_beam/typehints/schemas_test.py apache_beam/runners/portability/stager_test.py -m (uses_dill and not require_docker_in_docker)' ||
+ 'apache_beam/internal/pickler_test.py apache_beam/io/gcp/bigquery_file_loads_test.py apache_beam/options/pipeline_options_validator_test.py apache_beam/transforms/combinefn_lifecycle_test.py apache_beam/transforms/util_test.py apache_beam/transforms/ptransform_test.py apache_beam/typehints/schemas_test.py apache_beam/runners/portability/stager_test.py -m (uses_dill and require_docker_in_docker)'
Review Comment:
Why do we need this? Shouldn't filtering to uses_dill be enough
##########
sdks/python/apache_beam/coders/coders.py:
##########
@@ -1004,8 +1004,12 @@ def to_type_hint(self):
return Any
-def _should_force_use_dill(update_compat_version):
+def _should_force_use_dill(registry):
+ if getattr(registry, 'force_cloudpickle_deterministic_coders', False):
Review Comment:
Rather than doing this, should we provide a default value in __init__ like https://github.com/apache/beam/blob/6f31e56fcaca5cbb71b90f631c6562a1f1471c08/sdks/python/apache_beam/coders/typecoders.py#L87
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add flag for disabling dill check in coders. [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #36453:
URL: https://github.com/apache/beam/pull/36453#discussion_r2418018720
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -874,6 +874,18 @@ def _add_argparse_args(cls, parser):
'their condition met. Some operations, such as GroupByKey, disallow '
'this. This exists for cases where such loss is acceptable and for '
'backwards compatibility. See BEAM-9487.')
+ parser.add_argument(
+ '--force_cloudpickle_deterministic_coders',
Review Comment:
But to answer the original question: I made this a flag so that if someone knows they arent triggering the code path where the type is pickled using coders and are doing a pipeline update they don't need dill 0.3.1.1 installed. This can happen if they have a DeterministicFastPrimitivesCoder in the pipeline but their key is not a "special type" e.g. https://github.com/apache/beam/blob/6f31e56fcaca5cbb71b90f631c6562a1f1471c08/sdks/python/apache_beam/coders/coder_impl.py#L451
This can still be done but it is a bit more complicated and probably not worth it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add flag for disabling dill check in coders. [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #36453:
URL: https://github.com/apache/beam/pull/36453#discussion_r2426416013
##########
sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py:
##########
@@ -411,13 +406,6 @@ def test_partition_files_dofn_size_split(self):
label='CheckSinglePartition')
-def maybe_skip(compat_version):
Review Comment:
```
self = <apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads testMethod=test_triggering_frequency_3>
is_streaming = True, with_auto_sharding = False, compat_version = '2.64.0'
@parameterized.expand([
param(is_streaming=False, with_auto_sharding=False, compat_version=None),
param(is_streaming=True, with_auto_sharding=False, compat_version=None),
param(is_streaming=True, with_auto_sharding=True, compat_version=None),
param(
is_streaming=True, with_auto_sharding=False, compat_version="2.64.0"),
param(
is_streaming=True, with_auto_sharding=True, compat_version="2.64.0"),
])
def test_triggering_frequency(
self, is_streaming, with_auto_sharding, compat_version):
> maybe_skip(compat_version)
E NameError: name 'maybe_skip' is not defined
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add flag for disabling dill check in coders. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36453:
URL: https://github.com/apache/beam/pull/36453#issuecomment-3387140185
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add flag for disabling dill check in coders. [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #36453:
URL: https://github.com/apache/beam/pull/36453#discussion_r2418008474
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -874,6 +874,18 @@ def _add_argparse_args(cls, parser):
'their condition met. Some operations, such as GroupByKey, disallow '
'this. This exists for cases where such loss is acceptable and for '
'backwards compatibility. See BEAM-9487.')
+ parser.add_argument(
+ '--force_cloudpickle_deterministic_coders',
Review Comment:
Actually as I was typing out the explanation I realized this flag wont work in practice (because this change will result in a new coder name and therefore failing update compat check in dataflow.
I added it so that tests that use update compat flags (like reshuffle tests) that for some reason have a DeterministicFastPrimitivesCoder (but dont actually trigger the pickle code path) fail because dill is not installed can be run in non-dill test suites.
Maybe it should not be a flag/pipeline option, and just a property that we hack on in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add flag for disabling dill check in coders. [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #36453:
URL: https://github.com/apache/beam/pull/36453#discussion_r2418008474
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -874,6 +874,18 @@ def _add_argparse_args(cls, parser):
'their condition met. Some operations, such as GroupByKey, disallow '
'this. This exists for cases where such loss is acceptable and for '
'backwards compatibility. See BEAM-9487.')
+ parser.add_argument(
+ '--force_cloudpickle_deterministic_coders',
Review Comment:
Actually as I was typing out the explanation I realized this flag wont work in practice.
I added it so that tests that use update compat flags (like reshuffle tests) that for some reason have a DeterministicFastPrimitivesCoder (but dont actually trigger the pickle code path) fail because dill is not installed. These test have to be run in the dill test suite even though they dont really need dill.
Maybe it should not be a flag, and just a property that we hack on in the test.
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -874,6 +874,18 @@ def _add_argparse_args(cls, parser):
'their condition met. Some operations, such as GroupByKey, disallow '
'this. This exists for cases where such loss is acceptable and for '
'backwards compatibility. See BEAM-9487.')
+ parser.add_argument(
+ '--force_cloudpickle_deterministic_coders',
Review Comment:
Actually as I was typing out the explanation I realized this flag wont work in practice.
I added it so that tests that use update compat flags (like reshuffle tests) that for some reason have a DeterministicFastPrimitivesCoder (but dont actually trigger the pickle code path) fail because dill is not installed. These test have to be run in the dill test suite even though they dont really need dill.
Maybe it should not be a flag/pipeline option, and just a property that we hack on in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add flag for disabling dill check in coders. [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #36453:
URL: https://github.com/apache/beam/pull/36453#discussion_r2418010009
##########
.github/workflows/beam_PreCommit_Python_Dill.yml:
##########
@@ -106,9 +106,9 @@ jobs:
arguments: |
-Pposargs="${{
contains(matrix.os, 'self-hosted') &&
- 'apache_beam/internal/ apache_beam/ml/ apache_beam/transforms/ apache_beam/typehints/ apache_beam/runners/portability/ -m (uses_dill and not require_docker_in_docker)' ||
- 'apache_beam/internal/ apache_beam/ml/ apache_beam/transforms/ apache_beam/typehints/ apache_beam/runners/portability/ -m (uses_dill and require_docker_in_docker)'
- }}" \
+ 'apache_beam/internal/pickler_test.py apache_beam/io/gcp/bigquery_file_loads_test.py apache_beam/options/pipeline_options_validator_test.py apache_beam/transforms/combinefn_lifecycle_test.py apache_beam/transforms/util_test.py apache_beam/transforms/ptransform_test.py apache_beam/typehints/schemas_test.py apache_beam/runners/portability/stager_test.py -m (uses_dill and not require_docker_in_docker)' ||
+ 'apache_beam/internal/pickler_test.py apache_beam/io/gcp/bigquery_file_loads_test.py apache_beam/options/pipeline_options_validator_test.py apache_beam/transforms/combinefn_lifecycle_test.py apache_beam/transforms/util_test.py apache_beam/transforms/ptransform_test.py apache_beam/typehints/schemas_test.py apache_beam/runners/portability/stager_test.py -m (uses_dill and require_docker_in_docker)'
Review Comment:
I did this because it took forever to collect tests in my dev environment so this is more efficient
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add flag for disabling dill check in coders. [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #36453:
URL: https://github.com/apache/beam/pull/36453#discussion_r2418018720
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -874,6 +874,18 @@ def _add_argparse_args(cls, parser):
'their condition met. Some operations, such as GroupByKey, disallow '
'this. This exists for cases where such loss is acceptable and for '
'backwards compatibility. See BEAM-9487.')
+ parser.add_argument(
+ '--force_cloudpickle_deterministic_coders',
Review Comment:
But to answer the original question: I made this a flag so that if someone knows they arent triggering the code path where the type is pickled using coders and are doing a pipeline update they don't need dill 0.3.1.1 installed.
This can still be done but it is a bit more complicated and probably not worth it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add flag for disabling dill check in coders. [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #36453:
URL: https://github.com/apache/beam/pull/36453#discussion_r2418018720
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -874,6 +874,18 @@ def _add_argparse_args(cls, parser):
'their condition met. Some operations, such as GroupByKey, disallow '
'this. This exists for cases where such loss is acceptable and for '
'backwards compatibility. See BEAM-9487.')
+ parser.add_argument(
+ '--force_cloudpickle_deterministic_coders',
Review Comment:
But to answer the original question: I made this a flag so that if someone knows they arent triggering the code path where the type is pickled using coders and are doing a pipeline update they don't need dill 0.3.1.1 installed. This can happen if they have a DeterministicFastPrimitivesCoder in the pipeline but their key is not a "special type" e.g. https://github.com/apache/beam/blob/6f31e56fcaca5cbb71b90f631c6562a1f1471c08/sdks/python/apache_beam/coders/coder_impl.py#L451
This can still be done (the coder name should still be DeterministicFastPrimitivesCoder, but the dill import + version check needs to be skipped)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org