You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "nikitagrover19 (via GitHub)" <gi...@apache.org> on 2026/01/19 05:34:53 UTC
[PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
nikitagrover19 opened a new pull request, #37345:
URL: https://github.com/apache/beam/pull/37345
Fixes #36201
This PR fixes the `WriteToPubSub` transform in the Python SDK to properly pass the `ordering_key` parameter to the Google Cloud Pub/Sub `publish()` method.
**The Problem**
While the `ordering_key` field was correctly serialized into the `PubsubMessage` protobuf, it was not being extracted and passed to `PublisherClient.publish()` in the `_PubSubWriteDoFn._flush()` method. This caused message ordering to not work even when users explicitly specified an `ordering_key`.
**The Solution:**
Modified `_PubSubWriteDoFn._flush()` to:
- Extract `ordering_key` from the deserialized `PubsubMessage`
- Pass it to `publish()` in both code paths (with attributes and data-only)
- Handle empty/missing `ordering_key` for backward compatibility
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r2703311041
##########
sdks/python/apache_beam/io/gcp/pubsub_test.py:
##########
@@ -1098,6 +1097,71 @@ def test_write_to_pubsub_with_attributes_no_overwrite(self, unused_mock):
Lineage.query(p.result.metrics(), Lineage.SINK),
set(["pubsub:topic:fakeprj.a_topic"]))
+ def test_write_messages_with_ordering_key(self, mock_pubsub):
+ """Test WriteToPubSub with ordering_key in messages."""
+ data = b'data'
+ ordering_key = 'order-123'
+ attributes = {'key': 'value'}
+ payloads = [PubsubMessage(data, attributes, ordering_key=ordering_key)]
+
+ options = PipelineOptions([])
+ options.view_as(StandardOptions).streaming = True
+ with TestPipeline(options=options) as p:
+ _ = (
+ p
+ | Create(payloads)
+ | WriteToPubSub(
+ 'projects/fakeprj/topics/a_topic', with_attributes=True))
+
+ # Verify that publish was called with ordering_key
+ mock_pubsub.return_value.publish.assert_called()
+ call_args = mock_pubsub.return_value.publish.call_args
+
+ # Check that ordering_key was passed as a keyword argument
+ self.assertIn('ordering_key', call_args.kwargs)
+ self.assertEqual(call_args.kwargs['ordering_key'], ordering_key)
+
+ def test_write_messages_with_ordering_key_no_attributes(self, mock_pubsub):
+ """Test WriteToPubSub with ordering_key but no attributes."""
+ data = b'data'
+ ordering_key = 'order-456'
+ payloads = [PubsubMessage(data, None, ordering_key=ordering_key)]
+
+ options = PipelineOptions([])
+ options.view_as(StandardOptions).streaming = True
+ with TestPipeline(options=options) as p:
+ _ = (
+ p
+ | Create(payloads)
+ | WriteToPubSub(
+ 'projects/fakeprj/topics/a_topic', with_attributes=True))
+
+ # Verify that publish was called with ordering_key
+ mock_pubsub.return_value.publish.assert_called()
+ call_args = mock_pubsub.return_value.publish.call_args
+
+ # Check that ordering_key was passed
+ self.assertIn('ordering_key', call_args.kwargs)
+ self.assertEqual(call_args.kwargs['ordering_key'], ordering_key)
+
+ def test_write_messages_without_ordering_key(self, mock_pubsub):
+ """Test WriteToPubSub without ordering_key (backward compatibility)."""
+ data = b'data'
+ attributes = {'key': 'value'}
+ payloads = [PubsubMessage(data, attributes)] # No ordering_key
+
+ options = PipelineOptions([])
+ options.view_as(StandardOptions).streaming = True
+ with TestPipeline(options=options) as p:
+ _ = (
+ p
+ | Create(payloads)
+ | WriteToPubSub(
+ 'projects/fakeprj/topics/a_topic', with_attributes=True))
+
+ # Verify that publish was called
+ mock_pubsub.return_value.publish.assert_called()
Review Comment:

To make this test more robust, you should also verify that `ordering_key` is not passed to the `publish` method when no ordering key is provided in the `PubsubMessage`. You can do this by checking the `call_args` of the mock.
```suggestion
# Verify that publish was called
mock_pubsub.return_value.publish.assert_called()
call_args = mock_pubsub.return_value.publish.call_args
self.assertNotIn('ordering_key', call_args.kwargs)
```
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,51 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
+ def test_batch_write_with_ordering_key(self):
+ """Test WriteToPubSub in batch mode with ordering keys."""
+ from apache_beam.options.pipeline_options import PipelineOptions
+ from apache_beam.options.pipeline_options import StandardOptions
+ from apache_beam.transforms import Create
+
+ # Create test messages with ordering keys
+ test_messages = [
+ PubsubMessage(
+ b'order_data001', {'attr': 'value1'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data002', {'attr': 'value2'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data003', {'attr': 'value3'}, ordering_key='key2')
+ ]
+
+ pipeline_options = PipelineOptions()
+ pipeline_options.view_as(StandardOptions).streaming = False
+
+ with TestPipeline(options=pipeline_options) as p:
+ messages = p | 'CreateMessages' >> Create(test_messages)
+ _ = messages | 'WriteToPubSub' >> WriteToPubSub(
+ self.output_topic.name, with_attributes=True)
+
+ # Verify messages were published
+ time.sleep(10)
+
+ response = self.sub_client.pull(
+ request={
+ "subscription": self.output_sub.name,
+ "max_messages": 10,
+ })
+
+ self.assertEqual(len(response.received_messages), len(test_messages))
+
+ # Verify ordering keys were preserved
+ for received_message in response.received_messages:
+ self.assertIn('ordering_key', dir(received_message.message))
+ self.sub_client.acknowledge(
+ request={
+ "subscription": self.output_sub.name,
+ "ack_ids": [received_message.ack_id],
+ })
Review Comment:

The current verification for `ordering_key` only checks for its presence. To make the test more robust, you should assert that the `ordering_key` has the expected value for each message.
Additionally, you can improve efficiency by acknowledging all messages in a single batch request after pulling them, rather than one by one inside the loop.
```suggestion
received_map = {msg.message.data: msg for msg in response.received_messages}
self.assertEqual(received_map[b'order_data001'].message.ordering_key, 'key1')
self.assertEqual(received_map[b'order_data002'].message.ordering_key, 'key1')
self.assertEqual(received_map[b'order_data003'].message.ordering_key, 'key2')
ack_ids = [msg.ack_id for msg in response.received_messages]
if ack_ids:
self.sub_client.acknowledge(
request={
"subscription": self.output_sub.name,
"ack_ids": ack_ids,
})
```
##########
sdks/python/apache_beam/io/gcp/pubsub.py:
##########
@@ -658,12 +658,22 @@ def _flush(self):
# Deserialize the protobuf to get the original PubsubMessage
pubsub_msg = PubsubMessage._from_proto_str(elem)
- # Publish with the correct data and attributes
+ # Publish with the correct data, attributes, and ordering_key
if self.with_attributes and pubsub_msg.attributes:
future = self._pub_client.publish(
- self._topic, pubsub_msg.data, **pubsub_msg.attributes)
+ self._topic,
+ pubsub_msg.data,
+ ordering_key=pubsub_msg.ordering_key
+ if pubsub_msg.ordering_key else '',
+ **pubsub_msg.attributes)
else:
- future = self._pub_client.publish(self._topic, pubsub_msg.data)
+ if pubsub_msg.ordering_key:
+ future = self._pub_client.publish(
+ self._topic,
+ pubsub_msg.data,
+ ordering_key=pubsub_msg.ordering_key)
+ else:
+ future = self._pub_client.publish(self._topic, pubsub_msg.data)
Review Comment:

The logic for publishing messages can be simplified to improve readability and reduce code duplication. You can determine the attributes and `ordering_key` to use and then have a single call to `self._pub_client.publish`. This makes the code easier to maintain.
```python
attributes = {}
if self.with_attributes and pubsub_msg.attributes:
attributes = pubsub_msg.attributes
if pubsub_msg.ordering_key:
future = self._pub_client.publish(
self._topic,
pubsub_msg.data,
ordering_key=pubsub_msg.ordering_key,
**attributes)
else:
future = self._pub_client.publish(
self._topic, pubsub_msg.data, **attributes)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-3800195642
assign set of reviewers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-3766521906
/gemini review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3011857315
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java:
##########
@@ -1727,7 +1727,10 @@ public void startBundle(StartBundleContext c) throws IOException {
this.pubsubClient =
getPubsubClientFactory()
.newClient(
- getTimestampAttribute(), null, c.getPipelineOptions().as(PubsubOptions.class));
+ getTimestampAttribute(),
+ null,
+ c.getPipelineOptions().as(PubsubOptions.class),
+ Write.this.getPubsubRootUrl());
Review Comment:
okay, thanks for checking, no changes needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3326594886
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,51 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
+ def test_batch_write_with_ordering_key(self):
+ """Test WriteToPubSub in batch mode with ordering keys."""
+ from apache_beam.options.pipeline_options import PipelineOptions
+ from apache_beam.options.pipeline_options import StandardOptions
+ from apache_beam.transforms import Create
+
+ # Create test messages with ordering keys
+ test_messages = [
+ PubsubMessage(
+ b'order_data001', {'attr': 'value1'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data002', {'attr': 'value2'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data003', {'attr': 'value3'}, ordering_key='key2')
+ ]
+
+ pipeline_options = PipelineOptions()
+ pipeline_options.view_as(StandardOptions).streaming = False
+
+ with TestPipeline(options=pipeline_options) as p:
+ messages = p | 'CreateMessages' >> Create(test_messages)
+ _ = messages | 'WriteToPubSub' >> WriteToPubSub(
+ self.output_topic.name, with_attributes=True)
+
+ # Verify messages were published
+ time.sleep(10)
+
+ response = self.sub_client.pull(
+ request={
+ "subscription": self.output_sub.name,
+ "max_messages": 10,
+ })
+
+ self.assertEqual(len(response.received_messages), len(test_messages))
Review Comment:
if not enough messages were received, could we try to wait and pull again (with some guardrails to eventually stop) , in case waiting 10 seconds was not sufficient ? This could help prevent test flakiness.
##########
sdks/python/apache_beam/io/gcp/pubsub.py:
##########
@@ -628,7 +638,10 @@ def __init__(self, transform):
def setup(self):
from google.cloud import pubsub
- self._pub_client = pubsub.PublisherClient()
+ self._pub_client = pubsub.PublisherClient(
+ publisher_options=pubsub.types.PublisherOptions(
+ enable_message_ordering=True,
Review Comment:
should enable_message_ordering be conditionally-enabled here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-3772432343
FYI: CI failures look flaky (PortableRunner gRPC `DEADLINE_EXCEEDED` / `Socket closed`).
I ran these locally and they pass:
* `apache_beam/yaml/yaml_transform_unit_test.py::ExpandPipelineTest::test_expand_pipeline_with_pipeline_key_only`
* `apache_beam/dataframe/convert_test.py`
* `ConvertTest::test_convert`
* `ConvertTest::test_convert_memoization_clears_cache`
* (full file run also passes, including with `--runner=PortableRunner`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn merged PR #37345:
URL: https://github.com/apache/beam/pull/37345
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3336132764
##########
sdks/python/apache_beam/io/gcp/pubsub.py:
##########
@@ -647,21 +660,29 @@ def _flush(self):
if not self._buffer:
return
- import time
-
# The elements in buffer are serialized protobuf bytes from the previous
# transforms. We need to deserialize them to extract data and attributes.
futures = []
for elem in self._buffer:
# Deserialize the protobuf to get the original PubsubMessage
pubsub_msg = PubsubMessage._from_proto_str(elem)
- # Publish with the correct data and attributes
+ # Publish with the correct data, attributes, and ordering_key
if self.with_attributes and pubsub_msg.attributes:
future = self._pub_client.publish(
- self._topic, pubsub_msg.data, **pubsub_msg.attributes)
+ self._topic,
+ pubsub_msg.data,
+ ordering_key=pubsub_msg.ordering_key
+ if pubsub_msg.ordering_key else '',
+ **pubsub_msg.attributes)
else:
- future = self._pub_client.publish(self._topic, pubsub_msg.data)
+ if pubsub_msg.ordering_key:
+ future = self._pub_client.publish(
+ self._topic,
+ pubsub_msg.data,
+ ordering_key=pubsub_msg.ordering_key)
+ else:
+ future = self._pub_client.publish(self._topic, pubsub_msg.data)
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3017691530
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,51 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
Review Comment:
> would you prefer it at pipeline construction time
We need to show the error at job submission, so at construction time
I think if we put the warning inside _PubSubWriteFun, then the warning emiting code will be serialized and become part of the pipeline definition, but won't get executed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-4268084774
Stopping reviewer notifications for this pull request: requested by reviewer. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r2997203614
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java:
##########
@@ -1727,7 +1727,10 @@ public void startBundle(StartBundleContext c) throws IOException {
this.pubsubClient =
getPubsubClientFactory()
.newClient(
- getTimestampAttribute(), null, c.getPipelineOptions().as(PubsubOptions.class));
+ getTimestampAttribute(),
+ null,
+ c.getPipelineOptions().as(PubsubOptions.class),
+ Write.this.getPubsubRootUrl());
Review Comment:
if `Write.this.getPubsubRootUrl` somehow returned Null, would newClient still initialize with the default (as per pipeline options)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r2997203614
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java:
##########
@@ -1727,7 +1727,10 @@ public void startBundle(StartBundleContext c) throws IOException {
this.pubsubClient =
getPubsubClientFactory()
.newClient(
- getTimestampAttribute(), null, c.getPipelineOptions().as(PubsubOptions.class));
+ getTimestampAttribute(),
+ null,
+ c.getPipelineOptions().as(PubsubOptions.class),
+ Write.this.getPubsubRootUrl());
Review Comment:
if `Write.this.getPubsubRootUrl` somehow returned Null, would newClient still initialize with the default (pipeline options)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-4229416160
Reminder, please take a look at this pr: @tvalentyn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-3840995890
Reminder, please take a look at this pr: @tvalentyn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3023958632
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,62 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
+ def test_batch_write_with_ordering_key(self):
+ """Test WriteToPubSub in batch mode with ordering keys.
+ This test only applies to the Direct Runner. The ordering key fix in
Review Comment:
> The ordering key fix in _PubSubWriteDoFn._flush() does not apply to Dataflow,
Wording suggestion: I'd avoid low-level details and say smth like:
Dataflow Native Pubsub Sink doesn't support ordering_key: <link with details/open issue>, therefore this test only applies to runners that use Beam's Python WriteToPubSub Sink.
To use ordering_key, Dataflow users should use the XLang WriteToPubSub path instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r2996712381
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java:
##########
@@ -1727,7 +1727,10 @@ public void startBundle(StartBundleContext c) throws IOException {
this.pubsubClient =
getPubsubClientFactory()
.newClient(
- getTimestampAttribute(), null, c.getPipelineOptions().as(PubsubOptions.class));
+ getTimestampAttribute(),
+ null,
+ c.getPipelineOptions().as(PubsubOptions.class),
+ Write.this.getPubsubRootUrl());
Review Comment:
for my education, why was this necessary?
##########
sdks/python/apache_beam/io/external/gcp/pubsub.py:
##########
@@ -150,18 +152,24 @@ def __init__(
in a ReadFromPubSub PTransform to deduplicate messages.
timestamp_attribute: If set, will set an attribute for each Cloud Pub/Sub
message with the given name and the message's publish time as the value.
+ publish_with_ordering_key: If True, enables ordering key support when
+ publishing messages. The ordering key must be set on each
+ PubsubMessage via the ``ordering_key`` attribute. Requires
+ messages to be routed to the same region.
"""
self.params = WriteToPubsubSchema(
topic=topic,
id_label=id_label,
# with_attributes=with_attributes,
- timestamp_attribute=timestamp_attribute)
+ timestamp_attribute=timestamp_attribute,
+ publish_with_ordering_key=publish_with_ordering_key)
self.expansion_service = expansion_service
self.with_attributes = with_attributes
def expand(self, pvalue):
if self.with_attributes:
- pcoll = pvalue | 'ToProto' >> Map(pubsub.WriteToPubSub.to_proto_str)
+ pcoll = pvalue | 'ToProto' >> Map(
Review Comment:
Re: `Map(pubsub.WriteToPubSub.to_proto_str)` - was this a typo that was meant to be `Map(pubsub.WriteToPubSub.message_to_proto_str)` ? The latter seems to have some type checking which probably wouldn't hurt.
##########
sdks/python/apache_beam/io/external/gcp/pubsub.py:
##########
@@ -150,18 +152,24 @@ def __init__(
in a ReadFromPubSub PTransform to deduplicate messages.
timestamp_attribute: If set, will set an attribute for each Cloud Pub/Sub
message with the given name and the message's publish time as the value.
+ publish_with_ordering_key: If True, enables ordering key support when
+ publishing messages. The ordering key must be set on each
+ PubsubMessage via the ``ordering_key`` attribute. Requires
Review Comment:
> Requires
messages to be routed to the same region.
What does this mean?
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,51 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
Review Comment:
this will work only in direct runner, but not in dataflow runner - correct?
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,51 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
Review Comment:
also wondering, if we can have a warning for Dataflow that that advises users to use xlang version if they wish to use ordering key. IIRC we have something like that in Java SDK?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3422749075
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,96 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
+ def test_batch_write_with_ordering_key(self):
+ """Test WriteToPubSub in batch mode with ordering keys.
+
+ Dataflow's Native Pub/Sub Sink does not support ordering_key
+ (see https://github.com/apache/beam/issues/36201), so this test
+ only applies to runners using Beam's Python WriteToPubSub Sink.
+ Dataflow users should use the XLang WriteToPubSub path instead
+ (apache_beam.io.external.gcp.pubsub.WriteToPubSub with
+ publish_with_ordering_key=True).
+ """
+ if self.runner_name == 'TestDataflowRunner':
+ self.skipTest(
+ 'Dataflow Native PubSub Sink does not support ordering_key '
+ '(see https://github.com/apache/beam/issues/36201). '
+ 'Use apache_beam.io.external.gcp.pubsub.WriteToPubSub '
+ 'with publish_with_ordering_key=True instead.')
+ from google.pubsub_v1.types import Subscription
+
+ from apache_beam.options.pipeline_options import PipelineOptions
+ from apache_beam.options.pipeline_options import StandardOptions
+ from apache_beam.transforms import Create
+
+ ordering_topic = self.pub_client.create_topic(
+ name=self.pub_client.topic_path(
+ self.project, 'psit_topic_ordering' + self.uuid))
+ ordering_sub = self.sub_client.create_subscription(
+ request=Subscription(
+ name=self.sub_client.subscription_path(
+ self.project, 'psit_sub_ordering' + self.uuid),
+ topic=ordering_topic.name,
+ enable_message_ordering=True,
+ ))
+ time.sleep(10)
+
+ try:
+ test_messages = [
+ PubsubMessage(
+ b'order_data001', {'attr': 'value1'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data002', {'attr': 'value2'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data003', {'attr': 'value3'}, ordering_key='key2'),
+ ]
+
+ pipeline_options = PipelineOptions()
+ pipeline_options.view_as(StandardOptions).streaming = False
+
+ with TestPipeline(options=pipeline_options) as p:
+ messages = p | 'CreateMessages' >> Create(test_messages)
+ _ = messages | 'WriteToPubSub' >> WriteToPubSub(
+ ordering_topic.name, with_attributes=True)
Review Comment:
makes sense, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-4268017222
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @damccorm for label python.
R: @ahmedabu98 for label java.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3336246780
##########
sdks/python/apache_beam/io/gcp/pubsub.py:
##########
@@ -573,6 +573,7 @@ def __init__(self, transform):
self.id_label = transform.id_label
self.timestamp_attribute = transform.timestamp_attribute
self.with_attributes = transform.with_attributes
+ self.with_ordering = True
Review Comment:
should this be initialized based on the fact whether user attemts to use this feature or requests it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-4547112958
The 5 failing tests (VertexAITextEmbeddingsTest, VertexAIImageEmbeddingsTest, VertexAIInference) are all failing with ERROR: No matching distribution found for tensorflow>=2.12.0, this is a pre-existing CI infrastructure issue unrelated to this PR. None of the files I modified are in apache_beam/ml/.
The PreCommit Python Lint check is now passing.
Happy to address any other reviewer feedback. @tvalentyn could you take a look when you get a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-4536254343
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/37345?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.00000%` with `2 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 58.73%. Comparing base ([`1ca1faf`](https://app.codecov.io/gh/apache/beam/commit/1ca1fafcd62b0e3f1bd6a071842cab7566366d1c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7434bdb`](https://app.codecov.io/gh/apache/beam/commit/7434bdb1bb9d204afa40a477e683fc0522cb9e5c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 55 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/beam/pull/37345?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [sdks/python/apache\_beam/io/external/gcp/pubsub.py](https://app.codecov.io/gh/apache/beam/pull/37345?src=pr&el=tree&filepath=sdks%2Fpython%2Fapache_beam%2Fio%2Fexternal%2Fgcp%2Fpubsub.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZXh0ZXJuYWwvZ2NwL3B1YnN1Yi5weQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/beam/pull/37345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [sdks/python/apache\_beam/io/gcp/pubsub.py](https://app.codecov.io/gh/apache/beam/pull/37345?src=pr&el=tree&filepath=sdks%2Fpython%2Fapache_beam%2Fio%2Fgcp%2Fpubsub.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | 88.88% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/beam/pull/37345?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #37345 +/- ##
=========================================
Coverage 58.73% 58.73%
Complexity 7418 7418
=========================================
Files 1649 1649
Lines 212369 212444 +75
Branches 6330 6330
=========================================
+ Hits 124727 124782 +55
- Misses 83095 83115 +20
Partials 4547 4547
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/37345/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [python](https://app.codecov.io/gh/apache/beam/pull/37345/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.88% <80.00%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/beam/pull/37345?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3415677614
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,96 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
+ def test_batch_write_with_ordering_key(self):
+ """Test WriteToPubSub in batch mode with ordering keys.
+
+ Dataflow's Native Pub/Sub Sink does not support ordering_key
+ (see https://github.com/apache/beam/issues/36201), so this test
+ only applies to runners using Beam's Python WriteToPubSub Sink.
+ Dataflow users should use the XLang WriteToPubSub path instead
+ (apache_beam.io.external.gcp.pubsub.WriteToPubSub with
+ publish_with_ordering_key=True).
+ """
+ if self.runner_name == 'TestDataflowRunner':
+ self.skipTest(
+ 'Dataflow Native PubSub Sink does not support ordering_key '
+ '(see https://github.com/apache/beam/issues/36201). '
+ 'Use apache_beam.io.external.gcp.pubsub.WriteToPubSub '
+ 'with publish_with_ordering_key=True instead.')
+ from google.pubsub_v1.types import Subscription
+
+ from apache_beam.options.pipeline_options import PipelineOptions
+ from apache_beam.options.pipeline_options import StandardOptions
+ from apache_beam.transforms import Create
+
+ ordering_topic = self.pub_client.create_topic(
+ name=self.pub_client.topic_path(
+ self.project, 'psit_topic_ordering' + self.uuid))
+ ordering_sub = self.sub_client.create_subscription(
+ request=Subscription(
+ name=self.sub_client.subscription_path(
+ self.project, 'psit_sub_ordering' + self.uuid),
+ topic=ordering_topic.name,
+ enable_message_ordering=True,
+ ))
+ time.sleep(10)
+
+ try:
+ test_messages = [
+ PubsubMessage(
+ b'order_data001', {'attr': 'value1'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data002', {'attr': 'value2'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data003', {'attr': 'value3'}, ordering_key='key2'),
+ ]
+
+ pipeline_options = PipelineOptions()
+ pipeline_options.view_as(StandardOptions).streaming = False
+
+ with TestPipeline(options=pipeline_options) as p:
+ messages = p | 'CreateMessages' >> Create(test_messages)
+ _ = messages | 'WriteToPubSub' >> WriteToPubSub(
+ ordering_topic.name, with_attributes=True)
Review Comment:
thanks, do you have thoughts on why the test didn't catch this omission before?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3390670517
##########
sdks/python/apache_beam/io/gcp/pubsub.py:
##########
@@ -404,9 +406,13 @@ def __init__(
in a ReadFromPubSub PTransform to deduplicate messages.
timestamp_attribute: If set, will set an attribute for each Cloud Pub/Sub
message with the given name and the message's publish time as the value.
+ enable_message_ordering: If True, enables message ordering on the
Review Comment:
let's use consistent naming, sounds like we are using `publish_with_ordering_key` in Java and XLang so we can stick with that.
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,96 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
+ def test_batch_write_with_ordering_key(self):
+ """Test WriteToPubSub in batch mode with ordering keys.
+
+ Dataflow's Native Pub/Sub Sink does not support ordering_key
+ (see https://github.com/apache/beam/issues/36201), so this test
+ only applies to runners using Beam's Python WriteToPubSub Sink.
+ Dataflow users should use the XLang WriteToPubSub path instead
+ (apache_beam.io.external.gcp.pubsub.WriteToPubSub with
+ publish_with_ordering_key=True).
+ """
+ if self.runner_name == 'TestDataflowRunner':
+ self.skipTest(
+ 'Dataflow Native PubSub Sink does not support ordering_key '
Review Comment:
nit: given this is a test, we can stop after the first line.
##########
sdks/python/apache_beam/io/gcp/pubsub_test.py:
##########
@@ -1098,6 +1097,71 @@ def test_write_to_pubsub_with_attributes_no_overwrite(self, unused_mock):
Lineage.query(p.result.metrics(), Lineage.SINK),
set(["pubsub:topic:fakeprj.a_topic"]))
+ def test_write_messages_with_ordering_key(self, mock_pubsub):
+ """Test WriteToPubSub with ordering_key in messages."""
+ data = b'data'
+ ordering_key = 'order-123'
+ attributes = {'key': 'value'}
+ payloads = [PubsubMessage(data, attributes, ordering_key=ordering_key)]
+
+ options = PipelineOptions([])
+ options.view_as(StandardOptions).streaming = True
+ with TestPipeline(options=options) as p:
+ _ = (
+ p
+ | Create(payloads)
+ | WriteToPubSub(
+ 'projects/fakeprj/topics/a_topic', with_attributes=True))
+
+ # Verify that publish was called with ordering_key
+ mock_pubsub.return_value.publish.assert_called()
+ call_args = mock_pubsub.return_value.publish.call_args
+
+ # Check that ordering_key was passed as a keyword argument
+ self.assertIn('ordering_key', call_args.kwargs)
+ self.assertEqual(call_args.kwargs['ordering_key'], ordering_key)
+
+ def test_write_messages_with_ordering_key_no_attributes(self, mock_pubsub):
+ """Test WriteToPubSub with ordering_key but no attributes."""
+ data = b'data'
+ ordering_key = 'order-456'
+ payloads = [PubsubMessage(data, None, ordering_key=ordering_key)]
+
+ options = PipelineOptions([])
+ options.view_as(StandardOptions).streaming = True
+ with TestPipeline(options=options) as p:
+ _ = (
+ p
+ | Create(payloads)
+ | WriteToPubSub(
+ 'projects/fakeprj/topics/a_topic', with_attributes=True))
+
+ # Verify that publish was called with ordering_key
+ mock_pubsub.return_value.publish.assert_called()
+ call_args = mock_pubsub.return_value.publish.call_args
+
+ # Check that ordering_key was passed
+ self.assertIn('ordering_key', call_args.kwargs)
+ self.assertEqual(call_args.kwargs['ordering_key'], ordering_key)
+
+ def test_write_messages_without_ordering_key(self, mock_pubsub):
+ """Test WriteToPubSub without ordering_key (backward compatibility)."""
+ data = b'data'
+ attributes = {'key': 'value'}
+ payloads = [PubsubMessage(data, attributes)] # No ordering_key
+
+ options = PipelineOptions([])
+ options.view_as(StandardOptions).streaming = True
+ with TestPipeline(options=options) as p:
+ _ = (
+ p
+ | Create(payloads)
+ | WriteToPubSub(
+ 'projects/fakeprj/topics/a_topic', with_attributes=True))
+
+ # Verify that publish was called
+ mock_pubsub.return_value.publish.assert_called()
Review Comment:
+1
##########
sdks/python/apache_beam/io/gcp/pubsub.py:
##########
@@ -430,7 +436,16 @@ def bytes_to_proto_str(element: Union[bytes, str]) -> bytes:
def expand(self, pcoll):
# Store pipeline options for use in DoFn
self.pipeline_options = pcoll.pipeline.options if pcoll.pipeline else None
-
+ # Warn Dataflow users to use the XLang path for ordering key support,
+ # since _PubSubWriteDoFn._flush() is not used by Dataflow's implementation.
+ runner = self.pipeline_options.get_all_options().get(
+ 'runner', '') if self.pipeline_options else ''
+ if 'Dataflow' in str(runner):
Review Comment:
should we print this warning only if message ordering is enabled?
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,96 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
+ def test_batch_write_with_ordering_key(self):
+ """Test WriteToPubSub in batch mode with ordering keys.
+
+ Dataflow's Native Pub/Sub Sink does not support ordering_key
+ (see https://github.com/apache/beam/issues/36201), so this test
+ only applies to runners using Beam's Python WriteToPubSub Sink.
+ Dataflow users should use the XLang WriteToPubSub path instead
+ (apache_beam.io.external.gcp.pubsub.WriteToPubSub with
+ publish_with_ordering_key=True).
+ """
+ if self.runner_name == 'TestDataflowRunner':
+ self.skipTest(
+ 'Dataflow Native PubSub Sink does not support ordering_key '
+ '(see https://github.com/apache/beam/issues/36201). '
+ 'Use apache_beam.io.external.gcp.pubsub.WriteToPubSub '
+ 'with publish_with_ordering_key=True instead.')
+ from google.pubsub_v1.types import Subscription
+
+ from apache_beam.options.pipeline_options import PipelineOptions
+ from apache_beam.options.pipeline_options import StandardOptions
+ from apache_beam.transforms import Create
+
+ ordering_topic = self.pub_client.create_topic(
+ name=self.pub_client.topic_path(
+ self.project, 'psit_topic_ordering' + self.uuid))
+ ordering_sub = self.sub_client.create_subscription(
+ request=Subscription(
+ name=self.sub_client.subscription_path(
+ self.project, 'psit_sub_ordering' + self.uuid),
+ topic=ordering_topic.name,
+ enable_message_ordering=True,
+ ))
+ time.sleep(10)
+
+ try:
+ test_messages = [
+ PubsubMessage(
+ b'order_data001', {'attr': 'value1'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data002', {'attr': 'value2'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data003', {'attr': 'value3'}, ordering_key='key2'),
+ ]
+
+ pipeline_options = PipelineOptions()
+ pipeline_options.view_as(StandardOptions).streaming = False
+
+ with TestPipeline(options=pipeline_options) as p:
+ messages = p | 'CreateMessages' >> Create(test_messages)
+ _ = messages | 'WriteToPubSub' >> WriteToPubSub(
+ ordering_topic.name, with_attributes=True)
Review Comment:
We need to enable ordering here, right? Does the test pass without it? If so, how? Maybe the test isn't running?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-4268075324
stop reviewer notifications
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-4272171627
Seeing this error in postcommit suites:
```
2026-04-14T11:05:36.1052305Z [31m[1mERROR [0m apache_beam.runners.common:common.py:1608 Cannot publish a message with an ordering key when message ordering is not enabled. [while running 'WriteToPubSub/ParDo(_PubSubWriteDoFn)']
2026-04-14T11:05:36.1053051Z Traceback (most recent call last):
2026-04-14T11:05:36.1053753Z File "/runner/_work/beam/beam/sdks/python/apache_beam/runners/common.py", line 1555, in _invoke_bundle_method
2026-04-14T11:05:36.1054272Z bundle_method()
2026-04-14T11:05:36.1054798Z File "/runner/_work/beam/beam/sdks/python/apache_beam/runners/common.py", line 622, in invoke_finish_bundle
2026-04-14T11:05:36.1055362Z self.signature.finish_bundle_method.method_value())
2026-04-14T11:05:36.1055896Z File "/runner/_work/beam/beam/sdks/python/apache_beam/io/gcp/pubsub.py", line 656, in finish_bundle
2026-04-14T11:05:36.1056356Z self._flush()
2026-04-14T11:05:36.1056773Z File "/runner/_work/beam/beam/sdks/python/apache_beam/io/gcp/pubsub.py", line 673, in _flush
2026-04-14T11:05:36.1057242Z future = self._pub_client.publish(
2026-04-14T11:05:36.1059887Z File "/runner/_work/beam/beam/build/gradleenv/417525523/lib/python3.10/site-packages/google/cloud/pubsub_v1/publisher/client.py", line 375, in publish
2026-04-14T11:05:36.1060538Z raise ValueError(
2026-04-14T11:05:36.1060972Z ValueError: Cannot publish a message with an ordering key when message ordering is not enabled.
2026-04-14T11:05:36.1061819Z [31m[1mERROR [0m apache_beam.runners.worker.sdk_worker:sdk_worker.py:320 Error processing instruction inst002. Original traceback is
2026-04-14T11:05:36.1062395Z Traceback (most recent call last):
2026-04-14T11:05:36.1062938Z File "/runner/_work/beam/beam/sdks/python/apache_beam/runners/common.py", line 1555, in _invoke_bundle_method
2026-04-14T11:05:36.1063570Z bundle_method()
2026-04-14T11:05:36.1064057Z File "/runner/_work/beam/beam/sdks/python/apache_beam/runners/common.py", line 622, in invoke_finish_bundle
2026-04-14T11:05:36.1064617Z self.signature.finish_bundle_method.method_value())
2026-04-14T11:05:36.1065236Z File "/runner/_work/beam/beam/sdks/python/apache_beam/io/gcp/pubsub.py", line 656, in finish_bundle
2026-04-14T11:05:36.1065695Z self._flush()
2026-04-14T11:05:36.1066112Z File "/runner/_work/beam/beam/sdks/python/apache_beam/io/gcp/pubsub.py", line 673, in _flush
2026-04-14T11:05:36.1066575Z future = self._pub_client.publish(
2026-04-14T11:05:36.1067326Z File "/runner/_work/beam/beam/build/gradleenv/417525523/lib/python3.10/site-packages/google/cloud/pubsub_v1/publisher/client.py", line 375, in publish
2026-04-14T11:05:36.1067994Z raise ValueError(
2026-04-14T11:05:36.1068424Z ValueError: Cannot publish a message with an ordering key when message ordering is not enabled.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-4229616751
@tvalentyn all CI failures are pre-existing and unrelated to this PR:
Python ML failures: Milvus server connection timeouts
Python Docker / Integration 3.14: infrastructure timeouts on self-hosted runners
Java GCP IO Direct / Java Examples Dataflow Java21: self-hosted runner lost communication
Python 3.11/3.12 / YAML Xlang: pre-existing gRPC socket flakes
All pubsub-related checks are passing. Also triggering the PostCommit Python suite as requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r2766449810
##########
sdks/python/apache_beam/io/gcp/pubsub.py:
##########
@@ -658,12 +658,22 @@ def _flush(self):
# Deserialize the protobuf to get the original PubsubMessage
pubsub_msg = PubsubMessage._from_proto_str(elem)
- # Publish with the correct data and attributes
+ # Publish with the correct data, attributes, and ordering_key
if self.with_attributes and pubsub_msg.attributes:
future = self._pub_client.publish(
- self._topic, pubsub_msg.data, **pubsub_msg.attributes)
+ self._topic,
+ pubsub_msg.data,
+ ordering_key=pubsub_msg.ordering_key
+ if pubsub_msg.ordering_key else '',
+ **pubsub_msg.attributes)
else:
- future = self._pub_client.publish(self._topic, pubsub_msg.data)
+ if pubsub_msg.ordering_key:
+ future = self._pub_client.publish(
+ self._topic,
+ pubsub_msg.data,
+ ordering_key=pubsub_msg.ordering_key)
+ else:
+ future = self._pub_client.publish(self._topic, pubsub_msg.data)
Review Comment:

The logic for publishing messages can be simplified to reduce code duplication and improve readability. Instead of separate code paths for messages with and without attributes, you can build a dictionary of keyword arguments for the `publish` call. This makes the code cleaner and easier to maintain.
```python
publish_kwargs = {}
if self.with_attributes and pubsub_msg.attributes:
publish_kwargs.update(pubsub_msg.attributes)
if pubsub_msg.ordering_key:
publish_kwargs['ordering_key'] = pubsub_msg.ordering_key
future = self._pub_client.publish(
self._topic, pubsub_msg.data, **publish_kwargs)
```
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,51 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
+ def test_batch_write_with_ordering_key(self):
+ """Test WriteToPubSub in batch mode with ordering keys."""
+ from apache_beam.options.pipeline_options import PipelineOptions
+ from apache_beam.options.pipeline_options import StandardOptions
+ from apache_beam.transforms import Create
+
+ # Create test messages with ordering keys
+ test_messages = [
+ PubsubMessage(
+ b'order_data001', {'attr': 'value1'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data002', {'attr': 'value2'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data003', {'attr': 'value3'}, ordering_key='key2')
+ ]
+
+ pipeline_options = PipelineOptions()
+ pipeline_options.view_as(StandardOptions).streaming = False
+
+ with TestPipeline(options=pipeline_options) as p:
+ messages = p | 'CreateMessages' >> Create(test_messages)
+ _ = messages | 'WriteToPubSub' >> WriteToPubSub(
+ self.output_topic.name, with_attributes=True)
+
+ # Verify messages were published
+ time.sleep(10)
+
+ response = self.sub_client.pull(
+ request={
+ "subscription": self.output_sub.name,
+ "max_messages": 10,
+ })
+
+ self.assertEqual(len(response.received_messages), len(test_messages))
+
+ # Verify ordering keys were preserved
+ for received_message in response.received_messages:
+ self.assertIn('ordering_key', dir(received_message.message))
+ self.sub_client.acknowledge(
+ request={
+ "subscription": self.output_sub.name,
+ "ack_ids": [received_message.ack_id],
+ })
Review Comment:

The assertion in this test is quite weak. It only checks for the presence of the `ordering_key` attribute on the message object using `dir()`, but doesn't verify its value. A more robust test would be to check that the received messages have the correct ordering keys and attributes corresponding to the messages that were sent. Also, acknowledging messages one by one in a loop is inefficient; it's better to collect all `ack_id`s and acknowledge them in a single call.
```suggestion
# Verify ordering keys and attributes were preserved
received_msgs_map = {
msg.message.data: msg.message
for msg in response.received_messages
}
for sent_msg in test_messages:
self.assertIn(sent_msg.data, received_msgs_map)
received_msg = received_msgs_map[sent_msg.data]
self.assertEqual(received_msg.ordering_key, sent_msg.ordering_key)
self.assertEqual(dict(received_msg.attributes), sent_msg.attributes)
# Acknowledge all messages at once for efficiency
ack_ids = [msg.ack_id for msg in response.received_messages]
if ack_ids:
self.sub_client.acknowledge(
request={
"subscription": self.output_sub.name,
"ack_ids": ack_ids,
})
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3011854800
##########
sdks/python/apache_beam/io/external/gcp/pubsub.py:
##########
@@ -150,18 +152,24 @@ def __init__(
in a ReadFromPubSub PTransform to deduplicate messages.
timestamp_attribute: If set, will set an attribute for each Cloud Pub/Sub
message with the given name and the message's publish time as the value.
+ publish_with_ordering_key: If True, enables ordering key support when
+ publishing messages. The ordering key must be set on each
+ PubsubMessage via the ``ordering_key`` attribute. Requires
+ messages to be routed to the same region.
"""
self.params = WriteToPubsubSchema(
topic=topic,
id_label=id_label,
# with_attributes=with_attributes,
- timestamp_attribute=timestamp_attribute)
+ timestamp_attribute=timestamp_attribute,
+ publish_with_ordering_key=publish_with_ordering_key)
self.expansion_service = expansion_service
self.with_attributes = with_attributes
def expand(self, pvalue):
if self.with_attributes:
- pcoll = pvalue | 'ToProto' >> Map(pubsub.WriteToPubSub.to_proto_str)
+ pcoll = pvalue | 'ToProto' >> Map(
Review Comment:
sounds good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-4268076910
(since this is still a draft)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-4693762854
waiting on author
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3343132120
##########
sdks/python/apache_beam/io/gcp/pubsub.py:
##########
@@ -638,10 +639,13 @@ def __init__(self, transform):
def setup(self):
from google.cloud import pubsub
- self._pub_client = pubsub.PublisherClient(
- publisher_options=pubsub.types.PublisherOptions(
- enable_message_ordering=True,
- ))
+ if self.with_attributes:
Review Comment:
Good catch, thanks!
Updated this to use `self.with_ordering` instead of `self.with_attributes` in `setup()`.
Also changed the initialization of `with_ordering` in `__init__` to derive from `transform.with_attributes`, so ordering is enabled only when the feature is actually requested rather than being hardcoded to True.
Let me know if you’d prefer a different signal for enabling ordering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-4721500055
LGTM, triggered tests suites, waiting for them to pass. will plan to merge afterwards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3305801314
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/ExternalWrite.java:
##########
@@ -85,6 +90,9 @@ public PTransform<PCollection<byte[]>, PDone> buildExternal(Configuration config
if (config.timestampAttribute != null) {
writeBuilder.setTimestampAttribute(config.timestampAttribute);
}
+ if (config.publishWithOrderingKey) {
+ writeBuilder.setPublishWithOrderingKey(true);
+ }
Review Comment:

Instead of conditionally setting the ordering key option only when it is true, we can directly pass the boolean configuration value to the builder. This simplifies the code and avoids unnecessary conditional blocks.
```java
writeBuilder.setPublishWithOrderingKey(config.publishWithOrderingKey);
```
##########
sdks/python/apache_beam/io/gcp/pubsub.py:
##########
@@ -647,21 +660,29 @@ def _flush(self):
if not self._buffer:
return
- import time
-
# The elements in buffer are serialized protobuf bytes from the previous
# transforms. We need to deserialize them to extract data and attributes.
futures = []
for elem in self._buffer:
# Deserialize the protobuf to get the original PubsubMessage
pubsub_msg = PubsubMessage._from_proto_str(elem)
- # Publish with the correct data and attributes
+ # Publish with the correct data, attributes, and ordering_key
if self.with_attributes and pubsub_msg.attributes:
future = self._pub_client.publish(
- self._topic, pubsub_msg.data, **pubsub_msg.attributes)
+ self._topic,
+ pubsub_msg.data,
+ ordering_key=pubsub_msg.ordering_key
+ if pubsub_msg.ordering_key else '',
+ **pubsub_msg.attributes)
else:
- future = self._pub_client.publish(self._topic, pubsub_msg.data)
+ if pubsub_msg.ordering_key:
+ future = self._pub_client.publish(
+ self._topic,
+ pubsub_msg.data,
+ ordering_key=pubsub_msg.ordering_key)
+ else:
+ future = self._pub_client.publish(self._topic, pubsub_msg.data)
Review Comment:

The current nested `if-else` blocks for publishing with/without attributes and ordering keys can be significantly simplified and made more maintainable by constructing a keyword arguments dictionary dynamically.
```python
kwargs = {}
if self.with_attributes and pubsub_msg.attributes:
kwargs.update(pubsub_msg.attributes)
if pubsub_msg.ordering_key:
kwargs['ordering_key'] = pubsub_msg.ordering_key
future = self._pub_client.publish(self._topic, pubsub_msg.data, **kwargs)
```
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,96 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
+ def test_batch_write_with_ordering_key(self):
+ """Test WriteToPubSub in batch mode with ordering keys.
+
+ Dataflow's Native Pub/Sub Sink does not support ordering_key
+ (see https://github.com/apache/beam/issues/36201), so this test
+ only applies to runners using Beam's Python WriteToPubSub Sink.
+ Dataflow users should use the XLang WriteToPubSub path instead
+ (apache_beam.io.external.gcp.pubsub.WriteToPubSub with
+ publish_with_ordering_key=True).
+ """
+ if self.runner_name == 'TestDataflowRunner':
+ self.skipTest(
+ 'Dataflow Native PubSub Sink does not support ordering_key '
+ '(see https://github.com/apache/beam/issues/36201). '
+ 'Use apache_beam.io.external.gcp.pubsub.WriteToPubSub '
+ 'with publish_with_ordering_key=True instead.')
+ from google.pubsub_v1.types import Subscription
+
+ from apache_beam.options.pipeline_options import PipelineOptions
+ from apache_beam.options.pipeline_options import StandardOptions
+ from apache_beam.transforms import Create
+
+ ordering_topic = self.pub_client.create_topic(
+ name=self.pub_client.topic_path(
+ self.project, 'psit_topic_ordering' + self.uuid))
+ ordering_sub = self.sub_client.create_subscription(
+ request=Subscription(
+ name=self.sub_client.subscription_path(
+ self.project, 'psit_sub_ordering' + self.uuid),
+ topic=ordering_topic.name,
+ enable_message_ordering=True,
+ ))
+ time.sleep(10)
+
+ try:
+ test_messages = [
+ PubsubMessage(
+ b'order_data001', {'attr': 'value1'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data002', {'attr': 'value2'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data003', {'attr': 'value3'}, ordering_key='key2'),
+ ]
+
+ pipeline_options = PipelineOptions()
+ pipeline_options.view_as(StandardOptions).streaming = False
+
+ with TestPipeline(options=pipeline_options) as p:
+ messages = p | 'CreateMessages' >> Create(test_messages)
+ _ = messages | 'WriteToPubSub' >> WriteToPubSub(
+ ordering_topic.name, with_attributes=True)
+
+ time.sleep(10)
+
+ # Retry pulling to handle PubSub delivery delays
+ received_messages = []
+ deadline = time.time() + 60 # wait up to 60 seconds
+ while time.time() < deadline:
+ response = self.sub_client.pull(
+ request={
+ 'subscription': ordering_sub.name,
+ 'max_messages': 10,
+ })
+ received_messages.extend(response.received_messages)
+ if len(received_messages) >= len(test_messages):
+ break
+ time.sleep(5)
+
+ self.assertEqual(len(received_messages), len(test_messages))
+
+ received_map = {
+ msg.message.data: msg.message
+ for msg in received_messages
+ }
+ self.assertEqual(received_map[b'order_data001'].ordering_key, 'key1')
+ self.assertEqual(received_map[b'order_data002'].ordering_key, 'key1')
+ self.assertEqual(received_map[b'order_data003'].ordering_key, 'key2')
+
+ ack_ids = [msg.ack_id for msg in received_messages]
+ self.sub_client.acknowledge(
Review Comment:

To make the integration test robust against potential duplicate messages delivered by Pub/Sub (which could cause the pull loop to exit prematurely before all unique messages are received), we can track received messages in a dictionary keyed by message data. This also simplifies the assertions and acknowledgment logic.
```python
received_messages = {}
deadline = time.time() + 60 # wait up to 60 seconds
while time.time() < deadline:
response = self.sub_client.pull(
request={
'subscription': ordering_sub.name,
'max_messages': 10,
})
for msg in response.received_messages:
received_messages[msg.message.data] = msg
if len(received_messages) >= len(test_messages):
break
time.sleep(5)
self.assertEqual(len(received_messages), len(test_messages))
self.assertEqual(received_messages[b'order_data001'].message.ordering_key, 'key1')
self.assertEqual(received_messages[b'order_data002'].message.ordering_key, 'key1')
self.assertEqual(received_messages[b'order_data003'].message.ordering_key, 'key2')
ack_ids = [msg.ack_id for msg in received_messages.values()]
self.sub_client.acknowledge(
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-4579378706
Overall LGTM, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-3800203722
Assigning reviewers:
R: @tvalentyn for label python.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3326597895
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/ExternalWrite.java:
##########
@@ -85,6 +90,9 @@ public PTransform<PCollection<byte[]>, PDone> buildExternal(Configuration config
if (config.timestampAttribute != null) {
writeBuilder.setTimestampAttribute(config.timestampAttribute);
}
+ if (config.publishWithOrderingKey) {
+ writeBuilder.setPublishWithOrderingKey(true);
+ }
Review Comment:
+1
##########
sdks/python/apache_beam/io/gcp/pubsub.py:
##########
@@ -647,21 +660,29 @@ def _flush(self):
if not self._buffer:
return
- import time
-
# The elements in buffer are serialized protobuf bytes from the previous
# transforms. We need to deserialize them to extract data and attributes.
futures = []
for elem in self._buffer:
# Deserialize the protobuf to get the original PubsubMessage
pubsub_msg = PubsubMessage._from_proto_str(elem)
- # Publish with the correct data and attributes
+ # Publish with the correct data, attributes, and ordering_key
if self.with_attributes and pubsub_msg.attributes:
future = self._pub_client.publish(
- self._topic, pubsub_msg.data, **pubsub_msg.attributes)
+ self._topic,
+ pubsub_msg.data,
+ ordering_key=pubsub_msg.ordering_key
+ if pubsub_msg.ordering_key else '',
+ **pubsub_msg.attributes)
else:
- future = self._pub_client.publish(self._topic, pubsub_msg.data)
+ if pubsub_msg.ordering_key:
+ future = self._pub_client.publish(
+ self._topic,
+ pubsub_msg.data,
+ ordering_key=pubsub_msg.ordering_key)
+ else:
+ future = self._pub_client.publish(self._topic, pubsub_msg.data)
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-3766483676
## Summary of Changes
Hello @nikitagrover19, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request resolves a critical bug in the Apache Beam Python SDK's `WriteToPubSub` transform, which previously failed to correctly utilize the `ordering_key` for Google Cloud Pub/Sub messages. The fix ensures that messages published through Beam pipelines respect the specified ordering, which is crucial for applications requiring strict message sequence. The changes involve targeted modifications to the publishing logic and the addition of robust test cases to confirm the functionality and prevent regressions.
### Highlights
* **Fix for Pub/Sub Ordering Key**: The `WriteToPubSub` transform in the Python SDK now correctly passes the `ordering_key` to the Google Cloud Pub/Sub `publish()` method, resolving an issue where message ordering was not working as expected.
* **_PubSubWriteDoFn._flush() Modification**: The `_flush()` method has been updated to extract the `ordering_key` from the deserialized `PubsubMessage` and include it in the `publish()` call, ensuring proper handling for both attribute-rich and data-only messages.
* **Backward Compatibility**: The implementation includes logic to gracefully handle cases where `ordering_key` is empty or missing, maintaining backward compatibility for existing pipelines.
* **Comprehensive Testing**: New integration and unit tests have been added to validate the correct propagation and usage of the `ordering_key` in various scenarios, including with and without attributes, and to ensure backward compatibility.
🧠 **New Feature in Public Preview:** You can now enable **Memory** to help **Gemini Code Assist** learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. **Click [here](https://codeassist.google/code-review/login) to enable Memory in your admin console.**
<details>
<summary><b>Using Gemini Code Assist</b></summary>
<br>
The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.
<b>Invoking Gemini</b>
You can request assistance from Gemini at any point by creating a comment using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands on the current page.
Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help | `/gemini help` | Displays a list of available commands.
<b>Customization</b>
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).
<b>Limitations & Feedback</b>
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up [here](https://google.qualtrics.com/jfe/form/SV_2cyuGuTWsEw84yG).
<b>You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the [Gemini Code Assist IDE Extension](https://cloud.google.com/products/gemini/code-assist).</b>
</details>
[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3017730204
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,51 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
Review Comment:
if somehow the pipeline options to determine the runner are not available, we can access them like so: https://github.com/apache/beam/blob/4da3e5537508124ce8922c1877d7ea774f37ee33/sdks/python/apache_beam/coders/coders.py#L1012 during constriction time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3004726901
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java:
##########
@@ -1727,7 +1727,10 @@ public void startBundle(StartBundleContext c) throws IOException {
this.pubsubClient =
getPubsubClientFactory()
.newClient(
- getTimestampAttribute(), null, c.getPipelineOptions().as(PubsubOptions.class));
+ getTimestampAttribute(),
+ null,
+ c.getPipelineOptions().as(PubsubOptions.class),
+ Write.this.getPubsubRootUrl());
Review Comment:
Both PubsubJsonClient and PubsubGrpcClient use MoreObjects.firstNonNull(rootUrlOverride, options.getPubsubRootUrl()) — so if Write.this.getPubsubRootUrl() returns null, it falls back to options.getPubsubRootUrl() which has @Default.String("https://pubsub.googleapis.com"). So yes, the default is preserved correctly in the null case. No change needed here unless you'd prefer an explicit null guard for clarity?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3422516299
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,96 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
+ def test_batch_write_with_ordering_key(self):
+ """Test WriteToPubSub in batch mode with ordering keys.
+
+ Dataflow's Native Pub/Sub Sink does not support ordering_key
+ (see https://github.com/apache/beam/issues/36201), so this test
+ only applies to runners using Beam's Python WriteToPubSub Sink.
+ Dataflow users should use the XLang WriteToPubSub path instead
+ (apache_beam.io.external.gcp.pubsub.WriteToPubSub with
+ publish_with_ordering_key=True).
+ """
+ if self.runner_name == 'TestDataflowRunner':
+ self.skipTest(
+ 'Dataflow Native PubSub Sink does not support ordering_key '
+ '(see https://github.com/apache/beam/issues/36201). '
+ 'Use apache_beam.io.external.gcp.pubsub.WriteToPubSub '
+ 'with publish_with_ordering_key=True instead.')
+ from google.pubsub_v1.types import Subscription
+
+ from apache_beam.options.pipeline_options import PipelineOptions
+ from apache_beam.options.pipeline_options import StandardOptions
+ from apache_beam.transforms import Create
+
+ ordering_topic = self.pub_client.create_topic(
+ name=self.pub_client.topic_path(
+ self.project, 'psit_topic_ordering' + self.uuid))
+ ordering_sub = self.sub_client.create_subscription(
+ request=Subscription(
+ name=self.sub_client.subscription_path(
+ self.project, 'psit_sub_ordering' + self.uuid),
+ topic=ordering_topic.name,
+ enable_message_ordering=True,
+ ))
+ time.sleep(10)
+
+ try:
+ test_messages = [
+ PubsubMessage(
+ b'order_data001', {'attr': 'value1'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data002', {'attr': 'value2'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data003', {'attr': 'value3'}, ordering_key='key2'),
+ ]
+
+ pipeline_options = PipelineOptions()
+ pipeline_options.view_as(StandardOptions).streaming = False
+
+ with TestPipeline(options=pipeline_options) as p:
+ messages = p | 'CreateMessages' >> Create(test_messages)
+ _ = messages | 'WriteToPubSub' >> WriteToPubSub(
+ ordering_topic.name, with_attributes=True)
Review Comment:
The test verified that messages were received with the correct ordering keys on the subscriber side, but it didn't explicitly require ordering to be enabled on the publisher. For small batches like the 3 test messages, PubSub may deliver them in order even without enable_message_ordering=True on the PublisherClient, so the test passed without catching the omission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3011848251
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,51 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
Review Comment:
> Will add a comment to the integration test clarifying this scope.
We need to configure the test to run in direct runner suite, but not in dataflow suite.
```
if self.runner_name == 'TestDataflowRunner':
self.skipTest('<Reasons>')
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3415658689
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,96 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
+ def test_batch_write_with_ordering_key(self):
+ """Test WriteToPubSub in batch mode with ordering keys.
+
+ Dataflow's Native Pub/Sub Sink does not support ordering_key
+ (see https://github.com/apache/beam/issues/36201), so this test
+ only applies to runners using Beam's Python WriteToPubSub Sink.
+ Dataflow users should use the XLang WriteToPubSub path instead
+ (apache_beam.io.external.gcp.pubsub.WriteToPubSub with
+ publish_with_ordering_key=True).
+ """
+ if self.runner_name == 'TestDataflowRunner':
+ self.skipTest(
+ 'Dataflow Native PubSub Sink does not support ordering_key '
+ '(see https://github.com/apache/beam/issues/36201). '
+ 'Use apache_beam.io.external.gcp.pubsub.WriteToPubSub '
+ 'with publish_with_ordering_key=True instead.')
+ from google.pubsub_v1.types import Subscription
+
+ from apache_beam.options.pipeline_options import PipelineOptions
+ from apache_beam.options.pipeline_options import StandardOptions
+ from apache_beam.transforms import Create
+
+ ordering_topic = self.pub_client.create_topic(
+ name=self.pub_client.topic_path(
+ self.project, 'psit_topic_ordering' + self.uuid))
+ ordering_sub = self.sub_client.create_subscription(
+ request=Subscription(
+ name=self.sub_client.subscription_path(
+ self.project, 'psit_sub_ordering' + self.uuid),
+ topic=ordering_topic.name,
+ enable_message_ordering=True,
+ ))
+ time.sleep(10)
+
+ try:
+ test_messages = [
+ PubsubMessage(
+ b'order_data001', {'attr': 'value1'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data002', {'attr': 'value2'}, ordering_key='key1'),
+ PubsubMessage(
+ b'order_data003', {'attr': 'value3'}, ordering_key='key2'),
+ ]
+
+ pipeline_options = PipelineOptions()
+ pipeline_options.view_as(StandardOptions).streaming = False
+
+ with TestPipeline(options=pipeline_options) as p:
+ messages = p | 'CreateMessages' >> Create(test_messages)
+ _ = messages | 'WriteToPubSub' >> WriteToPubSub(
+ ordering_topic.name, with_attributes=True)
Review Comment:
Good catch, added publish_with_ordering_key=True to the WriteToPubSub call in the integration test.
##########
sdks/python/apache_beam/io/gcp/pubsub_test.py:
##########
@@ -1098,6 +1097,71 @@ def test_write_to_pubsub_with_attributes_no_overwrite(self, unused_mock):
Lineage.query(p.result.metrics(), Lineage.SINK),
set(["pubsub:topic:fakeprj.a_topic"]))
+ def test_write_messages_with_ordering_key(self, mock_pubsub):
+ """Test WriteToPubSub with ordering_key in messages."""
+ data = b'data'
+ ordering_key = 'order-123'
+ attributes = {'key': 'value'}
+ payloads = [PubsubMessage(data, attributes, ordering_key=ordering_key)]
+
+ options = PipelineOptions([])
+ options.view_as(StandardOptions).streaming = True
+ with TestPipeline(options=options) as p:
+ _ = (
+ p
+ | Create(payloads)
+ | WriteToPubSub(
+ 'projects/fakeprj/topics/a_topic', with_attributes=True))
+
+ # Verify that publish was called with ordering_key
+ mock_pubsub.return_value.publish.assert_called()
+ call_args = mock_pubsub.return_value.publish.call_args
+
+ # Check that ordering_key was passed as a keyword argument
+ self.assertIn('ordering_key', call_args.kwargs)
+ self.assertEqual(call_args.kwargs['ordering_key'], ordering_key)
+
+ def test_write_messages_with_ordering_key_no_attributes(self, mock_pubsub):
+ """Test WriteToPubSub with ordering_key but no attributes."""
+ data = b'data'
+ ordering_key = 'order-456'
+ payloads = [PubsubMessage(data, None, ordering_key=ordering_key)]
+
+ options = PipelineOptions([])
+ options.view_as(StandardOptions).streaming = True
+ with TestPipeline(options=options) as p:
+ _ = (
+ p
+ | Create(payloads)
+ | WriteToPubSub(
+ 'projects/fakeprj/topics/a_topic', with_attributes=True))
+
+ # Verify that publish was called with ordering_key
+ mock_pubsub.return_value.publish.assert_called()
+ call_args = mock_pubsub.return_value.publish.call_args
+
+ # Check that ordering_key was passed
+ self.assertIn('ordering_key', call_args.kwargs)
+ self.assertEqual(call_args.kwargs['ordering_key'], ordering_key)
+
+ def test_write_messages_without_ordering_key(self, mock_pubsub):
+ """Test WriteToPubSub without ordering_key (backward compatibility)."""
+ data = b'data'
+ attributes = {'key': 'value'}
+ payloads = [PubsubMessage(data, attributes)] # No ordering_key
+
+ options = PipelineOptions([])
+ options.view_as(StandardOptions).streaming = True
+ with TestPipeline(options=options) as p:
+ _ = (
+ p
+ | Create(payloads)
+ | WriteToPubSub(
+ 'projects/fakeprj/topics/a_topic', with_attributes=True))
+
+ # Verify that publish was called
+ mock_pubsub.return_value.publish.assert_called()
Review Comment:
Applied, test_write_messages_without_ordering_key now asserts ordering_key is not in call_args.kwargs, and the ordering key tests assert it is present with the correct value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-3766822447
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-3892717647
responded on the issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3336242479
##########
sdks/python/apache_beam/io/gcp/pubsub.py:
##########
@@ -638,10 +639,13 @@ def __init__(self, transform):
def setup(self):
from google.cloud import pubsub
- self._pub_client = pubsub.PublisherClient(
- publisher_options=pubsub.types.PublisherOptions(
- enable_message_ordering=True,
- ))
+ if self.with_attributes:
Review Comment:
should this be: `if self.with_ordering` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3336134263
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/ExternalWrite.java:
##########
@@ -85,6 +90,9 @@ public PTransform<PCollection<byte[]>, PDone> buildExternal(Configuration config
if (config.timestampAttribute != null) {
writeBuilder.setTimestampAttribute(config.timestampAttribute);
}
+ if (config.publishWithOrderingKey) {
+ writeBuilder.setPublishWithOrderingKey(true);
+ }
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3343661859
##########
sdks/python/apache_beam/io/gcp/pubsub.py:
##########
@@ -638,10 +639,13 @@ def __init__(self, transform):
def setup(self):
from google.cloud import pubsub
- self._pub_client = pubsub.PublisherClient(
- publisher_options=pubsub.types.PublisherOptions(
- enable_message_ordering=True,
- ))
+ if self.with_attributes:
Review Comment:
> Also changed the initialization of with_ordering in __init__ to derive from transform.with_attributes, so ordering is enabled only when the feature is actually requested rather than being hardcoded to True.
Is it feasible to derive based on whether ordering was requested? with_attributes seems more general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3033789400
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,51 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
Review Comment:
could you please trigger this test suite and let me know once this change is ready for review? thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3004736332
##########
sdks/python/apache_beam/io/external/gcp/pubsub.py:
##########
@@ -150,18 +152,24 @@ def __init__(
in a ReadFromPubSub PTransform to deduplicate messages.
timestamp_attribute: If set, will set an attribute for each Cloud Pub/Sub
message with the given name and the message's publish time as the value.
+ publish_with_ordering_key: If True, enables ordering key support when
+ publishing messages. The ordering key must be set on each
+ PubsubMessage via the ``ordering_key`` attribute. Requires
+ messages to be routed to the same region.
"""
self.params = WriteToPubsubSchema(
topic=topic,
id_label=id_label,
# with_attributes=with_attributes,
- timestamp_attribute=timestamp_attribute)
+ timestamp_attribute=timestamp_attribute,
+ publish_with_ordering_key=publish_with_ordering_key)
self.expansion_service = expansion_service
self.with_attributes = with_attributes
def expand(self, pvalue):
if self.with_attributes:
- pcoll = pvalue | 'ToProto' >> Map(pubsub.WriteToPubSub.to_proto_str)
+ pcoll = pvalue | 'ToProto' >> Map(
Review Comment:
Yes, `message_to_proto_str` is the right choice - it has proper type checking. The original `to_proto_str `reference was a pre-existing bug (the method doesn't exist on `WriteToPubSub`), which is why `with_attributes=True` was broken before this PR. Will switch to `message_to_proto_str`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-4127611773
@tvalentyn
Following up on our earlier discussion - since there wasn’t further feedback on the issue, I went ahead and implemented the XLang-based approach you suggested.
The latest commit reflects this, wiring `publish_with_ordering_key` through the Python external transform to Java PubSub IO, along with a couple of related fixes discovered during testing (including `pubsubRootUrl` handling and a `with_attributes` issue).
The PR currently also retains the direct `_flush` fix in the Python SDK for completeness. Happy to scope this down if you'd prefer focusing solely on the XLang approach.
Would appreciate your feedback on whether this direction looks good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3415652509
##########
sdks/python/apache_beam/io/gcp/pubsub.py:
##########
@@ -404,9 +406,13 @@ def __init__(
in a ReadFromPubSub PTransform to deduplicate messages.
timestamp_attribute: If set, will set an attribute for each Cloud Pub/Sub
message with the given name and the message's publish time as the value.
+ enable_message_ordering: If True, enables message ordering on the
Review Comment:
Done, renamed to publish_with_ordering_key to match Java and XLang naming.
##########
sdks/python/apache_beam/io/gcp/pubsub.py:
##########
@@ -430,7 +436,16 @@ def bytes_to_proto_str(element: Union[bytes, str]) -> bytes:
def expand(self, pcoll):
# Store pipeline options for use in DoFn
self.pipeline_options = pcoll.pipeline.options if pcoll.pipeline else None
-
+ # Warn Dataflow users to use the XLang path for ordering key support,
+ # since _PubSubWriteDoFn._flush() is not used by Dataflow's implementation.
+ runner = self.pipeline_options.get_all_options().get(
+ 'runner', '') if self.pipeline_options else ''
+ if 'Dataflow' in str(runner):
Review Comment:
Fixed, warning now only fires when publish_with_ordering_key=True.
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,96 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
+ def test_batch_write_with_ordering_key(self):
+ """Test WriteToPubSub in batch mode with ordering keys.
+
+ Dataflow's Native Pub/Sub Sink does not support ordering_key
+ (see https://github.com/apache/beam/issues/36201), so this test
+ only applies to runners using Beam's Python WriteToPubSub Sink.
+ Dataflow users should use the XLang WriteToPubSub path instead
+ (apache_beam.io.external.gcp.pubsub.WriteToPubSub with
+ publish_with_ordering_key=True).
+ """
+ if self.runner_name == 'TestDataflowRunner':
+ self.skipTest(
+ 'Dataflow Native PubSub Sink does not support ordering_key '
Review Comment:
Done, shortened to just the first line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3023909835
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,51 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
Review Comment:
also, to trigger a particular postcommit suite, you can trivially modify a corresponding trigger file, such as https://github.com/apache/beam/blob/master/.github/trigger_files/beam_PostCommit_Python.json
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-3890664608
Reminder, please take a look at this pr: @tvalentyn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r2997185916
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java:
##########
@@ -1727,7 +1727,10 @@ public void startBundle(StartBundleContext c) throws IOException {
this.pubsubClient =
getPubsubClientFactory()
.newClient(
- getTimestampAttribute(), null, c.getPipelineOptions().as(PubsubOptions.class));
+ getTimestampAttribute(),
+ null,
+ c.getPipelineOptions().as(PubsubOptions.class),
+ Write.this.getPubsubRootUrl());
Review Comment:
Found it:
> PubsubIO.java: fixed a pre-existing bug where PubsubBoundedWriter.startBundle ignored the transform-level pubsubRootUrl, always falling back to pipeline options
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3004726901
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java:
##########
@@ -1727,7 +1727,10 @@ public void startBundle(StartBundleContext c) throws IOException {
this.pubsubClient =
getPubsubClientFactory()
.newClient(
- getTimestampAttribute(), null, c.getPipelineOptions().as(PubsubOptions.class));
+ getTimestampAttribute(),
+ null,
+ c.getPipelineOptions().as(PubsubOptions.class),
+ Write.this.getPubsubRootUrl());
Review Comment:
Both `PubsubJsonClient ` and `PubsubGrpcClient` use `MoreObjects.firstNonNull(rootUrlOverride, options.getPubsubRootUrl())` - so if `Write.this.getPubsubRootUrl()` returns null, it falls back to `options.getPubsubRootUrl()` which has `@Default.String("https://pubsub.googleapis.com")` . So yes, the default is preserved correctly in the null case. No change needed here unless you'd prefer an explicit null guard for clarity?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3004737738
##########
sdks/python/apache_beam/io/gcp/pubsub_integration_test.py:
##########
@@ -305,6 +305,51 @@ def test_batch_write_with_attributes(self):
"""Test WriteToPubSub in batch mode with attributes."""
self._test_batch_write(with_attributes=True)
+ @pytest.mark.it_postcommit
Review Comment:
Correct - the `_flush` fix in `_PubSubWriteDoFn` only applies to Direct Runner since Dataflow overrides this with its own implementation. Will add a comment to the integration test clarifying this scope. For the warning - I'll add a `logging.warning` in `_PubSubWriteDoFn` when `ordering_key` is set on any message, advising Dataflow users to use the WriteToPubSub XLang path instead. Does that approach sound right, or would you prefer it at pipeline construction time (e.g., in `WriteToPubSub.expand()`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3004733236
##########
sdks/python/apache_beam/io/external/gcp/pubsub.py:
##########
@@ -150,18 +152,24 @@ def __init__(
in a ReadFromPubSub PTransform to deduplicate messages.
timestamp_attribute: If set, will set an attribute for each Cloud Pub/Sub
message with the given name and the message's publish time as the value.
+ publish_with_ordering_key: If True, enables ordering key support when
+ publishing messages. The ordering key must be set on each
+ PubsubMessage via the ``ordering_key`` attribute. Requires
Review Comment:
That sentence was inaccurate - it was leftover from an earlier draft and doesn't apply here. Will remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-3850303844
/gemini review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3390769476
##########
sdks/python/apache_beam/io/gcp/pubsub_test.py:
##########
@@ -1098,6 +1097,71 @@ def test_write_to_pubsub_with_attributes_no_overwrite(self, unused_mock):
Lineage.query(p.result.metrics(), Lineage.SINK),
set(["pubsub:topic:fakeprj.a_topic"]))
+ def test_write_messages_with_ordering_key(self, mock_pubsub):
+ """Test WriteToPubSub with ordering_key in messages."""
+ data = b'data'
+ ordering_key = 'order-123'
+ attributes = {'key': 'value'}
+ payloads = [PubsubMessage(data, attributes, ordering_key=ordering_key)]
+
+ options = PipelineOptions([])
+ options.view_as(StandardOptions).streaming = True
+ with TestPipeline(options=options) as p:
+ _ = (
+ p
+ | Create(payloads)
+ | WriteToPubSub(
+ 'projects/fakeprj/topics/a_topic', with_attributes=True))
+
+ # Verify that publish was called with ordering_key
+ mock_pubsub.return_value.publish.assert_called()
+ call_args = mock_pubsub.return_value.publish.call_args
+
+ # Check that ordering_key was passed as a keyword argument
+ self.assertIn('ordering_key', call_args.kwargs)
+ self.assertEqual(call_args.kwargs['ordering_key'], ordering_key)
+
+ def test_write_messages_with_ordering_key_no_attributes(self, mock_pubsub):
+ """Test WriteToPubSub with ordering_key but no attributes."""
+ data = b'data'
+ ordering_key = 'order-456'
+ payloads = [PubsubMessage(data, None, ordering_key=ordering_key)]
+
+ options = PipelineOptions([])
+ options.view_as(StandardOptions).streaming = True
+ with TestPipeline(options=options) as p:
+ _ = (
+ p
+ | Create(payloads)
+ | WriteToPubSub(
+ 'projects/fakeprj/topics/a_topic', with_attributes=True))
+
+ # Verify that publish was called with ordering_key
+ mock_pubsub.return_value.publish.assert_called()
+ call_args = mock_pubsub.return_value.publish.call_args
+
+ # Check that ordering_key was passed
+ self.assertIn('ordering_key', call_args.kwargs)
+ self.assertEqual(call_args.kwargs['ordering_key'], ordering_key)
+
+ def test_write_messages_without_ordering_key(self, mock_pubsub):
+ """Test WriteToPubSub without ordering_key (backward compatibility)."""
+ data = b'data'
+ attributes = {'key': 'value'}
+ payloads = [PubsubMessage(data, attributes)] # No ordering_key
+
+ options = PipelineOptions([])
+ options.view_as(StandardOptions).streaming = True
+ with TestPipeline(options=options) as p:
+ _ = (
+ p
+ | Create(payloads)
+ | WriteToPubSub(
+ 'projects/fakeprj/topics/a_topic', with_attributes=True))
+
+ # Verify that publish was called
+ mock_pubsub.return_value.publish.assert_called()
Review Comment:
+1 to the gemini-code-assist comment on Jan 18
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #37345:
URL: https://github.com/apache/beam/pull/37345#issuecomment-4725120370
Thanks for your contribution, @nikitagrover19 !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3336131563
##########
sdks/python/apache_beam/io/gcp/pubsub.py:
##########
@@ -628,7 +638,10 @@ def __init__(self, transform):
def setup(self):
from google.cloud import pubsub
- self._pub_client = pubsub.PublisherClient()
+ self._pub_client = pubsub.PublisherClient(
+ publisher_options=pubsub.types.PublisherOptions(
+ enable_message_ordering=True,
Review Comment:
Good point - fixed. `enable_message_ordering` is now only set when `with_attributes=True`, since ordering keys are only meaningful in that path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix WriteToPubSub to pass ordering_key to publish() method [beam]
Posted by "nikitagrover19 (via GitHub)" <gi...@apache.org>.
nikitagrover19 commented on code in PR #37345:
URL: https://github.com/apache/beam/pull/37345#discussion_r3350751370
##########
sdks/python/apache_beam/io/gcp/pubsub.py:
##########
@@ -638,10 +639,13 @@ def __init__(self, transform):
def setup(self):
from google.cloud import pubsub
- self._pub_client = pubsub.PublisherClient(
- publisher_options=pubsub.types.PublisherOptions(
- enable_message_ordering=True,
- ))
+ if self.with_attributes:
Review Comment:
You're right, `with_attributes` was too broad a signal.
I've updated this to derive `with_ordering` from a new explicit `enable_message_ordering` parameter on `WriteToPubSub`, so it's now tied directly to whether ordering was requested rather than inferred from attributes.
This avoids enabling ordering unnecessarily when attributes are used without an `ordering_key`.
Let me know if this approach looks good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org