You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Derek Hao Hu <ph...@gmail.com> on 2017/10/02 16:01:10 UTC
Re: How to get the PublishTime of PubsubMessages?
Thanks Reinier! Is it easy to get the [MessageId](
http://googleapis.github.io/googleapis/java/proto-google-cloud-pubsub-v1/0.1.19/apidocs/com/google/pubsub/v1/PubsubMessage.html#getMessageId--)
of the PubsubMessages?
PS: If you are familiar with this part of the Beam implementation, can you
also point out where it gets the timestamp attribute? I'm trying to
understand the implementation as well. :)
Cheers,
Derek
On Sat, Sep 30, 2017 at 5:58 AM, Reinier Kip <rk...@bol.com> wrote:
> Hi Derek,
>
>
> The publish time is the element's timestamp, retrievable via
> DoFn.ProcessContext#timestamp, unless you've configured a timestamp
> attribute with PubsubIO.Read#withTimestampAttribute. As far as I know,
> payload, attributes and publish time is all you get. What other information
> from Google's PubsubMessage are you looking for?
>
>
> Reinier
> ------------------------------
> *From:* Derek Hao Hu <ph...@gmail.com>
> *Sent:* 30 September 2017 03:18:13
> *To:* user@beam.apache.org
> *Subject:* How to get the PublishTime of PubsubMessages?
>
> Hi,
>
> I'm trying to use [PubsubIO](https://beam.apache.org/documentation/sdks/
> javadoc/2.1.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html) to read
> [PubsubMessage](https://beam.apache.org/documentation/sdks/
> javadoc/2.1.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessage.html).
>
> I've noticed that the PubsubMessage class in Beam only exposes Attribute
> and Payload but the [PubSubMessage in Google Cloud](http://googleapis.
> github.io/googleapis/java/proto-google-cloud-pubsub-v1/
> 0.1.19/apidocs/com/google/pubsub/v1/PubsubMessage.html?is-external=true)
> exposes a lot more information like `getPublishTime()`.
>
> Is there a way for us to get the same amount information in Beam easily?
> --
> Derek Hao Hu
>
> Software Engineer | Snapchat
> Snap Inc.
>
--
Derek Hao Hu
Software Engineer | Snapchat
Snap Inc.
Re: How to get the PublishTime of PubsubMessages?
Posted by Derek Hao Hu <ph...@gmail.com>.
That is really helpful. Thanks for your help again Reinier!
Derek
On Mon, Oct 2, 2017 at 9:45 AM, Reinier Kip <rk...@bol.com> wrote:
> Unless someone else knows I think timestamp, attributes, and payload
> are all you're gonna get.
>
>
> I'm definitely not familiar with the implementation, but the timestamp
> part is here:
>
>
> https://github.com/apache/beam/blob/ea7940d88d9d5ddb09fabc5c8993d8
> 395c59abce/sdks/java/io/google-cloud-platform/src/
> main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java#L101-L115
>
>
> Good luck!
>
> <https://github.com/apache/beam/blob/ea7940d88d9d5ddb09fabc5c8993d8395c59abce/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java#L101-L115>
>
> <https://github.com/apache/beam/blob/ea7940d88d9d5ddb09fabc5c8993d8395c59abce/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java#L101-L115>
> apache/beam
> <https://github.com/apache/beam/blob/ea7940d88d9d5ddb09fabc5c8993d8395c59abce/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java#L101-L115>
> github.com
> beam - Mirror of Apache Beam
>
>
> ------------------------------
> *From:* Derek Hao Hu <ph...@gmail.com>
> *Sent:* 02 October 2017 18:01:10
> *To:* user@beam.apache.org
> *Subject:* Re: How to get the PublishTime of PubsubMessages?
>
> Thanks Reinier! Is it easy to get the [MessageId](http://googleapis.
> github.io/googleapis/java/proto-google-cloud-pubsub-v1/
> 0.1.19/apidocs/com/google/pubsub/v1/PubsubMessage.html#getMessageId--) of
> the PubsubMessages?
>
> PS: If you are familiar with this part of the Beam implementation, can you
> also point out where it gets the timestamp attribute? I'm trying to
> understand the implementation as well. :)
>
> Cheers,
>
> Derek
>
> On Sat, Sep 30, 2017 at 5:58 AM, Reinier Kip <rk...@bol.com> wrote:
>
>> Hi Derek,
>>
>>
>> The publish time is the element's timestamp, retrievable via
>> DoFn.ProcessContext#timestamp, unless you've configured a timestamp
>> attribute with PubsubIO.Read#withTimestampAttribute. As far as I know,
>> payload, attributes and publish time is all you get. What other information
>> from Google's PubsubMessage are you looking for?
>>
>>
>> Reinier
>> ------------------------------
>> *From:* Derek Hao Hu <ph...@gmail.com>
>> *Sent:* 30 September 2017 03:18:13
>> *To:* user@beam.apache.org
>> *Subject:* How to get the PublishTime of PubsubMessages?
>>
>> Hi,
>>
>> I'm trying to use [PubsubIO](https://beam.apach
>> e.org/documentation/sdks/javadoc/2.1.0/org/apache/beam/sdk/
>> io/gcp/pubsub/PubsubIO.html) to read [PubsubMessage](https://beam.a
>> pache.org/documentation/sdks/javadoc/2.1.0/org/apache/beam/s
>> dk/io/gcp/pubsub/PubsubMessage.html).
>>
>> I've noticed that the PubsubMessage class in Beam only exposes Attribute
>> and Payload but the [PubSubMessage in Google Cloud](
>> http://googleapis.github.io/googleapis/java/proto-
>> google-cloud-pubsub-v1/0.1.19/apidocs/com/google/pubsub/v1/
>> PubsubMessage.html?is-external=true) exposes a lot more information like
>> `getPublishTime()`.
>>
>> Is there a way for us to get the same amount information in Beam easily?
>> --
>> Derek Hao Hu
>>
>> Software Engineer | Snapchat
>> Snap Inc.
>>
>
>
>
> --
> Derek Hao Hu
>
> Software Engineer | Snapchat
> Snap Inc.
>
--
Derek Hao Hu
Software Engineer | Snapchat
Snap Inc.
Re: How to get the PublishTime of PubsubMessages?
Posted by Reinier Kip <rk...@bol.com>.
Unless someone else knows I think timestamp, attributes, and payload are all you're gonna get.
I'm definitely not familiar with the implementation, but the timestamp part is here:
https://github.com/apache/beam/blob/ea7940d88d9d5ddb09fabc5c8993d8395c59abce/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java#L101-L115
Good luck!
<https://github.com/apache/beam/blob/ea7940d88d9d5ddb09fabc5c8993d8395c59abce/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java#L101-L115>
[https://avatars3.githubusercontent.com/u/47359?v=4&s=400]<https://github.com/apache/beam/blob/ea7940d88d9d5ddb09fabc5c8993d8395c59abce/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java#L101-L115>
apache/beam<https://github.com/apache/beam/blob/ea7940d88d9d5ddb09fabc5c8993d8395c59abce/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java#L101-L115>
github.com
beam - Mirror of Apache Beam
________________________________
From: Derek Hao Hu <ph...@gmail.com>
Sent: 02 October 2017 18:01:10
To: user@beam.apache.org
Subject: Re: How to get the PublishTime of PubsubMessages?
Thanks Reinier! Is it easy to get the [MessageId](http://googleapis.github.io/googleapis/java/proto-google-cloud-pubsub-v1/0.1.19/apidocs/com/google/pubsub/v1/PubsubMessage.html#getMessageId--) of the PubsubMessages?
PS: If you are familiar with this part of the Beam implementation, can you also point out where it gets the timestamp attribute? I'm trying to understand the implementation as well. :)
Cheers,
Derek
On Sat, Sep 30, 2017 at 5:58 AM, Reinier Kip <rk...@bol.com>> wrote:
Hi Derek,
The publish time is the element's timestamp, retrievable via DoFn.ProcessContext#timestamp, unless you've configured a timestamp attribute with PubsubIO.Read#withTimestampAttribute. As far as I know, payload, attributes and publish time is all you get. What other information from Google's PubsubMessage are you looking for?
Reinier
________________________________
From: Derek Hao Hu <ph...@gmail.com>>
Sent: 30 September 2017 03:18:13
To: user@beam.apache.org<ma...@beam.apache.org>
Subject: How to get the PublishTime of PubsubMessages?
Hi,
I'm trying to use [PubsubIO](https://beam.apache.org/documentation/sdks/javadoc/2.1.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html) to read [PubsubMessage](https://beam.apache.org/documentation/sdks/javadoc/2.1.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessage.html).
I've noticed that the PubsubMessage class in Beam only exposes Attribute and Payload but the [PubSubMessage in Google Cloud](http://googleapis.github.io/googleapis/java/proto-google-cloud-pubsub-v1/0.1.19/apidocs/com/google/pubsub/v1/PubsubMessage.html?is-external=true) exposes a lot more information like `getPublishTime()`.
Is there a way for us to get the same amount information in Beam easily?
--
Derek Hao Hu
Software Engineer | Snapchat
Snap Inc.
--
Derek Hao Hu
Software Engineer | Snapchat
Snap Inc.