You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by vicky fazlurrahman <vi...@gmail.com> on 2017/11/13 08:08:49 UTC
PubsubIO unable to set topic and subscription
Hi all,
I am using beam sdk 2.1.0, trying to run streaming pipeline, using
dataflow-runner that consume pub sub stream with the following config :
p.apply(PubsubIO.readStrings()
.fromTopic("projects/my-gcp-project/topics/publisher-test")
.fromSubscription("projects/my-gcp-project/subscriptions/subscibe-test"))
The build is failed because it unable to set the topic and subscription
[DEBUG] Setting accessibility to true in order to invoke main().
Nov 13, 2017 2:23:03 PM org.apache.beam.runners.dataflow.DataflowRunner
fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files
from the classpath: will stage 111 files. Enable logging at DEBUG level to
see which files will be staged.
[WARNING]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Can't set both the topic and
the subscription for a PubsubIO.Read transform
at
org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Read.expand(PubsubIO.java:702)
at
org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Read.expand(PubsubIO.java:536)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:514)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:454)
at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:165)
at com.example.drain.DrainPubsub.main(DrainPubsub.java:67)
... 6 more
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
A bit confused, about the cause of the exception and what is the proper
config for PubsubIO.
Because seems, it can't get topic and subscription although given the
correct format :
projects/<project_id>/topics/<topic_name>
projects/<project_id>/subscriptions/<subscription_name>
Any help would be appreciated.
--
Vicky
Re: PubsubIO unable to set topic and subscription
Posted by Vicky Fazlurrahman <vi...@go-jek.com>.
I already have a subscription for a pub sub topic. I just want to read from
the created subscription.
I See. My mistake. Its because i set both of fromTopic and fromSubscription at
the same time.
Thank you very much Eugene.
Regards,
Vicky
On Mon, Nov 13, 2017 at 3:58 PM, Eugene Kirpichov <ki...@google.com>
wrote:
> The error says "Can't set both the topic and the subscription": PubSub
> subscribers read from a subscription, and messages sent to a topic are sent
> to all subscriptions bound to this topic. That's why, when reading from
> PubSub in Beam you can specify either a topic (then a new subscription to
> this topic will be created specifically for this pipeline) or a
> subscription (if you already have a subscription you want to use for this
> pipeline). What are you trying to accomplish by specifying both at the same
> time?
>
> On Mon, Nov 13, 2017, 12:08 AM vicky fazlurrahman <
> vicky.fazlurrahman@gmail.com> wrote:
>
>>
>> Hi all,
>>
>> I am using beam sdk 2.1.0, trying to run streaming pipeline, using
>> dataflow-runner that consume pub sub stream with the following config :
>>
>> p.apply(PubsubIO.readStrings()
>> .fromTopic("projects/my-gcp-project/topics/publisher-test")
>> .fromSubscription("projects/my-gcp-project/subscriptions/
>> subscibe-test"))
>>
>> The build is failed because it unable to set the topic and subscription
>>
>> [DEBUG] Setting accessibility to true in order to invoke main().
>>
>> Nov 13, 2017 2:23:03 PM org.apache.beam.runners.dataflow.DataflowRunner
>> fromOptions
>> INFO: PipelineOptions.filesToStage was not specified. Defaulting to files
>> from the classpath: will stage 111 files. Enable logging at DEBUG level to
>> see which files will be staged.
>> [WARNING]
>> java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(
>> NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497)
>> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(
>> ExecJavaMojo.java:293)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.IllegalStateException: Can't set both the topic and
>> the subscription for a PubsubIO.Read transform
>> at org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Read.expand(
>> PubsubIO.java:702)
>> at org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Read.expand(
>> PubsubIO.java:536)
>> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:514)
>> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:454)
>> at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
>> at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:165)
>> at com.example.drain.DrainPubsub.main(DrainPubsub.java:67)
>> ... 6 more
>> [INFO] ------------------------------------------------------------
>> ------------
>> [INFO] BUILD FAILURE
>> [INFO] ------------------------------------------------------------
>> ------------
>>
>> A bit confused, about the cause of the exception and what is the proper
>> config for PubsubIO.
>> Because seems, it can't get topic and subscription although given the
>> correct format :
>>
>> projects/<project_id>/topics/<topic_name>
>>
>> projects/<project_id>/subscriptions/<subscription_name>
>>
>> Any help would be appreciated.
>>
>> --
>> Vicky
>>
>
--
Best Regards,
*Vicky Fazlurrahman*
*GO-JEK INDONESIA*
E-mail vicky@go-jek.com
Website www.go-jek.com
Phone +62 85729516006 <+62+85729516006>
Re: PubsubIO unable to set topic and subscription
Posted by Eugene Kirpichov <ki...@google.com>.
The error says "Can't set both the topic and the subscription": PubSub
subscribers read from a subscription, and messages sent to a topic are sent
to all subscriptions bound to this topic. That's why, when reading from
PubSub in Beam you can specify either a topic (then a new subscription to
this topic will be created specifically for this pipeline) or a
subscription (if you already have a subscription you want to use for this
pipeline). What are you trying to accomplish by specifying both at the same
time?
On Mon, Nov 13, 2017, 12:08 AM vicky fazlurrahman <
vicky.fazlurrahman@gmail.com> wrote:
>
> Hi all,
>
> I am using beam sdk 2.1.0, trying to run streaming pipeline, using
> dataflow-runner that consume pub sub stream with the following config :
>
> p.apply(PubsubIO.readStrings()
> .fromTopic("projects/my-gcp-project/topics/publisher-test")
>
> .fromSubscription("projects/my-gcp-project/subscriptions/subscibe-test"))
>
> The build is failed because it unable to set the topic and subscription
>
> [DEBUG] Setting accessibility to true in order to invoke main().
>
> Nov 13, 2017 2:23:03 PM org.apache.beam.runners.dataflow.DataflowRunner
> fromOptions
> INFO: PipelineOptions.filesToStage was not specified. Defaulting to files
> from the classpath: will stage 111 files. Enable logging at DEBUG level to
> see which files will be staged.
> [WARNING]
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: Can't set both the topic and
> the subscription for a PubsubIO.Read transform
> at
> org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Read.expand(PubsubIO.java:702)
> at
> org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Read.expand(PubsubIO.java:536)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:514)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:454)
> at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
> at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:165)
> at com.example.drain.DrainPubsub.main(DrainPubsub.java:67)
> ... 6 more
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
>
> A bit confused, about the cause of the exception and what is the proper
> config for PubsubIO.
> Because seems, it can't get topic and subscription although given the
> correct format :
>
> projects/<project_id>/topics/<topic_name>
>
> projects/<project_id>/subscriptions/<subscription_name>
>
> Any help would be appreciated.
>
> --
> Vicky
>