You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Bayu Aldi Yansyah <ba...@gmail.com> on 2018/01/05 07:24:50 UTC
beam.io.ReadStringsFromPubSub Unexpected behaviour
Hey guys,
I use the latest version apache beam package and found a strange behaviour.
I'm not sure this is bug or not, because 2 hours ago I ran the same code
and it's fine.
Now it's error.
If I use a topic name as the parameter:
events = (pipeline
| 'Read' >> ReadStringsFromPubSub(
topic=FQ_TOPIC_NAME,
id_label='event_id')
It runs as expected:
- It create a subscription: like projects/PROJECT_ID/
subscriptions/beam_1515134413_fbbb1a48
- Pull the events
But If I use the subscription as the parameter:
events = (pipeline
| 'Read' >> ReadStringsFromPubSub(
subscription=FQ_SUBSCRIPTION,
id_label='event_id')
it raise the following exception:
google.gax.errors.RetryError: RetryError(Exception occurred in
retry method that was not classified as transient, caused by <_Rendezvous
of RPC that terminated with (StatusCode.UNAUTHENTICATED, Deadline
Exceeded)>)
If I use the random topic name or subscription url, it will throw a
StatusCode.PERMISSION_DENIED
which is expected.
If i upgrade the pubsub client pip install --upgrade google-cloud-pubsub
it will throw an error like the following:
class Field(six.with_metaclass(_FieldMeta, object)):
TypeError: Error when calling the metaclass bases
metaclass conflict: the metaclass of a derived class must be a
(non-strict) subclass of the metaclasses of all its bases
Below is my pip freeze output:
apache-beam==2.2.0
avro==1.8.2
cachetools==2.0.1
certifi==2017.11.5
chardet==3.0.4
crcmod==1.7
dill==0.2.6
enum34==1.1.6
funcsigs==1.0.2
future==0.16.0
futures==3.2.0
gapic-google-cloud-pubsub-v1==0.15.4
google-apitools==0.5.11
google-auth==1.2.1
google-auth-httplib2==0.0.3
google-cloud-bigquery==0.25.0
google-cloud-core==0.25.0
google-cloud-pubsub==0.26.0
google-gax==0.15.16
googleapis-common-protos==1.5.3
googledatastore==7.0.1
grpc-google-iam-v1==0.11.4
grpcio==1.8.3
httplib2==0.9.2
idna==2.6
mock==2.0.0
oauth2client==3.0.0
pbr==3.1.1
ply==3.8
proto-google-cloud-datastore-v1==0.90.4
proto-google-cloud-pubsub-v1==0.15.4
protobuf==3.3.0
pyasn1==0.4.2
pyasn1-modules==0.2.1
PyYAML==3.12
requests==2.18.4
rsa==3.4.2
six==1.10.0
typing==3.6.2
urllib3==1.22
What should I do?
Thanks
Re: beam.io.ReadStringsFromPubSub Unexpected behaviour
Posted by Bayu Aldi Yansyah <ba...@gmail.com>.
Hmm I got a chance to test my code again and it worked perfectly without
any changes.
Is there any thoughts on why this is happen? So I can make sure that this
is not happen again in the future.
Thanks
— bay
On Fri, 5 Jan 2018 at 14.24 Bayu Aldi Yansyah <ba...@gmail.com>
wrote:
> Hey guys,
>
> I use the latest version apache beam package and found a strange behaviour.
> I'm not sure this is bug or not, because 2 hours ago I ran the same code
> and it's fine.
> Now it's error.
>
> If I use a topic name as the parameter:
>
> events = (pipeline
> | 'Read' >> ReadStringsFromPubSub(
> topic=FQ_TOPIC_NAME,
> id_label='event_id')
>
> It runs as expected:
> - It create a subscription: like
> projects/PROJECT_ID/subscriptions/beam_1515134413_fbbb1a48
> - Pull the events
>
> But If I use the subscription as the parameter:
>
> events = (pipeline
> | 'Read' >> ReadStringsFromPubSub(
> subscription=FQ_SUBSCRIPTION,
> id_label='event_id')
> it raise the following exception:
>
> google.gax.errors.RetryError: RetryError(Exception occurred in
> retry method that was not classified as transient, caused by <_Rendezvous
> of RPC that terminated with (StatusCode.UNAUTHENTICATED, Deadline
> Exceeded)>)
>
>
> If I use the random topic name or subscription url, it will throw a
> StatusCode.PERMISSION_DENIED
> which is expected.
>
> If i upgrade the pubsub client pip install --upgrade google-cloud-pubsub
> it will throw an error like the following:
>
> class Field(six.with_metaclass(_FieldMeta, object)):
> TypeError: Error when calling the metaclass bases
> metaclass conflict: the metaclass of a derived class must be a
> (non-strict) subclass of the metaclasses of all its bases
>
>
> Below is my pip freeze output:
>
> apache-beam==2.2.0
> avro==1.8.2
> cachetools==2.0.1
> certifi==2017.11.5
> chardet==3.0.4
> crcmod==1.7
> dill==0.2.6
> enum34==1.1.6
> funcsigs==1.0.2
> future==0.16.0
> futures==3.2.0
> gapic-google-cloud-pubsub-v1==0.15.4
> google-apitools==0.5.11
> google-auth==1.2.1
> google-auth-httplib2==0.0.3
> google-cloud-bigquery==0.25.0
> google-cloud-core==0.25.0
> google-cloud-pubsub==0.26.0
> google-gax==0.15.16
> googleapis-common-protos==1.5.3
> googledatastore==7.0.1
> grpc-google-iam-v1==0.11.4
> grpcio==1.8.3
> httplib2==0.9.2
> idna==2.6
> mock==2.0.0
> oauth2client==3.0.0
> pbr==3.1.1
> ply==3.8
> proto-google-cloud-datastore-v1==0.90.4
> proto-google-cloud-pubsub-v1==0.15.4
> protobuf==3.3.0
> pyasn1==0.4.2
> pyasn1-modules==0.2.1
> PyYAML==3.12
> requests==2.18.4
> rsa==3.4.2
> six==1.10.0
> typing==3.6.2
> urllib3==1.22
>
> What should I do?
> Thanks
>
--
Bayu Aldi Yansyah,
Universitas Airlangga
https://github.com/pyk