You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Udi Meiri (Jira)" <ji...@apache.org> on 2019/09/11 00:49:00 UTC
[jira] [Commented] (BEAM-7611) Python BigTableIO IT is not running
in any test suites
[ https://issues.apache.org/jira/browse/BEAM-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927153#comment-16927153 ]
Udi Meiri commented on BEAM-7611:
---------------------------------
I attempted to make the test run on DirectRunner, and ran into issues with pickling enums and self.client itself (client version 1.0.0).
The pipeline code needs to be refactored into its own module, so we can avoid pickling the client used for setting up the test.
Another question arises:
Why was this test placed in the examples directory? Was the intent to write one?
Pickling error:
{code}
======================================================================
ERROR: test_bigtable_write (apache_beam.examples.cookbook.bigtableio_it_test.BigtableIOWriteTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/examples/cookbook/bigtableio_it_test.py", line 176, in test_bigtable_write
| 'Generate Direct Rows' >> GenerateTestRows(number, **config_data))
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/transforms/ptransform.py", line 905, in __ror__
return self.transform.__ror__(pvalueish, self.label)
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/transforms/ptransform.py", line 514, in __ror__
result = p.apply(self, pvalueish, label)
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/pipeline.py", line 481, in apply
return self.apply(transform, pvalueish)
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/pipeline.py", line 517, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/runner.py", line 175, in apply
return m(transform, input, options)
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/runner.py", line 181, in apply_PTransform
return transform.expand(input)
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/examples/cookbook/bigtableio_it_test.py", line 97, in expand
beam_options['table_id']))
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/pvalue.py", line 113, in __or__
return self.pipeline.apply(ptransform, self)
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/pipeline.py", line 517, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/runner.py", line 175, in apply
return m(transform, input, options)
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/runner.py", line 181, in apply_PTransform
return transform.expand(input)
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/io/gcp/bigtableio.py", line 143, in expand
beam_options['table_id'])))
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/transforms/core.py", line 1107, in __init__
super(ParDo, self).__init__(fn, *args, **kwargs)
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/transforms/ptransform.py", line 702, in __init__
self.fn = pickler.loads(pickler.dumps(self.fn))
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/internal/pickler.py", line 237, in dumps
s = dill.dumps(o)
File "/usr/local/google/home/ehudm/src/beam/build/gradleenv/45127155/local/lib/python2.7/site-packages/dill/_dill.py", line 294, in dumps
dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
File "/usr/local/google/home/ehudm/src/beam/build/gradleenv/45127155/local/lib/python2.7/site-packages/dill/_dill.py", line 287, in dump
pik.dump(obj)
File "/usr/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
save(state)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/internal/pickler.py", line 198, in new_save_module_dict
return old_save_module_dict(pickler, obj)
File "/usr/local/google/home/ehudm/src/beam/build/gradleenv/45127155/local/lib/python2.7/site-packages/dill/_dill.py", line 910, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
save(v)
File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
save(state)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/internal/pickler.py", line 198, in new_save_module_dict
return old_save_module_dict(pickler, obj)
File "/usr/local/google/home/ehudm/src/beam/build/gradleenv/45127155/local/lib/python2.7/site-packages/dill/_dill.py", line 910, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
save(v)
File "/usr/lib/python2.7/pickle.py", line 306, in save
rv = reduce(self.proto)
File "/usr/local/google/home/ehudm/src/beam/build/gradleenv/45127155/local/lib/python2.7/site-packages/google/cloud/client.py", line 144, in __getstate__
"Clients have non-trivial state that is local and unpickleable.",
PicklingError: Pickling client objects is explicitly not supported.
Clients have non-trivial state that is local and unpickleable.
{code}
> Python BigTableIO IT is not running in any test suites
> ------------------------------------------------------
>
> Key: BEAM-7611
> URL: https://issues.apache.org/jira/browse/BEAM-7611
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp, testing
> Reporter: Chamikara Jayalath
> Assignee: Solomon Duskis
> Priority: Major
>
> We added an integration test here: [https://github.com/apache/beam/pull/7367]
>
> But this currently does not get picked up by any test suites (and get skipped by some due to missing dependencies) hence BigTable sink is largely untested.
>
> First attempt to enable it: [https://github.com/apache/beam/pull/8886]
>
> Solomon assigning to you since I cannot find Juan's (PR author) Jira ID.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)