[jira] [Commented] (BEAM-3342) Create a Cloud Bigtable Python connector

["Solomon Duskis (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289655#comment-16289655 ] 

Solomon Duskis commented on BEAM-3342:
--------------------------------------

I started with a simple pipeline that writes to Cloud Bigtable via the google.cloud bigtable package, which works locally with google.cloud installed, but doesn't work when I use a dataflow runner.  Here's what I get:

==
  message:  "Not processing workitem 2633526545277283048 since a deferred exception was found: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 706, in run
    self._load_main_session(self.local_staging_directory)
  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 446, in _load_main_session
    pickler.load_session(session_file)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line 247, in load_session
    return dill.load_session(file_path)
  File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 363, in load_session
    module = unpickler.load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1133, in load_reduce
    value = func(*args)
  File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 767, in _import_module
    return getattr(__import__(module, None, None, [obj]), obj)
AttributeError: 'module' object has no attribute 'bigtable'
==

Can I use the standard google.cloud bigtable client?  If so, how, and why don't BigQuery and Storage use the google.cloud clients?

> Create a Cloud Bigtable Python connector
> ----------------------------------------
>
>                 Key: BEAM-3342
>                 URL: https://issues.apache.org/jira/browse/BEAM-3342
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py-core
>            Reporter: Solomon Duskis
>            Assignee: Ahmet Altay
>
> I would like to create a Cloud Bigtable python connector.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)