You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Pablo Estrada (Jira)" <ji...@apache.org> on 2020/04/29 02:47:00 UTC
[jira] [Assigned] (BEAM-7885) DoFn.setup() don't run for streaming
jobs on DirectRunner.
[ https://issues.apache.org/jira/browse/BEAM-7885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pablo Estrada reassigned BEAM-7885:
-----------------------------------
Assignee: Pablo Estrada
> DoFn.setup() don't run for streaming jobs on DirectRunner.
> -----------------------------------------------------------
>
> Key: BEAM-7885
> URL: https://issues.apache.org/jira/browse/BEAM-7885
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Affects Versions: 2.14.0
> Environment: Python
> Reporter: niklas Hansson
> Assignee: Pablo Estrada
> Priority: Minor
>
> From version 2.14.0 Python have introduced setup and teardown for DoFn in order to "Called to prepare an instance for processing bundles of elements.This is a good place to initialize transient in-memory resources, such as network connections."
> However when trying to use it for a unbounded job (pubsub source) it seams like the DoFn.setup() is never called and the resources are never initialize. [UPDATE] it is working for Dataflow runner but not for DirectRunner. For the Dataflow runner the DoFn.Setup seams to be called multiple times but then never again when the pipeline is processing elements [UPDATE] . For the direct runner I get:
> """"
> AttributeError: 'NoneType' object has no attribute 'predict' [while running 'transform the data']
> """
> My source code: [https://github.com/NikeNano/DataflowSklearnStreaming]
>
> I am happy to contribute with example code for how to use setup as soon as I get it running :)
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)