You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Daniel Halperin (JIRA)" <ji...@apache.org> on 2016/03/18 16:43:33 UTC
[jira] [Updated] (BEAM-53) PubSubIO: reimplement in Java
[ https://issues.apache.org/jira/browse/BEAM-53?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Halperin updated BEAM-53:
--------------------------------
Assignee: Mark Shields
> PubSubIO: reimplement in Java
> -----------------------------
>
> Key: BEAM-53
> URL: https://issues.apache.org/jira/browse/BEAM-53
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-gcp
> Reporter: Daniel Halperin
> Assignee: Mark Shields
> Priority: Minor
>
> PubSubIO is currently only partially implemented in Java: the DirectPipelineRunner uses a non-scalable API in a single-threaded manner.
> In contrast, the DataflowPipelineRunner uses an entirely different code path implemented in the Google Cloud Dataflow service.
> We need to reimplement PubSubIO in Java in order to support other runners in a scalable way.
> Additionally, we can take this opportunity to add new features:
> * getting timestamp from an arbitrary lambda in arbitrary formats rather than from a message attribute in only 2 formats.
> * exposing metadata and attributes in the elements produced by PubSubIO.Read
> * setting metadata and attributes in the messages written by PubSubIO.Write
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)