You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Martin Kleppmann (JIRA)" <ji...@apache.org> on 2015/01/05 08:59:34 UTC
[jira] [Assigned] (SAMZA-212) Explore using PostgreSQL changelog as
input stream
[ https://issues.apache.org/jira/browse/SAMZA-212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Martin Kleppmann reassigned SAMZA-212:
--------------------------------------
Assignee: Martin Kleppmann
I am working on this.
> Explore using PostgreSQL changelog as input stream
> --------------------------------------------------
>
> Key: SAMZA-212
> URL: https://issues.apache.org/jira/browse/SAMZA-212
> Project: Samza
> Issue Type: New Feature
> Reporter: Martin Kleppmann
> Assignee: Martin Kleppmann
> Labels: project
>
> It would be good to be able to capture changes to a PostgreSQL database, and use it as an input to Samza (like SAMZA-200 but for Postgres instead of MySQL).
> On a high level, it seems like the options are:
> * [Creating a trigger|https://gist.github.com/fbeauchamp/9879820], and using [LISTEN|http://www.postgresql.org/docs/9.3/static/sql-listen.html]/[NOTIFY|http://www.postgresql.org/docs/9.3/static/sql-notify.html] to send changes to an external process. ([Example|https://www.evernote.com/shard/s17/sh/cfbad111-2725-41ef-bf4c-58ac13a026d6/1a6f422f7e64831d7b2d7995a49ee48e])
> * [Logical WAL decoding|http://www.anarazel.de/2ndquadrant/pgcon-2013-05-23/], which has been [committed|http://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=commitdiff;h=5eeedd55b2d7e53b5fdcdab6a8e74bb666d75bcc] and should be in the upcoming Postgres 9.4 release.
> [@selenamarie|https://twitter.com/selenamarie] [points out|https://twitter.com/selenamarie/status/450373310367268864]: "problem with it is that LISTEN/NOTIFY are stateless. if you want reliable transport, will need logical replication".
> [This Twitter conversation|https://twitter.com/selenamarie/status/450360318204448768] has some more detail.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)