[jira] [Created] (SAMZA-212) Explore using PostgreSQL changelog as input stream

["Martin Kleppmann (JIRA)" <ji...@apache.org>]

Martin Kleppmann created SAMZA-212:
--------------------------------------

             Summary: Explore using PostgreSQL changelog as input stream
                 Key: SAMZA-212
                 URL: https://issues.apache.org/jira/browse/SAMZA-212
             Project: Samza
          Issue Type: New Feature
            Reporter: Martin Kleppmann


It would be good to be able to capture changes to a PostgreSQL database, and use it as an input to Samza (like SAMZA-200 but for Postgres instead of MySQL).

On a high level, it seems like the options are:
* [Creating a trigger|https://gist.github.com/fbeauchamp/9879820], and using [LISTEN|http://www.postgresql.org/docs/9.3/static/sql-listen.html]/[NOTIFY|http://www.postgresql.org/docs/9.3/static/sql-notify.html] to send changes to an external process. ([Example|https://www.evernote.com/shard/s17/sh/cfbad111-2725-41ef-bf4c-58ac13a026d6/1a6f422f7e64831d7b2d7995a49ee48e])
* [Logical WAL decoding|http://www.anarazel.de/2ndquadrant/pgcon-2013-05-23/], which has been [committed|http://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=commitdiff;h=5eeedd55b2d7e53b5fdcdab6a8e74bb666d75bcc] and should be in the upcoming Postgres 9.4 release.

[@selenamarie|https://twitter.com/selenamarie] [points out|https://twitter.com/selenamarie/status/450373310367268864]: "problem with it is that LISTEN/NOTIFY are stateless. if you want reliable transport, will need logical replication".

[This Twitter conversation|https://twitter.com/selenamarie/status/450360318204448768] has some more detail.



--
This message was sent by Atlassian JIRA
(v6.2#6252)