[jira] [Commented] (SAMZA-423) Add document store to Samza

["Robert Žuljević (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272239#comment-15272239 ] 

Robert Žuljević commented on SAMZA-423:
---------------------------------------

Hi, [~nickpan47], I know that I basically rose this issue from the dead, but did you get a chance to look a this?

> Add document store to Samza
> ---------------------------
>
>                 Key: SAMZA-423
>                 URL: https://issues.apache.org/jira/browse/SAMZA-423
>             Project: Samza
>          Issue Type: New Feature
>            Reporter: Martin Kleppmann
>            Assignee: Robert Žuljević
>              Labels: design, project
>         Attachments: DESIGN-SAMZA-423-0.pdf, DESIGN-SAMZA-423-1.pdf, SAMZA-423.0.patch
>
>
> At the moment Samza only has a key-value storage engine (LevelDB/RocksDB/in-memory), but Samza's state abstraction is designed to be more general than that. In particular, we've discussed integrating Lucene in order to support full-text indexes.
> There are two modes of using Lucene which would make sense in a stream processing system:
> * Treat incoming messages as documents to be added to an index. This would be akin to the key-value storage model, but with much richer indexing capabilities. It would enable joining streams not just on a single key, but on more complex criteria (arbitrary boolean expressions for joins) and stuff like deduplicating similar documents (which you may want e.g. if implementing a web crawler).
> * Match incoming messages against a set of queries, where the queries are more or less fixed (perhaps the queries are updated via another input stream). In this case, the message isn't added to a persistent index, but it's only analysed and matched against a query as it flows through the stream processor. This is useful for monitoring an activity stream for events of interest ("notify me whenever a news article mentions my company name"). It's perhaps comparable to ElasticSearch's [percolator|http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-percolate.html].
> I was chatting to [~romseygeek] yesterday, a committer on Lucene/Solr. We have a vague plan to hack on a proof of concept to see what an integration of Lucene and Samza could look like. This is a placeholder ticket for collecting any stuff related to that experiment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)