[jira] [Comment Edited] (SOLR-10086) Add Streaming Expression for Kafka Streams

["Dennis Gove (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-10086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15896023#comment-15896023 ] 

Dennis Gove edited comment on SOLR-10086 at 3/5/17 2:01 AM:
------------------------------------------------------------

I'm developing this under a different project, so I dunno if the conversation should occur here or elsewhere, but to keep everyone in the loop, here's what I'm thinking about for the expression for a Kafka Producer. 

{code}
kafkaProducer(
  <incoming stream>, 
  topic=<evaluator>, 
  bootstrapServers=<kafka servers>,
  key=<evaluator>, // optional, if not provided then no key used when sending record
  keyType=[string,long,double,boolean], // only required if using key and can't figure out from key param (knowing long or double is tricky)
  value=<evaluator> // optional, if not provided then whole tuple as json string is used
  valueType=[string,long,double,boolean], // only required if can't figure out from value param (knowing long or double is tricky)
  partition=<evaluator>, // optional, if not provided then no partition used when sending record
)
{code}

Anything that is <evaluator> is allowed to be any valid StreamEvaluator, such as
{code}
raw("foo") // raw value "foo"
"foo" // value of field foo
add(foo, bar) // value of foo + value of bar
{code}

There are other parameters allowed with a Kafka v0.10.x producer, such as 
{code}
 props.put("acks", "all");
 props.put("retries", 0);
 props.put("batch.size", 16384);
 props.put("linger.ms", 1);
 props.put("buffer.memory", 33554432);
{code}

These could be handled like the parameters in a Search Stream, where any parameters not explicitly known are just passed along. We could do that here so people can set whatever parameters they want on the publisher and we don't have to keep updating our supported list as the Kafka lib version changes.


was (Author: dpgove):
I'm developing this under a different project, so I dunno if the conversation should occur here or elsewhere, but to keep everyone in the loop, here's what I'm thinking about for the expression for a Kafka Producer. 

{code}
kafkaProducer(
  <incoming stream>, 
  topic=<evaluator>, 
  bootstrapServers=<kafka servers>,
  key=<evaluator>, // optional, if not provided then no key used when sending record
  keyType=[string,long,double,boolean], // only required if using key and can't figure out from key param (knowing long or double is tricky)
  value=<evaluator> // optional, if not provided then whole tuple as json string is used
  valueType=[string,long,double,boolean], // only required if can't figure out from value param (knowing long or double is tricky)
  partition=<evaluator>, // optional, if not provided then no partition used when sending record
)
{code}

Anything that is <evaluator> is allowed to be any valid StreamEvaluator, such as
{code}
raw("foo") // raw value "foo"
"foo" // value of field foo
add(foo, bar) // value of foo + value of bar
{code}

There are other parameters allowed with a Kafka v0.10.x producer, such as 
{code}
 props.put("acks", "all");
 props.put("retries", 0);
 props.put("batch.size", 16384);
 props.put("linger.ms", 1);
 props.put("buffer.memory", 33554432);
{code}

These could be handled like the parameters in a Search Stream, where any parameters no explicitly known are just passed along. We could do that here so people can set whatever parameters they want on the publisher and we don't have to keep updating our supported list as the Kafka lib version changes.

> Add Streaming Expression for Kafka Streams
> ------------------------------------------
>
>                 Key: SOLR-10086
>                 URL: https://issues.apache.org/jira/browse/SOLR-10086
>             Project: Solr
>          Issue Type: New Feature
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: SolrJ
>            Reporter: Susheel Kumar
>            Priority: Minor
>
> This is being asked to have SolrCloud pull data from Kafka topic periodically using DataImport Handler. 
> Adding streaming expression support to pull data from Kafka would be good feature to have.  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org