You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Dan <da...@gmail.com> on 2015/04/14 20:46:48 UTC

Extra Systems and other extensions.

Hey,

At state.com we've started to write some generic extensions to Samza that
we think would be more generally useful. We've got
a ElasticsearchSystemProducer/Factory to output to an Elasticsearch index
and EnvironmentConfigRewriter to modify config from environment variable.

What's the best way for us to add things like this? Do you want more
modules in the main project or should we just create some separate projects
on github?

It would be good to get core extensions like these shared to be tested and
used by more people

Thanks,
Dan

Re: Extra Systems and other extensions.

Posted by Dan <da...@gmail.com>.
https://issues.apache.org/jira/browse/SAMZA-654 - Elasticsearch Producer

Please comment there if you have any opinions on requirements for the
clients / configuration / serialization with using this producer. It's
currently fit for our use case but I know there will be more.

https://issues.apache.org/jira/browse/SAMZA-655 - Env variable rewriter

Again any comments on the format of the variables names welcome!

 - Dan


On 15 April 2015 at 17:38, Chinmay Soman <ch...@gmail.com> wrote:

> +1 ! I was going to do this for my use case as well. Would love to have
> this !
>
> On Wed, Apr 15, 2015 at 9:24 AM, Roger Hoover <ro...@gmail.com>
> wrote:
>
> > Dan,
> >
> > This is great.  Would love to have a common ElasticSearch system
> producer.
> >
> > Cheers,
> >
> > Roger
> >
> > On Tue, Apr 14, 2015 at 1:34 PM, Dan <da...@gmail.com> wrote:
> >
> > > Thanks Jakob, I agree they'll be more maintained and tested if they're
> in
> > > the main repo so that's great.
> > >
> > > I'll sort out Jira's and get some patches of what we've got working now
> > out
> > > for review.
> > >
> > >  - Dan
> > >
> > >
> > > On 14 April 2015 at 19:56, Jakob Homan <jg...@gmail.com> wrote:
> > >
> > > > Hey Dan-
> > > > I'd love for the Elastic Search stuff to be added to the main code,
> as
> > > > a separate module.  Keeping these in the main source code keeps them
> > > > more likely to be maintained and correct.
> > > >
> > > > The EvironemtnConfigRewriter can likely go in the same place as the
> > > > ConfigRewriter interface, since it doesn't depend on Kafka as the
> > > > current RegexTopicRewriter does.
> > > >
> > > > If you could open JIRAs for these, that would be great.  Happy to
> > > > shepard the code in.
> > > > -Jakob
> > > >
> > > > On 14 April 2015 at 11:46, Dan <da...@gmail.com> wrote:
> > > > > Hey,
> > > > >
> > > > > At state.com we've started to write some generic extensions to
> Samza
> > > > that
> > > > > we think would be more generally useful. We've got
> > > > > a ElasticsearchSystemProducer/Factory to output to an Elasticsearch
> > > index
> > > > > and EnvironmentConfigRewriter to modify config from environment
> > > variable.
> > > > >
> > > > > What's the best way for us to add things like this? Do you want
> more
> > > > > modules in the main project or should we just create some separate
> > > > projects
> > > > > on github?
> > > > >
> > > > > It would be good to get core extensions like these shared to be
> > tested
> > > > and
> > > > > used by more people
> > > > >
> > > > > Thanks,
> > > > > Dan
> > > >
> > >
> >
>
>
>
> --
> Thanks and regards
>
> Chinmay Soman
>

Re: Extra Systems and other extensions.

Posted by Chinmay Soman <ch...@gmail.com>.
+1 ! I was going to do this for my use case as well. Would love to have
this !

On Wed, Apr 15, 2015 at 9:24 AM, Roger Hoover <ro...@gmail.com>
wrote:

> Dan,
>
> This is great.  Would love to have a common ElasticSearch system producer.
>
> Cheers,
>
> Roger
>
> On Tue, Apr 14, 2015 at 1:34 PM, Dan <da...@gmail.com> wrote:
>
> > Thanks Jakob, I agree they'll be more maintained and tested if they're in
> > the main repo so that's great.
> >
> > I'll sort out Jira's and get some patches of what we've got working now
> out
> > for review.
> >
> >  - Dan
> >
> >
> > On 14 April 2015 at 19:56, Jakob Homan <jg...@gmail.com> wrote:
> >
> > > Hey Dan-
> > > I'd love for the Elastic Search stuff to be added to the main code, as
> > > a separate module.  Keeping these in the main source code keeps them
> > > more likely to be maintained and correct.
> > >
> > > The EvironemtnConfigRewriter can likely go in the same place as the
> > > ConfigRewriter interface, since it doesn't depend on Kafka as the
> > > current RegexTopicRewriter does.
> > >
> > > If you could open JIRAs for these, that would be great.  Happy to
> > > shepard the code in.
> > > -Jakob
> > >
> > > On 14 April 2015 at 11:46, Dan <da...@gmail.com> wrote:
> > > > Hey,
> > > >
> > > > At state.com we've started to write some generic extensions to Samza
> > > that
> > > > we think would be more generally useful. We've got
> > > > a ElasticsearchSystemProducer/Factory to output to an Elasticsearch
> > index
> > > > and EnvironmentConfigRewriter to modify config from environment
> > variable.
> > > >
> > > > What's the best way for us to add things like this? Do you want more
> > > > modules in the main project or should we just create some separate
> > > projects
> > > > on github?
> > > >
> > > > It would be good to get core extensions like these shared to be
> tested
> > > and
> > > > used by more people
> > > >
> > > > Thanks,
> > > > Dan
> > >
> >
>



-- 
Thanks and regards

Chinmay Soman

Re: Extra Systems and other extensions.

Posted by Roger Hoover <ro...@gmail.com>.
Dan,

This is great.  Would love to have a common ElasticSearch system producer.

Cheers,

Roger

On Tue, Apr 14, 2015 at 1:34 PM, Dan <da...@gmail.com> wrote:

> Thanks Jakob, I agree they'll be more maintained and tested if they're in
> the main repo so that's great.
>
> I'll sort out Jira's and get some patches of what we've got working now out
> for review.
>
>  - Dan
>
>
> On 14 April 2015 at 19:56, Jakob Homan <jg...@gmail.com> wrote:
>
> > Hey Dan-
> > I'd love for the Elastic Search stuff to be added to the main code, as
> > a separate module.  Keeping these in the main source code keeps them
> > more likely to be maintained and correct.
> >
> > The EvironemtnConfigRewriter can likely go in the same place as the
> > ConfigRewriter interface, since it doesn't depend on Kafka as the
> > current RegexTopicRewriter does.
> >
> > If you could open JIRAs for these, that would be great.  Happy to
> > shepard the code in.
> > -Jakob
> >
> > On 14 April 2015 at 11:46, Dan <da...@gmail.com> wrote:
> > > Hey,
> > >
> > > At state.com we've started to write some generic extensions to Samza
> > that
> > > we think would be more generally useful. We've got
> > > a ElasticsearchSystemProducer/Factory to output to an Elasticsearch
> index
> > > and EnvironmentConfigRewriter to modify config from environment
> variable.
> > >
> > > What's the best way for us to add things like this? Do you want more
> > > modules in the main project or should we just create some separate
> > projects
> > > on github?
> > >
> > > It would be good to get core extensions like these shared to be tested
> > and
> > > used by more people
> > >
> > > Thanks,
> > > Dan
> >
>

Re: Extra Systems and other extensions.

Posted by Dan <da...@gmail.com>.
Thanks Jakob, I agree they'll be more maintained and tested if they're in
the main repo so that's great.

I'll sort out Jira's and get some patches of what we've got working now out
for review.

 - Dan


On 14 April 2015 at 19:56, Jakob Homan <jg...@gmail.com> wrote:

> Hey Dan-
> I'd love for the Elastic Search stuff to be added to the main code, as
> a separate module.  Keeping these in the main source code keeps them
> more likely to be maintained and correct.
>
> The EvironemtnConfigRewriter can likely go in the same place as the
> ConfigRewriter interface, since it doesn't depend on Kafka as the
> current RegexTopicRewriter does.
>
> If you could open JIRAs for these, that would be great.  Happy to
> shepard the code in.
> -Jakob
>
> On 14 April 2015 at 11:46, Dan <da...@gmail.com> wrote:
> > Hey,
> >
> > At state.com we've started to write some generic extensions to Samza
> that
> > we think would be more generally useful. We've got
> > a ElasticsearchSystemProducer/Factory to output to an Elasticsearch index
> > and EnvironmentConfigRewriter to modify config from environment variable.
> >
> > What's the best way for us to add things like this? Do you want more
> > modules in the main project or should we just create some separate
> projects
> > on github?
> >
> > It would be good to get core extensions like these shared to be tested
> and
> > used by more people
> >
> > Thanks,
> > Dan
>

Re: Extra Systems and other extensions.

Posted by Jakob Homan <jg...@gmail.com>.
Hey Dan-
I'd love for the Elastic Search stuff to be added to the main code, as
a separate module.  Keeping these in the main source code keeps them
more likely to be maintained and correct.

The EvironemtnConfigRewriter can likely go in the same place as the
ConfigRewriter interface, since it doesn't depend on Kafka as the
current RegexTopicRewriter does.

If you could open JIRAs for these, that would be great.  Happy to
shepard the code in.
-Jakob

On 14 April 2015 at 11:46, Dan <da...@gmail.com> wrote:
> Hey,
>
> At state.com we've started to write some generic extensions to Samza that
> we think would be more generally useful. We've got
> a ElasticsearchSystemProducer/Factory to output to an Elasticsearch index
> and EnvironmentConfigRewriter to modify config from environment variable.
>
> What's the best way for us to add things like this? Do you want more
> modules in the main project or should we just create some separate projects
> on github?
>
> It would be good to get core extensions like these shared to be tested and
> used by more people
>
> Thanks,
> Dan