You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Eric Shieh <da...@gmail.com> on 2019/10/10 16:46:13 UTC
Samza App Deployment Topology - Best Practices
Hi,
I am new to Samza, I am evaluating Samza as the backbone for my streaming
CEP requirement. I have:
1. Multiple data enrichment and ETL jobs
2. Multiple domain specific CEP rulesets
3. Common/shared system services like consuming topics/streams and
persisting the messages in ElasticSearch and HDFS.
My questions are:
1. Can I, or is it recommended to, package multiple jobs as 1 deployment
with 1 properties file or keep each app separated? Based on the
documentation, it appears to support 1 app/job within a single
configuration as there is no mechanism to assign multiple app classes and
given each a name unless I am mistaken.
2. If only 1 app per config+deployment, what is the best way to handle
requirement #3 - common/shared system services as there is no app or job
per say, I just need to specify the streams and output system (ie
org.apache.samza.system.hdfs.writer.
BinarySequenceFileHdfsWriter or
org.apache.samza.system.elasticsearch.indexrequest.DefaultIndexRequestFactory).
Given it's a common shared system service not tied to specific jobs, can it
be deployed without an app?
Thank you in advance for your help, looking forward to learning more about
Samza and developing this critical feature using Samza!
Regards,
Eric
Re: Samza App Deployment Topology - Best Practices
Posted by Eric Shieh <da...@gmail.com>.
Hi Bharath,
Thank you very much, this is extremely helpful! I had not originally
contemplated using beam but will definitely look into it following your
recommendation.
Based on your response and upon further reading of the source code, I
realized my original understanding of the samza-elasticsearch
system/connector and the overall Samza "system/connector" was incorrect.
Elasticsearch system/connector, unlike Kafka or HDFS systems, does not have
input nor output descriptors; furthermore since it maps stream name to
Elasticsearch index and doc_type name, I incorrectly assumed that Samza
internally used Kafka topics as the sources to Elasticsearch
"system/connector" so that writes to Elasticsearch can be buffered through
Kafka. I now understand the Elasticsearch "system/connector" is simply a
lower-level API that can be called by the jobs as a message stream sink
which means I need to implement a generic job to consume 1 or more Kafka
topics and use the API to write to Elasticsearch.
Regards,
Eric
On Tue, Oct 22, 2019 at 9:31 AM Bharath Kumara Subramanian <
codin.martial@gmail.com> wrote:
> Hi Eric,
>
> Thanks for additional clarification. I now have a better understanding of
> your use case.
> Your use case be accomplished using various approaches.
>
>
> 1. One Samza application that consumes all of your input streams (A and
> D) and also your intermediate output streams (B) and routes it to HDFS
> or
> ElasticSearch. With this approach,
> 1. You may have to scale up or scale down your entire job to react to
> changes to either of your inputs (A, D & B as well). It might
> potentially
> result in under utilization of resources.
> 2. A failure in one component could impact other components. For
> example, it is possible that there are other writers to output
> stream B and
> you don't want to disrupt them because of a bug in the
> transformation logic
> of input stream A.
> 3. On the similar lines of 2, it also possible that due to a HDFS or
> elastic search outage, the back pressure causes output B to grow
> which
> might potentially impact the time to catch up and also impact your
> job's
> performance (e.g. this can happen if you have priorities setup
> across your
> streams)
> 2. Another approach is to split them into multiple jobs; one for
> consuming input sources (A and D) and routing to appropriate
> destinations
> (B & C), another for consuming the output from the previous job (B) and
> routing it to HDFS or ElasticSearch. With this approach
> 1. You isolate the common/shared logic to write to HDFS or
> ElasticSearch into its own job which allows you to manage it
> independently
> including scale up/down.
> 2. During HDFS/ElasticSearch outages, other components are
> non-impacted and the back pressure causes your stream B to grow
> which Kafka
> can handle well.
>
>
> Our recommend API to write Samza application is Apache Beam
> <https://beam.apache.org/>. Examples on how to write a sample application
> using Beam API and run it using Samza can be found:
> https://github.com/apache/samza-beam-examples
>
> Please reach out to us if you have more questions.
>
> Thanks,
> Bharath
>
> On Wed, Oct 16, 2019 at 7:31 AM Eric Shieh <da...@gmail.com> wrote:
>
> > Thank you Bharath. Regarding my 2nd question, perhaps the following
> > scenario can help to illustrate what I am looking to achieve:
> >
> > Input stream A -> Job 1 -> Output stream B (Kafka Topic B)
> > Input stream A -> Job 2 -> Output stream C
> > Input stream D -> Job 3 -> Output stream B (Kafka Topic B)
> > Input stream B (Kafka Topic B) -> Elasticsearch (or write to HDFS)
> >
> > In the case of "Input stream B (Kafka Topic B) -> Elasticsearch (or write
> > to HDFS)" this is what I was referring to as "Common/shared system
> > services" that does not have any transformation logic except message sink
> > to either Elasticsearch or HDFS using Samza's systems/connectors. In
> other
> > words, Job 1 and Job 3 both output to "Output stream B" expecting
> messages
> > will be persisted in Elasticsearch or HDFS, would I need to specify the
> > system/connector configuration separately in Job 1 and Job 3? Is there a
> > way to have "Input stream B (Kafka Topic B) -> Elasticsearch (or write
> to
> > HDFS)" as its own stand-alone job so I can have the following:
> >
> > RESTful web services (or other none Samza services/applications) as Kafka
> > producer -> Input stream B (Kafka Topic B) -> Elasticsearch (or write to
> > HDFS)
> >
> > Regards,
> >
> > Eric
> >
> > On Mon, Oct 14, 2019 at 8:35 PM Bharath Kumara Subramanian <
> > codin.martial@gmail.com> wrote:
> >
> > > Hi Eric,
> > >
> > > Answers to your questions are as follows
> > >
> > >
> > > >
> > > >
> > > >
> > > > *Can I, or is it recommended to, package multiple jobs as 1
> > > deploymentwith
> > > > 1 properties file or keep each app separated? Based on
> > thedocumentation,
> > > > it appears to support 1 app/job within a singleconfiguration as there
> > is
> > > no
> > > > mechanism to assign multiple app classes andgiven each a name unless
> I
> > am
> > > > mistaken*
> > > >
> > >
> > > *app.class* is a single valued configuration and your understanding
> > about
> > > it based on the documentation is correct.
> > >
> > >
> > > >
> > > >
> > > > *If only 1 app per config+deployment, what is the best way to
> > > > handlerequirement #3 - common/shared system services as there is no
> app
> > > or
> > > > jobper say, I just need to specify the streams and output system
> > > > (ieorg.apache.samza.system.hdfs.writer*
> > > >
> > >
> > > There are couple of options to achieve your #3 requirement.
> > >
> > > 1. If there is enough commonality between your jobs, you could have
> > one
> > > application class that describes the logic and have the different
> > > configurations to modify the behavior of the application logic. This
> > > does
> > > come with some of the following considerations
> > > 1. Your deployment system needs to support deploying the same
> > > application with different configs.
> > > 2. Potential duplication of configuration if you configuration
> > system
> > > doesn't support hierarchies and overrides.
> > > 3. Potentially unmanageable for evolution, since the change in
> > > application affects multiple jobs and requires extensive testing
> > > across
> > > different configurations.
> > > 2. You could potentially have libraries to perform some piece of
> > > business logic and have your different jobs leverage them using
> > > composition. Some things to consider with this option
> > > 1. Your application and configuration stay isolated.
> > > 2. You could still leverage some of the common configurations if
> > your
> > > configuration system supports hierarchies and overrides
> > > 3. Alleviates concerns over evolution and testing as long as the
> > > changes are application specific.
> > >
> > >
> > > I am still unclear about the second part of your 2nd question.
> > > Do you mean to say all your jobs consume from same sources and write to
> > > sources and only your processing logic is different?
> > >
> > >
> > > > *common/shared system services as there is no app or jobper say, I
> just
> > > > need to specify the streams and output system*
> > >
> > >
> > > Also, I am not sure I follow what do you mean by "*there is no app or
> > > job"*.
> > > You still have 1 app per config + deployment, right?
> > >
> > > Thanks,
> > > Bharath
> > >
> > > On Thu, Oct 10, 2019 at 9:46 AM Eric Shieh <da...@gmail.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > I am new to Samza, I am evaluating Samza as the backbone for my
> > streaming
> > > > CEP requirement. I have:
> > > >
> > > > 1. Multiple data enrichment and ETL jobs
> > > > 2. Multiple domain specific CEP rulesets
> > > > 3. Common/shared system services like consuming topics/streams and
> > > > persisting the messages in ElasticSearch and HDFS.
> > > >
> > > > My questions are:
> > > >
> > > > 1. Can I, or is it recommended to, package multiple jobs as 1
> > deployment
> > > > with 1 properties file or keep each app separated? Based on the
> > > > documentation, it appears to support 1 app/job within a single
> > > > configuration as there is no mechanism to assign multiple app classes
> > and
> > > > given each a name unless I am mistaken.
> > > > 2. If only 1 app per config+deployment, what is the best way to
> handle
> > > > requirement #3 - common/shared system services as there is no app or
> > job
> > > > per say, I just need to specify the streams and output system (ie
> > > > org.apache.samza.system.hdfs.writer.
> > > > BinarySequenceFileHdfsWriter or
> > > >
> > > >
> > >
> >
> org.apache.samza.system.elasticsearch.indexrequest.DefaultIndexRequestFactory).
> > > > Given it's a common shared system service not tied to specific jobs,
> > can
> > > it
> > > > be deployed without an app?
> > > >
> > > > Thank you in advance for your help, looking forward to learning more
> > > about
> > > > Samza and developing this critical feature using Samza!
> > > >
> > > > Regards,
> > > >
> > > > Eric
> > > >
> > >
> >
>
Re: Samza App Deployment Topology - Best Practices
Posted by Bharath Kumara Subramanian <co...@gmail.com>.
Hi Eric,
Thanks for additional clarification. I now have a better understanding of
your use case.
Your use case be accomplished using various approaches.
1. One Samza application that consumes all of your input streams (A and
D) and also your intermediate output streams (B) and routes it to HDFS or
ElasticSearch. With this approach,
1. You may have to scale up or scale down your entire job to react to
changes to either of your inputs (A, D & B as well). It might potentially
result in under utilization of resources.
2. A failure in one component could impact other components. For
example, it is possible that there are other writers to output
stream B and
you don't want to disrupt them because of a bug in the
transformation logic
of input stream A.
3. On the similar lines of 2, it also possible that due to a HDFS or
elastic search outage, the back pressure causes output B to grow which
might potentially impact the time to catch up and also impact your job's
performance (e.g. this can happen if you have priorities setup
across your
streams)
2. Another approach is to split them into multiple jobs; one for
consuming input sources (A and D) and routing to appropriate destinations
(B & C), another for consuming the output from the previous job (B) and
routing it to HDFS or ElasticSearch. With this approach
1. You isolate the common/shared logic to write to HDFS or
ElasticSearch into its own job which allows you to manage it
independently
including scale up/down.
2. During HDFS/ElasticSearch outages, other components are
non-impacted and the back pressure causes your stream B to grow
which Kafka
can handle well.
Our recommend API to write Samza application is Apache Beam
<https://beam.apache.org/>. Examples on how to write a sample application
using Beam API and run it using Samza can be found:
https://github.com/apache/samza-beam-examples
Please reach out to us if you have more questions.
Thanks,
Bharath
On Wed, Oct 16, 2019 at 7:31 AM Eric Shieh <da...@gmail.com> wrote:
> Thank you Bharath. Regarding my 2nd question, perhaps the following
> scenario can help to illustrate what I am looking to achieve:
>
> Input stream A -> Job 1 -> Output stream B (Kafka Topic B)
> Input stream A -> Job 2 -> Output stream C
> Input stream D -> Job 3 -> Output stream B (Kafka Topic B)
> Input stream B (Kafka Topic B) -> Elasticsearch (or write to HDFS)
>
> In the case of "Input stream B (Kafka Topic B) -> Elasticsearch (or write
> to HDFS)" this is what I was referring to as "Common/shared system
> services" that does not have any transformation logic except message sink
> to either Elasticsearch or HDFS using Samza's systems/connectors. In other
> words, Job 1 and Job 3 both output to "Output stream B" expecting messages
> will be persisted in Elasticsearch or HDFS, would I need to specify the
> system/connector configuration separately in Job 1 and Job 3? Is there a
> way to have "Input stream B (Kafka Topic B) -> Elasticsearch (or write to
> HDFS)" as its own stand-alone job so I can have the following:
>
> RESTful web services (or other none Samza services/applications) as Kafka
> producer -> Input stream B (Kafka Topic B) -> Elasticsearch (or write to
> HDFS)
>
> Regards,
>
> Eric
>
> On Mon, Oct 14, 2019 at 8:35 PM Bharath Kumara Subramanian <
> codin.martial@gmail.com> wrote:
>
> > Hi Eric,
> >
> > Answers to your questions are as follows
> >
> >
> > >
> > >
> > >
> > > *Can I, or is it recommended to, package multiple jobs as 1
> > deploymentwith
> > > 1 properties file or keep each app separated? Based on
> thedocumentation,
> > > it appears to support 1 app/job within a singleconfiguration as there
> is
> > no
> > > mechanism to assign multiple app classes andgiven each a name unless I
> am
> > > mistaken*
> > >
> >
> > *app.class* is a single valued configuration and your understanding
> about
> > it based on the documentation is correct.
> >
> >
> > >
> > >
> > > *If only 1 app per config+deployment, what is the best way to
> > > handlerequirement #3 - common/shared system services as there is no app
> > or
> > > jobper say, I just need to specify the streams and output system
> > > (ieorg.apache.samza.system.hdfs.writer*
> > >
> >
> > There are couple of options to achieve your #3 requirement.
> >
> > 1. If there is enough commonality between your jobs, you could have
> one
> > application class that describes the logic and have the different
> > configurations to modify the behavior of the application logic. This
> > does
> > come with some of the following considerations
> > 1. Your deployment system needs to support deploying the same
> > application with different configs.
> > 2. Potential duplication of configuration if you configuration
> system
> > doesn't support hierarchies and overrides.
> > 3. Potentially unmanageable for evolution, since the change in
> > application affects multiple jobs and requires extensive testing
> > across
> > different configurations.
> > 2. You could potentially have libraries to perform some piece of
> > business logic and have your different jobs leverage them using
> > composition. Some things to consider with this option
> > 1. Your application and configuration stay isolated.
> > 2. You could still leverage some of the common configurations if
> your
> > configuration system supports hierarchies and overrides
> > 3. Alleviates concerns over evolution and testing as long as the
> > changes are application specific.
> >
> >
> > I am still unclear about the second part of your 2nd question.
> > Do you mean to say all your jobs consume from same sources and write to
> > sources and only your processing logic is different?
> >
> >
> > > *common/shared system services as there is no app or jobper say, I just
> > > need to specify the streams and output system*
> >
> >
> > Also, I am not sure I follow what do you mean by "*there is no app or
> > job"*.
> > You still have 1 app per config + deployment, right?
> >
> > Thanks,
> > Bharath
> >
> > On Thu, Oct 10, 2019 at 9:46 AM Eric Shieh <da...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I am new to Samza, I am evaluating Samza as the backbone for my
> streaming
> > > CEP requirement. I have:
> > >
> > > 1. Multiple data enrichment and ETL jobs
> > > 2. Multiple domain specific CEP rulesets
> > > 3. Common/shared system services like consuming topics/streams and
> > > persisting the messages in ElasticSearch and HDFS.
> > >
> > > My questions are:
> > >
> > > 1. Can I, or is it recommended to, package multiple jobs as 1
> deployment
> > > with 1 properties file or keep each app separated? Based on the
> > > documentation, it appears to support 1 app/job within a single
> > > configuration as there is no mechanism to assign multiple app classes
> and
> > > given each a name unless I am mistaken.
> > > 2. If only 1 app per config+deployment, what is the best way to handle
> > > requirement #3 - common/shared system services as there is no app or
> job
> > > per say, I just need to specify the streams and output system (ie
> > > org.apache.samza.system.hdfs.writer.
> > > BinarySequenceFileHdfsWriter or
> > >
> > >
> >
> org.apache.samza.system.elasticsearch.indexrequest.DefaultIndexRequestFactory).
> > > Given it's a common shared system service not tied to specific jobs,
> can
> > it
> > > be deployed without an app?
> > >
> > > Thank you in advance for your help, looking forward to learning more
> > about
> > > Samza and developing this critical feature using Samza!
> > >
> > > Regards,
> > >
> > > Eric
> > >
> >
>
Re: Samza App Deployment Topology - Best Practices
Posted by Eric Shieh <da...@gmail.com>.
Thank you Bharath. Regarding my 2nd question, perhaps the following
scenario can help to illustrate what I am looking to achieve:
Input stream A -> Job 1 -> Output stream B (Kafka Topic B)
Input stream A -> Job 2 -> Output stream C
Input stream D -> Job 3 -> Output stream B (Kafka Topic B)
Input stream B (Kafka Topic B) -> Elasticsearch (or write to HDFS)
In the case of "Input stream B (Kafka Topic B) -> Elasticsearch (or write
to HDFS)" this is what I was referring to as "Common/shared system
services" that does not have any transformation logic except message sink
to either Elasticsearch or HDFS using Samza's systems/connectors. In other
words, Job 1 and Job 3 both output to "Output stream B" expecting messages
will be persisted in Elasticsearch or HDFS, would I need to specify the
system/connector configuration separately in Job 1 and Job 3? Is there a
way to have "Input stream B (Kafka Topic B) -> Elasticsearch (or write to
HDFS)" as its own stand-alone job so I can have the following:
RESTful web services (or other none Samza services/applications) as Kafka
producer -> Input stream B (Kafka Topic B) -> Elasticsearch (or write to
HDFS)
Regards,
Eric
On Mon, Oct 14, 2019 at 8:35 PM Bharath Kumara Subramanian <
codin.martial@gmail.com> wrote:
> Hi Eric,
>
> Answers to your questions are as follows
>
>
> >
> >
> >
> > *Can I, or is it recommended to, package multiple jobs as 1
> deploymentwith
> > 1 properties file or keep each app separated? Based on thedocumentation,
> > it appears to support 1 app/job within a singleconfiguration as there is
> no
> > mechanism to assign multiple app classes andgiven each a name unless I am
> > mistaken*
> >
>
> *app.class* is a single valued configuration and your understanding about
> it based on the documentation is correct.
>
>
> >
> >
> > *If only 1 app per config+deployment, what is the best way to
> > handlerequirement #3 - common/shared system services as there is no app
> or
> > jobper say, I just need to specify the streams and output system
> > (ieorg.apache.samza.system.hdfs.writer*
> >
>
> There are couple of options to achieve your #3 requirement.
>
> 1. If there is enough commonality between your jobs, you could have one
> application class that describes the logic and have the different
> configurations to modify the behavior of the application logic. This
> does
> come with some of the following considerations
> 1. Your deployment system needs to support deploying the same
> application with different configs.
> 2. Potential duplication of configuration if you configuration system
> doesn't support hierarchies and overrides.
> 3. Potentially unmanageable for evolution, since the change in
> application affects multiple jobs and requires extensive testing
> across
> different configurations.
> 2. You could potentially have libraries to perform some piece of
> business logic and have your different jobs leverage them using
> composition. Some things to consider with this option
> 1. Your application and configuration stay isolated.
> 2. You could still leverage some of the common configurations if your
> configuration system supports hierarchies and overrides
> 3. Alleviates concerns over evolution and testing as long as the
> changes are application specific.
>
>
> I am still unclear about the second part of your 2nd question.
> Do you mean to say all your jobs consume from same sources and write to
> sources and only your processing logic is different?
>
>
> > *common/shared system services as there is no app or jobper say, I just
> > need to specify the streams and output system*
>
>
> Also, I am not sure I follow what do you mean by "*there is no app or
> job"*.
> You still have 1 app per config + deployment, right?
>
> Thanks,
> Bharath
>
> On Thu, Oct 10, 2019 at 9:46 AM Eric Shieh <da...@gmail.com> wrote:
>
> > Hi,
> >
> > I am new to Samza, I am evaluating Samza as the backbone for my streaming
> > CEP requirement. I have:
> >
> > 1. Multiple data enrichment and ETL jobs
> > 2. Multiple domain specific CEP rulesets
> > 3. Common/shared system services like consuming topics/streams and
> > persisting the messages in ElasticSearch and HDFS.
> >
> > My questions are:
> >
> > 1. Can I, or is it recommended to, package multiple jobs as 1 deployment
> > with 1 properties file or keep each app separated? Based on the
> > documentation, it appears to support 1 app/job within a single
> > configuration as there is no mechanism to assign multiple app classes and
> > given each a name unless I am mistaken.
> > 2. If only 1 app per config+deployment, what is the best way to handle
> > requirement #3 - common/shared system services as there is no app or job
> > per say, I just need to specify the streams and output system (ie
> > org.apache.samza.system.hdfs.writer.
> > BinarySequenceFileHdfsWriter or
> >
> >
> org.apache.samza.system.elasticsearch.indexrequest.DefaultIndexRequestFactory).
> > Given it's a common shared system service not tied to specific jobs, can
> it
> > be deployed without an app?
> >
> > Thank you in advance for your help, looking forward to learning more
> about
> > Samza and developing this critical feature using Samza!
> >
> > Regards,
> >
> > Eric
> >
>
Re: Samza App Deployment Topology - Best Practices
Posted by Bharath Kumara Subramanian <co...@gmail.com>.
Hi Eric,
Answers to your questions are as follows
>
>
>
> *Can I, or is it recommended to, package multiple jobs as 1 deploymentwith
> 1 properties file or keep each app separated? Based on thedocumentation,
> it appears to support 1 app/job within a singleconfiguration as there is no
> mechanism to assign multiple app classes andgiven each a name unless I am
> mistaken*
>
*app.class* is a single valued configuration and your understanding about
it based on the documentation is correct.
>
>
> *If only 1 app per config+deployment, what is the best way to
> handlerequirement #3 - common/shared system services as there is no app or
> jobper say, I just need to specify the streams and output system
> (ieorg.apache.samza.system.hdfs.writer*
>
There are couple of options to achieve your #3 requirement.
1. If there is enough commonality between your jobs, you could have one
application class that describes the logic and have the different
configurations to modify the behavior of the application logic. This does
come with some of the following considerations
1. Your deployment system needs to support deploying the same
application with different configs.
2. Potential duplication of configuration if you configuration system
doesn't support hierarchies and overrides.
3. Potentially unmanageable for evolution, since the change in
application affects multiple jobs and requires extensive testing across
different configurations.
2. You could potentially have libraries to perform some piece of
business logic and have your different jobs leverage them using
composition. Some things to consider with this option
1. Your application and configuration stay isolated.
2. You could still leverage some of the common configurations if your
configuration system supports hierarchies and overrides
3. Alleviates concerns over evolution and testing as long as the
changes are application specific.
I am still unclear about the second part of your 2nd question.
Do you mean to say all your jobs consume from same sources and write to
sources and only your processing logic is different?
> *common/shared system services as there is no app or jobper say, I just
> need to specify the streams and output system*
Also, I am not sure I follow what do you mean by "*there is no app or job"*.
You still have 1 app per config + deployment, right?
Thanks,
Bharath
On Thu, Oct 10, 2019 at 9:46 AM Eric Shieh <da...@gmail.com> wrote:
> Hi,
>
> I am new to Samza, I am evaluating Samza as the backbone for my streaming
> CEP requirement. I have:
>
> 1. Multiple data enrichment and ETL jobs
> 2. Multiple domain specific CEP rulesets
> 3. Common/shared system services like consuming topics/streams and
> persisting the messages in ElasticSearch and HDFS.
>
> My questions are:
>
> 1. Can I, or is it recommended to, package multiple jobs as 1 deployment
> with 1 properties file or keep each app separated? Based on the
> documentation, it appears to support 1 app/job within a single
> configuration as there is no mechanism to assign multiple app classes and
> given each a name unless I am mistaken.
> 2. If only 1 app per config+deployment, what is the best way to handle
> requirement #3 - common/shared system services as there is no app or job
> per say, I just need to specify the streams and output system (ie
> org.apache.samza.system.hdfs.writer.
> BinarySequenceFileHdfsWriter or
>
> org.apache.samza.system.elasticsearch.indexrequest.DefaultIndexRequestFactory).
> Given it's a common shared system service not tied to specific jobs, can it
> be deployed without an app?
>
> Thank you in advance for your help, looking forward to learning more about
> Samza and developing this critical feature using Samza!
>
> Regards,
>
> Eric
>