You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Justin Maltat <ju...@gmail.com> on 2014/08/19 00:53:29 UTC
Using kafka in non million users environment
Hello,
I'm managing a study to explore possibilities for migrating a monolith
architecture IT to a service oriented one.
That said, the company i'm working for is not a web based company, so
the number of user (i.e: the load) is not the heart of the issue, but
data/services diversity is.
I'm truly interested in Kafka as the main data pipeline because it'll
enable us abstracting storages from services, and as a consequence
having a better data management.
I would like to engineer a proof of concept around Kafka and Samza but
before going further into digging, I would like to have your advice
regarding this project in order to know weather your software will be
of use to a more 'common' company IT structure.
Thanks in advance for your insights.
Best regards
Justin Maltat
Re: Using kafka in non million users environment
Posted by Philip O'Toole <ph...@yahoo.com.INVALID>.
>>- we have a low data traffic compared to your figures: around 30 GB a
day. Will it be an issue?
I have personal experience that Kafka deals extremely well with very low-volumes, as well as very high. I have used Kafka for small integration-test setups, as well as large production systems. Kafka -- and the associated Producer-Consumer framework code -- has just the right configuration options so it can be tuned for a wide-variety of deployments.
Philip
-----------------------------------------
http://www.philipotoole.com
Re: Using kafka in non million users environment
Posted by Justin Maltat <ju...@gmail.com>.
Thanks Philip and Anand for the hints.
I fill more comfortable going further now.
On Wed, Aug 20, 2014 at 6:49 PM, Anand Nalya <an...@gmail.com> wrote:
> For operating kafka across multiple data centers have a look at
> https://kafka.apache.org/08/ops.html and MirrorMaker (
> https://kafka.apache.org/08/tools.html)
>
>
> On 20 August 2014 04:09, Justin Maltat <ju...@gmail.com> wrote:
>
>> Hi,
>>
>> As of today, our company IT is mainly composed of domain specific
>> software (proprietary and homemade). We could like to migrate them one
>> after another to a microservice architecture with Kafka as the data
>> pipeline. With the system now in place it's quite difficult to have a
>> common data flow because there is numerous and various input types and
>> all of them have their own integration within the IT infrastructure.
>>
>> According to our readings, Kafka has been built for managing huge data
>> trafic so its functionnalities have been thought this way.
>> Hence, some questions arised from our reflection:
>> - we have a low data traffic compared to your figures: around 30 GB a
>> day. Will it be an issue?
>> - we live in a country with low network bandwith, so partitioning over
>> WAN links is a hight risk and we have multiple access points scattered
>> around the country. Would it be possible to configure Kafka to work
>> with such an environment: during network patition, an isolated node
>> becomes its own master, on partition ending data merges, merge quality
>> would be ensured by ensuring access point only write data that he
>> managed. Or is there a better way to deal with this issue?
>>
>> Also, we would like to know what are the minimal server hardware
>> requirements for using Kafka with our data trafic?
>>
>> For information, the idea for this project came to us after reading
>> this enlightening blog post:
>>
>> http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
>>
>> Thanks for your time
>>
>> Best regards
>>
>> On Tue, Aug 19, 2014 at 10:18 PM, Daniel Compton <de...@danielcompton.net>
>> wrote:
>> > Hi Justin
>> >
>> > It sounds like Kafka could be a good fit for your environment. Are you
>> able to tell us more about the kinds of applications you will be running?
>> >
>> > Daniel.
>> >
>> >> On 19/08/2014, at 10:53 am, Justin Maltat <ju...@gmail.com>
>> wrote:
>> >>
>> >> Hello,
>> >>
>> >> I'm managing a study to explore possibilities for migrating a monolith
>> >> architecture IT to a service oriented one.
>> >> That said, the company i'm working for is not a web based company, so
>> >> the number of user (i.e: the load) is not the heart of the issue, but
>> >> data/services diversity is.
>> >>
>> >> I'm truly interested in Kafka as the main data pipeline because it'll
>> >> enable us abstracting storages from services, and as a consequence
>> >> having a better data management.
>> >> I would like to engineer a proof of concept around Kafka and Samza but
>> >> before going further into digging, I would like to have your advice
>> >> regarding this project in order to know weather your software will be
>> >> of use to a more 'common' company IT structure.
>> >>
>> >> Thanks in advance for your insights.
>> >>
>> >>
>> >> Best regards
>> >>
>> >> Justin Maltat
>>
Re: Using kafka in non million users environment
Posted by Anand Nalya <an...@gmail.com>.
For operating kafka across multiple data centers have a look at
https://kafka.apache.org/08/ops.html and MirrorMaker (
https://kafka.apache.org/08/tools.html)
On 20 August 2014 04:09, Justin Maltat <ju...@gmail.com> wrote:
> Hi,
>
> As of today, our company IT is mainly composed of domain specific
> software (proprietary and homemade). We could like to migrate them one
> after another to a microservice architecture with Kafka as the data
> pipeline. With the system now in place it's quite difficult to have a
> common data flow because there is numerous and various input types and
> all of them have their own integration within the IT infrastructure.
>
> According to our readings, Kafka has been built for managing huge data
> trafic so its functionnalities have been thought this way.
> Hence, some questions arised from our reflection:
> - we have a low data traffic compared to your figures: around 30 GB a
> day. Will it be an issue?
> - we live in a country with low network bandwith, so partitioning over
> WAN links is a hight risk and we have multiple access points scattered
> around the country. Would it be possible to configure Kafka to work
> with such an environment: during network patition, an isolated node
> becomes its own master, on partition ending data merges, merge quality
> would be ensured by ensuring access point only write data that he
> managed. Or is there a better way to deal with this issue?
>
> Also, we would like to know what are the minimal server hardware
> requirements for using Kafka with our data trafic?
>
> For information, the idea for this project came to us after reading
> this enlightening blog post:
>
> http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
>
> Thanks for your time
>
> Best regards
>
> On Tue, Aug 19, 2014 at 10:18 PM, Daniel Compton <de...@danielcompton.net>
> wrote:
> > Hi Justin
> >
> > It sounds like Kafka could be a good fit for your environment. Are you
> able to tell us more about the kinds of applications you will be running?
> >
> > Daniel.
> >
> >> On 19/08/2014, at 10:53 am, Justin Maltat <ju...@gmail.com>
> wrote:
> >>
> >> Hello,
> >>
> >> I'm managing a study to explore possibilities for migrating a monolith
> >> architecture IT to a service oriented one.
> >> That said, the company i'm working for is not a web based company, so
> >> the number of user (i.e: the load) is not the heart of the issue, but
> >> data/services diversity is.
> >>
> >> I'm truly interested in Kafka as the main data pipeline because it'll
> >> enable us abstracting storages from services, and as a consequence
> >> having a better data management.
> >> I would like to engineer a proof of concept around Kafka and Samza but
> >> before going further into digging, I would like to have your advice
> >> regarding this project in order to know weather your software will be
> >> of use to a more 'common' company IT structure.
> >>
> >> Thanks in advance for your insights.
> >>
> >>
> >> Best regards
> >>
> >> Justin Maltat
>
Re: Using kafka in non million users environment
Posted by Justin Maltat <ju...@gmail.com>.
Hi,
As of today, our company IT is mainly composed of domain specific
software (proprietary and homemade). We could like to migrate them one
after another to a microservice architecture with Kafka as the data
pipeline. With the system now in place it's quite difficult to have a
common data flow because there is numerous and various input types and
all of them have their own integration within the IT infrastructure.
According to our readings, Kafka has been built for managing huge data
trafic so its functionnalities have been thought this way.
Hence, some questions arised from our reflection:
- we have a low data traffic compared to your figures: around 30 GB a
day. Will it be an issue?
- we live in a country with low network bandwith, so partitioning over
WAN links is a hight risk and we have multiple access points scattered
around the country. Would it be possible to configure Kafka to work
with such an environment: during network patition, an isolated node
becomes its own master, on partition ending data merges, merge quality
would be ensured by ensuring access point only write data that he
managed. Or is there a better way to deal with this issue?
Also, we would like to know what are the minimal server hardware
requirements for using Kafka with our data trafic?
For information, the idea for this project came to us after reading
this enlightening blog post:
http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
Thanks for your time
Best regards
On Tue, Aug 19, 2014 at 10:18 PM, Daniel Compton <de...@danielcompton.net> wrote:
> Hi Justin
>
> It sounds like Kafka could be a good fit for your environment. Are you able to tell us more about the kinds of applications you will be running?
>
> Daniel.
>
>> On 19/08/2014, at 10:53 am, Justin Maltat <ju...@gmail.com> wrote:
>>
>> Hello,
>>
>> I'm managing a study to explore possibilities for migrating a monolith
>> architecture IT to a service oriented one.
>> That said, the company i'm working for is not a web based company, so
>> the number of user (i.e: the load) is not the heart of the issue, but
>> data/services diversity is.
>>
>> I'm truly interested in Kafka as the main data pipeline because it'll
>> enable us abstracting storages from services, and as a consequence
>> having a better data management.
>> I would like to engineer a proof of concept around Kafka and Samza but
>> before going further into digging, I would like to have your advice
>> regarding this project in order to know weather your software will be
>> of use to a more 'common' company IT structure.
>>
>> Thanks in advance for your insights.
>>
>>
>> Best regards
>>
>> Justin Maltat
Re: Using kafka in non million users environment
Posted by Daniel Compton <de...@danielcompton.net>.
Hi Justin
It sounds like Kafka could be a good fit for your environment. Are you able to tell us more about the kinds of applications you will be running?
Daniel.
> On 19/08/2014, at 10:53 am, Justin Maltat <ju...@gmail.com> wrote:
>
> Hello,
>
> I'm managing a study to explore possibilities for migrating a monolith
> architecture IT to a service oriented one.
> That said, the company i'm working for is not a web based company, so
> the number of user (i.e: the load) is not the heart of the issue, but
> data/services diversity is.
>
> I'm truly interested in Kafka as the main data pipeline because it'll
> enable us abstracting storages from services, and as a consequence
> having a better data management.
> I would like to engineer a proof of concept around Kafka and Samza but
> before going further into digging, I would like to have your advice
> regarding this project in order to know weather your software will be
> of use to a more 'common' company IT structure.
>
> Thanks in advance for your insights.
>
>
> Best regards
>
> Justin Maltat