Advice for Kafka project in Africa...

[cedric sende lubuele <se...@hotmail.com>]

Let me introduce myself, my name is Cedric and I am a network engineer passionate about new technologies and as part of my new activity, I am interested in Big Data. Currently, I live in Africa (Congo) and as everyone knows, Africa is very late in terms of IT infrastructure (the Cloud is difficult, we work a lot on premise).

While doing some research, I came across Kai Waehner's article (Kafka replace database?<https://www.kai-waehner.de/blog/2020/03/12/can-apache-kafka-replace-database-acid-storage-%20transactions-sql-nosql-data-lake%20/>) and I would like to be able to get an idea about the possibilities of Kafka.

Let me explain, I am working on a project for integrating several databases (MySQL, MongoDB, Neo4j, ...) and I have to develop with my team, an alert system which must detect anomalies on different criteria linked to a person in the various departments of the company.
Would Kafka be a good solution in order to centralize all the data and create several analysis scripts to detect an anomaly and send back an alert message such as for example a suspect wanted by the police?

Thank you in advance



Sende Cedric / Network IT
sende.cedric@hotmail.com<ma...@hotmail.com> / 082/8446954

UPSAIL GROUP
http://upsail.co/<https://htmlsig.com/t/000001BFBBXF>

[http://upsail.co/wp-content/themes/upsail/images/logo.png]

Re: Advice for Kafka project in Africa...

[Dave Klein <da...@usa.net>]

I’m no expert, but I think Kafka, Kafka Connect and Kafka Streams could be a great fit for your use case.  

Use Kafka Connect to pull data from the various data sources, into a topic or topics and then use Kafka Streams to do the joins, enrichment, and / or analysis needed to determine a record of interest. Send those records to another topic and have a simple consumer app watching that topic and sending notifications.   Sounds like a fun project! 

Dave

> On Sep 2, 2020, at 11:01 AM, cedric sende lubuele <se...@hotmail.com> wrote:
> 
> Let me introduce myself, my name is Cedric and I am a network engineer passionate about new technologies and as part of my new activity, I am interested in Big Data. Currently, I live in Africa (Congo) and as everyone knows, Africa is very late in terms of IT infrastructure (the Cloud is difficult, we work a lot on premise).
> 
> While doing some research, I came across Kai Waehner's article (Kafka replace database?<https://www.kai-waehner.de/blog/2020/03/12/can-apache-kafka-replace-database-acid-storage-%20transactions-sql-nosql-data-lake%20/>) and I would like to be able to get an idea about the possibilities of Kafka.
> 
> Let me explain, I am working on a project for integrating several databases (MySQL, MongoDB, Neo4j, ...) and I have to develop with my team, an alert system which must detect anomalies on different criteria linked to a person in the various departments of the company.
> Would Kafka be a good solution in order to centralize all the data and create several analysis scripts to detect an anomaly and send back an alert message such as for example a suspect wanted by the police?
> 
> Thank you in advance
> 
> 
> 
> Sende Cedric / Network IT
> sende.cedric@hotmail.com<ma...@hotmail.com> / 082/8446954
> 
> UPSAIL GROUP
> http://upsail.co/<https://htmlsig.com/t/000001BFBBXF>
> 
> [http://upsail.co/wp-content/themes/upsail/images/logo.png]

Re: Advice for Kafka project in Africa...

[jan <rt...@googlemail.com.INVALID>]

Everything's down to requirements. I'm unclear on yours. I tried to
look at your website to see if I could pick up any clues but upsail.co
(and just in case, upsail.com) don't exist, so I dunno.

Some questions, rhetorical really:

Is there any reason a standard SQL database would not do - what does
kafka offer that you think is useful? I presume you know SQL already,

What are you trying to do, more precisely? Do you have a clear idea of
the inputs and outputs? What is the amount of data you expect to
process? Is there some tight time limit?

What is your uptime requirement? What is your budget?

From my experience, mentioning big data at this point is a major red
flag. Make sure you really can't do it on a laptop or a server (both
in money and business requirements) before you start scaling out
because it gets messy. A decent server can crunch a *lot* of data if
well configured.

Also consider your infrastructure such as power supplies and
networking protection as Africa may not be too stable in those
regards. Also perhaps physical protection.
If you skip your own hardware and go for 'the cloud' then you are
totally in the hands of the networking gods.

These are points to mull over. Doubt I can suggest anything further. Good luck.

jan

On 02/09/2020, cedric sende lubuele <se...@hotmail.com> wrote:
> Let me introduce myself, my name is Cedric and I am a network engineer
> passionate about new technologies and as part of my new activity, I am
> interested in Big Data. Currently, I live in Africa (Congo) and as everyone
> knows, Africa is very late in terms of IT infrastructure (the Cloud is
> difficult, we work a lot on premise).
>
> While doing some research, I came across Kai Waehner's article (Kafka
> replace
> database?<https://www.kai-waehner.de/blog/2020/03/12/can-apache-kafka-replace-database-acid-storage-%20transactions-sql-nosql-data-lake%20/>)
> and I would like to be able to get an idea about the possibilities of
> Kafka.
>
> Let me explain, I am working on a project for integrating several databases
> (MySQL, MongoDB, Neo4j, ...) and I have to develop with my team, an alert
> system which must detect anomalies on different criteria linked to a person
> in the various departments of the company.
> Would Kafka be a good solution in order to centralize all the data and
> create several analysis scripts to detect an anomaly and send back an alert
> message such as for example a suspect wanted by the police?
>
> Thank you in advance
>
>
>
> Sende Cedric / Network IT
> sende.cedric@hotmail.com<ma...@hotmail.com> / 082/8446954
>
> UPSAIL GROUP
> http://upsail.co/<https://htmlsig.com/t/000001BFBBXF>
>
> [http://upsail.co/wp-content/themes/upsail/images/logo.png]
>

Re: Advice for Kafka project in Africa...

[Malcolm McFarland <mm...@cavulus.com>]

Hi Cedric,

I'm not sure if this is directly applicable, but you might get a feel for
how Kafka relates to more traditional databases from this post (afaik this
is one of the seminal talks relating the two ideas):
https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/

You can think of Kafka as a durable record of updates to the database, but
not a full view of the current database state itself. Kafka would be a
reliable place to store events that you want to record in multiple places,
and you could then add a processor for these events (ie Kafka Streams or
Samza) to actually update a database state from this event data. However,
the events themselves are not the full database.

I haven't explored ksqlDB at all; perhaps somebody familiar with it could
weigh in here.

Cheers,
Malcolm McFarland
Cavulus


On Thu, Sep 3, 2020 at 12:58 AM Gaurav Bajaj <ga...@gmail.com> wrote:

> Hello Cedric,
>
> My 2 cents :
>
> We use Kafka alot but mostly for messaging and event streaming purpose.
> Using Kafka as Database is not a usecase i would look Kafka for. Ofcourse
> you can use it to store some intermediate states but having it as "system
> of records" would be stretched use case for Kafka.
>
> Best Regards,
> Gaurav
>
> On Wed, Sep 2, 2020, 9:31 PM cedric sende lubuele <
> sende.cedric@hotmail.com>
> wrote:
>
> > Let me introduce myself, my name is Cedric and I am a network engineer
> > passionate about new technologies and as part of my new activity, I am
> > interested in Big Data. Currently, I live in Africa (Congo) and as
> everyone
> > knows, Africa is very late in terms of IT infrastructure (the Cloud is
> > difficult, we work a lot on premise).
> >
> > While doing some research, I came across Kai Waehner's article (Kafka
> > replace database?<
> >
> https://www.kai-waehner.de/blog/2020/03/12/can-apache-kafka-replace-database-acid-storage-%20transactions-sql-nosql-data-lake%20/
> >)
> > and I would like to be able to get an idea about the possibilities of
> Kafka.
> >
> > Let me explain, I am working on a project for integrating several
> > databases (MySQL, MongoDB, Neo4j, ...) and I have to develop with my
> team,
> > an alert system which must detect anomalies on different criteria linked
> to
> > a person in the various departments of the company.
> > Would Kafka be a good solution in order to centralize all the data and
> > create several analysis scripts to detect an anomaly and send back an
> alert
> > message such as for example a suspect wanted by the police?
> >
> > Thank you in advance
> >
> >
> >
> > Sende Cedric / Network IT
> > sende.cedric@hotmail.com<ma...@hotmail.com> / 082/8446954
> >
> > UPSAIL GROUP
> > http://upsail.co/<https://htmlsig.com/t/000001BFBBXF>
> >
> > [http://upsail.co/wp-content/themes/upsail/images/logo.png]
> >
>

Re: Advice for Kafka project in Africa...

[Gaurav Bajaj <ga...@gmail.com>]

Hello Cedric,

My 2 cents :

We use Kafka alot but mostly for messaging and event streaming purpose.
Using Kafka as Database is not a usecase i would look Kafka for. Ofcourse
you can use it to store some intermediate states but having it as "system
of records" would be stretched use case for Kafka.

Best Regards,
Gaurav

On Wed, Sep 2, 2020, 9:31 PM cedric sende lubuele <se...@hotmail.com>
wrote:

> Let me introduce myself, my name is Cedric and I am a network engineer
> passionate about new technologies and as part of my new activity, I am
> interested in Big Data. Currently, I live in Africa (Congo) and as everyone
> knows, Africa is very late in terms of IT infrastructure (the Cloud is
> difficult, we work a lot on premise).
>
> While doing some research, I came across Kai Waehner's article (Kafka
> replace database?<
> https://www.kai-waehner.de/blog/2020/03/12/can-apache-kafka-replace-database-acid-storage-%20transactions-sql-nosql-data-lake%20/>)
> and I would like to be able to get an idea about the possibilities of Kafka.
>
> Let me explain, I am working on a project for integrating several
> databases (MySQL, MongoDB, Neo4j, ...) and I have to develop with my team,
> an alert system which must detect anomalies on different criteria linked to
> a person in the various departments of the company.
> Would Kafka be a good solution in order to centralize all the data and
> create several analysis scripts to detect an anomaly and send back an alert
> message such as for example a suspect wanted by the police?
>
> Thank you in advance
>
>
>
> Sende Cedric / Network IT
> sende.cedric@hotmail.com<ma...@hotmail.com> / 082/8446954
>
> UPSAIL GROUP
> http://upsail.co/<https://htmlsig.com/t/000001BFBBXF>
>
> [http://upsail.co/wp-content/themes/upsail/images/logo.png]
>