You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Chanchal Chatterji <ch...@infosys.com> on 2018/09/12 07:31:48 UTC
Need info
Hi,
In the process of mainframe modernization, we are attempting to stream Mainframe data to AWS Cloud , using Kafka. We are planning to use Kafka 'Producer API' at mainframe side and 'Connector API' on the cloud side.
Since our data is processed by a module called 'Central dispatch' located in Mainframe and is sent to Kafka. We want to know what is rate of volume Kafka can handle. The other end of Kafka is connected to an AWS S3 Bucket
As sink. Please help us to provide this info or else please help to connect with relevant person who can help us to understand this.
Thanks and Regards
Chanchal Chatterji
Principal Consultant,
Infosys Ltd.
Electronic city Phase-1,
Bangalore-560100
Contacts : 9731141606/ 8105120766
RE: Need info
Posted by "Tauzell, Dave" <Da...@surescripts.com>.
If you size your cluster right, you can send large messages of many megabytes. We send lots (millions per day) of medium sized messages (5-10k) without any issues.
-Dave
-----Original Message-----
From: Chanchal Chatterji [mailto:chanchal.chatterji@infosys.com]
Sent: Wednesday, September 12, 2018 4:00 AM
To: users@kafka.apache.org
Subject: [External] RE: Need info
In simple words it is like :
We have MF application which is sending statement data to Kafka from internal data sources after some processing . Which would later be pushed to cloud ( through Kafka) and will be staged in Amazon S3 bucket in cloud.
The first time entire relevant data will be pushed ( By MF Application ) to the cloud and then from the next time onward, any data which would be updated ( Any new transaction in MF ) will be sent to the cloud through Kafka.
Can kafka support this type of streaming ? I am more concerned to understand what is the max sixe of intake of data Kafka can handle in use cases like this.
Please help!
Regards
Chanchal Chatterji
Principal Consultant,
Infosys Ltd.
Electronic city Phase-1,
Bangalore-560100
Contacts : 9731141606/ 8105120766
-----Original Message-----
From: Mich Talebzadeh <mi...@gmail.com>
Sent: Wednesday, September 12, 2018 2:16 PM
To: users@kafka.apache.org
Subject: Re: Need info
Hi,
As I understand you are trying to create an operational data store from your transactional database(s) upstream?
Do you have stats on the rate of DML in the primary source? These insert/update/deletes need to pass to Kafka as messages. Besides what Kafka can handle (largely depending on the architecture you have created and your Kafka cluster plus Zookeeper ensemble.), you will be constraint by the rate that you will be allowed to extract data from Mainframe and your shared network. Also for Kafka and Zookeeper you may think about deploying these as microservices.
HTH,
Dr Mich Talebzadeh
LinkedIn * https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fprofile%2Fview%3Fid%3DAAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw&data=01%7C01%7Cchanchal.chatterji%40infosys.com%7C9292c112fe0c4a30a77608d6188c60f2%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C1&sdata=uRUnySk5RbJAwbnomkiAY0yf28jADHZ%2B0%2FGLn0fQxjU%3D&reserved=0
<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fprofile%2Fview%3Fid%3DAAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw&data=01%7C01%7Cchanchal.chatterji%40infosys.com%7C9292c112fe0c4a30a77608d6188c60f2%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C1&sdata=uRUnySk5RbJAwbnomkiAY0yf28jADHZ%2B0%2FGLn0fQxjU%3D&reserved=0>*
https://apac01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftalebzadehmich.wordpress.com&data=01%7C01%7Cchanchal.chatterji%40infosys.com%7C9292c112fe0c4a30a77608d6188c60f2%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C1&sdata=bW0wyOvsEFfrvG%2F%2Bdf%2B72cIaB1kojxEPe7MrpeDvx58%3D&reserved=0
*Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On Wed, 12 Sep 2018 at 08:32, Chanchal Chatterji < chanchal.chatterji@infosys.com> wrote:
> Hi,
>
> In the process of mainframe modernization, we are attempting to stream
> Mainframe data to AWS Cloud , using Kafka. We are planning to use
> Kafka 'Producer API' at mainframe side and 'Connector API' on the cloud side.
> Since our data is processed by a module called 'Central dispatch'
> located in Mainframe and is sent to Kafka. We want to know what is
> rate of volume Kafka can handle. The other end of Kafka is connected
> to an AWS S3 Bucket As sink. Please help us to provide this info or
> else please help to connect with relevant person who can help us to understand this.
>
> Thanks and Regards
>
> Chanchal Chatterji
> Principal Consultant,
> Infosys Ltd.
> Electronic city Phase-1,
> Bangalore-560100
> Contacts : 9731141606/ 8105120766
>
>
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
Re: Need info
Posted by Liam Clarke <li...@adscale.co.nz>.
So you need to figure out your needs. Kafka can deliver near real time
streaming, and it can function as a data store. It can handle significantly
large messages if you want, but there are tradeoffs - you'd obviously need
more hardware.
I have no idea how many MB a bank transaction is, but you need to ask that
question, figure out an expected messaging rate per second and then you can
estimate the cluster size you need to get what you want.
On Wed, 12 Sep. 2018, 9:00 pm Chanchal Chatterji, <
chanchal.chatterji@infosys.com> wrote:
> In simple words it is like :
>
> We have MF application which is sending statement data to Kafka from
> internal data sources after some processing . Which would later be pushed
> to cloud ( through Kafka) and will be staged in Amazon S3 bucket in cloud.
> The first time entire relevant data will be pushed ( By MF Application )
> to the cloud and then from the next time onward, any data which would be
> updated ( Any new transaction in MF ) will be sent to the cloud through
> Kafka.
> Can kafka support this type of streaming ? I am more concerned to
> understand what is the max sixe of intake of data Kafka can handle in use
> cases like this.
>
> Please help!
>
> Regards
>
> Chanchal Chatterji
> Principal Consultant,
> Infosys Ltd.
> Electronic city Phase-1,
> Bangalore-560100
> Contacts : 9731141606/ 8105120766
>
>
>
>
> -----Original Message-----
> From: Mich Talebzadeh <mi...@gmail.com>
> Sent: Wednesday, September 12, 2018 2:16 PM
> To: users@kafka.apache.org
> Subject: Re: Need info
>
> Hi,
>
> As I understand you are trying to create an operational data store from
> your transactional database(s) upstream?
>
> Do you have stats on the rate of DML in the primary source? These
> insert/update/deletes need to pass to Kafka as messages. Besides what Kafka
> can handle (largely depending on the architecture you have created and your
> Kafka cluster plus Zookeeper ensemble.), you will be constraint by the rate
> that you will be allowed to extract data from Mainframe and your shared
> network. Also for Kafka and Zookeeper you may think about deploying these
> as microservices.
>
> HTH,
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn *
> https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fprofile%2Fview%3Fid%3DAAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw&data=01%7C01%7Cchanchal.chatterji%40infosys.com%7C9292c112fe0c4a30a77608d6188c60f2%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C1&sdata=uRUnySk5RbJAwbnomkiAY0yf28jADHZ%2B0%2FGLn0fQxjU%3D&reserved=0
> <
> https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fprofile%2Fview%3Fid%3DAAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw&data=01%7C01%7Cchanchal.chatterji%40infosys.com%7C9292c112fe0c4a30a77608d6188c60f2%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C1&sdata=uRUnySk5RbJAwbnomkiAY0yf28jADHZ%2B0%2FGLn0fQxjU%3D&reserved=0
> >*
>
>
>
>
> https://apac01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftalebzadehmich.wordpress.com&data=01%7C01%7Cchanchal.chatterji%40infosys.com%7C9292c112fe0c4a30a77608d6188c60f2%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C1&sdata=bW0wyOvsEFfrvG%2F%2Bdf%2B72cIaB1kojxEPe7MrpeDvx58%3D&reserved=0
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Wed, 12 Sep 2018 at 08:32, Chanchal Chatterji <
> chanchal.chatterji@infosys.com> wrote:
>
> > Hi,
> >
> > In the process of mainframe modernization, we are attempting to stream
> > Mainframe data to AWS Cloud , using Kafka. We are planning to use
> > Kafka 'Producer API' at mainframe side and 'Connector API' on the cloud
> side.
> > Since our data is processed by a module called 'Central dispatch'
> > located in Mainframe and is sent to Kafka. We want to know what is
> > rate of volume Kafka can handle. The other end of Kafka is connected
> > to an AWS S3 Bucket As sink. Please help us to provide this info or
> > else please help to connect with relevant person who can help us to
> understand this.
> >
> > Thanks and Regards
> >
> > Chanchal Chatterji
> > Principal Consultant,
> > Infosys Ltd.
> > Electronic city Phase-1,
> > Bangalore-560100
> > Contacts : 9731141606/ 8105120766
> >
> >
>
RE: Need info
Posted by Chanchal Chatterji <ch...@infosys.com>.
In simple words it is like :
We have MF application which is sending statement data to Kafka from internal data sources after some processing . Which would later be pushed to cloud ( through Kafka) and will be staged in Amazon S3 bucket in cloud.
The first time entire relevant data will be pushed ( By MF Application ) to the cloud and then from the next time onward, any data which would be updated ( Any new transaction in MF ) will be sent to the cloud through Kafka.
Can kafka support this type of streaming ? I am more concerned to understand what is the max sixe of intake of data Kafka can handle in use cases like this.
Please help!
Regards
Chanchal Chatterji
Principal Consultant,
Infosys Ltd.
Electronic city Phase-1,
Bangalore-560100
Contacts : 9731141606/ 8105120766
-----Original Message-----
From: Mich Talebzadeh <mi...@gmail.com>
Sent: Wednesday, September 12, 2018 2:16 PM
To: users@kafka.apache.org
Subject: Re: Need info
Hi,
As I understand you are trying to create an operational data store from your transactional database(s) upstream?
Do you have stats on the rate of DML in the primary source? These insert/update/deletes need to pass to Kafka as messages. Besides what Kafka can handle (largely depending on the architecture you have created and your Kafka cluster plus Zookeeper ensemble.), you will be constraint by the rate that you will be allowed to extract data from Mainframe and your shared network. Also for Kafka and Zookeeper you may think about deploying these as microservices.
HTH,
Dr Mich Talebzadeh
LinkedIn * https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fprofile%2Fview%3Fid%3DAAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw&data=01%7C01%7Cchanchal.chatterji%40infosys.com%7C9292c112fe0c4a30a77608d6188c60f2%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C1&sdata=uRUnySk5RbJAwbnomkiAY0yf28jADHZ%2B0%2FGLn0fQxjU%3D&reserved=0
<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fprofile%2Fview%3Fid%3DAAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw&data=01%7C01%7Cchanchal.chatterji%40infosys.com%7C9292c112fe0c4a30a77608d6188c60f2%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C1&sdata=uRUnySk5RbJAwbnomkiAY0yf28jADHZ%2B0%2FGLn0fQxjU%3D&reserved=0>*
https://apac01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftalebzadehmich.wordpress.com&data=01%7C01%7Cchanchal.chatterji%40infosys.com%7C9292c112fe0c4a30a77608d6188c60f2%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C1&sdata=bW0wyOvsEFfrvG%2F%2Bdf%2B72cIaB1kojxEPe7MrpeDvx58%3D&reserved=0
*Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On Wed, 12 Sep 2018 at 08:32, Chanchal Chatterji < chanchal.chatterji@infosys.com> wrote:
> Hi,
>
> In the process of mainframe modernization, we are attempting to stream
> Mainframe data to AWS Cloud , using Kafka. We are planning to use
> Kafka 'Producer API' at mainframe side and 'Connector API' on the cloud side.
> Since our data is processed by a module called 'Central dispatch'
> located in Mainframe and is sent to Kafka. We want to know what is
> rate of volume Kafka can handle. The other end of Kafka is connected
> to an AWS S3 Bucket As sink. Please help us to provide this info or
> else please help to connect with relevant person who can help us to understand this.
>
> Thanks and Regards
>
> Chanchal Chatterji
> Principal Consultant,
> Infosys Ltd.
> Electronic city Phase-1,
> Bangalore-560100
> Contacts : 9731141606/ 8105120766
>
>
Re: Need info
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi,
As I understand you are trying to create an operational data store from
your transactional database(s) upstream?
Do you have stats on the rate of DML in the primary source? These
insert/update/deletes need to pass to Kafka as messages. Besides what Kafka
can handle (largely depending on the architecture you have created and your
Kafka cluster plus Zookeeper ensemble.), you will be constraint by the rate
that you will be allowed to extract data from Mainframe and your shared
network. Also for Kafka and Zookeeper you may think about deploying these
as microservices.
HTH,
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Wed, 12 Sep 2018 at 08:32, Chanchal Chatterji <
chanchal.chatterji@infosys.com> wrote:
> Hi,
>
> In the process of mainframe modernization, we are attempting to stream
> Mainframe data to AWS Cloud , using Kafka. We are planning to use Kafka
> 'Producer API' at mainframe side and 'Connector API' on the cloud side.
> Since our data is processed by a module called 'Central dispatch' located
> in Mainframe and is sent to Kafka. We want to know what is rate of volume
> Kafka can handle. The other end of Kafka is connected to an AWS S3 Bucket
> As sink. Please help us to provide this info or else please help to
> connect with relevant person who can help us to understand this.
>
> Thanks and Regards
>
> Chanchal Chatterji
> Principal Consultant,
> Infosys Ltd.
> Electronic city Phase-1,
> Bangalore-560100
> Contacts : 9731141606/ 8105120766
>
>
RE: Need info
Posted by Chanchal Chatterji <ch...@infosys.com>.
We are planning to produce Bank statements out of data traversing through the Kafka. ( A simple example of it would be Bank statement for saving account / current account in printable format in our daily life.
So your three suggestions :
1. Build your cluster right
2. Size your Message right
3. Tune your producer right
You mean if we have correct size set for these three we can flow arbitrary data through Kafka?
Also if you have any observation on how much typical 'Bank Statement' message could be ( in MBs), and if you can share that , it would be great help.
Regards
Chanchal Chatterji
Principal Consultant,
Infosys Ltd.
Electronic city Phase-1,
Bangalore-560100
Contacts : 9731141606/ 8105120766
-----Original Message-----
From: Liam Clarke <li...@adscale.co.nz>
Sent: Wednesday, September 12, 2018 1:10 PM
To: users@kafka.apache.org
Subject: Re: Need info
The answer to your question is "It depends". You build your cluster right and size your messages right and tune your producers right, you can achieve near real time transport of terabytes of data a day.
There's been plenty of articles written about Kafka performance. E.g.,
https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fengineering.linkedin.com%2Fkafka%2Fbenchmarking-apache-kafka-2-million-writes-second-three-cheap-machines&data=01%7C01%7Cchanchal.chatterji%40infosys.com%7Cf7bece6231274ea7025e08d61882f2bf%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C1&sdata=Lp%2BsRFXrPRU3TXH9CP0bgLbj1AZ6cWYLge6sUN8aKJY%3D&reserved=0
Kind regards,
Liam Clarke
On Wed, 12 Sep. 2018, 7:32 pm Chanchal Chatterji, < chanchal.chatterji@infosys.com> wrote:
> Hi,
>
> In the process of mainframe modernization, we are attempting to stream
> Mainframe data to AWS Cloud , using Kafka. We are planning to use
> Kafka 'Producer API' at mainframe side and 'Connector API' on the cloud side.
> Since our data is processed by a module called 'Central dispatch'
> located in Mainframe and is sent to Kafka. We want to know what is
> rate of volume Kafka can handle. The other end of Kafka is connected
> to an AWS S3 Bucket As sink. Please help us to provide this info or
> else please help to connect with relevant person who can help us to understand this.
>
> Thanks and Regards
>
> Chanchal Chatterji
> Principal Consultant,
> Infosys Ltd.
> Electronic city Phase-1,
> Bangalore-560100
> Contacts : 9731141606/ 8105120766
>
>
Re: Need info
Posted by Liam Clarke <li...@adscale.co.nz>.
The answer to your question is "It depends". You build your cluster right
and size your messages right and tune your producers right, you can achieve
near real time transport of terabytes of data a day.
There's been plenty of articles written about Kafka performance. E.g.,
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
Kind regards,
Liam Clarke
On Wed, 12 Sep. 2018, 7:32 pm Chanchal Chatterji, <
chanchal.chatterji@infosys.com> wrote:
> Hi,
>
> In the process of mainframe modernization, we are attempting to stream
> Mainframe data to AWS Cloud , using Kafka. We are planning to use Kafka
> 'Producer API' at mainframe side and 'Connector API' on the cloud side.
> Since our data is processed by a module called 'Central dispatch' located
> in Mainframe and is sent to Kafka. We want to know what is rate of volume
> Kafka can handle. The other end of Kafka is connected to an AWS S3 Bucket
> As sink. Please help us to provide this info or else please help to
> connect with relevant person who can help us to understand this.
>
> Thanks and Regards
>
> Chanchal Chatterji
> Principal Consultant,
> Infosys Ltd.
> Electronic city Phase-1,
> Bangalore-560100
> Contacts : 9731141606/ 8105120766
>
>
RE: Need info
Posted by Chanchal Chatterji <ch...@infosys.com>.
Robin,
Thanks for writing back. I am aware of this Kafka Connector. Do you have any info on Kafka connecting to Mainframe?
Regards
Chanchal
-----Original Message-----
From: Robin Moffatt <ro...@confluent.io>
Sent: Friday, September 14, 2018 3:06 PM
To: users@kafka.apache.org
Subject: Re: Need info
As a side note from your question, I'd recommend looking into Kafka Connect. It is another API with Apache Kafka, and it simplifies the building of pipelines with Kafka such as the one you are describing There are pre-built connectors, including for S3 ( https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.confluent.io%2Fconnector%2Fkafka-connect-s3%2F&data=01%7C01%7Cchanchal.chatterji%40infosys.com%7C7382ed64a0214c778c8c08d61a25b0df%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C1&sdata=yc3Mo3h5JFD%2BpoGBDthgAgT7GxW52tLS4mZJxO4HKXs%3D&reserved=0). There are also options for pulling from the mainframe, depending on specific platform and system.
--
Robin Moffatt | Developer Advocate | robin@confluent.io | @rmoff
On Wed, 12 Sep 2018 at 08:32, Chanchal Chatterji < chanchal.chatterji@infosys.com> wrote:
> Hi,
>
> In the process of mainframe modernization, we are attempting to stream
> Mainframe data to AWS Cloud , using Kafka. We are planning to use
> Kafka 'Producer API' at mainframe side and 'Connector API' on the cloud side.
> Since our data is processed by a module called 'Central dispatch'
> located in Mainframe and is sent to Kafka. We want to know what is
> rate of volume Kafka can handle. The other end of Kafka is connected
> to an AWS S3 Bucket As sink. Please help us to provide this info or
> else please help to connect with relevant person who can help us to understand this.
>
> Thanks and Regards
>
> Chanchal Chatterji
> Principal Consultant,
> Infosys Ltd.
> Electronic city Phase-1,
> Bangalore-560100
> Contacts : 9731141606/ 8105120766
>
>
Re: Need info
Posted by Robin Moffatt <ro...@confluent.io>.
As a side note from your question, I'd recommend looking into Kafka
Connect. It is another API with Apache Kafka, and it simplifies the
building of pipelines with Kafka such as the one you are describing
There are pre-built connectors, including for S3 (
https://www.confluent.io/connector/kafka-connect-s3/). There are also
options for pulling from the mainframe, depending on specific platform and
system.
--
Robin Moffatt | Developer Advocate | robin@confluent.io | @rmoff
On Wed, 12 Sep 2018 at 08:32, Chanchal Chatterji <
chanchal.chatterji@infosys.com> wrote:
> Hi,
>
> In the process of mainframe modernization, we are attempting to stream
> Mainframe data to AWS Cloud , using Kafka. We are planning to use Kafka
> 'Producer API' at mainframe side and 'Connector API' on the cloud side.
> Since our data is processed by a module called 'Central dispatch' located
> in Mainframe and is sent to Kafka. We want to know what is rate of volume
> Kafka can handle. The other end of Kafka is connected to an AWS S3 Bucket
> As sink. Please help us to provide this info or else please help to
> connect with relevant person who can help us to understand this.
>
> Thanks and Regards
>
> Chanchal Chatterji
> Principal Consultant,
> Infosys Ltd.
> Electronic city Phase-1,
> Bangalore-560100
> Contacts : 9731141606/ 8105120766
>
>
Re: Need info
Posted by James Kwan <jw...@gmail.com>.
There are different customers getting data from zOS like DB2 or IMS to Kafka. They have written their own consumers to load the data into data warehouse like Teradata or other data warehouse platform. I am not sure if there is a particular customer using MongoDB, but there are some using MySQL and PostgrepSQL.
> On Sep 12, 2018, at 12:34 PM, Mich Talebzadeh <mi...@gmail.com> wrote:
>
> Hi James,
>
> As a matter of interest this streaming data is fed into some Operational
> Data Store ODS) like MongoDB?
>
> In general using this method will create a near real time snapshot for
> business users and customers.
>
> HTH
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Wed, 12 Sep 2018 at 17:58, James Kwan <jw...@gmail.com> wrote:
>
>> We have banking customers sending data from DB2 z to Kafka linux (not
>> cloud) with transaction rate 30K per seconds. Kafka can handle more than
>> this rate.
>>
>>> On Sep 12, 2018, at 2:31 AM, Chanchal Chatterji <
>> chanchal.chatterji@infosys.com> wrote:
>>>
>>> Hi,
>>>
>>> In the process of mainframe modernization, we are attempting to stream
>> Mainframe data to AWS Cloud , using Kafka. We are planning to use Kafka
>> 'Producer API' at mainframe side and 'Connector API' on the cloud side.
>>> Since our data is processed by a module called 'Central dispatch'
>> located in Mainframe and is sent to Kafka. We want to know what is rate of
>> volume Kafka can handle. The other end of Kafka is connected to an AWS S3
>> Bucket
>>> As sink. Please help us to provide this info or else please help to
>> connect with relevant person who can help us to understand this.
>>>
>>> Thanks and Regards
>>>
>>> Chanchal Chatterji
>>> Principal Consultant,
>>> Infosys Ltd.
>>> Electronic city Phase-1,
>>> Bangalore-560100
>>> Contacts : 9731141606/ 8105120766
>>>
>>
>>
Re: Need info
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi James,
As a matter of interest this streaming data is fed into some Operational
Data Store ODS) like MongoDB?
In general using this method will create a near real time snapshot for
business users and customers.
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Wed, 12 Sep 2018 at 17:58, James Kwan <jw...@gmail.com> wrote:
> We have banking customers sending data from DB2 z to Kafka linux (not
> cloud) with transaction rate 30K per seconds. Kafka can handle more than
> this rate.
>
> > On Sep 12, 2018, at 2:31 AM, Chanchal Chatterji <
> chanchal.chatterji@infosys.com> wrote:
> >
> > Hi,
> >
> > In the process of mainframe modernization, we are attempting to stream
> Mainframe data to AWS Cloud , using Kafka. We are planning to use Kafka
> 'Producer API' at mainframe side and 'Connector API' on the cloud side.
> > Since our data is processed by a module called 'Central dispatch'
> located in Mainframe and is sent to Kafka. We want to know what is rate of
> volume Kafka can handle. The other end of Kafka is connected to an AWS S3
> Bucket
> > As sink. Please help us to provide this info or else please help to
> connect with relevant person who can help us to understand this.
> >
> > Thanks and Regards
> >
> > Chanchal Chatterji
> > Principal Consultant,
> > Infosys Ltd.
> > Electronic city Phase-1,
> > Bangalore-560100
> > Contacts : 9731141606/ 8105120766
> >
>
>
Re: Need info
Posted by James Kwan <jw...@gmail.com>.
We have banking customers sending data from DB2 z to Kafka linux (not cloud) with transaction rate 30K per seconds. Kafka can handle more than this rate.
> On Sep 12, 2018, at 2:31 AM, Chanchal Chatterji <ch...@infosys.com> wrote:
>
> Hi,
>
> In the process of mainframe modernization, we are attempting to stream Mainframe data to AWS Cloud , using Kafka. We are planning to use Kafka 'Producer API' at mainframe side and 'Connector API' on the cloud side.
> Since our data is processed by a module called 'Central dispatch' located in Mainframe and is sent to Kafka. We want to know what is rate of volume Kafka can handle. The other end of Kafka is connected to an AWS S3 Bucket
> As sink. Please help us to provide this info or else please help to connect with relevant person who can help us to understand this.
>
> Thanks and Regards
>
> Chanchal Chatterji
> Principal Consultant,
> Infosys Ltd.
> Electronic city Phase-1,
> Bangalore-560100
> Contacts : 9731141606/ 8105120766
>