You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kailash Kota <Ka...@jda.com> on 2019/06/04 10:35:20 UTC
Live data streaming from Oracle to oracle using Kafka
Hi Users,
Here is our requirement :
We want to live stream data from Oracle DB and to Oracle DB as a target using Kafka and we do not want to use Oracle Golden Gate because of the extra license cost.
Is there a way we can read the redo logs and achieve this ?
If not, can you help us on the other ways that we can stream data. Please provide any documentation and procedure is available.
Thanks & Regards,
Kailash Kota
Product Development | JDA Software Pvt Ltd.
Ph: +91 80 6101 8649
Think BIG > Start small > Run fast! Download JDA's AI/ML buyers guide<http://bit.ly/2W68ef5> and get started on your autonomous SCM journey!
To the extent permitted by law, we may monitor electronic communications for the purposes of ensuring compliance with our legal and regulatory obligations and internal policies. We may also collect email traffic headers for analyzing patterns of network traffic and managing client relationships. For additional information see https://jda.com/privacy-policy.
RE: Live data streaming from Oracle to oracle using Kafka
Posted by Kailash Kota <Ka...@jda.com>.
Thanks for the info Joe ! Appreciate it
Thanks & Regards,
Kailash Kota
Product Development | JDA Software Pvt Ltd.
Ph: +91 80 6101 8649
-----Original Message-----
From: Joe Ammann <jo...@pyx.ch>
Sent: 04 June 2019 18:23
To: users@kafka.apache.org
Subject: Re: Live data streaming from Oracle to oracle using Kafka
CAUTION: External email. Please validate the sender before using the content. Do not select "reply all" unless everyone on the list is a valid recipient.
________________________________
Hi Kailash
On 6/4/19 12:35 PM, Kailash Kota wrote:
> We want to live stream data from Oracle DB and to Oracle DB as a target using Kafka and we do not want to use Oracle Golden Gate because of the extra license cost.
> Is there a way we can read the redo logs and achieve this ?
Debezium https://urldefense.proofpoint.com/v2/url?u=https-3A__debezium.io_&d=DwID-g&c=ToVbMNJY2W2sI6cHZmZYL1a1DSDW03K-K6TT0TaILp0&r=aGAW1WZThhVvUQAp92-lwMJA5-ISno7QC7uoFNhFRNo&m=WwpykHPxz1oLXCWQAc8N6OsoURKODnCwOynfOp9A0hY&s=xopkn1vSbRxPAV31yERJM9_G2U0yQ6ZVOXsSbhC64dM&e= could be an option, but the current Oracle implementation is based on XStream, which also requires a GoldenGate license. There is work underway to use Oracle LogMiner directly https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_debezium_debezium-2Dincubator_pull_87&d=DwID-g&c=ToVbMNJY2W2sI6cHZmZYL1a1DSDW03K-K6TT0TaILp0&r=aGAW1WZThhVvUQAp92-lwMJA5-ISno7QC7uoFNhFRNo&m=WwpykHPxz1oLXCWQAc8N6OsoURKODnCwOynfOp9A0hY&s=50E0z2X4n9mlkGTKwwOjHjEUSHuJWaogDJblSwZgjJU&e= , but not yet ready
> If not, can you help us on the other ways that we can stream data. Please provide any documentation and procedure is available.
Other options involve Kafka Connect with the JDBC connector, or DB based approaches like triggers or materialized views. Robin Moffatt has an execellent overview here https://urldefense.proofpoint.com/v2/url?u=https-3A__rmoff.net_2018_12_12_streaming-2Ddata-2Dfrom-2Doracle-2Dinto-2Dkafka-2Ddecember-2D2018_&d=DwID-g&c=ToVbMNJY2W2sI6cHZmZYL1a1DSDW03K-K6TT0TaILp0&r=aGAW1WZThhVvUQAp92-lwMJA5-ISno7QC7uoFNhFRNo&m=WwpykHPxz1oLXCWQAc8N6OsoURKODnCwOynfOp9A0hY&s=aSAKJPmsAdTsFEI4Jq1APMAqRQXtKGgltIO-l3YM-nk&e=
--
CU, Joe
To the extent permitted by law, we may monitor electronic communications for the purposes of ensuring compliance with our legal and regulatory obligations and internal policies. We may also collect email traffic headers for analyzing patterns of network traffic and managing client relationships. For additional information see https://jda.com/privacy-policy.
Re: Live data streaming from Oracle to oracle using Kafka
Posted by Joe Ammann <jo...@pyx.ch>.
Hi Kailash
On 6/11/19 9:24 AM, Kailash Kota wrote:
> I understand Oracle Golden Gate is a data replication tool which uses log-based technology to stream all changes to a database from source, to target. Can you please help me in understand what is the role of Kafka after the data is provided to it by OGG ?
The tool in question here is not "standard Oracle GoldenGate", but rather "Oracle GoldenGate for BigData" https://docs.oracle.com/goldengate/bd123010/gg-bd/. OGG-BD captures change records from the Oracle log (or other database) and publishes those changes to Kafka (or other typical "BigData technologies" like HDFS, HBase, Flume).
The alternatives that I'm aware of that do similar things are Attunity https://www.attunity.com/content/change-data-capture-cdc-oracle/ (commercial), StreamSets https://streamsets.com/blog/change-data-capture-from-oracle-with-streamsets-data-collector/ (partially OSS), Debezium https://debezium.io/ and Kafka Connect Oracle https://github.com/erdemcer/kafka-connect-oracle
There may be more.
> Also if we just need to data replicate our oracle DB, is there any other ways we can do it without Oracle Golden Gate ? Is Apache Storm/Flink a prospect for us to look into (though they are streaming tools).
As Robin Moffatt's article notes, besides log-based CDC there is also query based CDC. The tool choce there is much wider, and you could certainly use Storm of Flink to implement a solution.
Since most people I have seen doing this want a persistent store/buffer for the change records, instead of processing them directly without any buffering, most solutions I have seen push the records into Kafka, by using some Kafka Connect setup.
If your requirements are for direct processing, I would look into Flink for this.
--
CU, Joe
RE: Live data streaming from Oracle to oracle using Kafka
Posted by Kailash Kota <Ka...@jda.com>.
Hi Joe,
I understand Oracle Golden Gate is a data replication tool which uses log-based technology to stream all changes to a database from source, to target. Can you please help me in understand what is the role of Kafka after the data is provided to it by OGG ?
Also if we just need to data replicate our oracle DB, is there any other ways we can do it without Oracle Golden Gate ? Is Apache Storm/Flink a prospect for us to look into (though they are streaming tools).
Thanks & Regards,
Kailash Kota
Product Development | JDA Software Pvt Ltd.
Ph: +91 80 6101 8649
-----Original Message-----
From: Joe Ammann <jo...@pyx.ch>
Sent: 04 June 2019 18:23
To: users@kafka.apache.org
Subject: Re: Live data streaming from Oracle to oracle using Kafka
CAUTION: External email. Please validate the sender before using the content. Do not select "reply all" unless everyone on the list is a valid recipient.
________________________________
Hi Kailash
On 6/4/19 12:35 PM, Kailash Kota wrote:
> We want to live stream data from Oracle DB and to Oracle DB as a target using Kafka and we do not want to use Oracle Golden Gate because of the extra license cost.
> Is there a way we can read the redo logs and achieve this ?
Debezium https://urldefense.proofpoint.com/v2/url?u=https-3A__debezium.io_&d=DwID-g&c=ToVbMNJY2W2sI6cHZmZYL1a1DSDW03K-K6TT0TaILp0&r=aGAW1WZThhVvUQAp92-lwMJA5-ISno7QC7uoFNhFRNo&m=WwpykHPxz1oLXCWQAc8N6OsoURKODnCwOynfOp9A0hY&s=xopkn1vSbRxPAV31yERJM9_G2U0yQ6ZVOXsSbhC64dM&e= could be an option, but the current Oracle implementation is based on XStream, which also requires a GoldenGate license. There is work underway to use Oracle LogMiner directly https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_debezium_debezium-2Dincubator_pull_87&d=DwID-g&c=ToVbMNJY2W2sI6cHZmZYL1a1DSDW03K-K6TT0TaILp0&r=aGAW1WZThhVvUQAp92-lwMJA5-ISno7QC7uoFNhFRNo&m=WwpykHPxz1oLXCWQAc8N6OsoURKODnCwOynfOp9A0hY&s=50E0z2X4n9mlkGTKwwOjHjEUSHuJWaogDJblSwZgjJU&e= , but not yet ready
> If not, can you help us on the other ways that we can stream data. Please provide any documentation and procedure is available.
Other options involve Kafka Connect with the JDBC connector, or DB based approaches like triggers or materialized views. Robin Moffatt has an execellent overview here https://urldefense.proofpoint.com/v2/url?u=https-3A__rmoff.net_2018_12_12_streaming-2Ddata-2Dfrom-2Doracle-2Dinto-2Dkafka-2Ddecember-2D2018_&d=DwID-g&c=ToVbMNJY2W2sI6cHZmZYL1a1DSDW03K-K6TT0TaILp0&r=aGAW1WZThhVvUQAp92-lwMJA5-ISno7QC7uoFNhFRNo&m=WwpykHPxz1oLXCWQAc8N6OsoURKODnCwOynfOp9A0hY&s=aSAKJPmsAdTsFEI4Jq1APMAqRQXtKGgltIO-l3YM-nk&e=
--
CU, Joe
To the extent permitted by law, we may monitor electronic communications for the purposes of ensuring compliance with our legal and regulatory obligations and internal policies. We may also collect email traffic headers for analyzing patterns of network traffic and managing client relationships. For additional information see https://jda.com/privacy-policy.
Re: Live data streaming from Oracle to oracle using Kafka
Posted by Joe Ammann <jo...@pyx.ch>.
Hi Kailash
On 6/4/19 12:35 PM, Kailash Kota wrote:
> We want to live stream data from Oracle DB and to Oracle DB as a target using Kafka and we do not want to use Oracle Golden Gate because of the extra license cost.
> Is there a way we can read the redo logs and achieve this ?
Debezium https://debezium.io/ could be an option, but the current Oracle implementation is based on XStream, which also requires a GoldenGate license. There is work underway to use Oracle LogMiner directly https://github.com/debezium/debezium-incubator/pull/87, but not yet ready
> If not, can you help us on the other ways that we can stream data. Please provide any documentation and procedure is available.
Other options involve Kafka Connect with the JDBC connector, or DB based approaches like triggers or materialized views. Robin Moffatt has an execellent overview here https://rmoff.net/2018/12/12/streaming-data-from-oracle-into-kafka-december-2018/
--
CU, Joe