How to configure Ignite sink connector to use key-value?

[ignitedFox <bl...@gmail.com>]

Hi,
I have 2 connector and 2 worker configuration. The purpose of worker 1 and
connector 1 is to fetch data from oracle table and insert it into a Kafka
topic in JSON format. The purpose of worker 2 and connector 2 is to fetch
data from Kafka topic and insert it into Ignite cache. Below you can see the
configurations.









worker 1 and connector 1 works fine, and it push data to Kafka. In Kafka,
data resides like;



the schema of my table is like below;



I believe that the second worker and connector also works fine, coz the
number of ignite servers changes when its triggered. What I wanted is, I
want to insert this data in Kafa, to ignite, as key-value pair. My connector
2 is already inserting it to Ignite cache, but it is not as a key-value pair
format. I would like to have the **BUS_ID** as the key, and the rest as the
value, so that it will be searchable, and prevent duplication.

It will be really helpful if somebody kindly help me to achieve this using
Kafka connect. Thanks in advance.




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-to-configure-Ignite-sink-connector-to-use-key-value-tp13162.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: How to configure Ignite sink connector to use key-value?

[Roman Shtykh <rs...@yahoo.com>]

Hi,
Please see https://apacheignite-mix.readme.io/v2.0/docs/kafka-streamer

1. You can find the jars in the binary release of Ignite in lib/, for instance.2. To parse your json string and extract key and value, I recommend using Ignite 1.8 and up. With it, you can parse the incoming data with your extractor and decide what has to be your key. See this link for a simple extractor.https://github.com/apache/ignite/blob/master/modules/kafka/src/test/java/org/apache/ignite/stream/kafka/connect/IgniteSinkConnectorTest.java
-- Roman


    On Monday, May 29, 2017 10:29 PM, Humphrey <hm...@gmail.com> wrote:
 

 See answers in the following post.

http://apache-ignite-users.70518.x6.nabble.com/Kindly-tell-me-where-to-find-these-jar-files-td12649.html

I think in the last step you need to convert the data you getting to an
cache object, for example Annotation Based Configuration [1] so you can use
it to query your cache. Implementing the StreamSingleTupleExtractor will do
the conversion for you to the correct format.

[1]
https://apacheignite.readme.io/docs/indexes#annotation-based-configuration



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-to-configure-Ignite-sink-connector-to-use-key-value-tp13162p13198.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.


   

Re: How to configure Ignite sink connector to use key-value?

[Humphrey <hm...@gmail.com>]

See answers in the following post.

http://apache-ignite-users.70518.x6.nabble.com/Kindly-tell-me-where-to-find-these-jar-files-td12649.html

I think in the last step you need to convert the data you getting to an
cache object, for example Annotation Based Configuration [1] so you can use
it to query your cache. Implementing the StreamSingleTupleExtractor will do
the conversion for you to the correct format.

[1]
https://apacheignite.readme.io/docs/indexes#annotation-based-configuration



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-to-configure-Ignite-sink-connector-to-use-key-value-tp13162p13198.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.