You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by blasteralfred <bl...@gmail.com> on 2017/05/12 06:06:27 UTC
Kindly tell me where to find these jar files.
Hi all,
I am using *kafka - 0.10.2.0*, *ignite - 1.6.0*. Kafka is installed in my
linux box (confluent) and ignite in my windows, both are connected and up. I
would like to insert to ignite cache from Kafka topic, so that I have
followed `## Streaming Data to Ignite via Kafka Connect` described here
https://github.com/apache/ignite/tree/master/modules/kafka
I need following jars for accomplishing this;
- ignite-kafka-connect-x.x.x-SNAPSHOT.jar
- ignite-core-x.x.x-SNAPSHOT.jar
- ignite-spring-x.x.x-SNAPSHOT.jar
- cache-api-1.0.0.jar
- spring-aop-4.1.0.RELEASE.jar
- spring-beans-4.1.0.RELEASE.jar
- spring-context-4.1.0.RELEASE.jar
- spring-core-4.1.0.RELEASE.jar
- spring-expression-4.1.0.RELEASE.jar
- commons-logging-1.1.1.jar
Somebody kindly tell me where I can find these jar files (correct versions)
and where to place them?
Thanks you.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Kindly-tell-me-where-to-find-these-jar-files-tp12649.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Kindly tell me where to find these jar files.
Posted by Humphrey <hm...@gmail.com>.
I replied in the new thread you made above.
Look in Kafka how to extract the values you want from the SinkRecord you are
getting and map those to your Object you want to store in Ignite.
So for example you extract the BUS_ID from the SinkRecord (look in the kafka
api how to do that) and you map that to they key. Same you do for the other
fields and map them to the object you want to store in Ignite.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Kindly-tell-me-where-to-find-these-jar-files-tp12649p13228.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Kindly tell me where to find these jar files.
Posted by ignitedFox <bl...@gmail.com>.
Hi Humphrey,
I am getting when I try this solution. I have opened a thread here
http://apache-ignite-users.70518.x6.nabble.com/How-to-solve-SinkRecord-cannot-be-cast-exception-td13221.html
<http://apache-ignite-users.70518.x6.nabble.com/How-to-solve-SinkRecord-cannot-be-cast-exception-td13221.html>
Could you kindly tell me why this is happening? Is there any problem with my
JSON record? How can I fix this?
Thanks in advance.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Kindly-tell-me-where-to-find-these-jar-files-tp12649p13222.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Kindly tell me where to find these jar files.
Posted by ignitedFox <bl...@gmail.com>.
Hi Humphrey,
I am using Kafka Connect JDBC to fetch data from oracle table. You can find
the details in my post here
http://apache-ignite-users.70518.x6.nabble.com/How-to-configure-Ignite-sink-connector-to-use-key-value-td13162.html
. I want to use Kafka connect to fetch rows from oracle table to Kafka
topic, and then insert it into Ignite cache. I hope your solution will help
me on this.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Kindly-tell-me-where-to-find-these-jar-files-tp12649p13211.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Kindly tell me where to find these jar files.
Posted by Humphrey <hm...@gmail.com>.
I think it's not hard to do and once you start implement the interface you
will figure out by yourself.
- May I ask *what tool do you use* to parse the Schema that converts the
object to this json?
Here is what I did:
I created an object, called TulpeBus which would be the kind of messages
(objects) i'll be receiving from Kafka. When I receive this object from
kafka I'll extract the BusId and use that key for ignite and the value is
the rest of the content as one string, by calling my *specialized*
toString() method on the object.
In my example I'm passing an object. You probably will be getting an byte[]
from kafka which will need to be converted back to an object by the tool you
used to convert it in first place to byte[] before sending it to kafka.
*Here is myExtractor implementing the interface:*
public class MyExtractor implements StreamSingleTupleExtractor<TupleBUS,
Integer, String> {
@Override
public Entry<Integer, String> extract(TupleBUS msg) {
return new AbstractMap.SimpleEntry<Integer, String>(msg.getBusId(),
msg.toString());
}
}
*Here is my TupleBus with specialized toString() method:*
public class TupleBUS {
int busId;
int busNumber;
long busDate;
String route;
long lastUpdated;
public TupleBUS(int busId, int busNumber, long busDate, String route, long
lastUpdated) {
super();
this.busId = busId;
this.busNumber = busNumber;
this.busDate = busDate;
this.route = route;
this.lastUpdated = lastUpdated;
}
public int getBusId() {
return busId;
}
public int getBusNumber() {
return busNumber;
}
public long getBusDate() {
return busDate;
}
public String getRoute() {
return route;
}
public long getLastUpdated() {
return lastUpdated;
}
@Override
public String toString() {
return " [busNumber=" + busNumber + ", busDate=" + busDate + ", route=" +
route + ", lastUpdated=" + lastUpdated + "]";
}
}
*Here is a main method to testIt:*
public class TestMyExtractor {
public static void main(String[] args) {
TupleBUS tulpe = new TupleBUS(3, 100, 1484092800000L, "A-B",
1495650596335L);
MyExtractor extractor = new MyExtractor();
System.out.println(extractor.extract(tulpe));
}
}
*Output:*
3= [busNumber=100, busDate=1484092800000, route=A-B,
lastUpdated=1495650596335]
I hope this example helps.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Kindly-tell-me-where-to-find-these-jar-files-tp12649p13192.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Kindly tell me where to find these jar files.
Posted by ignitedFox <bl...@gmail.com>.
Hi Humphrey,
Need a small help. Could you kindly tell me how to implement custom
key-value pair in ignite sink connector? from the documentation here
<https://github.com/apache/ignite/tree/master/modules/kafka> , i got to
know that there need to implement "StreamSingleTupleExtractor and specify it
as 'singleTupleExtractorCls'". Could you kindly tell me how to do this?
Thank you.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Kindly-tell-me-where-to-find-these-jar-files-tp12649p13191.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Kindly tell me where to find these jar files.
Posted by Humphrey <hm...@gmail.com>.
Seems like you need to start first with the basics. This is not the correct
place to learn java.
Google online and do more research.
Here are some links explaining maven.
https://youtu.be/JK9oZfScQgg
https://youtu.be/uEYjXpMDJiU
There are more out there. Play a little around with maven and java first.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Kindly-tell-me-where-to-find-these-jar-files-tp12649p12884.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Kindly tell me where to find these jar files.
Posted by blasteralfred <bl...@gmail.com>.
Hi Humphrey, thank you so much for your kind help. As I am a beginner, and
using eclipse, till now, I have lots of errors in my project like `create
class`. Could you kindly tell me the imports that I have to do, in these
classes?
Thank you.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Kindly-tell-me-where-to-find-these-jar-files-tp12649p12871.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Kindly tell me where to find these jar files.
Posted by Humphrey <hm...@gmail.com>.
Here is some code that I used. I made the Kafka Ignite Streamer as a service
in Ignite and deploy it as a cluster singleton.
public class IgniteKafkaStreamerService implements Service {
private static final long serialVersionUID = 1L;
@IgniteInstanceResource
private Ignite ignite;
private KafkaStreamer<String, String> kafkaStreamer = new
KafkaStreamer<>();
private IgniteLogger logger;
@Override
public void init(ServiceContext ctx) throws Exception {
logger = ignite.log();
IgniteDataStreamer<String, String> stmr = ignite.dataStreamer(CACHE_NAME);
stmr.allowOverwrite(true);
stmr.autoFlushFrequency(1000);
kafkaStreamer.setIgnite(ignite);
kafkaStreamer.setStreamer(stmr);
kafkaStreamer.setThreads(4);
kafkaStreamer.setTopic(KAFKA_TOPIC);
Properties kafkaProps = Util.loadProperties("config/kafka.properties"); //
Some code to read in the kafka properties
kafkaStreamer.setConsumerConfig(new ConsumerConfig(kafkaProps));
kafkaStreamer.setSingleTupleExtractor(msg -> new
AbstractMap.SimpleEntry<String, String>(new String(msg.key()), new
String(msg.message())));
}
@Override
public void execute(ServiceContext ctx) throws Exception {
kafkaStreamer.start();
logger.info("KafkaStreamer started.");
}
@Override
public void cancel(ServiceContext ctx) {
kafkaStreamer.stop();
logger.info("KafkaStreamer stopped.");
}
}
Below the code to startup:
public class IgniteNodeStartup {
public static void main(String[] args) {
// Use to start up an Ignite server with default configuration
Ignite ignite = Ignition.start();
ignite.getOrCreateCache(getCacheConfiguration());
// Deploy data streamer service on the server nodes.
ClusterGroup forServers = ignite.cluster().forServers();
ignite.services(forServers).deployClusterSingleton("KafkaService", new
IgniteKafkaStreamerService());
}
private static CacheConfiguration<String, String> getCacheConfiguration() {
CacheConfiguration<String, String> cfg = new CacheConfiguration<String,
String>();
cfg.setName(CACHE_NAME);
cfg.setBackups(1);
return cfg;
}
}
I hope this helps.
Humphrey
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Kindly-tell-me-where-to-find-these-jar-files-tp12649p12836.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Kindly tell me where to find these jar files.
Posted by Humphrey Lopez <hm...@gmail.com>.
I'm not able to help you out today but Monday I can send you some code. I used the ignite Kafka streamer to get the messages from Kafka. So ignite is pulling from Kafka rather than Kafka pushing to ignite.
Humphrey
> On 12 May 2017, at 10:15, blasteralfred <bl...@gmail.com> wrote:
>
> Hi Humphrey, Thanks for your reply. I am trying to use Kafka Connect for pushing to Kafka, and from there, to Ignite, and that's why I have asked for jar files. Its okay, that I can try the Stream also. I have no idea on how to use streams, and the examples I have found were, not in a beginner level (for me to understand). Could you be kind enough to show me a sample code to push data to ignite, from kafka? I have pushed data to a kafka topic using Kafka connect's file stream source, and in Kafka, the data reside like this; {"schema":{"type":"string","optional":false},"payload":"hello world 6"} {"schema":{"type":"string","optional":false},"payload":"hello world 7"} {"schema":{"type":"string","optional":false},"payload":"hello world 8"} {"schema":{"type":"string","optional":false},"payload":"hello world 9"} {"schema":{"type":"string","optional":false},"payload":"hello world 10"} Can you show me a code to push this data to Ignite? Thanks in advance..
> View this message in context: Re: Kindly tell me where to find these jar files.
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Kindly tell me where to find these jar files.
Posted by blasteralfred <bl...@gmail.com>.
Hi Humphrey, Thanks for your reply.I am trying to use *Kafka Connect* for
pushing to Kafka, and from there, to Ignite, and that's why I have asked for
jar files. Its okay, that I can try the Stream also. I have no idea on how
to use streams, and the examples I have found were, not in a beginner level
(for me to understand). Could you be kind enough to show me a sample code to
push data to ignite, from kafka? I have pushed data to a kafka topic using
Kafka connect's file stream source, and in Kafka, the data reside like
this;{"schema":{"type":"string","optional":false},"payload":"hello world
6"}{"schema":{"type":"string","optional":false},"payload":"hello world
7"}{"schema":{"type":"string","optional":false},"payload":"hello world
8"}{"schema":{"type":"string","optional":false},"payload":"hello world
9"}{"schema":{"type":"string","optional":false},"payload":"hello world
10"}Can you show me a code to push this data to Ignite? Thanks in advance..
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Kindly-tell-me-where-to-find-these-jar-files-tp12649p12653.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Kindly tell me where to find these jar files.
Posted by Humphrey <hm...@gmail.com>.
I've done the same thing, but just with Ignite 1.9.1 and all on one Linux
machine, but that wouldn't matter. I used maven repository to get the jar
files for me. I can share my dependencies of pom.xml with you.
The jars are only needed on the machine where you start Ignite. Change the
ignite.version to 1.6.0 so it will download the correct version of ignite
for you.
<properties>
<ignite.version>1.9.1</ignite.version>
<kafka.version>0.10.2.0</kafka.version>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-kafka</artifactId>
<version>${ignite.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.kafka</groupId>
<artifactId>connect-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-spring</artifactId>
<version>${ignite.version}</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.12</artifactId>
<version>${kafka.version}</version>
</dependency>
<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-log4j</artifactId>
<version>${ignite.version}</version>
</dependency>
</dependencies>
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Kindly-tell-me-where-to-find-these-jar-files-tp12649p12650.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.