Student Blog about Apache Ignite & Questions how to efficiently handle data

[svonn <sv...@posteo.de>]

Hello!

I noticed that this community is pretty active, so there might be some
people that are interested in this:

For a university project, we're trying to compare different stream
processing engines. I decided to use Apache Ignite.
Since our professorship hasn't really worked with most of those engines,
we're supposed to write a Blog about our progressions - mostly focused on
the stuff that doesn't work. 
So if you're interested in following a newbie struggling with Apache Ignite,
you might like this:

https://wordpress.com/post/streambench.wordpress.com/1310

I am always super happy about recommendations, tips, and experience I can
get from others, so don't hesitate with feedback!

__________________________


Now, to the question part:

The following data is to be streamed to Ignite:

Measurement: A measurement is basically the biggest entity, containing both
AccelerationPoints as well as GpsPoints. 
The key sent by our produced consists of the deviceId concatinated with a
measurementId. This key is unique as works as primary key.
GpsPoint: A GpsPoint belongs to a Measurement and has the very same key,
deviceId:MeasurementId. Its value (serialized as byte[] in Kafka, I convert
it to a binary object for Ignite currently) contains a timestamp, which
could probably be enough in combination with the key provided by Kafka for a
primary key.
AccelerationPoint: Similar to a GpsPoint, only that we get about 200 of
those per GpsPoint - they need to be interpolated later.

The issue now is that Kafka Connect provides the deviceId:MeasurementId key
schema for all three entities - therefore I'm either running in key
conflicts or I'm forced to overwrite the data already in the cache.
How do I deal with those key issues? 
I was thinking about either adding the timestamp from the value to the key
or just add any new entity to a list of entities with the same key. For both
those approaches, I don't really know how to do it properly since it has to
happen somewhere between consuming the data from Kafka and writing it to
Ignite.

Any help would be appreciated!

Best regards,
Sven 




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Student Blog about Apache Ignite & Questions how to efficiently handle data

[svonn <sv...@posteo.de>]

Apparently I messed up the Blog link, here's the proper one:

https://streambench.wordpress.com/category/apache-ignite/

Best Regards 
Sven



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Student Blog about Apache Ignite & Questions how to efficiently handle data

[Denis Magda <dm...@apache.org>]

Hi Sven,

Feel free to send a link to the blog directly to my email. I’ll add it to Ignite’s blog feed:
https://ignite.apache.org/blogs.html <https://ignite.apache.org/blogs.html>

—
Denis

> On Dec 5, 2017, at 8:10 AM, svonn <sv...@posteo.de> wrote:
> 
> Hello Nikolai,
> 
> Since I'm already pretty lost in the deeps of Ignite I will start another
> topic with more information, code and questions soon - in this case, here's
> one fo the extractors I'm currently using:
> 
> public class AccelerationPointExtractor implements
> StreamSingleTupleExtractor<SinkRecord, IgniteKey, AccelerationPoint> {
>    @Override public Map.Entry<IgniteKey,
> AccelerationPoint>extract(SinkRecord msg) {
>        String key = (String) msg.key();
>        AccelerationPoint accelerationPoint = (AccelerationPoint)
> msg.value();
>        String deviceId = ((String) msg.key()).split(":")[0];
>        long measurementId = Long.parseLong(((String)
> msg.key()).split(":")[1]);
>        accelerationPoint.setDeviceId(deviceId);
>        accelerationPoint.setMeasurementId(measurementId);
>        return new AbstractMap.SimpleEntry<>(new IgniteKey(deviceId,
> measurementId, accelerationPoint.getTimestamp()), accelerationPoint);
>    }
> 
> }
> 
> Best regards,
> Sven
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Student Blog about Apache Ignite & Questions how to efficiently handle data

[svonn <sv...@posteo.de>]

Hello Nikolai,

Since I'm already pretty lost in the deeps of Ignite I will start another
topic with more information, code and questions soon - in this case, here's
one fo the extractors I'm currently using:

public class AccelerationPointExtractor implements
StreamSingleTupleExtractor<SinkRecord, IgniteKey, AccelerationPoint> {
    @Override public Map.Entry<IgniteKey,
AccelerationPoint>extract(SinkRecord msg) {
        String key = (String) msg.key();
        AccelerationPoint accelerationPoint = (AccelerationPoint)
msg.value();
        String deviceId = ((String) msg.key()).split(":")[0];
        long measurementId = Long.parseLong(((String)
msg.key()).split(":")[1]);
        accelerationPoint.setDeviceId(deviceId);
        accelerationPoint.setMeasurementId(measurementId);
        return new AbstractMap.SimpleEntry<>(new IgniteKey(deviceId,
measurementId, accelerationPoint.getTimestamp()), accelerationPoint);
    }

}

Best regards,
Sven



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Student Blog about Apache Ignite & Questions how to efficiently handle data

[Nikolai Tikhonov <nt...@apache.org>]

Hello Sven!

Glad hear that you solved the problem by yourself!
Any way, if you are able to share code snippet that shows how you use kafka
streamer then community can provide some additional suggest.

On Tue, Dec 5, 2017 at 2:45 PM, svonn <sv...@posteo.de> wrote:

> I solved the key issue with a singleTupleExtractor - For both GpsPoints and
> AccelerationPoints I'm simply adding a hashvalue over the Object.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Student Blog about Apache Ignite & Questions how to efficiently handle data

[svonn <sv...@posteo.de>]

I solved the key issue with a singleTupleExtractor - For both GpsPoints and
AccelerationPoints I'm simply adding a hashvalue over the Object. 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/