You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samoa.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/05/16 13:12:04 UTC
[jira] [Commented] (SAMOA-65) Apache Kafka integration components
for SAMOA
[ https://issues.apache.org/jira/browse/SAMOA-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16012339#comment-16012339 ]
ASF GitHub Bot commented on SAMOA-65:
-------------------------------------
GitHub user pwawrzyniak opened a pull request:
https://github.com/apache/incubator-samoa/pull/59
SAMOA-65
Initial version of Kafka components for SAMOA. Works with Apache Kafka 0.10, includes JSON and AVRO mappers to read/wtite instances to Kafka.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pwawrzyniak/incubator-samoa SAMOA-65
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-samoa/pull/59.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #59
----
commit 0e90fe8c107cd90a7baf16b5d4f334d1650a0afb
Author: pwawrzyniak <pi...@wawrzyniak.org.pl>
Date: 2017-03-14T16:43:25Z
Initial structure of Kafka components.
Initital code for Kafka Consumer
commit 2168400f86189605b06fb531511235733bf3b6ca
Author: pwawrzyniak <pi...@wawrzyniak.org.pl>
Date: 2017-03-17T10:05:14Z
Code for KafkaEntranceProcessor (consuming messages from Kafka)
commit cf2ff369989fa3560f1570dfaf32bcb746d51c25
Author: pwawrzyniak <pi...@wawrzyniak.org.pl>
Date: 2017-03-17T10:54:47Z
Updated comments
commit 26d86da6ebdd53e440e8b1e65f5ba5ed9f5f609e
Author: pwawrzyniak <pi...@wawrzyniak.org.pl>
Date: 2017-03-17T11:09:52Z
Sample serializer/deserializer for JSON and InstanceContentEvent
Updates in comments
commit 45b3546f9c525af385f646447a09d3683b70937d
Author: pwawrzyniak <pi...@wawrzyniak.org.pl>
Date: 2017-03-17T14:40:25Z
KafkaDestinationProcessor implementation (sending msg to Kafka
commit 2e6d502a2909731cb9859ba3403407e6a33a2fc2
Author: pwawrzyniak <pi...@wawrzyniak.org.pl>
Date: 2017-03-24T13:34:49Z
Tests for kafkaUtils and KafkaEntranceProcessor, minor changes in classes
commit 208febc0524ddc328ae1a785f0642650173f7101
Author: pwawrzyniak <pi...@wawrzyniak.org.pl>
Date: 2017-04-11T14:44:30Z
Tests for KafkaDestinationProcessor, minor changes in classes
commit ff6b84dc5403c102388646a31f120c11ee416063
Author: Jakub Jankowski <ja...@orange.com>
Date: 2017-04-28T15:08:29Z
Added Kafka Avro serializer with unit tests. Added Kafka task with unit test
commit 7997fd538c258e8844cbd1a81d4c6c43d8788aa4
Author: Jakub Jankowski <ja...@orange.com>
Date: 2017-05-01T09:23:00Z
Merge branch 'SAMOA-65' of https://github.com/pwawrzyniak/incubator-samoa into SAMOA-65
commit ae7889fce1f52afdaa6d7e6080cff54dd1b01b9b
Author: Jakub Jankowski <ja...@orange.com>
Date: 2017-05-10T12:37:25Z
Fixing avro deserialization
commit 107c4c5f72fcacc38d02bfc7cd1552a1593cdd5a
Author: Jakub Jankowski <ja...@orange.com>
Date: 2017-05-10T12:45:16Z
Fixing avro deserialization
commit cde0642ccc768f7492e8c5811b2f1232e58acec4
Author: pwawrzyniak <pi...@wawrzyniak.org.pl>
Date: 2017-05-12T15:17:27Z
Rebuild of Consumer-related classes (added separate thread to read from Kafka, blocking until sth received)
Major cleanup in tests
commit f20301a4c64c9487fdbcff17829b6a8018e9a00b
Author: pwawrzyniak <pi...@wawrzyniak.org.pl>
Date: 2017-05-12T15:22:54Z
Minor change in tests
commit b4ee754c54ac5be51f9846704947e7c19c186e73
Author: Jakub Jankowski <ja...@orange.com>
Date: 2017-05-16T12:11:36Z
Fixing avro deserializer (using ReflectDatumReader) to be able to read arrays in InstanceData objects
commit d636ba90851d8424a6d3e651d80c87d0a8d7394e
Author: pwawrzyniak <pi...@wawrzyniak.org.pl>
Date: 2017-05-16T13:00:33Z
Changes in JSON mapper
----
> Apache Kafka integration components for SAMOA
> ---------------------------------------------
>
> Key: SAMOA-65
> URL: https://issues.apache.org/jira/browse/SAMOA-65
> Project: SAMOA
> Issue Type: New Feature
> Components: SAMOA-API, SAMOA-Instances
> Reporter: Piotr Wawrzyniak
> Labels: kafka, sink, source, streaming
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> As of now Apache SAMOA includes no integration components for Apache Kafka, meaning in particular no possibility to read data coming from Kafka and write data with prediction results back to Kafka.
> The key assumptions for the development of Kafka-related components are as follows:
> 1) develop support for input data stream arriving to Apache Samoa via Apache Kafka
> 2) develop support for output data stream produced by Apache Samoa, including the results of stream mining and forwarded to Apache Kafka to be provided in this way to other modules consuming the stream.
> This makes the goal of this issue is to create the following components:
> 1) KafkaEntranceProcessor in samoa-api. This entrance processor will be able to accept incoming Kafka stream. It will require KafkaDeserializer interface implementation to be delivered. The role of Deserializer would be to translate incoming Apache Kafka messages into implementation of Instance interface of SAMOA.
> 2) KafkaDestinationProcessor in samoa-api. Similarly to the KafkaEntranceProcessor, this processor would require KafkaSerializer interface implementation to be delivered. The role of Serializer would be to create a Kafka message from the underlying Instance class.
> 3) KafkaStream, as the extension to existing streams (e.g. InstanceStream), would take similar role to other streams, and will provide the control over Instances flows in the entire topology.
> Moreover, the following assumptions are considered:
> 1) Components would be implemented with the use of most up-to-date version of Apache Kafka, i.e. 0.10
> 2) Samples of aforementioned Serializer and Deserializer would be delivered, both supporting AVRO and JSON serialization of Instance objects.
> 3) Sample testing classes providing reference use of Kafka source and destination would be included in the project as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)