You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/05/31 01:01:04 UTC
[jira] [Commented] (SAMZA-1312) Add Control Messages and
Intermediate Stream Serde
[ https://issues.apache.org/jira/browse/SAMZA-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16030455#comment-16030455 ]
ASF GitHub Bot commented on SAMZA-1312:
---------------------------------------
GitHub user xinyuiscool opened a pull request:
https://github.com/apache/samza/pull/207
SAMZA-1312: Add Control Messages and Intermediate Stream Serde
In this patch, we add the control message types which includes:
* EndOfStreamMessage
* WatermarkMessage
To support in-band data and control messages, we provide a wrapper serde (IntermediateMessageSerde) to serialize/deserialize data/control messages based on message type byte (first byte in the intermediate stream message). The format of the message is defined in SAMZA-1312. The patch integrates this serde with SerdeManager.
Tested in example jobs deployed locally and works as expected.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xinyuiscool/samza SAMZA-1312
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/207.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #207
----
commit f0bb67f26204fc5698f5d9b5fbdac8b90c70fabe
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-05-26T22:01:05Z
Add control message types and serde
commit 2d79ed7a8a6f4698904232899ef485c4f3a1e627
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-05-30T22:41:12Z
Clean up the ControlMessage class and some enhancements
commit f0e32d517942fb119253e2d4f06fc921c34ec0ba
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-05-31T00:52:49Z
Merge branch 'master' into SAMZA-1312
----
> Add Control Messages and Intermediate Stream Serde
> --------------------------------------------------
>
> Key: SAMZA-1312
> URL: https://issues.apache.org/jira/browse/SAMZA-1312
> Project: Samza
> Issue Type: Bug
> Reporter: Xinyu Liu
> Assignee: Xinyu Liu
>
> The format of the intermediate stream message:
> * IntermediateMessage => [MessageType MessageData]
> ** MessageType => byte
> ** MessageData => byte[]
>
> * MessageType => [0(UserMessage), 1(Watermark), 2(EndOfStream)]
> * MessageData => [UserMessage/ControlMessage]
> * ControlMessage =>
> ** Type => int
> ** Version => int
> ** Other Message Data (based on different types of control message)
> For user message, we will use the user provided serde (default is the system serde). For control message, we will use JSON serde since it is built in Samza and easy to parse.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)