You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2019/05/08 13:55:00 UTC

[jira] [Work logged] (BEAM-7240) Kinesis IO Watermark Computation Improvements

     [ https://issues.apache.org/jira/browse/BEAM-7240?focusedWorklogId=239213&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239213 ]

ASF GitHub Bot logged work on BEAM-7240:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 08/May/19 13:54
            Start Date: 08/May/19 13:54
    Worklog Time Spent: 10m 
      Work Description: ajothomas commented on issue #8513: [BEAM-7240] Kinesis IO Watermark Computation Improvements
URL: https://github.com/apache/beam/pull/8513#issuecomment-490495031
 
 
   Thanks, @aromanenko-dev. I have created a Jira ticket and changed the title and commit messages per your suggestion. Please review the PR whenever you find the time.
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

            Worklog Id:     (was: 239213)
            Time Spent: 10m
    Remaining Estimate: 0h

> Kinesis IO Watermark Computation Improvements
> ---------------------------------------------
>
>                 Key: BEAM-7240
>                 URL: https://issues.apache.org/jira/browse/BEAM-7240
>             Project: Beam
>          Issue Type: Improvement
>          Components: io-java-kinesis
>            Reporter: Ajo Thomas
>            Priority: Minor
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently, watermarks in kinesis IO are computed taking into account the record arrival time in a {{KinesisRecord}}. The arrival time might not always be the right representation of the event time. The user of the IO should be able to specify how they want to extract the event time from the KinesisRecord. 
> As the per current logic, the end user of the IO cannot control watermark computation in any way. A user should be able to control watermark computation through some custom heuristics or configurable params like time duration to advance the watermark if no data was received (could be due to a shard getting stalled.  The watermark should advance and not be stalled in that case).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)