You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Xinyu Liu (JIRA)" <ji...@apache.org> on 2017/11/28 22:22:00 UTC
[jira] [Updated] (SAMZA-1516) Another round of issues found by BEAM tests

     [ https://issues.apache.org/jira/browse/SAMZA-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xinyu Liu updated SAMZA-1516:
-----------------------------
    Description: 
A couple of more fixes:

1. fix a bug of identifying input streams for an operator. The logic is inside the traversal of operator graph inside OperatorImplGraph. Basically we need to traverse further to add the input stream to the operator even the operator is already created.

2. for partitionBy, right now if there is null key and null partition key, Kafka will do round-robin (https://kafka.apache.org/0102/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html). This is causing problems for BEAM reduce. Since BEAM assume records without key will go to a single reducer. Add a fix in partitonBy to set the partitionKey.

  was:
A couple of more fixes:

1. fix a bug of identifying input streams for an operator. The logic is inside the traversal of operator graph inside OperatorImplGraph. Basically we need to traverse further to add the input stream to the operator even the operator is already created.

2. for partitionBy, right now if there is null key and null partition key, Kafka will do round-robin. This is causing problems for BEAM reduce. Since BEAM assume records without key will go to a single reducer. Add a fix in partitonBy to set the partitionKey.


> Another round of issues found by BEAM tests
> -------------------------------------------
>
>                 Key: SAMZA-1516
>                 URL: https://issues.apache.org/jira/browse/SAMZA-1516
>             Project: Samza
>          Issue Type: Bug
>    Affects Versions: 0.14.0
>            Reporter: Xinyu Liu
>            Assignee: Xinyu Liu
>             Fix For: 0.14.0
>
>
> A couple of more fixes:
> 1. fix a bug of identifying input streams for an operator. The logic is inside the traversal of operator graph inside OperatorImplGraph. Basically we need to traverse further to add the input stream to the operator even the operator is already created.
> 2. for partitionBy, right now if there is null key and null partition key, Kafka will do round-robin (https://kafka.apache.org/0102/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html). This is causing problems for BEAM reduce. Since BEAM assume records without key will go to a single reducer. Add a fix in partitonBy to set the partitionKey.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)