You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Abhishek Tiwari (JIRA)" <ji...@apache.org> on 2017/08/22 08:10:00 UTC
[jira] [Updated] (GOBBLIN-19) dataset specific properties are
ignored&dropped by KafkaBiLevelWorkUnitPacker
[ https://issues.apache.org/jira/browse/GOBBLIN-19?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Abhishek Tiwari updated GOBBLIN-19:
-----------------------------------
Sprint: Apache Gobblin 170807, Apache Gobblin 170821 (was: Apache Gobblin 170807)
> dataset specific properties are ignored&dropped by KafkaBiLevelWorkUnitPacker
> -----------------------------------------------------------------------------
>
> Key: GOBBLIN-19
> URL: https://issues.apache.org/jira/browse/GOBBLIN-19
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: Clemens Valiente
> Assignee: Kuai Yu
>
> I failed to get dataset.specific.props to work on our jobs, and I think I found the reason:
> in KafkaSource.getWorkUnitForTopicPartition the properties are added correctly to the individual workunits.
> The KafkaBiLevelWorkUnitPacker then assigns the WorkUnits to their bins and combines them into one WorkUnit in squeezeMultiWorkUnit() but doesn't copy over the topicSpecificSettings.
> Using the KafkaSingleLevelWorkUnitPacker works fine with dataset.specific.props since it doesn't call squeezeMultiWorkUnit on non-empty workUnits.
>
> *Github Url* : https://github.com/linkedin/gobblin/issues/1901
> *Github Reporter* : [~cvaliente]
> *Github Created At* : 2017-05-26T09:25:54Z
> *Github Updated At* : 2017-05-31T06:39:04Z
> h3. Comments
> ----
> [~cvaliente] wrote on 2017-05-26T10:55:37Z : fix in #1903
>
> *Github Url* : https://github.com/linkedin/gobblin/issues/1901#issuecomment-304253329
> ----
> [~stakiar] wrote on 2017-05-30T17:42:07Z : Doesn't `KafkaSource#addTopicSpecificPropsToWorkUnits` handle adding dataset specific configuration? That method is run after the bin-packing is done. So if `dataset.specific.props` isn't working I would guess the bug would be in that method.
>
> *Github Url* : https://github.com/linkedin/gobblin/issues/1901#issuecomment-304953996
> ----
> [~cvaliente] wrote on 2017-05-31T06:39:04Z : You are right, that wasn't yet implemented in 0.9 and I forgot to check upstream.
>
> *Github Url* : https://github.com/linkedin/gobblin/issues/1901#issuecomment-305098396
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)