[jira] [Updated] (GOBBLIN-259) Support writing Kafka messages to db/table file path

["Zhixiong Chen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/GOBBLIN-259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zhixiong Chen updated GOBBLIN-259:
----------------------------------
    Summary: Support writing Kafka messages to db/table file path  (was: CLONE - Gobblin runOnce not working correctly)

> Support writing Kafka messages to db/table file path
> ----------------------------------------------------
>
>                 Key: GOBBLIN-259
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-259
>             Project: Apache Gobblin
>          Issue Type: Bug
>            Reporter: Zhixiong Chen
>            Assignee: Zhixiong Chen
>              Labels: Writer:HDFS
>
> runOnce feature of Gobblin does not work correctly. A job without a schedule will be re-run upon Gobblin restart or modifications on the file. Additionally, `*.done` files are not being written.
> Root cause:
> In JobScheduler, Gobblin checks whether the job has a schedule in the method `scheduleJob(Properties, JobListener, Map, Class)` and sets the key `ConfigurationKeys.JOB_RUN_ONCE_KEY` accordingly. On the other hand, the method `scheduleGeneralConfiguredJobs()` checks the key `ConfigurationKeys.JOB_RUN_ONCE_KEY` and if runonce, creates the `RunOnceJobListener` that creates the `*.done` file. However, `scheduleGeneralConfiguredJobs()` is called before `scheduleJob(Properties, JobListener, Map, Class)`, so the property has not been set yet, and the `*.done` file is never written.
> On Gobblin restart, Gobblin checks for presence of done files, and skips jobs that have already been executed. However, the done file is not present, so the job gets repeated.
>  
> *Github Url* : https://github.com/linkedin/gobblin/issues/1195 
> *Github Reporter* : [~ibuenros] 
> *Github Created At* : 2016-08-11T21:03:24Z 
> *Github Updated At* : 2017-01-12T04:59:43Z



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)