You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by Jay Sen <ja...@apache.org> on 2019/02/28 08:06:05 UTC
JobConfigurationManager ?
Hi Dev Team,
just wanted to confirm that there is no scheduled local job file
configuration manager that will periodically check the job conf path for
file updates/addition.
I only see scheduled and streaming job config manager for kafka only.
does LinkedIn store the job config in kafka from where the gobblin picks
from ?
can someone pls explain ?
thanks
Jay
Fwd: JobConfigurationManager ?
Posted by Jay Sen <ja...@apache.org>.
Hi Hung,
just saw you response over markmail but looks like I never received your
response on my forwarding email, due to some issue.
I will look into the REST Api way of submitting jobs in Gobblin as a
Service and also the FSJobCatalog.
Thanks for the info and confirming.
- Jay
---------- Forwarded message ---------
From: Jay Sen <ja...@apache.org>
Date: Thu, Feb 28, 2019 at 12:06 AM
Subject: JobConfigurationManager ?
To: <de...@gobblin.incubator.apache.org>
Hi Dev Team,
just wanted to confirm that there is no scheduled local job file
configuration manager that will periodically check the job conf path for
file updates/addition.
I only see scheduled and streaming job config manager for kafka only.
does LinkedIn store the job config in kafka from where the gobblin picks
from ?
can someone pls explain ?
thanks
Jay
Re: JobConfigurationManager ?
Posted by Hung Tran <hu...@linkedin.com>.
Hi Jay,
You can use Gobblin-as-a-Service with Gobblin-cluster. Jobs can be created in GaaS through a REST call or through a git monitor. GaaS will then send the job spec to the cluster through kafka.
On the cluster side the jobs received through kafka are put into a JobCatalog.
If you would like to use the cluster without kafka then you can change the JobCatalog type to one that monitors the job conf path.
The default for 'gobblin.cluster.job.catalog' is 'org.apache.gobblin.runtime.job_catalog.NonObservingFSJobCatalog' which does not monitor the path and is intended to be used with the kafka job monitor.
You can set 'gobblin.cluster.job.catalog' to 'org.apache.gobblin.runtime.job_catalog.FSJobCatalog' to use a job catalog that monitors an FS path.
I haven't tried the config above. We have only used the cluster only with an initial set of files in the job conf dir or dynamic jobs through kafka. Also, there is no way to disable the JobConfigurationManager, so it will submit the initial set of jobs on startup.
Hung.
________________________________
From: Jay Sen <ja...@apache.org>
Sent: Thursday, February 28, 2019 12:06:05 AM
To: dev@gobblin.incubator.apache.org
Subject: JobConfigurationManager ?
Hi Dev Team,
just wanted to confirm that there is no scheduled local job file
configuration manager that will periodically check the job conf path for
file updates/addition.
I only see scheduled and streaming job config manager for kafka only.
does LinkedIn store the job config in kafka from where the gobblin picks
from ?
can someone pls explain ?
thanks
Jay