Gobblin properties

[Jay Sen <ja...@apache.org>]

Hi Guys,

I noticed that the property usage and definition is bit confusing,

Ex; property to define path where all job configurations (.pull files) are
placed.
following are all ways for the same:
"job.config.path"
"jobconf.dir"
"jobconf.fullyQualifiedPath"
"gobblin.cluster.jobconf.fullyQualifiedPath"
"gobblin.cluster.job.conf.path"

Also question about the properties with "gobblin.cluster" prefix and
without it,  does it get automatically prefixed somewhere base don the mode
?

and should we standardize these properties, like have getJobPath() in
configUtil to centralize the decision making ?

comments / thoughts ?

Thanks
Jay

Re: Gobblin properties

[Zhixiong Chen <zh...@linkedin.com>]

Hi Jen,

We're thinking about creating a convention to define gobblin properties. Given we have a large number of and distributed properties, centralizing configuration management doesn't scale. I have an idea, of keeping tracking of configs ownership and hierarchy, that:

  *   Each top level configurable construct/class defines configuration: `<camel-cased class name>.<config name>`, for example, a `HttpSource` may have a config named `url`, whose full config name will be `httpSource.url`
     *   For sub level or nested construct,  the convention will be `<camel-cased class name>.<sub scope>.<config name>`, for example, a `HttpSource` may uses a `HttpExtractor`, which has a config named `dataSize`, whose full config name will be `httpSource.extractor.dataSize`

For rolling out this idea

  *   Newly defined construct should accept `Config` as a constructor parameter instead of any kind of `State`
  *   Leave old construct untouched, but its new configurations will be defined following this convention.

Zhixiong
________________________________
From: Jay Sen <ja...@apache.org>
Sent: Saturday, June 22, 2019 9:17 PM
To: dev@gobblin.incubator.apache.org
Subject: Gobblin properties

Hi Guys,

I noticed that the property usage and definition is bit confusing,

Ex; property to define path where all job configurations (.pull files) are
placed.
following are all ways for the same:
"job.config.path"
"jobconf.dir"
"jobconf.fullyQualifiedPath"
"gobblin.cluster.jobconf.fullyQualifiedPath"
"gobblin.cluster.job.conf.path"

Also question about the properties with "gobblin.cluster" prefix and
without it,  does it get automatically prefixed somewhere base don the mode
?

and should we standardize these properties, like have getJobPath() in
configUtil to centralize the decision making ?

comments / thoughts ?

Thanks
Jay