[jira] [Created] (GOBBLIN-1399) provide a way to speficy writer path and file name format via config

["Jay Sen (Jira)" <ji...@apache.org>]

Jay Sen created GOBBLIN-1399:
--------------------------------

             Summary: provide a way to speficy writer path and file name format via config
                 Key: GOBBLIN-1399
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1399
             Project: Apache Gobblin
          Issue Type: New Feature
          Components: gobblin-api
    Affects Versions: 0.15.0
            Reporter: Jay Sen
            Assignee: Hung Tran
             Fix For: 0.16.0


currently gobblin has hard coded specification for writer's path and file name

primarily it has namespace and tablename and default - 3 way to have writer's path and file name.
{code:java}
// code placeholder

switch (getWriterFilePathType(state)) {
  case NAMESPACE_TABLE:
    // writer.file.path.format = <extract.namespace>/<extract.table.name>/
    return getNamespaceTableWriterFilePath(state);
  case TABLENAME:
    // <extract.table.name>
    return WriterUtils.getTableNameWriterFilePath(state);
  default:
    return WriterUtils.getDefaultWriterFilePath(state, numBranches, branchId);
}

{code}
 

 Filename:
{code:java}
namespace.replaceAll("\\.", "/") + "/" + table + "/" + extractId + "_"
    + (isFull ? "full" : "append");
{code}
 

There is no way user can add any other parameters like version, batchId.

Also it would be awesome to have any configuration value to be part of the writer path, which can be defined by the format like this

 

{code: java}

extract.type = increments
writer.file.path.format="<extract.table.name>/<extract.extract.id>/<extract.type>"
writer.file.name.format="part.<writer_id>_batch_<dataset.batch_id>.<branch_id>.<format_extension>"

{code}

 

This will be enabled by the feature flag, so existing functionality can remains the same.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)