You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Joseph Allemandou (Jira)" <ji...@apache.org> on 2021/10/07 09:07:00 UTC
[jira] [Updated] (GOBBLIN-1558) Make it possible to overwrite
BaseDataPublisher behavior when parent-folder doesn't exist
[ https://issues.apache.org/jira/browse/GOBBLIN-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joseph Allemandou updated GOBBLIN-1558:
---------------------------------------
Description:
It's currently not possible to overwrite the behavior of the publisher when publishing to non-existing parent directory (first run of job type for instance). This ticket is about extracting a func tion in the the BaseDataPublisher to make behavior possible to override.
Extracted new method
{code:java}
addWriterOutputToNewDir{code}
that goes with the already existing
{code:java}
addWriterOutputToExistingDir{code}
The goal of this ticket is to NOT change the behavior of the current code.
was:
When publishing to non-existing parent directory (first run of job type for instance), the BaseDataPublisher doesn't recurse into written folders but moves the top written folder at once (line 429). While being correct, this prevent the state's PUBLISHER_DIRS value to contain the detailed published folder.
The update consists of a new method
{code:java}
addWriterOutputToNewDir{code}
that mimic the already existing
{code:java}
addWriterOutputToExistingDir{code}
and is applied when the parent folder is not present, making the PUBLISHER_DIRS value being filled with detailed folder value.
Summary: Make it possible to overwrite BaseDataPublisher behavior when parent-folder doesn't exist (was: Make BasePublisher record detailed folders in state when parent-folder doesn't exist)
> Make it possible to overwrite BaseDataPublisher behavior when parent-folder doesn't exist
> -----------------------------------------------------------------------------------------
>
> Key: GOBBLIN-1558
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1558
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-core
> Reporter: Joseph Allemandou
> Assignee: Abhishek Tiwari
> Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It's currently not possible to overwrite the behavior of the publisher when publishing to non-existing parent directory (first run of job type for instance). This ticket is about extracting a func tion in the the BaseDataPublisher to make behavior possible to override.
> Extracted new method
> {code:java}
> addWriterOutputToNewDir{code}
> that goes with the already existing
> {code:java}
> addWriterOutputToExistingDir{code}
> The goal of this ticket is to NOT change the behavior of the current code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)