[jira] [Commented] (FALCON-1102) Gather data transfer details of filesystem replication

["Peeyush Bishnoi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FALCON-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14904188#comment-14904188 ] 

Peeyush Bishnoi commented on FALCON-1102:
-----------------------------------------

Thanks [~sriksun] for taking time to review. I have attached the updated patch after incorporating your comments. 

{quote}
    This is a very useful feature and should be enabled for standard replication as well.
{quote}

I have enabled for standard replication as well along with HDFS DR recipe. 
For standard replication, if user is interested to capture the required job counters, parameter “job.counter" with value “true” need to be added in the Feed entity property section. 
{code:java}
<properties>
      <property name="job.counter" value="true" />
</properties>
{code}

Once this parameter is set, replication-action.xml will be updated with element counterLogDir that signify about capturing counters from replication job.
{code:java}
 <arg>-counterLogDir</arg>
 <arg>${logDir}/job-${nominalTime}/${srcClusterName == 'NA' ? '' : srcClusterName}</arg>
{code}

{quote}
    Besides BYTESCOPIED, BANDWIDTH, Time taken, num files etc are some useful counters. Likewise if mechanism exists for this to be saved in the instance graph, we can figure how to open this up for processes as well. (That of course can be taken up in a separate jira)
{quote}
Apart from BYTESCOPIED, I have added timetaken and number of files copied counters captured from replication job. Already, I am populating the counters to graph DB, so that captured counters should be displayed on lineage UI.

{quote}
    Not sure if adding an index on bytescopied is useful

    +        makeLongKey("BYTESCOPIED");
{quote}
Fixed.

{quote}
    Should this be proxied file system ?

    +        FileSystem sourceFs = HadoopClientFactory.get().createProxiedFileSystem(
    +                inPaths.get(0).toUri(), getConf());
{quote}
Fixed.

{quote}
    Eagerly deleting the counter file, might result in gaps if the graph update were to fail for any reason. Can we let this be cleaned up along with the regular process log deletion. Please confirm that it is indeed getting deleted.

    +    private static void addCounterToWF(WorkflowExecutionContext executionContext) throws FalconException {
    +        FileSystem fs = HadoopClientFactory.get().createProxiedFileSystem(
    +                new Path(executionContext.getLogDir()).toUri());
    +        Path counterFilePath = getCounterFilePath(executionContext.getLogDir());
    +        try {
    +            if (fs.exists(counterFilePath)) {
    +                String counters = readCounters(fs, counterFilePath);
    +                if (!StringUtils.isEmpty(counters)) {
    +                    executionContext.context.put(WorkflowExecutionArgs.COUNTERS, counters);
    +                }
    +            }
    +        } catch (IOException e) {
    +            throw new FalconException("Error in checking counter file :" + e);
    +        } finally {
    +            try {
    +                if (fs.exists(counterFilePath)) {
    +                    fs.delete(counterFilePath, false);
    +                }
    +                fs.close();
    +            } catch (IOException e) {
    +                LOG.error("unable to delete counter file: {}", e);
    +            }
    +        }
    +    }
    +
{quote}
This counter file is just intermediate file to store job counter information.  Also, I have tested and found that this file is actually getting deleted after COUNTERS arg in WorkflowExecution populated with counters string. As you have suggested, now I am creating this file in Falcon log directory and thus it will be cleaned up along with the regular process log deletion, if it does not get clean up earlier.

{quote}
    Data type on Counter value should be long and not string

    +    private void addCountersToInstance(String counter, Vertex vertex) {
    +        int index = counter.indexOf(":");
    +        String counterKey = counter.substring(0, index);
    +        String counterValue = counter.substring(index+1, counter.length());
    +        vertex.setProperty(counterKey, counterValue);
    +    }
    +
{quote}
Fixed.

{quote}
    Minor nits: There are some wildcard imports.
{quote}
Fixed

> Gather data transfer details of filesystem replication
> ------------------------------------------------------
>
>                 Key: FALCON-1102
>                 URL: https://issues.apache.org/jira/browse/FALCON-1102
>             Project: Falcon
>          Issue Type: Sub-task
>    Affects Versions: 0.7
>            Reporter: Peeyush Bishnoi
>            Assignee: Peeyush Bishnoi
>         Attachments: FALCON-1102.patch
>
>
> Falcon UI has a requirement to show data transfer details from replication job invoked through HDFS recipe. To carry out this, we need to capture the bytes transferred from source to destination of replication job and then populate to backend store from where UI can access.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)