You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@gobblin.apache.org by "Lei Sun (JIRA)" <ji...@apache.org> on 2018/09/25 19:05:00 UTC

[jira] [Created] (GOBBLIN-595) Filter non-jar file in addJar method

Lei Sun created GOBBLIN-595:
-------------------------------

             Summary: Filter non-jar file in addJar method
                 Key: GOBBLIN-595
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-595
             Project: Apache Gobblin
          Issue Type: Improvement
            Reporter: Lei Sun
            Assignee: Lei Sun


In the addJar method, we use file length to monitor if the uploading is in-progress or completed. And we don't overwrite existing file since there are multiple gobblin instance sharing the same directory.

 

However consider the case:

There's a file named as FILE, and it has been uploaded before on a sharedJar file path, in the path of hdfs:///tmp/FILE. When we compare them and saw length difference, we just regarded it in the process of uploading. But this is not true obviously especially for log file (file name stay the same after we append more logs into it), except that for .jar file whenever there's updates, the file name will be updated with version number.  But since this uploading is supposed to be for .jar file only, we could bypass it by only accepting .jar file. 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)