[jira] [Work logged] (GOBBLIN-759) DistCP files modified in last n days within a look back period

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/GOBBLIN-759?focusedWorklogId=398151&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-398151 ]

ASF GitHub Bot logged work on GOBBLIN-759:
------------------------------------------

                Author: ASF GitHub Bot
            Created on: 05/Mar/20 05:32
            Start Date: 05/Mar/20 05:32
    Worklog Time Spent: 10m 
      Work Description: sv2000 commented on pull request #2633: GOBBLIN-759: Added feature to support DistCP to copy files that were …
URL: https://github.com/apache/incubator-gobblin/pull/2633#discussion_r388085632
 
 

 ##########
 File path: gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/TimestampBasedCopyableDatasetTest.java
 ##########
 @@ -91,12 +110,82 @@ public void testConfigOptions() {
         TimeBasedCopyPolicyForTest.class.getName());
   }
 
+  @Test
+  public void testCopyWithFilter() throws IOException {
+
+    /** source setup **/
+    Path srcRoot = new Path(this.testTempPath, "src/data/dataset1/daily");
+
+    if (this.localFs.exists(srcRoot)) {
+      this.localFs.delete(srcRoot, true);
+    }
+
+    List<DateTime> dateTimeList = Lists.newArrayList();
+    IntStream.range(0, 4)
+        .forEach(
+            i -> dateTimeList.add(new DateTime(DateTimeZone.forID(ConfigurationKeys.PST_TIMEZONE_NAME)).minusDays(i)));
+
+    String datePattern = "yyyy/MM/dd";
+    DateTimeFormatter formatter = DateTimeFormat.forPattern(datePattern);
+
+    for (DateTime dt : dateTimeList) {
+      String srcVersionPathStr = formatter.print(dt);
+      Path srcVersionPath = new Path(srcRoot, srcVersionPathStr);
+      this.localFs.mkdirs(srcVersionPath);
+
+      Path srcfile = new Path(srcVersionPath, "file1.avro");
+      this.localFs.create(srcfile);
+    }
+
+    /** destination setup **/
+    Path destRoot = new Path(this.testTempPath, "dest/data/dataset1");
+    if (this.localFs.exists(destRoot)) {
+      this.localFs.delete(destRoot, true);
+    }
+    this.localFs.mkdirs(destRoot);
+
+    Properties props = new Properties();
+    props.setProperty(TimestampBasedCopyableDataset.COPY_POLICY, SelectBetweenTimeBasedPolicy.class.getName());
+    props.setProperty(TimestampBasedCopyableDataset.DATASET_VERSION_FINDER,
+        DateTimeDatasetVersionFinder.class.getName());
+    props.setProperty(SelectBetweenTimeBasedPolicy.TIME_BASED_SELECTION_MIN_LOOK_BACK_TIME_KEY, "1d");
+    props.setProperty(SelectBetweenTimeBasedPolicy.TIME_BASED_SELECTION_MAX_LOOK_BACK_TIME_KEY, "6d");
+    props.setProperty(DateTimeDatasetVersionFinder.DATE_TIME_PATTERN_KEY, "yyyy/MM/dd");
+    props.setProperty("gobblin.dataset.copyable.file.filter.class",
 
 Review comment:
   Make DatasetUtils.COPYABLE_FILE_FILTER_KEY package private and use it here?
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 398151)

> DistCP files modified in last n days within a look back period
> --------------------------------------------------------------
>
>                 Key: GOBBLIN-759
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-759
>             Project: Apache Gobblin
>          Issue Type: Improvement
>            Reporter: Karthik Amarnath
>            Priority: Major
>          Time Spent: 7h 10m
>  Remaining Estimate: 0h
>
> *Feature Request:*
>  # DistCP only the files modified in last n days within the look back window.
>  # DistCP will copy only the files modified even when the source file which were NOT modified in last n days in the destination directory.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)