You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by GitBox <gi...@apache.org> on 2020/10/18 21:29:14 UTC
[GitHub] [incubator-gobblin] sv2000 opened a new pull request #3131: GOBBLIN-1292: Undo hardcoded tmp directory locations for unit tests
sv2000 opened a new pull request #3131:
URL: https://github.com/apache/incubator-gobblin/pull/3131
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1292
### Description
- [x] Here are some details about my PR, including screenshots (if applicable):
Several unit tests in Gobblin make use of hardcoded tmp directory locations. If these locations are not cleaned up, they have the potential to cause test failures in subsequent builds. Further, hardcoded locations may potentially collide with other tests which may accidentally use the same location. This task modifies a host of tests to use dynamically generated temp directories which are automatically cleaned up when tests complete.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
This PR modifies existing unit tests.
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] sv2000 commented on pull request #3131: GOBBLIN-1292: Undo hardcoded tmp directory locations for unit tests
Posted by GitBox <gi...@apache.org>.
sv2000 commented on pull request #3131:
URL: https://github.com/apache/incubator-gobblin/pull/3131#issuecomment-712247761
> @sv2000 : This works.
> A suggestion: Did you consider factoring out the temp directory generator into another class (perhaps in the module: gobblin-test-utils) and have every test use it directly?
>
> Usage might look like:
>
> TestDirectoryManager dirMgr = new TestDirectoryManager(this.getClass()); // this sets up the root path, attaches to the calling class.
> this.taskStateFile = dirMgr.getDirectoryPath("/taskState/_RUNNING");
Thanks @shirshanka ! Yes, I did think of re-factoring the tmp dir creation. However, I did not see much of a benefit (in terms of avoiding code duplication) since the equivalent of the "TestDirectoryManager" needs to be duplicated whenever a tmp dir is created. Further, the tmp dir creation is really only 2 lines of code - one to create the directory itself and one for marking the dir for deletion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] shirshanka commented on pull request #3131: GOBBLIN-1292: Undo hardcoded tmp directory locations for unit tests
Posted by GitBox <gi...@apache.org>.
shirshanka commented on pull request #3131:
URL: https://github.com/apache/incubator-gobblin/pull/3131#issuecomment-712557981
@sv2000 : I think the standardization of how temporary directories are created and how directories from within them are handed out might be useful refactoring for future test writers. They won't need to remember how to create a temp directory, how to combine a temp directory with inner directories etc.
I know it seems like a nit, but I think it sets up the "testing harness" in a better way IMO.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] sv2000 commented on pull request #3131: GOBBLIN-1292: Undo hardcoded tmp directory locations for unit tests
Posted by GitBox <gi...@apache.org>.
sv2000 commented on pull request #3131:
URL: https://github.com/apache/incubator-gobblin/pull/3131#issuecomment-711444834
@abti @htran1 Please review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] aplex commented on a change in pull request #3131: GOBBLIN-1292: Undo hardcoded tmp directory locations for unit tests
Posted by GitBox <gi...@apache.org>.
aplex commented on a change in pull request #3131:
URL: https://github.com/apache/incubator-gobblin/pull/3131#discussion_r509700611
##########
File path: gobblin-cluster/src/test/java/org/apache/gobblin/cluster/FsJobConfigurationManagerTest.java
##########
@@ -91,6 +91,10 @@ public void setUp() throws IOException {
return null;
}).when(this.eventBus).post(Mockito.anyObject());
+ File tmpDir = Files.createTempDir();
+ tmpDir.deleteOnExit();
+ this.jobConfDir = tmpDir.getAbsolutePath() + "/" + this.getClass().getSimpleName() + "/jobCatalog";
Review comment:
Is it important to have a class name in this path? AFAIK, createTempDir will return a unique path on each call. (same in other test classes below)
##########
File path: gobblin-cluster/src/test/java/org/apache/gobblin/cluster/suite/IntegrationJobCancelSuite.java
##########
@@ -17,14 +17,22 @@
package org.apache.gobblin.cluster.suite;
+import java.io.File;
+
import org.junit.Assert;
+import com.google.common.io.Files;
import com.typesafe.config.Config;
public class IntegrationJobCancelSuite extends IntegrationBasicSuite {
public static final String JOB_ID = "job_HelloWorldTestJob_1234";
- public static final String TASK_STATE_FILE = "/tmp/IntegrationJobCancelSuite/taskState/_RUNNING";
+ public static final String TASK_STATE_FILE;
+ static {
+ File tmpDir = Files.createTempDir();
Review comment:
Does this need to happen in static block? Can it happen for the instance of a class? Accessing file system from static block looks strange to me. Also TASK_STATE_FILE seems to be like a constant, but it changes on every application run.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] sv2000 commented on pull request #3131: GOBBLIN-1292: Undo hardcoded tmp directory locations for unit tests
Posted by GitBox <gi...@apache.org>.
sv2000 commented on pull request #3131:
URL: https://github.com/apache/incubator-gobblin/pull/3131#issuecomment-712975542
@shirshanka Thanks for the comment. Will address.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org