You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "snmvaughan (via GitHub)" <gi...@apache.org> on 2023/02/02 17:37:21 UTC
[GitHub] [hadoop] snmvaughan opened a new pull request, #5343: HDFS-16905. Provide default hadoop.log.dir for tests
snmvaughan opened a new pull request, #5343:
URL: https://github.com/apache/hadoop/pull/5343
### Description of PR
Allow testing in other environments by providing a default setting for hadoop.log.dir
### How was this patch tested?
Tested using the Hadoop development environment Docker image and an IDE on Mac
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snmvaughan commented on pull request #5343: HDFS-16905. Provide default hadoop.log.dir for tests
Posted by "snmvaughan (via GitHub)" <gi...@apache.org>.
snmvaughan commented on PR #5343:
URL: https://github.com/apache/hadoop/pull/5343#issuecomment-1419321373
The value set in the `hadoop-project/pom.xml` only applies to executions performed through Maven. Test execution through the IDE and alternative forms of test execution won't see this setting, which causes problems.
I'll change the value to `target/log` as I had it originally, which matches this setting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-16905. Provide default hadoop.log.dir for tests [hadoop]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #5343: HDFS-16905. Provide default hadoop.log.dir for tests
URL: https://github.com/apache/hadoop/pull/5343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5343: HDFS-16905. Provide default hadoop.log.dir for tests
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5343:
URL: https://github.com/apache/hadoop/pull/5343#issuecomment-1423053155
> when i run tests in intellij idea I use its build (not maven) and set sysprops like -Dfs.s3a.scale.test.enabled=true for it. maybe that's needed here.
I see, but since "scale" is already a profile, I actually enable the profile in IDEA so that even `-Dfs.s3a.scale.test.enabled=true` is not required, it will be done by profile anyways.
<img width="280" alt="Screenshot 2023-02-08 at 10 17 00 AM" src="https://user-images.githubusercontent.com/34790606/217617224-afafad88-2e80-4d2d-96c4-1708f4fa940d.png">
> with this patch do hdfs tests log to the right place?
This patch changes are actually not taken into effect by IDE because hadoop-project pom `<hadoop.log.dir>${project.build.directory}/log</hadoop.log.dir>` value is loaded by IDE as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5343: HDFS-16905. Provide default hadoop.log.dir for tests
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5343:
URL: https://github.com/apache/hadoop/pull/5343#issuecomment-1418615144
Without this patch, do we not have `hadoop.log.dir` being set to `{module}/{sub-module}/target/log` by default?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5343: HDFS-16905. Provide default hadoop.log.dir for tests
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5343:
URL: https://github.com/apache/hadoop/pull/5343#issuecomment-1418618542
We have this being set in hadoop-project pom `<hadoop.log.dir>${project.build.directory}/log</hadoop.log.dir>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5343: HDFS-16905. Provide default hadoop.log.dir for tests
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #5343:
URL: https://github.com/apache/hadoop/pull/5343#discussion_r1095639304
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/test/resources/log4j.properties:
##########
@@ -16,6 +16,8 @@
#
# log4j configuration used during build and unit tests
+hadoop.log.dir=.
Review Comment:
where does this put the logs? in target/ ? should we have a subdir?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5343: HDFS-16905. Provide default hadoop.log.dir for tests
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5343:
URL: https://github.com/apache/hadoop/pull/5343#issuecomment-1419578697
I tried TestMetaSave only because this patch includes adding hadoop.log.dir in hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties.
The other thing is, regardless of whatever value of `hadoop.log.dir` I provide in above log4j.properties, there is no change in the outcome of `System.getProperty("hadoop.log.dir")`, it always prints `${dir}/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/log` location only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5343: HDFS-16905. Provide default hadoop.log.dir for tests
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5343:
URL: https://github.com/apache/hadoop/pull/5343#issuecomment-1419576009
Without applying this patch, I just ran `TestMetaSave` in my IDE and for all the tests in `TestMetaSave`, `System.getProperty("hadoop.log.dir")` prints `${dir}/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/log`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5343: HDFS-16905. Provide default hadoop.log.dir for tests
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #5343:
URL: https://github.com/apache/hadoop/pull/5343#issuecomment-1419526380
your ide will run it wherever your ide is configured for a base dir; they will need to set something
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snmvaughan commented on pull request #5343: HDFS-16905. Provide default hadoop.log.dir for tests
Posted by "snmvaughan (via GitHub)" <gi...@apache.org>.
snmvaughan commented on PR #5343:
URL: https://github.com/apache/hadoop/pull/5343#issuecomment-1416155276
I originally had it set to `./target/logs`, but changed it to `.` to match the other modules. I would think pointing it to `target` would make the most sense.
Perhaps we should update all the `hadoop.log.dir` for tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5343: HDFS-16905. Provide default hadoop.log.dir for tests
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5343:
URL: https://github.com/apache/hadoop/pull/5343#issuecomment-1419653059
I was just able to confirm that if we revert https://github.com/apache/hadoop/commit/90fe7bcc3b30f8497d7a5124c91c5aea28f2fccb commit and refresh IDE maven cache, we would run into issues, otherwise this commit has taken care of providing `hadoop.log.dir` value as `${project.build.directory}/log` that applies to all modules including hadoop-hdfs.
What I did realize is that we have some redundancy with setting `hadoop.log.dir` on multiple sub-modules, which should be removed because the value of `hadoop.log.dir` would anyways be set up by hadoop-project pom which is parent of all modules poms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-16905. Provide default hadoop.log.dir for tests [hadoop]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5343:
URL: https://github.com/apache/hadoop/pull/5343#issuecomment-3449071916
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5343: HDFS-16905. Provide default hadoop.log.dir for tests
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5343:
URL: https://github.com/apache/hadoop/pull/5343#issuecomment-1419508684
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 19m 20s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 72m 28s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | shadedclient | 22m 45s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 52s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 1m 18s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 104m 13s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5343/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5343 |
| Optional Tests | dupname asflicense unit codespell detsecrets |
| uname | Linux 1e3e11f36f4f 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1877c2128d6949a3829781e2ac2392731aebd9ca |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5343/2/testReport/ |
| Max. process+thread count | 670 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5343/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5343: HDFS-16905. Provide default hadoop.log.dir for tests
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #5343:
URL: https://github.com/apache/hadoop/pull/5343#issuecomment-1422986551
when i run tests in intellij idea I use its build (not maven) and set sysprops like `-Dfs.s3a.scale.test.enabled=true` for it. maybe that's needed here.
anyway, what now? with this patch do hdfs tests log to the right place?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5343: HDFS-16905. Provide default hadoop.log.dir for tests
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5343:
URL: https://github.com/apache/hadoop/pull/5343#issuecomment-1414249776
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 21m 20s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 74m 21s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | shadedclient | 22m 25s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 52s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 1m 19s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not generate ASF License warnings. |
| | | 106m 8s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5343/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5343 |
| Optional Tests | dupname asflicense unit codespell detsecrets |
| uname | Linux b8068cb1680d 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 77513b45d884cc0cd5fa45428911a9afb71a7f56 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5343/1/testReport/ |
| Max. process+thread count | 565 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5343/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org