You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/03/25 00:57:16 UTC

[GitHub] [ozone] neils-dev opened a new pull request #2081: Hdds 5007

neils-dev opened a new pull request #2081:
URL: https://github.com/apache/ozone/pull/2081


   ## What changes were proposed in this pull request?
   Added intellij support for ozone fs shell commands for osf://, osf3:// such as `-ls -R osf3` through an intellij run configuration.  Configuration file,  _OzoneFsShell.xml_ runs the `hadoop-ozone-filesystem` main class executing shell commands through program arguments such as `-ls` and `-rm -R`.
   
   Run configuration executing through `hadoop-ozone-filesystem` requires two items added to the java classpath:
   
   1. core-site.xml (_$PROJECT_DIR$/hadoop-ozone/dev-support/intellij_) resource
   2. commons-cli (_$MAVEN_REPOSITORY$/commons-cli/commons-cli/1.2/commons-cli-1.2.jar_) jar - this jar is **_already_ included** in the ozone build from _hadoop-hdds-tools/pom.xml_
   
   **Also** in addition the environment variable _OZONE_CONF_DIR_ is set to the location of the core-site.xml configuration in the _/hadoop-ozone/dev-support/intellij_ directory.
   
   The two additions the the java classpath are appended to the build classpath through the VM Arguments **_-Xbootclasspath/a_** for the ide environment.  In doing so, the classpath is set for the ide environment without modifying the ozone production/test build environment. 
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-5007
   
   ## How was this patch tested?
   
   Patch is manually tested through the intellij run configuration ide environment.  Test is setup by the following steps:
   
   - volume create /vol1
   - bucket create /vol1/bucket1
   - key put /vol1/bucket1/key1 /tmp/myfile
   
   OzoneFsShell run configuration is configured with program argument `-ls -R osf3://bucket1.vol1`.
   
   Run configuration, OzoneFsShell is run.  Expected output in intellij terminal window is:
   -rw-rw-rw-   3 neil neil         12 2021-03-24 17:49 o3fs://bucket1.vol1/key1
   
   Process finished with exit code 0
   
   Tested with both Java 8 and Java 11 compilers.
   
   See attached images.
   
   ![ozonefsshell_config1](https://user-images.githubusercontent.com/81126310/112402668-91149c80-8cd2-11eb-87ec-0182760cdc31.png)
   ![ozonefsshell_config2](https://user-images.githubusercontent.com/81126310/112402678-94a82380-8cd2-11eb-989b-0c8ce6823b11.png)
   ![test_output](https://user-images.githubusercontent.com/81126310/112402681-970a7d80-8cd2-11eb-9b98-006ed8c4f6f6.png)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] neils-dev edited a comment on pull request #2081: HDDS-5007. Intellij run configuration for ozonefs shell.

Posted by GitBox <gi...@apache.org>.
neils-dev edited a comment on pull request #2081:
URL: https://github.com/apache/ozone/pull/2081#issuecomment-807488552


   I've pushed changes to the OzoneFsShell.xml file following comments by **@avijayanhwx**  - Thanks!.  Also, thanks to **@elek** for his help with the configurations settings, _OZONE_CONF_DIR_, etc and for suggesting this PR. 
   
   **Note:** the added jar to the classpath in the run configuration is fully specified including versioning ie. **_commons-cli-1.2.jar_**.  Adding this jar file to the classpath is done explicitly here through the VM argument.  This can also be done internally should we change the _hadoop-ozone-filesystem_ build _pom.xml_  file to include commons-cli (the commons-cli dependency is not fully specified in this case). 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] neils-dev commented on pull request #2081: HDDS-5007. Intellij run configuration for ozonefs shell.

Posted by GitBox <gi...@apache.org>.
neils-dev commented on pull request #2081:
URL: https://github.com/apache/ozone/pull/2081#issuecomment-806313084


   Missing Apache license on newly added file. (rat.sh)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] neils-dev commented on pull request #2081: HDDS-5007. Intellij run configuration for ozonefs shell.

Posted by GitBox <gi...@apache.org>.
neils-dev commented on pull request #2081:
URL: https://github.com/apache/ozone/pull/2081#issuecomment-807488552


   I've pushed changes to the OzoneFsShell.xml file following comments by **@avijayanhwx**  - Thanks!.  Also, thanks to **@elek** for his help with the configurations settings, _OZONE_CONF_DIR_, etc and for suggesting this PR. 
   
   **Note:** the added jar to the classpath in the run configuration is fully specified including versioning ie. **_commons-cli-1.2.jar_**.  Adding this jar file to the classpath is done explicitly here through the VM argument.  This can also be done internally should we change the _hadoop-ozone-fs_ build _pom.xml_  file to include commons-cli (the commons-cli dependency is not fully specified in this case). 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] elek merged pull request #2081: HDDS-5007. Intellij run configuration for ozonefs shell.

Posted by GitBox <gi...@apache.org>.
elek merged pull request #2081:
URL: https://github.com/apache/ozone/pull/2081


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] neils-dev commented on a change in pull request #2081: HDDS-5007. Intellij run configuration for ozonefs shell.

Posted by GitBox <gi...@apache.org>.
neils-dev commented on a change in pull request #2081:
URL: https://github.com/apache/ozone/pull/2081#discussion_r603541363



##########
File path: hadoop-ozone/dev-support/intellij/runConfigurations/OzoneFsShell.xml
##########
@@ -0,0 +1,36 @@
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+<component name="ProjectRunConfigurationManager">
+  <configuration default="false" name="OzoneFsShell" type="Application" factoryName="Application">
+    <envs>
+      <env name="OZONE_CONF_DIR" value="$PROJECT_DIR$/hadoop-ozone/dev-support/intellij" />
+    </envs>
+    <option name="MAIN_CLASS_NAME" value="org.apache.hadoop.fs.ozone.OzoneFsShell" />
+    <module name="hadoop-ozone-filesystem" />

Review comment:
       I'm glad to see the explicit _commons-cli_ jar file included to be unnecessary - great!  Using the _hadoop-ozone-tools_ module instead of _hadoop-ozone-filesystem_ with the _OzoneFsShell_ main resolves our external jar issues, thanks for this **@elek**.  Much better.  Committing suggestions. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] elek commented on pull request #2081: HDDS-5007. Intellij run configuration for ozonefs shell.

Posted by GitBox <gi...@apache.org>.
elek commented on pull request #2081:
URL: https://github.com/apache/ozone/pull/2081#issuecomment-810850447


   Merging after the green build. Thanks the contribution.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] elek commented on a change in pull request #2081: HDDS-5007. Intellij run configuration for ozonefs shell.

Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #2081:
URL: https://github.com/apache/ozone/pull/2081#discussion_r603092202



##########
File path: hadoop-ozone/dev-support/intellij/runConfigurations/OzoneFsShell.xml
##########
@@ -0,0 +1,36 @@
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+<component name="ProjectRunConfigurationManager">
+  <configuration default="false" name="OzoneFsShell" type="Application" factoryName="Application">
+    <envs>
+      <env name="OZONE_CONF_DIR" value="$PROJECT_DIR$/hadoop-ozone/dev-support/intellij" />
+    </envs>
+    <option name="MAIN_CLASS_NAME" value="org.apache.hadoop.fs.ozone.OzoneFsShell" />
+    <module name="hadoop-ozone-filesystem" />
+    <option name="PROGRAM_PARAMETERS" value="-conf=hadoop-ozone/dev-support/intellij/ozone-site.xml -ls -R o3fs://bucket1.vol1/" />
+    <option name="VM_PARAMETERS" value="-Dlog4j.configuration=file:hadoop-ozone/dev-support/intellij/log4j.properties -Xbootclasspath/a:$PROJECT_DIR$/hadoop-ozone/dev-support/intellij -Xbootclasspath/a:$MAVEN_REPOSITORY$/commons-cli/commons-cli/1.2/commons-cli-1.2.jar" />

Review comment:
       ```suggestion
       <option name="VM_PARAMETERS" value="-Dlog4j.configuration=file:hadoop-ozone/dev-support/intellij/log4j.properties -Xbootclasspath/a:$PROJECT_DIR$/hadoop-ozone/dev-support/intellij" />
   ```

##########
File path: hadoop-ozone/dev-support/intellij/runConfigurations/OzoneFsShell.xml
##########
@@ -0,0 +1,36 @@
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+<component name="ProjectRunConfigurationManager">
+  <configuration default="false" name="OzoneFsShell" type="Application" factoryName="Application">
+    <envs>
+      <env name="OZONE_CONF_DIR" value="$PROJECT_DIR$/hadoop-ozone/dev-support/intellij" />
+    </envs>
+    <option name="MAIN_CLASS_NAME" value="org.apache.hadoop.fs.ozone.OzoneFsShell" />
+    <module name="hadoop-ozone-filesystem" />

Review comment:
       ```suggestion
       <module name="hadoop-ozone-tools" />
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] neils-dev commented on a change in pull request #2081: HDDS-5007. Intellij run configuration for ozonefs shell.

Posted by GitBox <gi...@apache.org>.
neils-dev commented on a change in pull request #2081:
URL: https://github.com/apache/ozone/pull/2081#discussion_r603544655



##########
File path: hadoop-ozone/dev-support/intellij/runConfigurations/OzoneFsShell.xml
##########
@@ -0,0 +1,36 @@
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+<component name="ProjectRunConfigurationManager">
+  <configuration default="false" name="OzoneFsShell" type="Application" factoryName="Application">
+    <envs>
+      <env name="OZONE_CONF_DIR" value="$PROJECT_DIR$/hadoop-ozone/dev-support/intellij" />
+    </envs>
+    <option name="MAIN_CLASS_NAME" value="org.apache.hadoop.fs.ozone.OzoneFsShell" />
+    <module name="hadoop-ozone-filesystem" />
+    <option name="PROGRAM_PARAMETERS" value="-conf=hadoop-ozone/dev-support/intellij/ozone-site.xml -ls -R o3fs://bucket1.vol1/" />
+    <option name="VM_PARAMETERS" value="-Dlog4j.configuration=file:hadoop-ozone/dev-support/intellij/log4j.properties -Xbootclasspath/a:$PROJECT_DIR$/hadoop-ozone/dev-support/intellij -Xbootclasspath/a:$MAVEN_REPOSITORY$/commons-cli/commons-cli/1.2/commons-cli-1.2.jar" />

Review comment:
       Committing suggestion to remove explicit inclusion of jar files that are now automatically included in classpath with change of module from '_ozone-filesystem_' to '_ozone-tools_'.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] elek commented on pull request #2081: HDDS-5007. Intellij run configuration for ozonefs shell.

Posted by GitBox <gi...@apache.org>.
elek commented on pull request #2081:
URL: https://github.com/apache/ozone/pull/2081#issuecomment-809175199


   ` -Xbootclasspath/a:$PROJECT_DIR$/hadoop-ozone/dev-support/intellij`
   
   BTW, It's a very nice trick, I didn't know about it... Earlier we made it possible to configure the location of ozone-site.xml with `--conf` just because we didn't know about this workaround.
   
   With the same approach we can create runners for secure Ozone clusters.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org