You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by IMS94 <gi...@git.apache.org> on 2017/07/10 14:27:30 UTC
[GitHub] oodt pull request #44: OODT-945 Rework OODT configuration to make use of Zoo...
GitHub user IMS94 opened a pull request:
https://github.com/apache/oodt/pull/44
OODT-945 Rework OODT configuration to make use of Zookeeper for distributed configuration management (GSoC 2017)
This PR includes the Distributed Configuration Management implementation for OODT [OODT-945] along with the distributed configuration publishing CLI [OODT-952].
Explanation on the design of this feature is available at [Rework OODT configuration to make use of Zookeeper for distributed configuration management](https://cwiki.apache.org/confluence/display/OODT/Rework+OODT+configuration+to+make+use+of+Zookeeper+for+distributed+configuration+management).
User documentation on how to use this distributed configuration management feature is available at [OODT Distributed Configuration Management](https://cwiki.apache.org/confluence/display/OODT/OODT+Distributed+Configuration+Management)
Corresponding GSoC project proposal is available at [https://cwiki.apache.org/confluence/display/OODT/Proposal](https://cwiki.apache.org/confluence/display/OODT/Proposal)
**Since this is the first glance of the distributed configuration management feature, there can be bugs and improvements required to be addressed. Please review this PR and provide me with your kind feedback so that I can improve on that.**
Thank you!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/IMS94/oodt feature/zookeeper-config
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/oodt/pull/44.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #44
----
commit ac7170bdf5c973374f141c6f3e031fa748e0af13
Author: Imesha Sudasingha <im...@gmail.com>
Date: 2017-04-01T07:00:55Z
Defined initial API for ConfigurationManager
commit 4b88f71497263675bb313462075fd552696ba717
Author: Imesha Sudasingha <im...@gmail.com>
Date: 2017-06-16T10:23:43Z
Implemented global configuration publishing mechanism.
commit fadf149cb77dd943af9e81c0f127ce74f203c478
Author: Imesha Sudasingha <im...@gmail.com>
Date: 2017-06-17T15:39:58Z
Altered configuration publishing mechanism to use spring configuration
commit 5fcf2220125b64afba080c6c9c6c5592daae2dea
Author: Imesha Sudasingha <im...@gmail.com>
Date: 2017-06-17T17:51:12Z
Implemented configuration fetching for DistributedConfigurationManager
commit 27a28daa3d3c448b462bd9a33eb6ac4fd859b75a
Author: Imesha Sudasingha <im...@gmail.com>
Date: 2017-07-02T15:23:49Z
Added zip packaging for distributed configuration publisher
commit a973c531fee5bf37ebf4448334470c75943af1ec
Author: Imesha Sudasingha <im...@gmail.com>
Date: 2017-07-03T10:25:22Z
Created test cases for distributed configuration manager and publisher
commit e697405223fe17c96739ae8f156639a0f5ef321c
Author: Imesha Sudasingha <im...@gmail.com>
Date: 2017-07-09T16:48:40Z
Minor improvements for configuration manager tests
commit 99b086b9887cafc8b89de3ff65fa61d3f80cc56c
Author: Imesha Sudasingha <im...@gmail.com>
Date: 2017-07-10T11:51:51Z
Improvements to file downloading from zookeeper and tests
commit a8092af5484d066a4dda623395bf21612d696d62
Author: Imesha Sudasingha <im...@gmail.com>
Date: 2017-07-10T14:24:28Z
Merge branch 'master' into feature/zookeeper-config
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt pull request #44: OODT-945 Rework OODT configuration to make use of Zoo...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on a diff in the pull request:
https://github.com/apache/oodt/pull/44#discussion_r126509940
--- Diff: filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java ---
@@ -52,29 +41,20 @@
import org.apache.oodt.cas.metadata.Metadata;
import org.apache.oodt.cas.metadata.exceptions.MetExtractionException;
import org.apache.oodt.commons.date.DateUtils;
+import org.apache.oodt.config.ConfigurationManager;
+import org.apache.oodt.config.ConfigurationManagerFactory;
+import org.apache.oodt.config.Constants;
import org.apache.xmlrpc.WebServer;
-import com.google.common.collect.Lists;
-
-import java.io.File;
-import java.io.FileInputStream;
-import java.io.FileNotFoundException;
-import java.io.FileOutputStream;
-import java.io.IOException;
+import java.io.*;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.URL;
-import java.util.Arrays;
-import java.util.Hashtable;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import java.util.Vector;
+import java.util.*;
--- End diff --
we don't use star imports and these changes are orthogonal
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt issue #44: OODT-945 Rework OODT configuration to make use of Zookeeper ...
Posted by IMS94 <gi...@git.apache.org>.
Github user IMS94 commented on the issue:
https://github.com/apache/oodt/pull/44
@chrismattmann Thanks for your support too. Did the tests pass for you without a problem?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt issue #44: OODT-945 Rework OODT configuration to make use of Zookeeper ...
Posted by IMS94 <gi...@git.apache.org>.
Github user IMS94 commented on the issue:
https://github.com/apache/oodt/pull/44
Hi @chrismattmann, No. Tests do not require any environment variable to be set. But if any of the above mentioned env variables are set, that could be a possible reason for your test failure (because I didn't encounter test failures). I'm using environment variables to determine each OODT component's HOME when downloading and storing configuration files from zookeeper.
Therefore, what I wanted know was whether you have any of those env vars set.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt issue #44: OODT-945 Rework OODT configuration to make use of Zookeeper ...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on the issue:
https://github.com/apache/oodt/pull/44
merged this thanks @IMS94
```
LMC-053601:oodt mattmann$ git push -u origin feature/zookeeper-config
Counting objects: 475, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (203/203), done.
Writing objects: 100% (475/475), 81.85 KiB | 0 bytes/s, done.
Total 475 (delta 155), reused 355 (delta 113)
remote: oodt git commit: Update log4j config.
remote: oodt git commit: Merge branch 'feature/zookeeper-config' of https://github.com/IMS94/oodt into feature/zookeeper-config
remote: oodt git commit: assert not null.
remote: oodt git commit: Fix for empty component home variable
remote: oodt git commit: Merge and apply robust changes for stack trace print in tests.
remote: oodt git commit: Merge branch 'feature/zookeeper-config' of https://github.com/IMS94/oodt into feature/zookeeper-config
remote: oodt git commit: fix for presence of component home environment variables in tests
remote: oodt git commit: Merge branch 'feature/zookeeper-config' of https://github.com/IMS94/oodt into feature/zookeeper-config
remote: oodt git commit: fix pom.xml
remote: oodt git commit: Fixed parent artifactId in config/pom.xml
remote: oodt git commit: Removed * imports. Improved code formatting and refactored code.
remote: oodt git commit: Merge branch 'master' of https://github.com/apache/oodt into feature/zookeeper-config
remote: oodt git commit: Update README.md
remote: oodt git commit: Merge branch 'OODT-847' of http://github.com/adhulipa/oodt
remote: oodt git commit: Record change for OODT-847.
remote: oodt git commit: Merge branch 'master' into feature/zookeeper-config
remote: oodt git commit: Improvements to file downloading from zookeeper and tests
remote: oodt git commit: Minor improvements for configuration manager tests
remote: oodt git commit: Created test cases for distributed configuration manager and publisher
remote: oodt git commit: Added zip packaging for distributed configuration publisher
remote: oodt git commit: Implemented configuration fetching for DistributedConfigurationManager
remote: oodt git commit: Altered configuration publishing mechanism to use spring configuration
remote: oodt git commit: Implemented global configuration publishing mechanism.
remote: oodt git commit: Defined initial API for ConfigurationManager
remote: oodt git commit: Fix for OODT-847 contributed by adhulipa & rverma
To https://git-wip-us.apache.org/repos/asf/oodt.git
ec5b201..3fb7660 feature/zookeeper-config -> feature/zookeeper-config
Branch feature/zookeeper-config set up to track remote branch feature/zookeeper-config from origin.
LMC-053601:oodt mattmann$
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt issue #44: OODT-945 Rework OODT configuration to make use of Zookeeper ...
Posted by IMS94 <gi...@git.apache.org>.
Github user IMS94 commented on the issue:
https://github.com/apache/oodt/pull/44
Hi @chrismattmann, can you check now? I had put wrong parent artifactId name in `config/pom.xml` earlier. It should build now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt pull request #44: OODT-945 Rework OODT configuration to make use of Zoo...
Posted by IMS94 <gi...@git.apache.org>.
Github user IMS94 commented on a diff in the pull request:
https://github.com/apache/oodt/pull/44#discussion_r126577697
--- Diff: README.md ---
@@ -53,8 +50,7 @@ To contribute a patch, follow these instructions (note that installing
```
-License (see also LICENSE.txt)
-==============================
+## License (see also LICENSE.txt)
--- End diff --
@chrismattmann Sure. But I think you have applied these changes to the Readme.md with the commit [update README.md](https://github.com/apache/oodt/commit/1b42cacacfc96e96b69f0c28d7eefcb89ae85856). Therefore, I will rollback this change
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt pull request #44: OODT-945 Rework OODT configuration to make use of Zoo...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on a diff in the pull request:
https://github.com/apache/oodt/pull/44#discussion_r126652721
--- Diff: README.md ---
@@ -53,8 +50,7 @@ To contribute a patch, follow these instructions (note that installing
```
-License (see also LICENSE.txt)
-==============================
+## License (see also LICENSE.txt)
--- End diff --
Hi @IMS94 I haven't applied *all of them*, just the *first one*, so can you please open a new PR with the other changes you did and remove/rollback this one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt pull request #44: OODT-945 Rework OODT configuration to make use of Zoo...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on a diff in the pull request:
https://github.com/apache/oodt/pull/44#discussion_r126674394
--- Diff: README.md ---
@@ -53,8 +50,7 @@ To contribute a patch, follow these instructions (note that installing
```
-License (see also LICENSE.txt)
-==============================
+## License (see also LICENSE.txt)
--- End diff --
hi @IMS94 I am saying that the rest of your heading formattings (for all else besides the first main heading) are also welcomed, so at a minimum:
1) remove them from this PR
2) if you like (and I am +1 for this so please do) open a new PR with *just those changes for the rest of the headings* and I will gladly commit it.
HTH.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt issue #44: OODT-945 Rework OODT configuration to make use of Zookeeper ...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on the issue:
https://github.com/apache/oodt/pull/44
hi @IMS94 got further, but failing on unit tests, thoughts?
```
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.oodt.config.distributed.DistributedConfigurationManagerTest
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.server.quorum.QuorumPeerConfig).
log4j:WARN Please initialize the log4j system properly.
Starting configuration publishing
Processing commands for component : FILE_MANAGER
Publishing configuration for : FILE_MANAGER
Published configuration for : FILE_MANAGER
Processing commands for component : RESOURCE_MANAGER
Publishing configuration for : RESOURCE_MANAGER
Published configuration for : RESOURCE_MANAGER
Starting configuration publishing
Processing commands for component : FILE_MANAGER
Clearing configuration for : FILE_MANAGER
Cleared configuration for : FILE_MANAGER
Processing commands for component : RESOURCE_MANAGER
Clearing configuration for : RESOURCE_MANAGER
Cleared configuration for : RESOURCE_MANAGER
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.209 sec <<< FAILURE! - in org.apache.oodt.config.distributed.DistributedConfigurationManagerTest
loadConfigurationTest(org.apache.oodt.config.distributed.DistributedConfigurationManagerTest) Time elapsed: 0.049 sec <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at org.apache.oodt.config.distributed.DistributedConfigurationManagerTest.loadConfigurationTest(DistributedConfigurationManagerTest.java:91)
Running org.apache.oodt.config.distributed.DistributedConfigurationPublisherTest
Starting configuration publishing
Processing commands for component : FILE_MANAGER
Publishing configuration for : FILE_MANAGER
Published configuration for : FILE_MANAGER
Verifying configuration for : FILE_MANAGER
OK... Configuration verified
Verified configuration for : FILE_MANAGER
Processing commands for component : RESOURCE_MANAGER
Publishing configuration for : RESOURCE_MANAGER
Published configuration for : RESOURCE_MANAGER
Verifying configuration for : RESOURCE_MANAGER
OK... Configuration verified
Verified configuration for : RESOURCE_MANAGER
Starting configuration publishing
Processing commands for component : FILE_MANAGER
Clearing configuration for : FILE_MANAGER
Cleared configuration for : FILE_MANAGER
Processing commands for component : RESOURCE_MANAGER
Clearing configuration for : RESOURCE_MANAGER
Cleared configuration for : RESOURCE_MANAGER
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.512 sec - in org.apache.oodt.config.distributed.DistributedConfigurationPublisherTest
Results :
Failed tests:
DistributedConfigurationManagerTest.loadConfigurationTest:91 null
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] OODT Core .......................................... SUCCESS [ 1.820 s]
[INFO] Common Utilities ................................... SUCCESS [ 17.797 s]
[INFO] CAS Command Line Interface ......................... SUCCESS [ 6.046 s]
[INFO] OODT - Configuration Management .................... FAILURE [ 5.426 s]
[INFO] Process Control System Input Data Package .......... SKIPPED
[INFO] Catalog and Archive Service Generic Multi-valued Metadata Container SKIPPED
[INFO] CAS Protocol ....................................... SKIPPED
[INFO] CAS Protocol FTP Implementation .................... SKIPPED
[INFO] CAS Protocol HTTP Implementation ................... SKIPPED
[INFO] CAS Protocol IMAPS Implementation .................. SKIPPED
[INFO] CAS Protocol SFTP Implementation ................... SKIPPED
[INFO] Query Expression ................................... SKIPPED
[INFO] OODT Single Sign On Security Package ............... SKIPPED
[INFO] Catalog and Archive File Management Component ...... SKIPPED
[INFO] OODT CAS Virtual Catalog and Integration Service. .. SKIPPED
[INFO] Catalog and Archive Resource Management Component .. SKIPPED
[INFO] Catalog and Archive Workflow Management Component .. SKIPPED
[INFO] Catalog and Archive Crawling Framework ............. SKIPPED
[INFO] CAS Curation Web Services .......................... SKIPPED
[INFO] Process Control System Core Package ................ SKIPPED
[INFO] OODT Wicket Web Components ......................... SKIPPED
[INFO] CAS Curation Interface ............................. SKIPPED
[INFO] CAS PGE Adaptor Framework .......................... SKIPPED
[INFO] CAS Installer Maven Mojo ........................... SKIPPED
[INFO] OODT :: Archetypes :: OpsUI ........................ SKIPPED
[INFO] OODT :: Archetypes :: RADiX ........................ SKIPPED
[INFO] OODT :: Archetypes ................................. SKIPPED
[INFO] CAS Push-Pull-Framework ............................ SKIPPED
[INFO] Product Service .................................... SKIPPED
[INFO] Profile Service .................................... SKIPPED
[INFO] OODT Web Grid ...................................... SKIPPED
[INFO] XML-configured, DBMS-based Product and Profile Server SKIPPED
[INFO] Apache OODT Configurable OPeNDAP Profile Server .... SKIPPED
[INFO] CAS File Manager Browser Web App ................... SKIPPED
[INFO] CAS Product Server Web Application ................. SKIPPED
[INFO] CAS Workflow Manager Monitor Web App ............... SKIPPED
[INFO] Catalog and Archive File Management Browser ........ SKIPPED
[INFO] Catalog and Archive Workflow Management GUI Editor . SKIPPED
[INFO] Process Control System Operator Interface Webapp ... SKIPPED
[INFO] OODT Process Control System JAX-RS service layer ... SKIPPED
[INFO] Apache OODT ........................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 31.600 s
[INFO] Finished at: 2017-07-13T15:38:14-07:00
[INFO] Final Memory: 53M/480M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project oodt-conf: There are test failures.
[ERROR]
[ERROR] Please refer to /Users/mattmann/git/oodt/config/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :oodt-conf
LMC-053601:oodt mattmann$
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt pull request #44: OODT-945 Rework OODT configuration to make use of Zoo...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on a diff in the pull request:
https://github.com/apache/oodt/pull/44#discussion_r126509639
--- Diff: config/src/main/resources/examples/filemgr/cmd-line-actions.xml ---
@@ -0,0 +1,123 @@
+<?xml version="1.0" encoding="UTF-8"?>
--- End diff --
maybe we should use maven-ant-tasks to copy these in the prepare-resources phase or in the compile phase (whatever you need them for) instead of physically copying the files?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt issue #44: OODT-945 Rework OODT configuration to make use of Zookeeper ...
Posted by IMS94 <gi...@git.apache.org>.
Github user IMS94 commented on the issue:
https://github.com/apache/oodt/pull/44
Hi @chrismattmann, Thanks for the invitation.
I have pushed an update. There's still a chance for the tests to fail because I store configuration files in `${COMPONENT_HOME}` directory if `${COMPONENT_HOME}` is present, else in the working directory. If it fails again, I will find a more convenient method for my implementation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt pull request #44: OODT-945 Rework OODT configuration to make use of Zoo...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on a diff in the pull request:
https://github.com/apache/oodt/pull/44#discussion_r126509803
--- Diff: config/src/main/resources/examples/filemgr/filemgr.properties ---
@@ -0,0 +1,114 @@
+#
--- End diff --
same why copy these when we can use some maven copy task
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt pull request #44: OODT-945 Rework OODT configuration to make use of Zoo...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on a diff in the pull request:
https://github.com/apache/oodt/pull/44#discussion_r126501802
--- Diff: README.md ---
@@ -53,8 +50,7 @@ To contribute a patch, follow these instructions (note that installing
```
-License (see also LICENSE.txt)
-==============================
+## License (see also LICENSE.txt)
--- End diff --
great changes, these are orthogonal though - can you separate them into a new PR apart from this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt pull request #44: OODT-945 Rework OODT configuration to make use of Zoo...
Posted by IMS94 <gi...@git.apache.org>.
Github user IMS94 commented on a diff in the pull request:
https://github.com/apache/oodt/pull/44#discussion_r126577969
--- Diff: config/src/main/resources/examples/filemgr/cmd-line-actions.xml ---
@@ -0,0 +1,123 @@
+<?xml version="1.0" encoding="UTF-8"?>
--- End diff --
@chrismattmann I will look into that. Thanks for pointing that out. Didn't know we can do that with a plugin before.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt issue #44: OODT-945 Rework OODT configuration to make use of Zookeeper ...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on the issue:
https://github.com/apache/oodt/pull/44
hi @IMS94 do I need to set those env vars? This will screw up tests since they should not have environment variable requirements, or if so, should be mock'ed in the tests themselves.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt issue #44: OODT-945 Rework OODT configuration to make use of Zookeeper ...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on the issue:
https://github.com/apache/oodt/pull/44
they did pass once I disabled tesseract
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt pull request #44: OODT-945 Rework OODT configuration to make use of Zoo...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on a diff in the pull request:
https://github.com/apache/oodt/pull/44#discussion_r126597857
--- Diff: config/src/main/resources/examples/filemgr/cmd-line-actions.xml ---
@@ -0,0 +1,123 @@
+<?xml version="1.0" encoding="UTF-8"?>
--- End diff --
thanks check out Maven Antrun plugin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt issue #44: OODT-945 Rework OODT configuration to make use of Zookeeper ...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on the issue:
https://github.com/apache/oodt/pull/44
hi @IMS94 I have errors building this locally can you check:
```
LMC-053601:oodt mattmann$ mvn clean install
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[WARNING] 'parent.relativePath' of POM org.apache.oodt:oodt-conf:[unknown-version] (/Users/mattmann/git/oodt/config/pom.xml) points at org.apache.oodt:oodt-core instead of org.apache.oodt:oodt, please verify your project structure @ org.apache.oodt:oodt-conf:[unknown-version], /Users/mattmann/git/oodt/config/pom.xml, line 5, column 13
[ERROR] 'dependencies.dependency.version' for org.apache.curator:curator-framework:jar is missing. @ org.apache.oodt:oodt-conf:[unknown-version], /Users/mattmann/git/oodt/config/pom.xml, line 19, column 21
[ERROR] 'dependencies.dependency.version' for args4j:args4j:jar is missing. @ org.apache.oodt:oodt-conf:[unknown-version], /Users/mattmann/git/oodt/config/pom.xml, line 39, column 21
[ERROR] 'dependencies.dependency.version' for org.apache.curator:curator-test:jar is missing. @ org.apache.oodt:oodt-conf:[unknown-version], /Users/mattmann/git/oodt/config/pom.xml, line 55, column 21
@
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project org.apache.oodt:oodt-conf:1.1-SNAPSHOT (/Users/mattmann/git/oodt/config/pom.xml) has 3 errors
[ERROR] 'dependencies.dependency.version' for org.apache.curator:curator-framework:jar is missing. @ org.apache.oodt:oodt-conf:[unknown-version], /Users/mattmann/git/oodt/config/pom.xml, line 19, column 21
[ERROR] 'dependencies.dependency.version' for args4j:args4j:jar is missing. @ org.apache.oodt:oodt-conf:[unknown-version], /Users/mattmann/git/oodt/config/pom.xml, line 39, column 21
[ERROR] 'dependencies.dependency.version' for org.apache.curator:curator-test:jar is missing. @ org.apache.oodt:oodt-conf:[unknown-version], /Users/mattmann/git/oodt/config/pom.xml, line 55, column 21
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt pull request #44: OODT-945 Rework OODT configuration to make use of Zoo...
Posted by IMS94 <gi...@git.apache.org>.
Github user IMS94 commented on a diff in the pull request:
https://github.com/apache/oodt/pull/44#discussion_r126662934
--- Diff: README.md ---
@@ -53,8 +50,7 @@ To contribute a patch, follow these instructions (note that installing
```
-License (see also LICENSE.txt)
-==============================
+## License (see also LICENSE.txt)
--- End diff --
Hi @chrismattmann What I had done on README.md was applying **# Heading 1** (markdown heading styles) for all the headings replacing
=============================================================
Welcome to Apache OODT <http://oodt.apache.org/>
=============================================================
type headings. As I saw, you had fixed the main heading (Apache OODT heading)'s formatting with **# Heading** style. Now the formatting of README is ok and my previous suggestion is not needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt issue #44: OODT-945 Rework OODT configuration to make use of Zookeeper ...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on the issue:
https://github.com/apache/oodt/pull/44
@IMS94
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt issue #44: OODT-945 Rework OODT configuration to make use of Zookeeper ...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on the issue:
https://github.com/apache/oodt/pull/44
So I checked, and I do have those environment variables set, except for ```RESMGR_HOME``` e.g.,
```
LMC-053601:oodt mattmann$ echo $FILEMGR_HOME
/usr/local/bigtranslate/filemgr
LMC-053601:oodt mattmann$ echo $FILEMGR_URL
http://localhost:9000
LMC-053601:oodt mattmann$
LMC-053601:oodt mattmann$ echo $RESMGR_URL
http://localhost:9002
LMC-053601:oodt mattmann$
```
I was thinking about your code - is it that if an environment variable from local config isn't that set - you are publishing a null value to ZK or there is a getPropertyOrNull call somewhere that is causing this? If so, we should probably have a sensibleDefaults() configuration which are all the sensible defaults in case the environment is Null. Even better would be to use PathUtils in o.a.cas.metadata.util to do EnvVarOrMetadata replacement on all paths you read from config.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt issue #44: OODT-945 Rework OODT configuration to make use of Zookeeper ...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on the issue:
https://github.com/apache/oodt/pull/44
please see my review comments above - great start!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt issue #44: OODT-945 Rework OODT configuration to make use of Zookeeper ...
Posted by IMS94 <gi...@git.apache.org>.
Github user IMS94 commented on the issue:
https://github.com/apache/oodt/pull/44
Hi @chrismattmann, Can let me know do you have any of previously set up `FILEMGR_HOME` or `RESMGR_HOME` environment variables available in your system?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt issue #44: OODT-945 Rework OODT configuration to make use of Zookeeper ...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on the issue:
https://github.com/apache/oodt/pull/44
OK @IMS94 I have invited you to the USC Data Science slack
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt issue #44: OODT-945 Rework OODT configuration to make use of Zookeeper ...
Posted by IMS94 <gi...@git.apache.org>.
Github user IMS94 commented on the issue:
https://github.com/apache/oodt/pull/44
Hi @chrismattmann, Thanks for the update. That is the reason for the test failure. Actually this is a case I should have addressed within my code. I have wrote the fix. Will push it now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt issue #44: OODT-945 Rework OODT configuration to make use of Zookeeper ...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on the issue:
https://github.com/apache/oodt/pull/44
@IMS94 feel free to request an invite to uscdatascience.slack.com I'll make a channel for slack chat if you want real time comm on this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt pull request #44: OODT-945 Rework OODT configuration to make use of Zoo...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/oodt/pull/44
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oodt pull request #44: OODT-945 Rework OODT configuration to make use of Zoo...
Posted by IMS94 <gi...@git.apache.org>.
Github user IMS94 commented on a diff in the pull request:
https://github.com/apache/oodt/pull/44#discussion_r126577868
--- Diff: filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java ---
@@ -52,29 +41,20 @@
import org.apache.oodt.cas.metadata.Metadata;
import org.apache.oodt.cas.metadata.exceptions.MetExtractionException;
import org.apache.oodt.commons.date.DateUtils;
+import org.apache.oodt.config.ConfigurationManager;
+import org.apache.oodt.config.ConfigurationManagerFactory;
+import org.apache.oodt.config.Constants;
import org.apache.xmlrpc.WebServer;
-import com.google.common.collect.Lists;
-
-import java.io.File;
-import java.io.FileInputStream;
-import java.io.FileNotFoundException;
-import java.io.FileOutputStream;
-import java.io.IOException;
+import java.io.*;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.URL;
-import java.util.Arrays;
-import java.util.Hashtable;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import java.util.Vector;
+import java.util.*;
--- End diff --
@chrismattmann Sorry, it was my fault. I was using IntelliJ IDEA's optimize imports command. I will fix this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---