You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ofbiz.apache.org by ta...@apache.org on 2017/08/23 18:56:16 UTC
svn commit: r1805947 [1/2] - in /ofbiz/ofbiz-framework/trunk: ./
framework/base/src/main/java/org/apache/ofbiz/base/container/
framework/base/src/main/java/org/apache/ofbiz/base/util/
framework/entity/src/main/java/org/apache/ofbiz/entity/util/ framewo...
Author: taher
Date: Wed Aug 23 18:56:15 2017
New Revision: 1805947
URL: http://svn.apache.org/viewvc?rev=1805947&view=rev
Log:
Refactor OFBiz containers and remove StartupCommandToArgsAdapter
(OFBIZ-9441)
A major refactoring patch that applies the following changes:
- Fully refactor the EntityDataLoadContainer
- Fully refactor the TestRunContainer
- In both above mentioned containers the logic is broken up into smaller
methods to enhance code readability. It also changes the signature of
the method calls to rely on StartupCommand instead of String[] arrays
- Remove most old documentation and commented out code where applicable
- Delete the StratupCommandToArgsAdapter as it is no longer needed
- Create a new feature in which OFBiz by default fails if any data file
fails to load. This can be overridden by passing a flag called
"continue-on-failure" e.g. ./gradlew "ofbiz --load-data continue-on-failure"
- Add a new property to --test called "loglevel" (old code but working properly)
- Add many new properties to the --load-data command including repair-columns,
try-inserts, maintain-txs, etc ... These properties are old, but somehow
did not make it into the command line arguments. This code is now exposed
through the command line to set a direction in the future of whether to
keep or delete these extra commands
- Update the documentation output of ./gradlew "ofbiz --help" to incorporate
the new mentioned properties
- Refactor README.md to incorporate the changes to the gradle commands for both
"-load-data" and "-test" properties. Also remove the gradle GUI
documentation as it is now deprecated.
- Also refactor README.md in other locations to cleanup and make it more
consistent. This icludes moving long notes into new sections and reducing
the verbosity of the security header. Furthermore, created a new header
called Miscellaneous documentation to house the newly created sections
- lots of small changes to remove EOL white space
Thanks: Jacques Le Roux for reviewing and testing
Removed:
ofbiz/ofbiz-framework/trunk/framework/base/src/main/java/org/apache/ofbiz/base/container/StartupCommandToArgsAdapter.java
Modified:
ofbiz/ofbiz-framework/trunk/README.md
ofbiz/ofbiz-framework/trunk/build.gradle
ofbiz/ofbiz-framework/trunk/framework/base/src/main/java/org/apache/ofbiz/base/util/UtilURL.java
ofbiz/ofbiz-framework/trunk/framework/entity/src/main/java/org/apache/ofbiz/entity/util/EntityDataLoader.java
ofbiz/ofbiz-framework/trunk/framework/entity/src/main/java/org/apache/ofbiz/entity/util/EntitySaxReader.java
ofbiz/ofbiz-framework/trunk/framework/entityext/src/main/java/org/apache/ofbiz/entityext/data/EntityDataLoadContainer.java
ofbiz/ofbiz-framework/trunk/framework/start/src/main/java/org/apache/ofbiz/base/start/StartupCommandUtil.java
ofbiz/ofbiz-framework/trunk/framework/testtools/src/main/java/org/apache/ofbiz/testtools/TestRunContainer.java
Modified: ofbiz/ofbiz-framework/trunk/README.md
URL: http://svn.apache.org/viewvc/ofbiz/ofbiz-framework/trunk/README.md?rev=1805947&r1=1805946&r2=1805947&view=diff
==============================================================================
--- ofbiz/ofbiz-framework/trunk/README.md (original)
+++ ofbiz/ofbiz-framework/trunk/README.md Wed Aug 23 18:56:15 2017
@@ -17,11 +17,11 @@ specific language governing permissions
under the License.
-->
Apache OFBiz®
-============
+=============
Welcome to __Apache OFBiz®__! A powerful top level Apache software project.
OFBiz is an Enterprise Resource Planning (ERP) System written in Java and
-houses a large set of libraries, entities, services and features to run
+houses a large set of libraries, entities, services and features to run
all aspects of your business.
For more details about OFBiz please visit the OFBiz Documentation page:
@@ -30,35 +30,25 @@ For more details about OFBiz please visi
[OFBiz License](http://www.apache.org/licenses/LICENSE-2.0)
->_Note_: since the trunk was split into **ofbiz-framework** and **ofbiz-plugins**,
-the specialpurpose and hot-deploy directories have disappeared.
-New components must be put in a plugins directory which works as was the hot-deploy directory.
-If you need to load the components in a specific order put a component-load.xml file in the plugins directory.</br>
-For OFBiz existing components, check them out using the Gradle tasks below.
-To check out a component, in a created on the fly plugins directory, use the **pullPluginSource** Gradle task.
-To get all components use **pullAllPluginsSource**. **Beware** this deletes a previously existing plugins directory.
+>_Note_: If you want to use Eclipse, read the "Setup eclipse project for OFBiz"
+section to set it up.
+>_Note_: If you want to use an external database like MySQL or PostgreSQL, read
+the "Setup an external database" section to set it up.
+
+> Note : The directory structure and repositories have changed. For more
+information read the "Repository and directory structure" section.
System requirements
-------------------
-The only requirement to run OFBiz is to have the Java Development Kit (JDK)
+The only requirement to run OFBiz is to have the Java Development Kit (JDK)
version 8 installed on your system (not just the JRE, but the full JDK) which
-you can download from the below link.
+you can download from the below link. Make sure of setting the $JAVA_HOME
+environment variable.
[JDK download](http://www.oracle.com/technetwork/java/javase/downloads/index.html)
->_Note_: if you are using Eclipse, make sure of running the appropriate Eclipse
-command `gradlew eclipse` before creating the project in Eclipse.
-This command will prepare OFBiz for Eclipse with the correct classpath and settings
-by creating the.classpath and .project files.
-
->_Note_: if you want to use an external DBMS, instead of the embedded Derby, you will need a JDBC driver.
-The only thing you need to do is add a dependency in build.gradle to the MySQL/PostgreSQL/whatever JDBC driver.
-Search in Jcenter for the database driver suitable for the database installed on your production system.
-For example, under the dependencies section you can add something like this for mysql: `runtime 'mysql:mysql-connector-java:5.1.36'`
-Of course you need to make sure the connector is compatible with _your_ version of the database installed!
-
Quick start
-----------
To quickly install and fire-up OFBiz, please follow the below instructions
@@ -84,8 +74,8 @@ MS Windows:
Unix-like OS:
`./gradlew ofbiz`
->_Note_: then ignore the % progress indicator because this task does not end as long as OFBiz is running.
-
+>_Note_: Ignore the % progress indicator because this task does not end as
+long as OFBiz is running.
### Visit OFBiz through your browser:
@@ -97,22 +87,15 @@ Unix-like OS:
You can log in with the user __admin__ and password __ofbiz__.
->_Note_: the default configuration uses an embedded Java database
-(Apache Derby) and embedded application server components such as
-Apache Tomcat®, Apache Geronimo (transaction manager), etc.
-
Security
--------------------
+--------
-You can trust the OFBiz Project Management Committee members and committers do their best to keep OFBiz secure from external exploits,
-and fix vulnerabilities as soon as they are known. Despite these efforts, if ever you find and want to report a security issue, please report
-at: security @ ofbiz.apache.org, before disclosing them in a public forum.
-
->_Note_: Be sure to read this Wiki page if ever you plan to use RMI, JNDI, JMX or Spring and maybe other Java classes OFBiz does not use
-Out Of The Box (OOTB): [The infamous Java serialization vulnerability](https://cwiki.apache.org/confluence/display/OFBIZ/The+infamous+Java+serialization+vulnerability)
-
-You can find more information about security in OFBiz at [Keeping OFBiz secure](https://cwiki.apache.org/confluence/display/OFBIZ/Keeping+OFBiz+secure)
+If you find a security issue, please report it to: security @ ofbiz.apache.org.
+Once proper mitigations to the security issues are complete the OFBiz team will
+disclose this information to the public mailing list.
+You can find more information about security in OFBiz at
+[Keeping OFBiz secure](https://cwiki.apache.org/confluence/display/OFBIZ/Keeping+OFBiz+secure)
* * * * * * * * * * * *
@@ -146,9 +129,9 @@ There are two types of tasks designed fo
- __ofbizDebug__ : server commands running in remote debug mode
- __ofbizBackground__ ; server commands running in a background forked process
-Tips:
+Tips:
-- OFBiz __server commands__ require __"quoting"__ the
+- OFBiz __server commands__ require __"quoting"__ the
commands. For example: `gradlew "ofbiz --help"`
- Shortcuts to task names can be used by writing the
@@ -176,7 +159,7 @@ Tips:
#### Example mixed tasks (standard and OFBiz server)
-`gradlew cleanAll loadAll "ofbiz --start"`
+`gradlew cleanAll loadAll "ofbiz --start --portoffset 10000"`
* * * * * * * * * * * *
@@ -185,7 +168,7 @@ Quick reference
You can use the below common list of tasks as a quick reference
for controlling the system. This document uses the windows task
-syntax, if you are on a Unix-like system, you need to add the
+syntax, if you are on a Unix-like system, you need to add the
`./` to gradlew i.e. `./gradlew`
* * * * * * * * * * * *
@@ -290,7 +273,7 @@ OFBiz contains the following data reader
- __seed__: OFBiz and External Seed Data - to be maintained along with source and
updated whenever a system deployment is updated
-- __seed-initial__: OFBiz and External Seed Data - to be maintained along with
+- __seed-initial__: OFBiz and External Seed Data - to be maintained along with
source like other seed data, but only loaded initially and not updated
when a system is updated except manually reviewing each line
- __demo__: OFBiz Only Demo Data
@@ -298,6 +281,35 @@ OFBiz contains the following data reader
- __ext-test__: External Test Data (custom)
- __ext-demo__: External Demo Data (custom)
+Available options for the --load-data server command are the following:
+
+- **readers=[name]**: only load data from certain readers separated by comma.
+ e.g. seed,seed-initial,ext
+- **file=[path]**: load a single file from location or several files separated
+ by commas. e.g. /my/file/1,/my/file/2
+- **dir=[path]**: load all data files found in directory
+- **component=[name]**: only load data from a specific component. e.g. base
+- **delegator=[name]**: use the defined delegator. Default is "default". If the
+ value passed is **"all-tenants"** then OFBiz will load the data for all defined
+ tenants in the system.
+- **group=[name]**: override the entity group (org.apache.ofbiz).
+ e.g. com.example.something
+- **timeout=[millis]**: timeout in milliseconds
+- **create-pks**: create primary keys
+- **drop-pks**: drop primary keys
+- **create-constraints**: create indexes and foreign keys after loading
+- **drop-constraints**: drop indexes and foreign keys before loading
+- **create-fks**: create dummy (placeholder) foreign keys
+- **maintain-txs**: maintain timestamps in data file
+- **try-inserts**: use mostly inserts
+- **repair-columns**: repair column sizes
+ (default is true w/ drop-constraints)
+- **continue-on-failure**: By default OFBiz will fail and stop if it is unable
+ to load any of the files it is attempting to load. By passing this property
+ OFBiz will ignore failures and continue loading all files
+
+#### Load specific OFBiz data
+
you can choose which data readers to pass in the following syntax:
`gradlew "ofbiz --load-data readers=<readers-here-comma-separated>"`
@@ -306,15 +318,12 @@ Example:
`gradlew "ofbiz --load-data readers=seed,seed-initial,ext,ext-demo"`
-#### Load default OFBiz data
-
-Loads default data set; meant for initial loading of generic OFBiz data.
-Can be applied for development, testing, demonstration, etc. purposes.
-Be aware that executing this task can result in your data being overwritten in your database of choice.
-Use with caution in production environments.
-The default data set is defined by datasource using the read-data attribute,
-followed by the name of the data set, into the datasource element of the 'entityengine.xml' file.
+#### Load all OFBiz data
+Loads all data sets; meant for initial loading of generic OFBiz data.
+Can be applied for development, testing, demonstration, etc. purposes.
+Be aware that executing this task can result in your data being overwritten
+in your database of choice. Use with caution in production environments.
`gradlew loadAll`
@@ -356,14 +365,14 @@ Load data from an XML file holding entit
#### create a new tenant
Create a new tenant in your environment, create the delegator, load
-initial data with admin-user and password (needs multitenant=Y in
+initial data with admin-user and password (needs multitenant=Y in
general.properties). The following project parameters are passed:
- tenantId: mandatory
- tenantName: optional, default is value of tenantId
- domainName: optional, default is org.apache.ofbiz
- tenantReaders: optional, default value is seed,seed-initial,demo
-- dbPlatform: optional, D(Derby), M(MySQL), O(Oracle), P(PostgreSQL) (default D)
+- dbPlatform: optional, D(Derby), M(MySQL), O(Oracle), P(PostgreSQL) (default D)
- dbIp: optional, ip address of the database
- dbUser: optional, username of the database
- dbPassword: optional, password of the database
@@ -408,42 +417,50 @@ OR
`gradlew 'ofbiz --test'`
+#### Execute integration tests with a different log level
+
+It is possible to start integration tests with a log level different from the
+default one. The log levels allowed are listed below from most verbose to
+least verbose:
+
+- always
+- verbose
+- timing
+- info
+- important
+- warning
+- error
+- fatal
+
+`gradlew "ofbiz --test loglevel=fatal"`
+
#### Execute an integration test case
run a test case, in this example the component is "entity" and the case
name is "entity-tests"
-`gradlew "ofbiz --test component=entity --test case=entity-tests"`
+`gradlew "ofbiz --test component=entity --test suitename=entitytests --test case=entity-query-tests"`
-#### Execute an integration test case in debug mode
+#### Execute an integration test case in debug mode with verbose log
listens on port __5005__
-`gradlew "ofbizDebug --test component=entity --test case=entity-tests"`
+`gradlew "ofbizDebug --test component=entity --test loglevel=verbose"`
#### Execute an integration test suite
-`gradlew "ofbiz --test component=widget --test suitename=widgettests"`
+`gradlew "ofbiz --test component=entity --test suitename=entitytests"`
#### Execute an integration test suite in debug mode
listens on port __5005__
-`gradlew "ofbizDebug --test component=widget --test suitename=widgettests"`
+`gradlew "ofbizDebug --test component=entity --test suitename=entitytests"`
* * * * * * * * * * * *
### Miscellaneous tasks
-#### Launch a graphical user interface of Gradle
-
-This is a very convenient feature of Gradle which
-allows the user to interact with Gradle through a
-swing GUI. You can save frequently used commands
-in a list of favorites for frequent reuse.
-
-`gradlew --gui`
-
#### Run all tests on a clean system
`gradlew cleanAll loadAll testIntegration`
@@ -476,14 +493,14 @@ The below command activates a gradle plu
and reports known vulnerabilities (CVEs) in OFBiz library dependencies.
This command takes a long time to execute because it needs to download
all plugin dependencies and the CVE identification process is also
-time consuming. But it's the only way to check OFBiz does not use
+time consuming. But it's the only way to check OFBiz does not use
vulnerable libraries.
`gradlew -PenableOwasp dependencyCheck`
#### Setup eclipse project for OFBiz
-Setting up OFBiz on eclipse is easy by simply running the below command
+Setting up OFBiz on eclipse is done by simply running the below command
and then importing the project to eclipse. This command will generate
the necessary __.classpath__ and __.project__ files for eclipse and it
will also make the source code for external libraries available in
@@ -579,7 +596,7 @@ file if it exists
Calls __uninstallPlugin__ on an existing plugin and then delete it from the file-system
-`gradlew removePlugin -PpluginId=myplugin`
+`gradlew removePlugin -PpluginId=myplugin`
### Create a new plugin
@@ -614,10 +631,51 @@ a maven repository. Currently, pushing i
* * * * * * * * * * * *
-Useful Tips
------------
+Miscellaneous Documentation
+---------------------------
+
+### Repository and directory structure
+
+OFBiz is split into two repositories:
+
+- **ofbiz-framework**: Contains the core framework and main applications in
+the system like accounting, party, order, etc
+
+- **ofbiz-plugins**: Renamed from "special-purpose" and contains optional
+components that are officially supported by the community
+
+Furthermore, the hot-deploy directory is removed as the plugins directory works
+as a replacement for both "special-purpose" and "hot-deploy".
+
+If you need to load the components in the plugins directory in a specific order
+place a component-load.xml file in the plugins directory listing the order.
+
+To check out a plugin from source control use the **pullPluginSource** Gradle task.
+To check out all plugins from source control use the **pullAllPluginsSource**.
+**Beware** this deletes a previously existing plugins directory.
-### Gradle tab-completion on Unix-like systems:
+### Setup an external database like MySQL, PostgreSQL, etc
+
+To setup an external database instead of the default embedded Apache Derby, you
+will need to follow the following instructions:
+
+1. Find the JDBC driver suitable for your database using one of the following
+ options:
+ * Search for the JDBC driver in [jcenter](https://bintray.com/bintray/jcenter)
+ and place it in build.gradle dependencies e.g.
+ `runtime 'mysql:mysql-connector-java:5.1.36'`
+
+ OR
+
+ * Download the JDBC driver jar and place it in $OFBIZ_HOME/lib or the lib
+ sub-directory of any component
+
+2. Modify the entityengine.xml file located in $OFBIZ_HOME/framework/entity/config
+ to switch the default database to the one you selected. For more details you
+ can read the relevant section in the
+ [technical setup guide](https://cwiki.apache.org/confluence/display/OFBIZ/Apache+OFBiz+Technical+Production+Setup+Guide)
+
+### Setup gradle tab-completion on Unix-like systems:
To get tab completion (auto complete gradle commands by pressing tab)
you can download the script from the below link and place it in the
@@ -627,7 +685,9 @@ appropriate location for your system.
For example, on debian based systems, you can use the following command:
-`sudo curl -L -s https://gist.github.com/Ea87/46401a96df31cd208a87/raw/gradle-tab-completion.bash -o /etc/bash_completion.d/gradle-tab-completion.bash`
+`sudo curl -L -s https://edub.me/gradle-completion-bash -o /etc/bash_completion.d/gradle-tab-completion.bash`
+
+* * * * * * * * * * * *
Crypto notice
-------------
Modified: ofbiz/ofbiz-framework/trunk/build.gradle
URL: http://svn.apache.org/viewvc/ofbiz/ofbiz-framework/trunk/build.gradle?rev=1805947&r1=1805946&r2=1805947&view=diff
==============================================================================
--- ofbiz/ofbiz-framework/trunk/build.gradle (original)
+++ ofbiz/ofbiz-framework/trunk/build.gradle Wed Aug 23 18:56:15 2017
@@ -860,14 +860,14 @@ task cleanAll(group: cleanupGroup, depen
tasks.addRule('Pattern: ofbiz <Commands>: Execute OFBiz startup commands') { String taskName ->
if (taskName ==~ /^ofbiz\s.*/ || taskName == 'ofbiz') {
- def arguments = (taskName - 'ofbiz').toLowerCase().tokenize(' ')
+ def arguments = (taskName - 'ofbiz').tokenize(' ')
createOfbizCommandTask(taskName, arguments, jvmArguments, false)
}
}
tasks.addRule('Pattern: ofbizDebug <Commands>: Execute OFBiz startup commands in remote debug mode') { String taskName ->
if (taskName ==~ /^ofbizDebug\s.*/ || taskName == 'ofbizDebug') {
- def arguments = (taskName - 'ofbizDebug').toLowerCase().tokenize(' ')
+ def arguments = (taskName - 'ofbizDebug').tokenize(' ')
createOfbizCommandTask(taskName, arguments, jvmArguments, true)
}
}
Modified: ofbiz/ofbiz-framework/trunk/framework/base/src/main/java/org/apache/ofbiz/base/util/UtilURL.java
URL: http://svn.apache.org/viewvc/ofbiz/ofbiz-framework/trunk/framework/base/src/main/java/org/apache/ofbiz/base/util/UtilURL.java?rev=1805947&r1=1805946&r2=1805947&view=diff
==============================================================================
--- ofbiz/ofbiz-framework/trunk/framework/base/src/main/java/org/apache/ofbiz/base/util/UtilURL.java (original)
+++ ofbiz/ofbiz-framework/trunk/framework/base/src/main/java/org/apache/ofbiz/base/util/UtilURL.java Wed Aug 23 18:56:15 2017
@@ -130,7 +130,7 @@ public final class UtilURL {
try {
if (file.exists()) url = file.toURI().toURL();
} catch (java.net.MalformedURLException e) {
- e.printStackTrace();
+ Debug.logError(e, "unable to retrieve URL for file: " + filename, module);
url = null;
}
return url;
Modified: ofbiz/ofbiz-framework/trunk/framework/entity/src/main/java/org/apache/ofbiz/entity/util/EntityDataLoader.java
URL: http://svn.apache.org/viewvc/ofbiz/ofbiz-framework/trunk/framework/entity/src/main/java/org/apache/ofbiz/entity/util/EntityDataLoader.java?rev=1805947&r1=1805946&r2=1805947&view=diff
==============================================================================
--- ofbiz/ofbiz-framework/trunk/framework/entity/src/main/java/org/apache/ofbiz/entity/util/EntityDataLoader.java (original)
+++ ofbiz/ofbiz-framework/trunk/framework/entity/src/main/java/org/apache/ofbiz/entity/util/EntityDataLoader.java Wed Aug 23 18:56:15 2017
@@ -228,6 +228,10 @@ public class EntityDataLoader {
}
public static int loadData(URL dataUrl, String helperName, Delegator delegator, List<Object> errorMessages, int txTimeout, boolean dummyFks, boolean maintainTxs, boolean tryInsert) throws GenericEntityException {
+ return loadData(dataUrl, helperName, delegator, errorMessages, txTimeout, false, false, false, true);
+ }
+
+ public static int loadData(URL dataUrl, String helperName, Delegator delegator, List<Object> errorMessages, int txTimeout, boolean dummyFks, boolean maintainTxs, boolean tryInsert, boolean continueOnFail) throws GenericEntityException {
int rowsChanged = 0;
if (dataUrl == null) {
@@ -254,11 +258,16 @@ public class EntityDataLoader {
}
reader.setCreateDummyFks(dummyFks);
reader.setMaintainTxStamps(maintainTxs);
+ reader.setContinueOnFail(continueOnFail);
rowsChanged += reader.parse(dataUrl);
} catch (Exception e) {
String xmlError = "[loadData]: Error loading XML Resource \"" + dataUrl.toExternalForm() + "\"; Error was: " + e.getMessage();
errorMessages.add(xmlError);
- Debug.logError(e, xmlError, module);
+ if (continueOnFail) {
+ Debug.logError(e, xmlError, module);
+ } else {
+ throw new GenericEntityException(xmlError, e);
+ }
}
return rowsChanged;
Modified: ofbiz/ofbiz-framework/trunk/framework/entity/src/main/java/org/apache/ofbiz/entity/util/EntitySaxReader.java
URL: http://svn.apache.org/viewvc/ofbiz/ofbiz-framework/trunk/framework/entity/src/main/java/org/apache/ofbiz/entity/util/EntitySaxReader.java?rev=1805947&r1=1805946&r2=1805947&view=diff
==============================================================================
--- ofbiz/ofbiz-framework/trunk/framework/entity/src/main/java/org/apache/ofbiz/entity/util/EntitySaxReader.java (original)
+++ ofbiz/ofbiz-framework/trunk/framework/entity/src/main/java/org/apache/ofbiz/entity/util/EntitySaxReader.java Wed Aug 23 18:56:15 2017
@@ -95,6 +95,7 @@ public class EntitySaxReader extends Def
private boolean maintainTxStamps = false;
private boolean createDummyFks = false;
private boolean checkDataOnly = false;
+ private boolean continueOnFail = false;
private enum Action {CREATE, CREATE_UPDATE, CREATE_REPLACE, DELETE};
private List<String> actionTags = UtilMisc.toList("create", "create-update", "create-replace", "delete");
private Action currentAction = Action.CREATE_UPDATE;
@@ -148,7 +149,11 @@ public class EntitySaxReader extends Def
public void setCheckDataOnly(boolean checkDataOnly) {
this.checkDataOnly = checkDataOnly;
}
-
+
+ public void setContinueOnFail(boolean continueOnFail) {
+ this.continueOnFail = continueOnFail;
+ }
+
public void setPlaceholderValues(Map<String,Object> placeholderValues) {
this.placeholderValues = placeholderValues;
}
@@ -549,7 +554,11 @@ public class EntitySaxReader extends Def
currentValue.setIsFromEntitySync(true);
}
} catch (Exception e) {
- Debug.logError(e, module);
+ if (continueOnFail) {
+ Debug.logError(e, module);
+ } else {
+ throw new SAXException(e);
+ }
}
if (currentValue != null) {
Modified: ofbiz/ofbiz-framework/trunk/framework/entityext/src/main/java/org/apache/ofbiz/entityext/data/EntityDataLoadContainer.java
URL: http://svn.apache.org/viewvc/ofbiz/ofbiz-framework/trunk/framework/entityext/src/main/java/org/apache/ofbiz/entityext/data/EntityDataLoadContainer.java?rev=1805947&r1=1805946&r2=1805947&view=diff
==============================================================================
--- ofbiz/ofbiz-framework/trunk/framework/entityext/src/main/java/org/apache/ofbiz/entityext/data/EntityDataLoadContainer.java (original)
+++ ofbiz/ofbiz-framework/trunk/framework/entityext/src/main/java/org/apache/ofbiz/entityext/data/EntityDataLoadContainer.java Wed Aug 23 18:56:15 2017
@@ -19,21 +19,25 @@
package org.apache.ofbiz.entityext.data;
import java.io.File;
-import java.net.MalformedURLException;
import java.net.URL;
import java.text.NumberFormat;
+import java.util.ArrayList;
+import java.util.Arrays;
import java.util.Collection;
-import java.util.LinkedList;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.TreeSet;
+import java.util.stream.Collectors;
import org.apache.ofbiz.base.component.ComponentConfig;
import org.apache.ofbiz.base.container.Container;
import org.apache.ofbiz.base.container.ContainerConfig;
+import org.apache.ofbiz.base.container.ContainerConfig.Configuration;
+import org.apache.ofbiz.base.container.ContainerConfig.Configuration.Property;
import org.apache.ofbiz.base.container.ContainerException;
-import org.apache.ofbiz.base.container.StartupCommandToArgsAdapter;
import org.apache.ofbiz.base.start.StartupCommand;
+import org.apache.ofbiz.base.start.StartupCommandUtil;
import org.apache.ofbiz.base.util.Debug;
import org.apache.ofbiz.base.util.StringUtil;
import org.apache.ofbiz.base.util.UtilURL;
@@ -53,268 +57,237 @@ import org.apache.ofbiz.entity.util.Enti
import org.apache.ofbiz.entity.util.EntityUtil;
import org.apache.ofbiz.service.ServiceDispatcher;
-
/**
- * Some utility routines for loading seed data.
+ * Container responsible for loading all types of data e.g. seed, seed-initial, etc.
+ * This container is the one used when the user issues --load-data commands.
*/
public class EntityDataLoadContainer implements Container {
public static final String module = EntityDataLoadContainer.class.getName();
-
- protected String overrideDelegator = null;
- protected String overrideGroup = null;
- protected String configFile = null;
- protected String readers = null;
- protected String directory = null;
- protected List<String> files = new LinkedList<String>();
- protected String component = null;
- protected boolean useDummyFks = false;
- protected boolean maintainTxs = false;
- protected boolean tryInserts = false;
- protected boolean repairColumns = false;
- protected boolean dropPks = false;
- protected boolean createPks = false;
- protected boolean dropConstraints = false;
- protected boolean createConstraints = false;
- protected int txTimeout = -1;
-
private String name;
- public EntityDataLoadContainer() {
- super();
- }
+ // possible command line properties passed by user
+ private static final String DATA_READERS = "readers";
+ private static final String DATA_FILE = "file";
+ private static final String DATA_DIR = "dir";
+ private static final String DATA_COMPONENT = "component";
+ private static final String DELEGATOR_NAME = "delegator";
+ private static final String DATA_GROUP = "group";
+ private static final String TIMEOUT = "timeout";
+ private static final String CREATE_P_KEYS = "create-pks";
+ private static final String DROP_P_KEYS = "drop-pks";
+ private static final String CREATE_CONSTRAINTS = "create-constraints";
+ private static final String DROP_CONSTRAINTS = "drop-constraints";
+ private static final String CREATE_F_KEYS = "create-fks";
+ private static final String MAINTAIN_TXS = "maintain-txs";
+ private static final String TRY_INSERTS = "try-inserts";
+ private static final String REPAIR_COLUMNS = "repair-columns";
+ private static final String CONTINUE_ON_FAIL = "continue-on-failure";
@Override
public void init(List<StartupCommand> ofbizCommands, String name, String configFile) throws ContainerException {
- // TODO: remove this hack and provide clean implementation
- String[] args = StartupCommandToArgsAdapter.adaptStartupCommandsToLoaderArgs(ofbizCommands);
-
this.name = name;
- this.configFile = configFile;
- // disable job scheduler, JMS listener and startup services
- // FIXME: This is not thread-safe.
+
+ // get the data-load properties passed by the user in the command line
+ Map<String, String> loadDataProps = ofbizCommands.stream()
+ .filter(command -> command.getName().equals(StartupCommandUtil.StartupOption.LOAD_DATA.getName()))
+ .map(command -> command.getProperties())
+ .findFirst().get();
+
+ /* disable job scheduler, JMS listener and startup services
+ * FIXME: This is not thread-safe. */
ServiceDispatcher.enableJM(false);
ServiceDispatcher.enableJMS(false);
ServiceDispatcher.enableSvcs(false);
- /*
- load-data arguments:
- readers (none, all, seed, demo, ext, etc - configured in entityengine.xml and associated via ofbiz-component.xml)
- timeout (transaction timeout default 7200)
- delegator (overrides the delegator name configured for the container)
- group (overrides the entity group name configured for the container)
- dir (imports all XML files in a directory)
- file (import a specific XML file)
-
- Example:
- $ java -jar build/libs/ofbiz.jar --load-data -readers=seed,demo,ext -timeout=7200 -delegator=default -group=org.apache.ofbiz
- $ java -jar build/libs/ofbiz.jar --load-data -file=/tmp/dataload.xml
- $ java -jar build/libs/ofbiz.jar --load-data -readers=seed,demo,ext -timeout=7200 -delegator=default -group=org.apache.ofbiz
- $ java -jar build/libs/ofbiz.jar --load-data -file=/tmp/dataload.xml
- */
- if (args != null) {
- for (String argument: args) {
- // arguments can prefix w/ a '-'. Just strip them off
- if (argument.startsWith("-")) {
- int subIdx = 1;
- if (argument.startsWith("--")) {
- subIdx = 2;
- }
- argument = argument.substring(subIdx);
- }
-
- // parse the arguments
- String argumentName;
- String argumentVal;
- if (argument.indexOf("=") != -1) {
- argumentName = argument.substring(0, argument.indexOf("="));
- argumentVal = argument.substring(argument.indexOf("=") + 1);
- } else {
- argumentName = argument;
- argumentVal = "";
- }
- Debug.logInfo("Install Argument - " + argumentName + " = " + argumentVal, module);
-
- if ("readers".equalsIgnoreCase(argumentName)) {
- this.readers = argumentVal;
- } else if ("timeout".equalsIgnoreCase(argumentName)) {
- try {
- this.txTimeout = Integer.parseInt(argumentVal);
- } catch (Exception e) {
- this.txTimeout = -1;
- }
- } else if ("component".equalsIgnoreCase(argumentName)) {
- this.component = argumentVal;
- } else if ("delegator".equalsIgnoreCase(argumentName)) {
- this.overrideDelegator = argumentVal;
- } else if ("group".equalsIgnoreCase(argumentName)) {
- this.overrideGroup = argumentVal;
- } else if ("file".equalsIgnoreCase(argumentName)) {
- this.files.addAll(StringUtil.split(argumentVal, ","));
- } else if ("dir".equalsIgnoreCase(argumentName)) {
- this.directory = argumentVal;
- } else if ("createfks".equalsIgnoreCase(argumentName)) {
- this.useDummyFks = "true".equalsIgnoreCase(argumentVal);
- } else if ("maintainTxs".equalsIgnoreCase(argumentName)) {
- this.maintainTxs = "true".equalsIgnoreCase(argumentVal);
- } else if ("inserts".equalsIgnoreCase(argumentName)) {
- this.tryInserts = "true".equalsIgnoreCase(argumentVal);
- } else if ("repair-columns".equalsIgnoreCase(argumentName)) {
- if (UtilValidate.isEmpty(argumentVal) || "true".equalsIgnoreCase(argumentVal)) {
- repairColumns = true;
- }
- } else if ("drop-pks".equalsIgnoreCase(argumentName)) {
- if (UtilValidate.isEmpty(argumentVal) || "true".equalsIgnoreCase(argumentVal)) {
- dropPks = true;
- }
- } else if ("create-pks".equalsIgnoreCase(argumentName)) {
- if (UtilValidate.isEmpty(argumentVal) || "true".equalsIgnoreCase(argumentVal)) {
- createPks = true;
- }
- } else if ("drop-constraints".equalsIgnoreCase(argumentName)) {
- if (UtilValidate.isEmpty(argumentVal) || "true".equalsIgnoreCase(argumentVal)) {
- dropConstraints = true;
- }
- } else if ("create-constraints".equalsIgnoreCase(argumentName)) {
- if (UtilValidate.isEmpty(argumentVal) || "true".equalsIgnoreCase(argumentVal)) {
- createConstraints = true;
- }
- } else if ("help".equalsIgnoreCase(argumentName)) {
- //"java -jar build/libs/ofbiz.jar --load-data [options]\n" +
- String helpStr = "\n--------------------------------------\n" +
- "java -jar build/libs/ofbiz.jar --load-data [options]\n" +
- "-component=[name] .... only load from a specific component\n" +
- "-delegator=[name] .... use the defined delegator (default-no-eca)\n" +
- "-group=[name] ........ override the entity group (org.apache.ofbiz)\n" +
- "-file=[path] ......... load a single file from location, several files separated by commas\n" +
- "-createfks ........... create dummy (placeholder) FKs\n" +
- "-maintainTxs ......... maintain timestamps in data file\n" +
- "-inserts ............. use mostly inserts option\n" +
- "-repair-columns ........... repair column sizes\n" +
- "-drop-pks ............ drop primary keys\n" +
- "-create-pks .......... create primary keys\n" +
- "-drop-constraints..... drop indexes and foreign keys before loading\n" +
- "-create-constraints... create indexes and foreign keys after loading (default is true w/ drop-constraints)\n" +
- "-help ................ display this information\n";
- throw new ContainerException(helpStr);
- }
-
- // special case
- if (this.readers == null && (!this.files.isEmpty() || this.directory != null)) {
- this.readers = "none";
- }
+ Configuration configuration = ContainerConfig.getConfiguration(name, configFile);
+ Property delegatorNameProp = configuration.getProperty("delegator-name");
+ String overrideDelegator = loadDataProps.get(DELEGATOR_NAME);
+
+ if ("all-tenants".equals(overrideDelegator)) {
+ // load data for all tenants
+ for (GenericValue tenant : getTenantList(delegatorNameProp)) {
+ String tenantDelegator = delegatorNameProp.value + "#" + tenant.getString("tenantId");
+ loadDataForDelegator(loadDataProps, configuration, delegatorNameProp, tenantDelegator);
}
+ } else {
+ // load data for a single delegator
+ loadDataForDelegator(loadDataProps, configuration, delegatorNameProp, overrideDelegator);
}
}
- /**
- * @see org.apache.ofbiz.base.container.Container#start()
- */
@Override
public boolean start() throws ContainerException {
- if("all-tenants".equals(this.overrideDelegator)) {
- if (!EntityUtil.isMultiTenantEnabled()) {
- Debug.logWarning("Multitenant is disabled. Please enable multitenant. (e.g. general.properties --> multitenant=Y)", module);
- return true;
- }
- ContainerConfig.Configuration cfg = ContainerConfig.getConfiguration(name, configFile);
- ContainerConfig.Configuration.Property delegatorNameProp = cfg.getProperty("delegator-name");
- String delegatorName = null;
- if (delegatorNameProp == null || UtilValidate.isEmpty(delegatorNameProp.value)) {
- throw new ContainerException("Invalid delegator-name defined in container configuration");
- } else {
- delegatorName = delegatorNameProp.value;
- }
- Delegator delegator = DelegatorFactory.getDelegator(delegatorName);
- if (delegator == null) {
- throw new ContainerException("Invalid delegator name!");
- }
- List<EntityExpr> expr = new LinkedList<EntityExpr>();
- expr.add(EntityCondition.makeCondition("disabled", EntityOperator.EQUALS, "N"));
- expr.add(EntityCondition.makeCondition("disabled", EntityOperator.EQUALS, null));
- List<GenericValue> tenantList;
- try {
- tenantList = EntityQuery.use(delegator).from("Tenant").where(expr, EntityOperator.OR).queryList();
- } catch (GenericEntityException e) {
- throw new ContainerException(e.getMessage());
- }
- for (GenericValue tenant : tenantList) {
- this.overrideDelegator = delegator.getDelegatorName() + "#" + tenant.getString("tenantId");
- loadContainer();
- }
- } else {
- loadContainer();
- }
return true;
}
- private void loadContainer() throws ContainerException{
- ContainerConfig.Configuration cfg = ContainerConfig.getConfiguration(name, configFile);
- ContainerConfig.Configuration.Property delegatorNameProp = cfg.getProperty("delegator-name");
- ContainerConfig.Configuration.Property entityGroupNameProp = cfg.getProperty("entity-group-name");
- String delegatorName = null;
- String entityGroupName = null;
+ @Override
+ public void stop() throws ContainerException {
+ }
- if (delegatorNameProp == null || UtilValidate.isEmpty(delegatorNameProp.value)) {
- throw new ContainerException("Invalid delegator-name defined in container configuration");
- } else {
- delegatorName = delegatorNameProp.value;
+ @Override
+ public String getName() {
+ return name;
+ }
+
+ private List<GenericValue> getTenantList(Property delegatorNameProp) throws ContainerException {
+ if (!EntityUtil.isMultiTenantEnabled()) {
+ throw new ContainerException("Multitenant is disabled, must be enabled in general.properties -> multitenant=Y");
+ }
+
+ Delegator delegator = getDelegator(delegatorNameProp, null);
+ List<EntityExpr> expr = new ArrayList<EntityExpr>();
+ expr.add(EntityCondition.makeCondition("disabled", EntityOperator.EQUALS, "N"));
+ expr.add(EntityCondition.makeCondition("disabled", EntityOperator.EQUALS, null));
+
+ try {
+ return EntityQuery.use(delegator).from("Tenant").where(expr, EntityOperator.OR).queryList();
+ } catch (GenericEntityException e) {
+ throw new ContainerException(e);
+ }
+ }
+
+ private void loadDataForDelegator(Map<String, String> loadDataProps, Configuration configuration,
+ Property delegatorNameProp, String overrideDelegator) throws ContainerException{
+
+ // prepare command line properties passed by user
+ boolean createPks = isPropertySet(loadDataProps, CREATE_P_KEYS);
+ boolean dropPks = isPropertySet(loadDataProps, DROP_P_KEYS);
+ boolean createConstraints = isPropertySet(loadDataProps, CREATE_CONSTRAINTS);
+ boolean dropConstraints = isPropertySet(loadDataProps, DROP_CONSTRAINTS);
+ boolean repairColumns = isPropertySet(loadDataProps, REPAIR_COLUMNS);
+ String entityGroup = getEntityGroupNameFromConfig(configuration, loadDataProps.get(DATA_GROUP));
+
+ // prepare objects needed for the data loading logic
+ Delegator delegator = getDelegator(delegatorNameProp, overrideDelegator);
+ Delegator baseDelegator = getBaseDelegator(delegator);
+ GenericHelperInfo helperInfo = getHelperInfo(delegator, entityGroup);
+ DatabaseUtil dbUtil = new DatabaseUtil(helperInfo);
+ Map<String, ModelEntity> modelEntities = getModelEntities(delegator, entityGroup);
+ TreeSet<String> modelEntityNames = new TreeSet<String>(modelEntities.keySet());
+ Collection<ComponentConfig> allComponents = ComponentConfig.getAllComponents();
+
+ // data loading logic starts here
+ createOrUpdateComponentEntities(baseDelegator, allComponents);
+
+ if (dropConstraints) {
+ dropDbConstraints(dbUtil, modelEntities, modelEntityNames);
+ }
+ if (dropPks) {
+ dropPrimaryKeys(dbUtil, modelEntities, modelEntityNames);
+ }
+ if (repairColumns) {
+ repairDbColumns(dbUtil, modelEntities);
}
- if (entityGroupNameProp == null || UtilValidate.isEmpty(entityGroupNameProp.value)) {
- throw new ContainerException("Invalid entity-group-name defined in container configuration");
+ loadData(delegator, baseDelegator, allComponents, helperInfo, loadDataProps);
+
+ if (createPks) {
+ createPrimaryKeys(dbUtil, modelEntities, modelEntityNames);
+ }
+ if (createConstraints) {
+ createDbConstraints(dbUtil, modelEntities, modelEntityNames);
+ }
+ }
+
+ /*
+ * If the user passed a flag, then make sure to set it to true if it has no
+ * value or its value is the string "true".
+ *
+ * key=true -> true
+ * key -> true
+ * key=false -> false
+ * (no-key) -> false
+ */
+ private boolean isPropertySet(Map<String, String> props, String key) {
+ String value = props.get(key);
+ if (props.containsKey(key) && (value == null || "true".equalsIgnoreCase(value))) {
+ return true;
} else {
- entityGroupName = entityGroupNameProp.value;
+ return false;
}
+ }
- // parse the pass in list of readers to use
- List<String> readerNames = null;
- if (this.readers != null && !"none".equalsIgnoreCase(this.readers)) {
- if (this.readers.indexOf(",") == -1) {
- readerNames = new LinkedList<String>();
- readerNames.add(this.readers);
+ /*
+ * Gets the default entity-group-name defined in the container definition
+ * unless overridden by the user
+ */
+ private String getEntityGroupNameFromConfig(Configuration cfg, String overrideGroup) throws ContainerException {
+ if (overrideGroup != null) {
+ return overrideGroup;
+ } else {
+ ContainerConfig.Configuration.Property entityGroupNameProp = cfg.getProperty("entity-group-name");
+ if (entityGroupNameProp == null || UtilValidate.isEmpty(entityGroupNameProp.value)) {
+ throw new ContainerException("Invalid entity-group-name defined in container configuration");
} else {
- readerNames = StringUtil.split(this.readers, ",");
+ return entityGroupNameProp.value;
}
}
+ }
+
+ /*
+ * Gets the default delegator defined in the container definition unless
+ * overridden by the user. This method will create all the tables, keys and
+ * indices if missing and hence might take a long time.
+ */
+ private Delegator getDelegator(Property delegatorNameProp, String overrideDelegator) throws ContainerException {
+ if (overrideDelegator != null) {
+ return DelegatorFactory.getDelegator(overrideDelegator);
+ } else {
+ return getDelegatorFromProp(delegatorNameProp);
+ }
+ }
- String delegatorNameToUse = overrideDelegator != null ? overrideDelegator : delegatorName;
- String groupNameToUse = overrideGroup != null ? overrideGroup : entityGroupName;
- Delegator delegator = DelegatorFactory.getDelegator(delegatorNameToUse);
- Delegator baseDelegator = null;
- if (delegator == null) {
- throw new ContainerException("Invalid delegator name!");
+ private Delegator getDelegatorFromProp(Property delegatorNameProp) throws ContainerException {
+ if (delegatorNameProp != null && UtilValidate.isNotEmpty(delegatorNameProp.value)) {
+ Delegator delegator = DelegatorFactory.getDelegator(delegatorNameProp.value);
+ if (delegator != null) {
+ return delegator;
+ } else {
+ throw new ContainerException("Invalid delegator name: " + delegatorNameProp.value);
+ }
+ } else {
+ throw new ContainerException("Invalid delegator name defined in container configuration");
}
+ }
+ private Delegator getBaseDelegator(Delegator delegator) {
if (delegator.getDelegatorTenantId() != null) {
- baseDelegator = DelegatorFactory.getDelegator(delegator.getDelegatorBaseName());
+ return DelegatorFactory.getDelegator(delegator.getDelegatorBaseName());
} else {
- baseDelegator = delegator;
+ return delegator;
}
+ }
- GenericHelperInfo helperInfo = delegator.getGroupHelperInfo(groupNameToUse);
+ private GenericHelperInfo getHelperInfo(Delegator delegator, String entityGroup) throws ContainerException {
+ GenericHelperInfo helperInfo = delegator.getGroupHelperInfo(entityGroup);
if (helperInfo == null) {
- throw new ContainerException("Unable to locate the datasource helper for the group [" + groupNameToUse + "]");
+ throw new ContainerException("Unable to locate the datasource helper for the group: " + entityGroup);
}
+ return helperInfo;
+ }
- // get the database util object
- DatabaseUtil dbUtil = new DatabaseUtil(helperInfo);
- Map<String, ModelEntity> modelEntities;
+ private Map<String, ModelEntity> getModelEntities(Delegator delegator,
+ String entityGroup) throws ContainerException {
try {
- modelEntities = delegator.getModelEntityMapByGroup(groupNameToUse);
+ return delegator.getModelEntityMapByGroup(entityGroup);
} catch (GenericEntityException e) {
- throw new ContainerException(e.getMessage(), e);
+ throw new ContainerException(e);
}
- TreeSet<String> modelEntityNames = new TreeSet<String>(modelEntities.keySet());
- // store all components
- Collection<ComponentConfig> allComponents = ComponentConfig.getAllComponents();
+ }
+
+ private void createOrUpdateComponentEntities(Delegator baseDelegator,
+ Collection<ComponentConfig> allComponents) {
+
for (ComponentConfig config : allComponents) {
- //Debug.logInfo("- Stored component : " + config.getComponentName(), module);
GenericValue componentEntry = baseDelegator.makeValue("Component");
componentEntry.set("componentName", config.getComponentName());
componentEntry.set("rootLocation", config.getRootLocation());
try {
- GenericValue componentCheck = EntityQuery.use(baseDelegator).from("Component").where("componentName", config.getComponentName()).queryOne();
+ GenericValue componentCheck = EntityQuery.use(baseDelegator)
+ .from("Component")
+ .where("componentName", config.getComponentName())
+ .queryOne();
if (UtilValidate.isEmpty(componentCheck)) {
componentEntry.create();
} else {
@@ -324,264 +297,308 @@ public class EntityDataLoadContainer imp
Debug.logError(e.getMessage(), module);
}
}
- // load specify components
- List<String> loadComponents = new LinkedList<String>();
- if (UtilValidate.isNotEmpty(delegator.getDelegatorTenantId()) && EntityUtil.isMultiTenantEnabled()) {
- try {
- if (UtilValidate.isEmpty(this.component)) {
- for (ComponentConfig config : allComponents) {
- loadComponents.add(config.getComponentName());
- }
- List<GenericValue> tenantComponents = EntityQuery.use(baseDelegator).from("TenantComponent").where("tenantId", delegator.getDelegatorTenantId()).orderBy("sequenceNum").queryList();
- for (GenericValue tenantComponent : tenantComponents) {
- loadComponents.add(tenantComponent.getString("componentName"));
- }
- Debug.logInfo("Loaded components by tenantId : " + delegator.getDelegatorTenantId() + ", " + tenantComponents.size() + " components", module);
- } else {
- List<GenericValue> tenantComponents = EntityQuery.use(baseDelegator).from("TenantComponent").where("tenantId", delegator.getDelegatorTenantId(), "componentName", this.component).orderBy("sequenceNum").queryList();
- for (GenericValue tenantComponent : tenantComponents) {
- loadComponents.add(tenantComponent.getString("componentName"));
- }
- Debug.logInfo("Loaded tenantId : " + delegator.getDelegatorTenantId() + " and component : " + this.component, module);
- }
- Debug.logInfo("Loaded : " + loadComponents.size() + " components", module);
- } catch (GenericEntityException e) {
- Debug.logError(e.getMessage(), module);
+ }
+
+ private void dropDbConstraints(DatabaseUtil dbUtil, Map<String, ModelEntity> modelEntities,
+ TreeSet<String> modelEntityNames) {
+
+ List<String> messages = new ArrayList<String>();
+
+ Debug.logImportant("Dropping foreign key indices...", module);
+ for (String entityName : modelEntityNames) {
+ ModelEntity modelEntity = modelEntities.get(entityName);
+ if (modelEntity != null) {
+ dbUtil.deleteForeignKeyIndices(modelEntity, messages);
}
}
- // check for drop index/fks
- if (dropConstraints) {
- List<String> messages = new LinkedList<String>();
- Debug.logImportant("Dropping foreign key indcies...", module);
- for (String entityName : modelEntityNames) {
- ModelEntity modelEntity = modelEntities.get(entityName);
- if (modelEntity != null) {
- dbUtil.deleteForeignKeyIndices(modelEntity, messages);
- }
+ Debug.logImportant("Dropping declared indices...", module);
+ for (String entityName : modelEntityNames) {
+ ModelEntity modelEntity = modelEntities.get(entityName);
+ if (modelEntity != null) {
+ dbUtil.deleteDeclaredIndices(modelEntity, messages);
}
+ }
- Debug.logImportant("Dropping declared indices...", module);
- for (String entityName : modelEntityNames) {
- ModelEntity modelEntity = modelEntities.get(entityName);
- if (modelEntity != null) {
- dbUtil.deleteDeclaredIndices(modelEntity, messages);
- }
+ Debug.logImportant("Dropping foreign keys...", module);
+ for (String entityName : modelEntityNames) {
+ ModelEntity modelEntity = modelEntities.get(entityName);
+ if (modelEntity != null) {
+ dbUtil.deleteForeignKeys(modelEntity, modelEntities, messages);
}
+ }
- Debug.logImportant("Dropping foreign keys...", module);
- for (String entityName : modelEntityNames) {
- ModelEntity modelEntity = modelEntities.get(entityName);
- if (modelEntity != null) {
- dbUtil.deleteForeignKeys(modelEntity, modelEntities, messages);
- }
- }
+ logMessageList(messages);
+ }
- if (messages.size() > 0) {
- if (Debug.infoOn()) {
- for (String message : messages) {
- Debug.logInfo(message, module);
- }
- }
+ private void createDbConstraints(DatabaseUtil dbUtil, Map<String, ModelEntity> modelEntities,
+ TreeSet<String> modelEntityNames) {
+
+ List<String> messages = new ArrayList<String>();
+
+ Debug.logImportant("Creating foreign keys...", module);
+ for (String entityName : modelEntityNames) {
+ ModelEntity modelEntity = modelEntities.get(entityName);
+ if (modelEntity != null) {
+ dbUtil.createForeignKeys(modelEntity, modelEntities, messages);
}
}
- // drop pks
- if (dropPks) {
- List<String> messages = new LinkedList<String>();
- Debug.logImportant("Dropping primary keys...", module);
- for (String entityName : modelEntityNames) {
- ModelEntity modelEntity = modelEntities.get(entityName);
- if (modelEntity != null) {
- dbUtil.deletePrimaryKey(modelEntity, messages);
- }
+ Debug.logImportant("Creating foreign key indcies...", module);
+ for (String entityName : modelEntityNames) {
+ ModelEntity modelEntity = modelEntities.get(entityName);
+ if (modelEntity != null) {
+ dbUtil.createForeignKeyIndices(modelEntity, messages);
}
+ }
- if (messages.size() > 0) {
- if (Debug.infoOn()) {
- for (String message : messages) {
- Debug.logInfo(message, module);
- }
- }
+ Debug.logImportant("Creating declared indices...", module);
+ for (String entityName : modelEntityNames) {
+ ModelEntity modelEntity = modelEntities.get(entityName);
+ if (modelEntity != null) {
+ dbUtil.createDeclaredIndices(modelEntity, messages);
}
}
- // repair columns
- if (repairColumns) {
- List<String> fieldsToRepair = new LinkedList<String>();
- List<String> messages = new LinkedList<String>();
- dbUtil.checkDb(modelEntities, fieldsToRepair, messages, false, false, false, false);
- if (fieldsToRepair.size() > 0) {
- messages = new LinkedList<String>();
- dbUtil.repairColumnSizeChanges(modelEntities, fieldsToRepair, messages);
- if (messages.size() > 0) {
- if (Debug.infoOn()) {
- for (String message : messages) {
- Debug.logInfo(message, module);
- }
- }
- }
+ logMessageList(messages);
+ }
+
+ private void dropPrimaryKeys(DatabaseUtil dbUtil, Map<String, ModelEntity> modelEntities,
+ TreeSet<String> modelEntityNames) {
+
+ List<String> messages = new ArrayList<String>();
+
+ Debug.logImportant("Dropping primary keys...", module);
+ for (String entityName : modelEntityNames) {
+ ModelEntity modelEntity = modelEntities.get(entityName);
+ if (modelEntity != null) {
+ dbUtil.deletePrimaryKey(modelEntity, messages);
}
}
- // get the reader name URLs first
- List<URL> urlList = null;
- if (UtilValidate.isNotEmpty(loadComponents)) {
- if (UtilValidate.isNotEmpty(readerNames)) {
- urlList = EntityDataLoader.getUrlByComponentList(helperInfo.getHelperBaseName(), loadComponents, readerNames);
- } else if (!"none".equalsIgnoreCase(this.readers)) {
- urlList = EntityDataLoader.getUrlByComponentList(helperInfo.getHelperBaseName(), loadComponents);
- }
- } else {
- if (UtilValidate.isNotEmpty(readerNames)) {
- urlList = EntityDataLoader.getUrlList(helperInfo.getHelperBaseName(), component, readerNames);
- } else if (!"none".equalsIgnoreCase(this.readers)) {
- urlList = EntityDataLoader.getUrlList(helperInfo.getHelperBaseName(), component);
- }
- }
- // need a list if it is empty
- if (urlList == null) {
- urlList = new LinkedList<URL>();
- }
-
- // add in the defined extra files
- for (String fileName: this.files) {
- URL fileUrl = UtilURL.fromResource(fileName);
- if (fileUrl != null) {
- urlList.add(fileUrl);
+ logMessageList(messages);
+ }
+
+ private void createPrimaryKeys(DatabaseUtil dbUtil, Map<String, ModelEntity> modelEntities,
+ TreeSet<String> modelEntityNames) {
+
+ List<String> messages = new ArrayList<String>();
+
+ Debug.logImportant("Creating primary keys...", module);
+ for (String entityName : modelEntityNames) {
+ ModelEntity modelEntity = modelEntities.get(entityName);
+ if (modelEntity != null) {
+ dbUtil.createPrimaryKey(modelEntity, messages);
}
}
- // next check for a directory of files
- if (this.directory != null) {
- File dir = new File(this.directory);
- if (dir.exists() && dir.isDirectory() && dir.canRead()) {
- File[] fileArray = dir.listFiles();
- if (fileArray != null && fileArray.length > 0) {
- for (File file: fileArray) {
- if (file.getName().toLowerCase().endsWith(".xml")) {
- try {
- urlList.add(file.toURI().toURL());
- } catch (MalformedURLException e) {
- Debug.logError(e, "Unable to load file (" + file.getName() + "); not a valid URL.", module);
- }
- }
- }
- }
- }
+ logMessageList(messages);
+ }
+
+ private void repairDbColumns(DatabaseUtil dbUtil, Map<String, ModelEntity> modelEntities) {
+ List<String> fieldsToRepair = new ArrayList<String>();
+ List<String> messages = new ArrayList<String>();
+ dbUtil.checkDb(modelEntities, fieldsToRepair, messages, false, false, false, false);
+ if (UtilValidate.isNotEmpty(fieldsToRepair)) {
+ dbUtil.repairColumnSizeChanges(modelEntities, fieldsToRepair, messages);
}
- // process the list of files
- NumberFormat changedFormat = NumberFormat.getIntegerInstance();
- changedFormat.setMinimumIntegerDigits(5);
- changedFormat.setGroupingUsed(false);
+ logMessageList(messages);
+ }
- List<Object> errorMessages = new LinkedList<Object>();
- List<String> infoMessages = new LinkedList<String>();
+ private void logMessageList(List<String> messages) {
+ if (Debug.infoOn()) {
+ messages.forEach(message -> Debug.logInfo(message, module));
+ }
+ }
+
+ private void loadData(Delegator delegator, Delegator baseDelegator,
+ Collection<ComponentConfig> allComponents,
+ GenericHelperInfo helperInfo,
+ Map<String, String> loadDataProps) throws ContainerException {
+
+ // prepare command line properties passed by user
+ int txTimeout = getTransactionTimeout(loadDataProps.get(TIMEOUT));
+ boolean useDummyFks = isPropertySet(loadDataProps, CREATE_F_KEYS);
+ boolean maintainTxs = isPropertySet(loadDataProps, MAINTAIN_TXS);
+ boolean tryInserts = isPropertySet(loadDataProps, TRY_INSERTS);
+ boolean continueOnFail = isPropertySet(loadDataProps, CONTINUE_ON_FAIL);
+
+ List<URL> urlList = prepareDataUrls(delegator, baseDelegator, allComponents, helperInfo, loadDataProps);
+ List<String> infoMessages = new ArrayList<String>();
+ List<Object> errorMessages = new ArrayList<Object>();
int totalRowsChanged = 0;
- if (UtilValidate.isNotEmpty(urlList)) {
- Debug.logImportant("=-=-=-=-=-=-= Doing a data load using delegator '" + delegator.getDelegatorName() + "' with the following files:", module);
- for (URL dataUrl: urlList) {
- Debug.logImportant(dataUrl.toExternalForm(), module);
- }
- Debug.logImportant("=-=-=-=-=-=-= Starting the data load...", module);
+ logDataLoadingPlan(urlList, delegator.getDelegatorName());
- for (URL dataUrl: urlList) {
- try {
- int rowsChanged = EntityDataLoader.loadData(dataUrl, helperInfo.getHelperBaseName(), delegator, errorMessages, txTimeout, useDummyFks, maintainTxs, tryInserts);
- totalRowsChanged += rowsChanged;
- infoMessages.add(changedFormat.format(rowsChanged) + " of " + changedFormat.format(totalRowsChanged) + " from " + dataUrl.toExternalForm());
- } catch (GenericEntityException e) {
+ for (URL dataUrl: urlList) {
+ try {
+ int rowsChanged = EntityDataLoader.loadData(dataUrl, helperInfo.getHelperBaseName(),
+ delegator, errorMessages, txTimeout, useDummyFks, maintainTxs, tryInserts, continueOnFail);
+ totalRowsChanged += rowsChanged;
+ infoMessages.add(createDataLoadMessage(dataUrl, rowsChanged, totalRowsChanged));
+ } catch (GenericEntityException e) {
+ if (continueOnFail) {
Debug.logError(e, "Error loading data file: " + dataUrl.toExternalForm(), module);
+ } else {
+ throw new ContainerException(e);
}
}
- } else {
- Debug.logImportant("=-=-=-=-=-=-= No data load files found.", module);
}
- if (infoMessages.size() > 0) {
- Debug.logImportant("=-=-=-=-=-=-= Here is a summary of the data load:", module);
- for (String message: infoMessages) {
- Debug.logImportant(message, module);
- }
+ logDataLoadingResults(infoMessages, errorMessages, totalRowsChanged);
+ }
+
+ private int getTransactionTimeout(String timeout) {
+ try {
+ return Integer.parseInt(timeout);
+ } catch (Exception e) {
+ return -1;
}
+ }
- if (errorMessages.size() > 0) {
- Debug.logImportant("The following errors occurred in the data load:", module);
- for (Object message: errorMessages) {
- Debug.logImportant(message.toString(), module);
+ private List<URL> prepareDataUrls(Delegator delegator, Delegator baseDelegator,
+ Collection<ComponentConfig> allComponents, GenericHelperInfo helperInfo,
+ Map<String, String> loadDataProps) throws ContainerException {
+
+ List<URL> urlList = new ArrayList<URL>();
+
+ // prepare command line properties passed by user
+ List<String> files = getLoadFiles(loadDataProps.get(DATA_FILE));
+ String directory = loadDataProps.get(DATA_DIR);
+ String component = loadDataProps.get(DATA_COMPONENT);
+ String readers = loadDataProps.get(DATA_READERS);
+
+ boolean readersEnabled = isDataReadersEnabled(files, directory, readers);
+ String helperBaseName = helperInfo.getHelperBaseName();
+ List<String> loadComponents = prepareTenantLoadComponents(delegator, baseDelegator, allComponents, component);
+ List<String> readerNames = StringUtil.split(readers, ",");
+
+ // retrieve URLs from readers
+ if (UtilValidate.isNotEmpty(loadComponents)) {
+ if (UtilValidate.isNotEmpty(readerNames)) {
+ urlList.addAll(EntityDataLoader.getUrlByComponentList(helperBaseName, loadComponents, readerNames));
+ } else if (readersEnabled) {
+ urlList.addAll(EntityDataLoader.getUrlByComponentList(helperBaseName, loadComponents));
+ }
+ } else {
+ if (UtilValidate.isNotEmpty(readerNames)) {
+ urlList.addAll(EntityDataLoader.getUrlList(helperBaseName, component, readerNames));
+ } else if (readersEnabled) {
+ urlList.addAll(EntityDataLoader.getUrlList(helperBaseName, component));
}
}
- Debug.logImportant("=-=-=-=-=-=-= Finished the data load with " + totalRowsChanged + " rows changed.", module);
+ // retrieve URLs from files
+ urlList.addAll(retireveDataUrlsFromFileList(files));
- // create primary keys
- if (createPks) {
- List<String> messages = new LinkedList<String>();
+ // retrieve URLs from all data files in "directory"
+ urlList.addAll(retrieveDataUrlsFromDirectory(directory));
- Debug.logImportant("Creating primary keys...", module);
- for (String entityName : modelEntityNames) {
- ModelEntity modelEntity = modelEntities.get(entityName);
- if (modelEntity != null) {
- dbUtil.createPrimaryKey(modelEntity, messages);
- }
- }
- if (messages.size() > 0) {
- if (Debug.infoOn()) {
- for (String message : messages) {
- Debug.logInfo(message, module);
- }
- }
- }
+ return urlList;
+ }
+
+ private List<String> getLoadFiles(String fileProp) {
+ List<String> fileList = new ArrayList<String>();
+ Optional.ofNullable(fileProp)
+ .ifPresent(props -> fileList.addAll(StringUtil.split(props, ",")));
+ return fileList;
+ }
+
+ private boolean isDataReadersEnabled(List<String> files, String directory, String readers) {
+ /* if files or directories are passed by the user and no readers are
+ * passed then set readers to "none" */
+ if (readers == null && (!files.isEmpty() || directory != null)) {
+ return false;
+ } else {
+ return true;
}
+ }
- // create constraints
- if (createConstraints) {
- List<String> messages = new LinkedList<String>();
+ private List<String> prepareTenantLoadComponents(Delegator delegator, Delegator baseDelegator,
+ Collection<ComponentConfig> allComponents, String component) {
- Debug.logImportant("Creating foreign keys...", module);
- for (String entityName : modelEntityNames) {
- ModelEntity modelEntity = modelEntities.get(entityName);
- if (modelEntity != null) {
- dbUtil.createForeignKeys(modelEntity, modelEntities, messages);
- }
- }
+ List<String> loadComponents = new ArrayList<String>();
+ List<EntityCondition> queryConditions = new ArrayList<EntityCondition>();
- Debug.logImportant("Creating foreign key indcies...", module);
- for (String entityName : modelEntityNames) {
- ModelEntity modelEntity = modelEntities.get(entityName);
- if (modelEntity != null) {
- dbUtil.createForeignKeyIndices(modelEntity, messages);
- }
+ if (UtilValidate.isNotEmpty(delegator.getDelegatorTenantId()) && EntityUtil.isMultiTenantEnabled()) {
+
+ queryConditions.add(EntityCondition.makeCondition("tenantId", delegator.getDelegatorTenantId()));
+ if (UtilValidate.isEmpty(component)) {
+ allComponents.forEach(config -> loadComponents.add(config.getComponentName()));
+ } else {
+ queryConditions.add(EntityCondition.makeCondition("componentName", component));
}
- Debug.logImportant("Creating declared indices...", module);
- for (String entityName : modelEntityNames) {
- ModelEntity modelEntity = modelEntities.get(entityName);
- if (modelEntity != null) {
- dbUtil.createDeclaredIndices(modelEntity, messages);
- }
+ try {
+ List<GenericValue> tenantComponents = EntityQuery.use(baseDelegator)
+ .from("TenantComponent")
+ .where(queryConditions)
+ .orderBy("sequenceNum")
+ .queryList();
+ tenantComponents.forEach(comp -> loadComponents.add(comp.getString("componentName")));
+ Debug.logInfo("Loaded : " + loadComponents.size() + " components", module);
+ } catch (GenericEntityException e) {
+ Debug.logError(e.getMessage(), module);
}
+ }
+ return loadComponents;
+ }
- if (messages.size() > 0) {
- if (Debug.infoOn()) {
- for (String message : messages) {
- Debug.logInfo(message, module);
- }
- }
+ private List<URL> retireveDataUrlsFromFileList(List<String> files) throws ContainerException {
+ List<URL> fileUrls = new ArrayList<URL>();
+ for(String file: files) {
+ URL url = UtilURL.fromResource(file);
+ if (url == null) {
+ throw new ContainerException("Unable to locate data file: " + file);
+ } else {
+ fileUrls.add(url);
}
}
+ return fileUrls;
}
- /**
- * @see org.apache.ofbiz.base.container.Container#stop()
- */
- @Override
- public void stop() throws ContainerException {
+
+ private List<URL> retrieveDataUrlsFromDirectory(String directory) {
+ return Optional.ofNullable(directory)
+ .map(dir -> Arrays.asList(new File(dir).listFiles()).stream()
+ .filter(file -> file.getName().toLowerCase().endsWith(".xml"))
+ .map(file -> UtilURL.fromFilename(file.getPath()))
+ .collect(Collectors.toList()))
+ .orElse(new ArrayList<URL>());
}
- @Override
- public String getName() {
- return name;
+ private void logDataLoadingPlan(List<URL> urlList, String delegatorName) {
+ if (UtilValidate.isNotEmpty(urlList)) {
+ Debug.logImportant("=-=-=-=-=-=-= Doing a data load using delegator '"
+ + delegatorName + "' with the following files:", module);
+ urlList.forEach(dataUrl -> Debug.logImportant(dataUrl.toExternalForm(), module));
+ Debug.logImportant("=-=-=-=-=-=-= Starting the data load...", module);
+ } else {
+ Debug.logImportant("=-=-=-=-=-=-= No data load files found.", module);
+ }
+ }
+
+ private String createDataLoadMessage(URL dataUrl, int rowsChanged, int totalRowsChanged) {
+ NumberFormat formatter = NumberFormat.getIntegerInstance();
+ formatter.setMinimumIntegerDigits(5);
+ formatter.setGroupingUsed(false);
+ return formatter.format(rowsChanged)
+ + " of " + formatter.format(totalRowsChanged)
+ + " from " + dataUrl.toExternalForm();
+ }
+
+ private void logDataLoadingResults(List<String> infoMessages,
+ List<Object> errorMessages, int totalRowsChanged) {
+
+ if (UtilValidate.isNotEmpty(infoMessages)) {
+ Debug.logImportant("=-=-=-=-=-=-= Here is a summary of the data load:", module);
+ infoMessages.forEach(message -> Debug.logImportant(message, module));
+ }
+ if (UtilValidate.isNotEmpty(errorMessages)) {
+ Debug.logImportant("The following errors occurred in the data load:", module);
+ errorMessages.forEach(message -> Debug.logImportant(message.toString(), module));
+ }
+ Debug.logImportant("=-=-=-=-=-=-= Finished the data load with "
+ + totalRowsChanged + " rows changed.", module);
}
}
Modified: ofbiz/ofbiz-framework/trunk/framework/start/src/main/java/org/apache/ofbiz/base/start/StartupCommandUtil.java
URL: http://svn.apache.org/viewvc/ofbiz/ofbiz-framework/trunk/framework/start/src/main/java/org/apache/ofbiz/base/start/StartupCommandUtil.java?rev=1805947&r1=1805946&r2=1805947&view=diff
==============================================================================
--- ofbiz/ofbiz-framework/trunk/framework/start/src/main/java/org/apache/ofbiz/base/start/StartupCommandUtil.java (original)
+++ ofbiz/ofbiz-framework/trunk/framework/start/src/main/java/org/apache/ofbiz/base/start/StartupCommandUtil.java Wed Aug 23 18:56:15 2017
@@ -39,21 +39,21 @@ import org.apache.commons.cli.ParseExcep
/**
* A utility class for processing OFBiz command line arguments
- *
+ *
* <p>
* Defines OFBiz startup options called through main e.g. --load-data or --help
* in addition to utility methods for parsing and handling these options
- * </p>
+ * </p>
*/
public final class StartupCommandUtil {
- /*
+ /*
* Make sure of defining the same set of values in:
- *
+ *
* - The StartupOptions in the StartupOption enum
* - The commons-cli options (e.g. HELP, START, etc ...)
* - The getOfbizStartupOptions method
- *
+ *
* Keeping these items in sync means that OFBiz behaves correctly
* while being decoupled from the commons-cli library and the only
* place where commons-cli is used is in this class
@@ -67,7 +67,7 @@ public final class StartupCommandUtil {
START("start"),
STATUS("status"),
TEST("test");
-
+
private String name;
private StartupOption(String name) {
this.name = name;
@@ -88,15 +88,35 @@ public final class StartupCommandUtil {
+ System.lineSeparator()
+ "-l readers=seed,demo,ext"
+ System.lineSeparator()
- + "-l timeout=7200"
+ + "-l file=/tmp/dataload1.xml,/tmp/dataload2.xml"
+ + System.lineSeparator()
+ + "-l dir=directory/of/files"
+ + System.lineSeparator()
+ + "-l component=base"
+ System.lineSeparator()
+ "-l delegator=default"
+ System.lineSeparator()
+ "-l group=org.apache.ofbiz"
+ System.lineSeparator()
- + "-l dir=directory/of/files"
+ + "-l timeout=7200"
+ + System.lineSeparator()
+ + "-l create-pks"
+ + System.lineSeparator()
+ + "-l drop-pks"
+ + System.lineSeparator()
+ + "-l create-constraints"
+ + System.lineSeparator()
+ + "-l drop-constraints"
+ + System.lineSeparator()
+ + "-l create-fks"
+ + System.lineSeparator()
+ + "-l maintain-txs"
+ + System.lineSeparator()
+ + "-l try-inserts"
+ + System.lineSeparator()
+ + "-l repair-columns"
+ System.lineSeparator()
- + "-l file=/tmp/dataload.xml")
+ + "-l continue-on-failure" )
.numberOfArgs(2)
.valueSeparator('=')
.optionalArg(true)
@@ -127,12 +147,13 @@ public final class StartupCommandUtil {
private static final Option TEST = Option.builder("t")
.longOpt(StartupOption.TEST.getName())
.desc("Runs the selected test or all if none selected e.g.: "
+ + "--test component=entity"
+ System.lineSeparator()
- + "--test component=base --test case=somecase"
+ + "--test suitename=entitytests"
+ System.lineSeparator()
- + "or"
+ + "--test case=entity-query-tests"
+ System.lineSeparator()
- + "--test component=base --test suitename=xyz")
+ + "--test loglevel=warning")
.numberOfArgs(2)
.valueSeparator('=')
.optionalArg(true)
@@ -195,7 +216,7 @@ public final class StartupCommandUtil {
}
private static final List<StartupCommand> mapCommonsCliOptionsToStartupCommands(final CommandLine commandLine) {
- Set<Option> uniqueOptions = new HashSet<Option>(Arrays.asList(commandLine.getOptions()));
+ Set<Option> uniqueOptions = new HashSet<Option>(Arrays.asList(commandLine.getOptions()));
return uniqueOptions.stream()
.map(option -> new StartupCommand.Builder(option.getLongOpt())
.properties(populateMapFromProperties(commandLine.getOptionProperties(option.getLongOpt())))