[jira] [Created] (OFBIZ-5042) Apache Solr Implementation

["Paul Piper (JIRA)" <ji...@apache.org>]

Paul Piper created OFBIZ-5042:
---------------------------------

             Summary: Apache Solr Implementation
                 Key: OFBIZ-5042
                 URL: https://issues.apache.org/jira/browse/OFBIZ-5042
             Project: OFBiz
          Issue Type: Improvement
          Components: ALL COMPONENTS
            Reporter: Paul Piper


Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community. With my agency, ilscipio, we have done several Apache Solr implementations in the past, and we felt that we could contribute this feature to the community. Currently, Solr is greatly used within syracus.net (our standalone store) and serves to generate the categories & search functionalities. From our team mainly Pascal Proulx, Jonatan Soto and I were involved in this feature (albeit more having contributed to it. We created a standalone component which can be integrated into any running ofbiz implementation. It was tested against OFBiz 09.04 - 12.04.. 


=====================================================
Solr Component for Apache Ofbiz
=====================================================

This document describes the Ofbiz solr component, an Ofbiz (http://ofbiz.apache.org/) 
implementation of the Apache Solr search platform (http://lucene.apache.org/solr/).
The solr component includes an Ofbiz service-based wrapper layer to the Apache Solr
webapp queries as well as the native Apache Solr web interface itself.

Currently, the solr component focuses on Product data.

Note: This document is a work in progress; information is subject to change.

Contents:

1. Installation
2. Configuration
3. Data Indexing
4. Data Querying
5. Implementation Concerns
6. Known Bugs, Limitations and Issues


-----------------------------------------------------
1. Installation
-----------------------------------------------------

To install solr in an Ofbiz setting, simply extract the solr directory and
and sub-folders to the hot-deploy folder.

Afterward, the solr home system property (solr.solr.home) must be set to the value
hot-deploy/solr manually using one of the following methods:


* solr.solr.home in batch/script file:

Add the parameter "-Dsolr.solr.home=hot-deploy/solr" to the Java command
invocation for ofbiz.jar.
e.g.:
"%JAVA_HOME%\bin\java" -Xms128M -Xmx512M -XX:MaxPermSize=512m -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar


* solr.solr.home in Ant build configuration:

In your root Ofbiz build.xml file, add the element "<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>"
to the "<java jar="ofbiz.jar"...>" invocation of the appropriate Ant target(s) (run, start, run-install, etc.).
e.g.:
<target name="start" description="Start OFBiz">
	<java jar="ofbiz.jar" fork="true">
		<jvmarg value="${memory.initial.param}"/>
		<jvmarg value="${memory.max.param}"/>
		<jvmarg value="${memory.maxpermsize.param}"/>
		<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>
	</java>
</target>

***

It may be possible to specify solr home using other methods (JNDI, web.xml), but at the time
of this writing, this was the most reliable method known.


-----------------------------------------------------
2. Configuration
-----------------------------------------------------

The solr component can run out-of-the-box without configuration, but many
files, settings and interfaces allow custom settings. Some of these include:

* Ofbiz configurations:
** System properties:
*** ofbiz.solr.eca.enabled - Global solr ECA toggling boolean (true/false, see Data Indexing)
** Config files:
*** solr/config/solrconfig.properties - Ofbiz solr service behavior control
*** ofbiz-component.xml - Standard Ofbiz component config

* Apache Solr configurations:
** System properties:
*** solr.solr.home - Solr home (see Installation)
** Config files:
*** solr.xml - Base solr config
*** conf/schema.xml - Solr index schema
*** webapp/WEB-INF/web.xml - Dual Ofbiz/Solr webapp config
** Interfaces:
*** /solr/admin/ - Webapp admin interface (see below)

***

It is possible to set extensive native Solr configuration using
the admin webapp interface noted above. It should be accessible at the address:
http://localhost:8080/solr/admin/
(where 8080 is your server's http port)
Please refer to the Apache Solr documentation for usage of this interface
and other native Solr configuration details.


-----------------------------------------------------
3. Data Indexing
-----------------------------------------------------

The solr component indexes data such as Products into the Apache Solr database
using services defined in the file:
servicesdef/solrservices.xml
The initial indexing may need to be performed or scheduled manually, but subsequent indexing
may be partially or fully automated, though automated methods are disabled by default and must be enabled.
Note that in general, solr services can only successfully run in contexts where the solr webapp is 
loaded and accessible.

There are two methods for indexing data:


* Index rebuilding service (rebuildSolrIndex):

The rebuildSolrIndex is the most important data import service. It reindexes
all Ofbiz Products existing in the system into the solr index. rebuildSolrIndex MUST be run 
AT LEAST once after installation and also following any data load operation that loads new products using the Ofbiz 
"install" starting mode (run-install, load-demo, etc.).

To do this, one can simply use the Webtools backend to invoke the service manually
(e.g., /webtools/control/setSyncServiceParameters?SERVICE_NAME=rebuildSolrIndex&POOL_NAME=pool&_RUN_SYNC_=Y)
or summon the Webtools Job Scheduler.

Once the initial indexing has been performed, one can then use the Job Scheduler to
invoke rebuildSolrIndex on a regular basis (every hour, every midnight, etc.) to update the Solr index.


* ECAs/SECAs (addToSolr, for Product data):

Although the rebuildSolrIndex is always necessary for the initial data import, one may also
use ECAs and SECAs to import subsequent data changes automatically at every individual data (e.g. Product)
update instead of running rebuildSolrIndex periodically. This is done by defining ECAs or SECAs that
trigger the addToSolr service.

The addToSolr service simply accepts a single "instance" parameter, a GenericValue. At the time of this writing,
any entity value having a valid "productId" field designating a Product value may be passed; this will trigger reindexing
for the specific product.

By default, the addToSolr service implementation (and any service intended for use with ECAs) is disabled globally and
succeeds silently. It can be enabled by setting the "solr.eca.enabled=true" in "config/solrconfig.properties".
This property is provided for easy toggling. It can also be specified on the command line or using ant through 
the true/false "ofbiz.solr.eca.enabled" system property (which has priority over file-based "solr.eca.enabled") in the same
way described for the solr home system property (see Installation); this allows toggling them per Ant target.

Some stock/default/example ECAs are provided and are functional once enabled. They can be found in:
entitydef/eecas.xml
Custom ECAs may also be added to this file, and custom SECAs may be added to the file:
servicedef/secas.xml

Special care may need to be taken to ensure the addToSolr is not triggered during "install" sequences
(run-install/load-demo) because it requires the solr webapp to be accessible. addToSolr will automatically
attempt to prevent reindexing in cases where the solr webapp is unavailable. However, it is possible that
this behavior results in needlessly prevented reindexing (and thus lost updates) in some unknown complex server
configurations. If this is the case, this behavior can be disabled by setting the option
"solr.eca.useSolrWebappLoadedCheck=false" in the properties file "config/solrconfig.properties".
 
However, if the above useSolrWebappLoadedCheck option is specified, you will have to manually disable or comment the 
ECA definitions/services to prevent them from executing during data-load operations; the "solr.eca.enabled" property and "ofbiz.solr.eca.enabled" system property can be used for this purpose. One easy way is to set "solr.eca.enabled=true" in 
the property files and then specify "-Dofbiz.solr.eca.enabled=false" for any relevant 
Ant build install/run-install/load-demo targets.

Note that simply commenting the ECAs/SECAs or preventing their inclusion using
ofbiz-component.xml may eliminate some service call overhead compared to using the property toggling. Ideally,
a method exploiting ofbiz-component.xml or eecas.xml to prevent ECAs from running in inappropriate contexts
would be ideal; however, at the time of this writing, such a method could not be established.

***

To reset and clear the solr index completely, it is sufficient to shut down the server and delete the folders:
solr/data/index
solr/data/spellchecker
and any other such folders (must delete the whole folders, not only contents).


-----------------------------------------------------
4. Data Querying
-----------------------------------------------------

Solr queries can be done using two methods:


* Solr Ofbiz services:

Simply invoke (manually or in code) the query services found in the file:
servicesdef/solrservices.xml
These include solrProductsSearch, solrKeywordSearch and others. Note that in general, 
solr services can only successfully run in contexts where the solr webapp is loaded and accessible.


* Solr native admin webapp interface:

One can also perform native Solr queries and diagnostics using the standard
admin interface, accessible as described under Configuration. Please refer
to the Apache Solr documentation for usage of this interface.


-----------------------------------------------------
5. Implementation Concerns
-----------------------------------------------------

* The structure of the solr/webapp directory closely mirrors the contents of the solr.war
distribution; however, please note that it contains some Ofbiz-specific modifications such
as those found in web.xml. The same is true for other folders such as solr/conf. Therefore,
any library updates must be done with care to preserve these modifications.


-----------------------------------------------------
6. Known Bugs, Limitations and Issues
-----------------------------------------------------

* In general, solr services can only successfully run in contexts where the solr webapp 
is loaded and accessible.
* ECA indexing may be problematic due to data loading and unknown complex server configurations; 
various toggling options are provided to work around this issue (see Data Indexing for details).
In the future - and ideally - this could be addressed using ofbiz-component.xml (or eecas.xml) directives.
* The indexing services often produce the log warning "Problem reading product features" due to their
use of the stock Ofbiz getProductFeatureSet service.

***

Please report any other issues encountered. Thank you.




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (OFBIZ-5042) Apache Solr Implementation

["Tom Burns (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OFBIZ-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466508#comment-13466508 ] 

Tom Burns commented on OFBIZ-5042:
----------------------------------

Hi Paul,

I was thinking of using Solar / Tika for processing resumes in the HR application. Your work is a great help in jump starting what I have in mind. Thanks!

I ran into one problem. I work in Eclipse and invoke OFBiz using Eclipse Dbug with the following configuration.

-Xms128M -Xmx512M -XX:MaxPermSize=512M
-Dcom.sun.management.jmxremote.port=3333 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

When I added -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar to the configuration  it throws:
 
throw (StartupException) new StartupException("Couldn't not fetch config instance").initCause(e);

I altered the root build xml as described above and was able to run in Eclipse by using the ant start and stop commands but I would like to run in a debug configuration.

Do you have a VM Arguments command for Eclipse that starts OFBiz with Solar?

Tom

                
> Apache Solr Implementation
> --------------------------
>
>                 Key: OFBIZ-5042
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-5042
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: ALL COMPONENTS
>            Reporter: Paul Piper
>              Labels: component, feature, solr
>         Attachments: solr-v1.00-r4284.zip
>
>
> Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community. With my agency, ilscipio, we have done several Apache Solr implementations in the past, and we felt that we could contribute this feature to the community. Currently, Solr is greatly used within syracus.net (our standalone store) and serves to generate the categories & search functionalities. From our team mainly Pascal Proulx, Jonatan Soto and I  were involved in this feature (albeit more having contributed to it. We created a standalone component which can be integrated into any running ofbiz implementation. It was tested against OFBiz 09.04 - 12.04.. 
> =====================================================
> Solr Component for Apache Ofbiz
> =====================================================
> This document describes the Ofbiz solr component, an Ofbiz (http://ofbiz.apache.org/) 
> implementation of the Apache Solr search platform (http://lucene.apache.org/solr/).
> The solr component includes an Ofbiz service-based wrapper layer to the Apache Solr
> webapp queries as well as the native Apache Solr web interface itself.
> Currently, the solr component focuses on Product data.
> Note: This document is a work in progress; information is subject to change.
> Contents:
> 1. Installation
> 2. Configuration
> 3. Data Indexing
> 4. Data Querying
> 5. Implementation Concerns
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> 1. Installation
> -----------------------------------------------------
> To install solr in an Ofbiz setting, simply extract the solr directory and
> and sub-folders to the hot-deploy folder.
> Afterward, the solr home system property (solr.solr.home) must be set to the value
> hot-deploy/solr manually using one of the following methods:
> * solr.solr.home in batch/script file:
> Add the parameter "-Dsolr.solr.home=hot-deploy/solr" to the Java command
> invocation for ofbiz.jar.
> e.g.:
> "%JAVA_HOME%\bin\java" -Xms128M -Xmx512M -XX:MaxPermSize=512m -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar
> * solr.solr.home in Ant build configuration:
> In your root Ofbiz build.xml file, add the element "<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>"
> to the "<java jar="ofbiz.jar"...>" invocation of the appropriate Ant target(s) (run, start, run-install, etc.).
> e.g.:
> <target name="start" description="Start OFBiz">
> 	<java jar="ofbiz.jar" fork="true">
> 		<jvmarg value="${memory.initial.param}"/>
> 		<jvmarg value="${memory.max.param}"/>
> 		<jvmarg value="${memory.maxpermsize.param}"/>
> 		<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>
> 	</java>
> </target>
> ***
> It may be possible to specify solr home using other methods (JNDI, web.xml), but at the time
> of this writing, this was the most reliable method known.
> -----------------------------------------------------
> 2. Configuration
> -----------------------------------------------------
> The solr component can run out-of-the-box without configuration, but many
> files, settings and interfaces allow custom settings. Some of these include:
> * Ofbiz configurations:
> ** System properties:
> *** ofbiz.solr.eca.enabled - Global solr ECA toggling boolean (true/false, see Data Indexing)
> ** Config files:
> *** solr/config/solrconfig.properties - Ofbiz solr service behavior control
> *** ofbiz-component.xml - Standard Ofbiz component config
> * Apache Solr configurations:
> ** System properties:
> *** solr.solr.home - Solr home (see Installation)
> ** Config files:
> *** solr.xml - Base solr config
> *** conf/schema.xml - Solr index schema
> *** webapp/WEB-INF/web.xml - Dual Ofbiz/Solr webapp config
> ** Interfaces:
> *** /solr/admin/ - Webapp admin interface (see below)
> ***
> It is possible to set extensive native Solr configuration using
> the admin webapp interface noted above. It should be accessible at the address:
> http://localhost:8080/solr/admin/
> (where 8080 is your server's http port)
> Please refer to the Apache Solr documentation for usage of this interface
> and other native Solr configuration details.
> -----------------------------------------------------
> 3. Data Indexing
> -----------------------------------------------------
> The solr component indexes data such as Products into the Apache Solr database
> using services defined in the file:
> servicesdef/solrservices.xml
> The initial indexing may need to be performed or scheduled manually, but subsequent indexing
> may be partially or fully automated, though automated methods are disabled by default and must be enabled.
> Note that in general, solr services can only successfully run in contexts where the solr webapp is 
> loaded and accessible.
> There are two methods for indexing data:
> * Index rebuilding service (rebuildSolrIndex):
> The rebuildSolrIndex is the most important data import service. It reindexes
> all Ofbiz Products existing in the system into the solr index. rebuildSolrIndex MUST be run 
> AT LEAST once after installation and also following any data load operation that loads new products using the Ofbiz 
> "install" starting mode (run-install, load-demo, etc.).
> To do this, one can simply use the Webtools backend to invoke the service manually
> (e.g., /webtools/control/setSyncServiceParameters?SERVICE_NAME=rebuildSolrIndex&POOL_NAME=pool&_RUN_SYNC_=Y)
> or summon the Webtools Job Scheduler.
> Once the initial indexing has been performed, one can then use the Job Scheduler to
> invoke rebuildSolrIndex on a regular basis (every hour, every midnight, etc.) to update the Solr index.
> * ECAs/SECAs (addToSolr, for Product data):
> Although the rebuildSolrIndex is always necessary for the initial data import, one may also
> use ECAs and SECAs to import subsequent data changes automatically at every individual data (e.g. Product)
> update instead of running rebuildSolrIndex periodically. This is done by defining ECAs or SECAs that
> trigger the addToSolr service.
> The addToSolr service simply accepts a single "instance" parameter, a GenericValue. At the time of this writing,
> any entity value having a valid "productId" field designating a Product value may be passed; this will trigger reindexing
> for the specific product.
> By default, the addToSolr service implementation (and any service intended for use with ECAs) is disabled globally and
> succeeds silently. It can be enabled by setting the "solr.eca.enabled=true" in "config/solrconfig.properties".
> This property is provided for easy toggling. It can also be specified on the command line or using ant through 
> the true/false "ofbiz.solr.eca.enabled" system property (which has priority over file-based "solr.eca.enabled") in the same
> way described for the solr home system property (see Installation); this allows toggling them per Ant target.
> Some stock/default/example ECAs are provided and are functional once enabled. They can be found in:
> entitydef/eecas.xml
> Custom ECAs may also be added to this file, and custom SECAs may be added to the file:
> servicedef/secas.xml
> Special care may need to be taken to ensure the addToSolr is not triggered during "install" sequences
> (run-install/load-demo) because it requires the solr webapp to be accessible. addToSolr will automatically
> attempt to prevent reindexing in cases where the solr webapp is unavailable. However, it is possible that
> this behavior results in needlessly prevented reindexing (and thus lost updates) in some unknown complex server
> configurations. If this is the case, this behavior can be disabled by setting the option
> "solr.eca.useSolrWebappLoadedCheck=false" in the properties file "config/solrconfig.properties".
>  
> However, if the above useSolrWebappLoadedCheck option is specified, you will have to manually disable or comment the 
> ECA definitions/services to prevent them from executing during data-load operations; the "solr.eca.enabled" property and "ofbiz.solr.eca.enabled" system property can be used for this purpose. One easy way is to set "solr.eca.enabled=true" in 
> the property files and then specify "-Dofbiz.solr.eca.enabled=false" for any relevant 
> Ant build install/run-install/load-demo targets.
> Note that simply commenting the ECAs/SECAs or preventing their inclusion using
> ofbiz-component.xml may eliminate some service call overhead compared to using the property toggling. Ideally,
> a method exploiting ofbiz-component.xml or eecas.xml to prevent ECAs from running in inappropriate contexts
> would be ideal; however, at the time of this writing, such a method could not be established.
> ***
> To reset and clear the solr index completely, it is sufficient to shut down the server and delete the folders:
> solr/data/index
> solr/data/spellchecker
> and any other such folders (must delete the whole folders, not only contents).
> -----------------------------------------------------
> 4. Data Querying
> -----------------------------------------------------
> Solr queries can be done using two methods:
> * Solr Ofbiz services:
> Simply invoke (manually or in code) the query services found in the file:
> servicesdef/solrservices.xml
> These include solrProductsSearch, solrKeywordSearch and others. Note that in general, 
> solr services can only successfully run in contexts where the solr webapp is loaded and accessible.
> * Solr native admin webapp interface:
> One can also perform native Solr queries and diagnostics using the standard
> admin interface, accessible as described under Configuration. Please refer
> to the Apache Solr documentation for usage of this interface.
> -----------------------------------------------------
> 5. Implementation Concerns
> -----------------------------------------------------
> * The structure of the solr/webapp directory closely mirrors the contents of the solr.war
> distribution; however, please note that it contains some Ofbiz-specific modifications such
> as those found in web.xml. The same is true for other folders such as solr/conf. Therefore,
> any library updates must be done with care to preserve these modifications.
> -----------------------------------------------------
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> * In general, solr services can only successfully run in contexts where the solr webapp 
> is loaded and accessible.
> * ECA indexing may be problematic due to data loading and unknown complex server configurations; 
> various toggling options are provided to work around this issue (see Data Indexing for details).
> In the future - and ideally - this could be addressed using ofbiz-component.xml (or eecas.xml) directives.
> * The indexing services often produce the log warning "Problem reading product features" due to their
> use of the stock Ofbiz getProductFeatureSet service.
> ***
> Please report any other issues encountered. Thank you.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (OFBIZ-5042) Apache Solr Implementation

["Jacques Le Roux (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OFBIZ-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499377#comment-13499377 ] 

Jacques Le Roux commented on OFBIZ-5042:
----------------------------------------

Sure there is no hurry, but we need to discuss it and get ahead ;) I agree we have 1st to discuss about existing speicalpurpose components. Some are obvious to me, like moving ebay and google components to Extras, but maybe others will be against, etc. I hope though, we will quickly find a common ground. By consensus I mean, to avoid commits wars like we had in the past :(, and even votes if possible.
                
> Apache Solr Implementation
> --------------------------
>
>                 Key: OFBIZ-5042
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-5042
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: ALL COMPONENTS
>            Reporter: Paul Piper
>              Labels: component, feature, solr
>         Attachments: solr-v1.00-r4284.zip
>
>
> Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community. With my agency, ilscipio, we have done several Apache Solr implementations in the past, and we felt that we could contribute this feature to the community. Currently, Solr is greatly used within syracus.net (our standalone store) and serves to generate the categories & search functionalities. From our team mainly Pascal Proulx, Jonatan Soto and I  were involved in this feature (albeit more having contributed to it. We created a standalone component which can be integrated into any running ofbiz implementation. It was tested against OFBiz 09.04 - 12.04.. 
> =====================================================
> Solr Component for Apache Ofbiz
> =====================================================
> This document describes the Ofbiz solr component, an Ofbiz (http://ofbiz.apache.org/) 
> implementation of the Apache Solr search platform (http://lucene.apache.org/solr/).
> The solr component includes an Ofbiz service-based wrapper layer to the Apache Solr
> webapp queries as well as the native Apache Solr web interface itself.
> Currently, the solr component focuses on Product data.
> Note: This document is a work in progress; information is subject to change.
> Contents:
> 1. Installation
> 2. Configuration
> 3. Data Indexing
> 4. Data Querying
> 5. Implementation Concerns
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> 1. Installation
> -----------------------------------------------------
> To install solr in an Ofbiz setting, simply extract the solr directory and
> and sub-folders to the hot-deploy folder.
> Afterward, the solr home system property (solr.solr.home) must be set to the value
> hot-deploy/solr manually using one of the following methods:
> * solr.solr.home in batch/script file:
> Add the parameter "-Dsolr.solr.home=hot-deploy/solr" to the Java command
> invocation for ofbiz.jar.
> e.g.:
> "%JAVA_HOME%\bin\java" -Xms128M -Xmx512M -XX:MaxPermSize=512m -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar
> * solr.solr.home in Ant build configuration:
> In your root Ofbiz build.xml file, add the element "<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>"
> to the "<java jar="ofbiz.jar"...>" invocation of the appropriate Ant target(s) (run, start, run-install, etc.).
> e.g.:
> <target name="start" description="Start OFBiz">
> 	<java jar="ofbiz.jar" fork="true">
> 		<jvmarg value="${memory.initial.param}"/>
> 		<jvmarg value="${memory.max.param}"/>
> 		<jvmarg value="${memory.maxpermsize.param}"/>
> 		<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>
> 	</java>
> </target>
> ***
> It may be possible to specify solr home using other methods (JNDI, web.xml), but at the time
> of this writing, this was the most reliable method known.
> -----------------------------------------------------
> 2. Configuration
> -----------------------------------------------------
> The solr component can run out-of-the-box without configuration, but many
> files, settings and interfaces allow custom settings. Some of these include:
> * Ofbiz configurations:
> ** System properties:
> *** ofbiz.solr.eca.enabled - Global solr ECA toggling boolean (true/false, see Data Indexing)
> ** Config files:
> *** solr/config/solrconfig.properties - Ofbiz solr service behavior control
> *** ofbiz-component.xml - Standard Ofbiz component config
> * Apache Solr configurations:
> ** System properties:
> *** solr.solr.home - Solr home (see Installation)
> ** Config files:
> *** solr.xml - Base solr config
> *** conf/schema.xml - Solr index schema
> *** webapp/WEB-INF/web.xml - Dual Ofbiz/Solr webapp config
> ** Interfaces:
> *** /solr/admin/ - Webapp admin interface (see below)
> ***
> It is possible to set extensive native Solr configuration using
> the admin webapp interface noted above. It should be accessible at the address:
> http://localhost:8080/solr/admin/
> (where 8080 is your server's http port)
> Please refer to the Apache Solr documentation for usage of this interface
> and other native Solr configuration details.
> -----------------------------------------------------
> 3. Data Indexing
> -----------------------------------------------------
> The solr component indexes data such as Products into the Apache Solr database
> using services defined in the file:
> servicesdef/solrservices.xml
> The initial indexing may need to be performed or scheduled manually, but subsequent indexing
> may be partially or fully automated, though automated methods are disabled by default and must be enabled.
> Note that in general, solr services can only successfully run in contexts where the solr webapp is 
> loaded and accessible.
> There are two methods for indexing data:
> * Index rebuilding service (rebuildSolrIndex):
> The rebuildSolrIndex is the most important data import service. It reindexes
> all Ofbiz Products existing in the system into the solr index. rebuildSolrIndex MUST be run 
> AT LEAST once after installation and also following any data load operation that loads new products using the Ofbiz 
> "install" starting mode (run-install, load-demo, etc.).
> To do this, one can simply use the Webtools backend to invoke the service manually
> (e.g., /webtools/control/setSyncServiceParameters?SERVICE_NAME=rebuildSolrIndex&POOL_NAME=pool&_RUN_SYNC_=Y)
> or summon the Webtools Job Scheduler.
> Once the initial indexing has been performed, one can then use the Job Scheduler to
> invoke rebuildSolrIndex on a regular basis (every hour, every midnight, etc.) to update the Solr index.
> * ECAs/SECAs (addToSolr, for Product data):
> Although the rebuildSolrIndex is always necessary for the initial data import, one may also
> use ECAs and SECAs to import subsequent data changes automatically at every individual data (e.g. Product)
> update instead of running rebuildSolrIndex periodically. This is done by defining ECAs or SECAs that
> trigger the addToSolr service.
> The addToSolr service simply accepts a single "instance" parameter, a GenericValue. At the time of this writing,
> any entity value having a valid "productId" field designating a Product value may be passed; this will trigger reindexing
> for the specific product.
> By default, the addToSolr service implementation (and any service intended for use with ECAs) is disabled globally and
> succeeds silently. It can be enabled by setting the "solr.eca.enabled=true" in "config/solrconfig.properties".
> This property is provided for easy toggling. It can also be specified on the command line or using ant through 
> the true/false "ofbiz.solr.eca.enabled" system property (which has priority over file-based "solr.eca.enabled") in the same
> way described for the solr home system property (see Installation); this allows toggling them per Ant target.
> Some stock/default/example ECAs are provided and are functional once enabled. They can be found in:
> entitydef/eecas.xml
> Custom ECAs may also be added to this file, and custom SECAs may be added to the file:
> servicedef/secas.xml
> Special care may need to be taken to ensure the addToSolr is not triggered during "install" sequences
> (run-install/load-demo) because it requires the solr webapp to be accessible. addToSolr will automatically
> attempt to prevent reindexing in cases where the solr webapp is unavailable. However, it is possible that
> this behavior results in needlessly prevented reindexing (and thus lost updates) in some unknown complex server
> configurations. If this is the case, this behavior can be disabled by setting the option
> "solr.eca.useSolrWebappLoadedCheck=false" in the properties file "config/solrconfig.properties".
>  
> However, if the above useSolrWebappLoadedCheck option is specified, you will have to manually disable or comment the 
> ECA definitions/services to prevent them from executing during data-load operations; the "solr.eca.enabled" property and "ofbiz.solr.eca.enabled" system property can be used for this purpose. One easy way is to set "solr.eca.enabled=true" in 
> the property files and then specify "-Dofbiz.solr.eca.enabled=false" for any relevant 
> Ant build install/run-install/load-demo targets.
> Note that simply commenting the ECAs/SECAs or preventing their inclusion using
> ofbiz-component.xml may eliminate some service call overhead compared to using the property toggling. Ideally,
> a method exploiting ofbiz-component.xml or eecas.xml to prevent ECAs from running in inappropriate contexts
> would be ideal; however, at the time of this writing, such a method could not be established.
> ***
> To reset and clear the solr index completely, it is sufficient to shut down the server and delete the folders:
> solr/data/index
> solr/data/spellchecker
> and any other such folders (must delete the whole folders, not only contents).
> -----------------------------------------------------
> 4. Data Querying
> -----------------------------------------------------
> Solr queries can be done using two methods:
> * Solr Ofbiz services:
> Simply invoke (manually or in code) the query services found in the file:
> servicesdef/solrservices.xml
> These include solrProductsSearch, solrKeywordSearch and others. Note that in general, 
> solr services can only successfully run in contexts where the solr webapp is loaded and accessible.
> * Solr native admin webapp interface:
> One can also perform native Solr queries and diagnostics using the standard
> admin interface, accessible as described under Configuration. Please refer
> to the Apache Solr documentation for usage of this interface.
> -----------------------------------------------------
> 5. Implementation Concerns
> -----------------------------------------------------
> * The structure of the solr/webapp directory closely mirrors the contents of the solr.war
> distribution; however, please note that it contains some Ofbiz-specific modifications such
> as those found in web.xml. The same is true for other folders such as solr/conf. Therefore,
> any library updates must be done with care to preserve these modifications.
> -----------------------------------------------------
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> * In general, solr services can only successfully run in contexts where the solr webapp 
> is loaded and accessible.
> * ECA indexing may be problematic due to data loading and unknown complex server configurations; 
> various toggling options are provided to work around this issue (see Data Indexing for details).
> In the future - and ideally - this could be addressed using ofbiz-component.xml (or eecas.xml) directives.
> * The indexing services often produce the log warning "Problem reading product features" due to their
> use of the stock Ofbiz getProductFeatureSet service.
> ***
> Please report any other issues encountered. Thank you.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (OFBIZ-5042) Apache Solr Implementation

["Jacques Le Roux (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OFBIZ-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470955#comment-13470955 ] 

Jacques Le Roux edited comment on OFBIZ-5042 at 10/6/12 9:05 AM:
-----------------------------------------------------------------

== BETTER COMMENT: WE == COMMUNITY ==
Since the community decided to move most specialpurpose components to Apache Extra ([see slim down effort|https://issues.apache.org/jira/browse/OFBIZ/fixforversion/12320551]), the best place for this component is in [Apache OFBiz Extra|http://code.google.com/a/apache-extras.org/hosting/search?q=label%3aOFBiz]. Please read [Apache Extra guidelines|http://community.apache.org/apache-extras/guidelines.html]. 

Note the point 6:
{quote}
Each Apache Extras project is encouraged to publish their releases and complete the project tagging information available on Apache Extras. 
{quote}

As explained in the guidelines, The Apache OFBiz Extra projects will not be official part of the Apache OFBiz project. The goals are pragmatic (at least as I see it):
* Components which are not mandatory (not supposed to be used in each project powered by OFBiz) can be moved to Apache Extra. 
* With limited Apache OFBiz project human resources, it's a way to delegate the code and releases responsabilities to specific communities

But this does not mean that Apache OFBiz Extra projects will be forgotten or neglected by the Apache OFBiz project. Now I feel something is missing in the scheme so far: how to relate an Apache OFBiz Extra project release to an Apache OFBiz release? I think it should part of the Apache OFBiz Extra project community to indicate which OFBiz release/s (trunk included) are compatible with the component. This is kind of part of the guidelines point 6 in my opininon. This is correctly maintained could be used by the Apache OFBiz project when releasing, we could at least have a link to the [Apache OFBiz Extra projects page|http://code.google.com/a/apache-extras.org/hosting/search?q=label%3aOFBiz]. Same from the [Apache OFBiz project site main page|http://ofbiz.apache.org/], a reference should be added. I will do so soon, but will wait that at least a complete usable Apache OFBiz Extra project exists there. The Neogia team made a first effor but nothing was committed yet, the Solr component could be the first to engage the movement. With the ApacheConEu coming it could be also a good way to share about this point.

Also I'd like to have a way to know the value of an Apache OFBiz Extra project, as experienced by its users. For the moment I see only one way to do this: recommend it on Google + (see in each component Project information). Of course this is a very limited way, but at least a mean.

So it would be great if you Paul, Pascal and Jonatan (at least for now, the community will hopefully grow from comments above) could move the Solr component into the Apache OFBiz Extra projects. This would begin the dynamic, completing the pending work initiated by the Neogia team. Hopefully more will come later. For instance I read a lof about Big Fish, why not sharing in Apache OFBiz Extra projects also?
                
      was (Author: jacques.le.roux):
    Since we decided to move most specialpurpose components to Apache Extra ([see slim down effort|https://issues.apache.org/jira/browse/OFBIZ/fixforversion/12320551]), the best place for this component is in [Apache OFBiz Extra|http://code.google.com/a/apache-extras.org/hosting/search?q=label%3aOFBiz]. Please read [Apache Extra guidelines|http://community.apache.org/apache-extras/guidelines.html]. 

Note the point 6:
{quote}
Each Apache Extras project is encouraged to publish their releases and complete the project tagging information available on Apache Extras. 
{quote}

As explained in the guidelines, The Apache OFBiz Extra projects will not be official part of the Apache OFBiz project. The goals are pragmatic (at least as I see it):
* Components which are not mandatory (not supposed to be used in each project powered by OFBiz) can be moved to Apache Extra. 
* With limited Apache OFBiz project human resources, it's a way to delegate the code and releases responsabilities to specific communities

But this does not mean that Apache OFBiz Extra projects will be forgotten or neglected by the Apache OFBiz project. Now I feel something is missing in the scheme so far: how to relate an Apache OFBiz Extra project release to an Apache OFBiz release? I think it should part of the Apache OFBiz Extra project community to indicate which OFBiz release/s (trunk included) are compatible with the component. This is kind of part of the guidelines point 6 in my opininon. This is correctly maintained could be used by the Apache OFBiz project when releasing, we could at least have a link to the [Apache OFBiz Extra projects page|http://code.google.com/a/apache-extras.org/hosting/search?q=label%3aOFBiz]. Same from the [Apache OFBiz project site main page|http://ofbiz.apache.org/], a reference should be added. I will do so soon, but will wait that at least a complete usable Apache OFBiz Extra project exists there. The Neogia team made a first effor but nothing was committed yet, the Solr component could be the first to engage the movement. With the ApacheConEu coming it could be also a good way to share about this point.

Also I'd like to have a way to know the value of an Apache OFBiz Extra project, as experienced by its users. For the moment I see only one way to do this: recommend it on Google + (see in each component Project information). Of course this is a very limited way, but at least a mean.

So it would be great if you Paul, Pascal and Jonatan (at least for now, the community will hopefully grow from comments above) could move the Solr component into the Apache OFBiz Extra projects. This would begin the dynamic, completing the pending work initiated by the Neogia team. Hopefully more will come later. For instance I read a lof about Big Fish, why not sharing in Apache OFBiz Extra projects also?
                  
> Apache Solr Implementation
> --------------------------
>
>                 Key: OFBIZ-5042
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-5042
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: ALL COMPONENTS
>            Reporter: Paul Piper
>              Labels: component, feature, solr
>         Attachments: solr-v1.00-r4284.zip
>
>
> Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community. With my agency, ilscipio, we have done several Apache Solr implementations in the past, and we felt that we could contribute this feature to the community. Currently, Solr is greatly used within syracus.net (our standalone store) and serves to generate the categories & search functionalities. From our team mainly Pascal Proulx, Jonatan Soto and I  were involved in this feature (albeit more having contributed to it. We created a standalone component which can be integrated into any running ofbiz implementation. It was tested against OFBiz 09.04 - 12.04.. 
> =====================================================
> Solr Component for Apache Ofbiz
> =====================================================
> This document describes the Ofbiz solr component, an Ofbiz (http://ofbiz.apache.org/) 
> implementation of the Apache Solr search platform (http://lucene.apache.org/solr/).
> The solr component includes an Ofbiz service-based wrapper layer to the Apache Solr
> webapp queries as well as the native Apache Solr web interface itself.
> Currently, the solr component focuses on Product data.
> Note: This document is a work in progress; information is subject to change.
> Contents:
> 1. Installation
> 2. Configuration
> 3. Data Indexing
> 4. Data Querying
> 5. Implementation Concerns
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> 1. Installation
> -----------------------------------------------------
> To install solr in an Ofbiz setting, simply extract the solr directory and
> and sub-folders to the hot-deploy folder.
> Afterward, the solr home system property (solr.solr.home) must be set to the value
> hot-deploy/solr manually using one of the following methods:
> * solr.solr.home in batch/script file:
> Add the parameter "-Dsolr.solr.home=hot-deploy/solr" to the Java command
> invocation for ofbiz.jar.
> e.g.:
> "%JAVA_HOME%\bin\java" -Xms128M -Xmx512M -XX:MaxPermSize=512m -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar
> * solr.solr.home in Ant build configuration:
> In your root Ofbiz build.xml file, add the element "<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>"
> to the "<java jar="ofbiz.jar"...>" invocation of the appropriate Ant target(s) (run, start, run-install, etc.).
> e.g.:
> <target name="start" description="Start OFBiz">
> 	<java jar="ofbiz.jar" fork="true">
> 		<jvmarg value="${memory.initial.param}"/>
> 		<jvmarg value="${memory.max.param}"/>
> 		<jvmarg value="${memory.maxpermsize.param}"/>
> 		<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>
> 	</java>
> </target>
> ***
> It may be possible to specify solr home using other methods (JNDI, web.xml), but at the time
> of this writing, this was the most reliable method known.
> -----------------------------------------------------
> 2. Configuration
> -----------------------------------------------------
> The solr component can run out-of-the-box without configuration, but many
> files, settings and interfaces allow custom settings. Some of these include:
> * Ofbiz configurations:
> ** System properties:
> *** ofbiz.solr.eca.enabled - Global solr ECA toggling boolean (true/false, see Data Indexing)
> ** Config files:
> *** solr/config/solrconfig.properties - Ofbiz solr service behavior control
> *** ofbiz-component.xml - Standard Ofbiz component config
> * Apache Solr configurations:
> ** System properties:
> *** solr.solr.home - Solr home (see Installation)
> ** Config files:
> *** solr.xml - Base solr config
> *** conf/schema.xml - Solr index schema
> *** webapp/WEB-INF/web.xml - Dual Ofbiz/Solr webapp config
> ** Interfaces:
> *** /solr/admin/ - Webapp admin interface (see below)
> ***
> It is possible to set extensive native Solr configuration using
> the admin webapp interface noted above. It should be accessible at the address:
> http://localhost:8080/solr/admin/
> (where 8080 is your server's http port)
> Please refer to the Apache Solr documentation for usage of this interface
> and other native Solr configuration details.
> -----------------------------------------------------
> 3. Data Indexing
> -----------------------------------------------------
> The solr component indexes data such as Products into the Apache Solr database
> using services defined in the file:
> servicesdef/solrservices.xml
> The initial indexing may need to be performed or scheduled manually, but subsequent indexing
> may be partially or fully automated, though automated methods are disabled by default and must be enabled.
> Note that in general, solr services can only successfully run in contexts where the solr webapp is 
> loaded and accessible.
> There are two methods for indexing data:
> * Index rebuilding service (rebuildSolrIndex):
> The rebuildSolrIndex is the most important data import service. It reindexes
> all Ofbiz Products existing in the system into the solr index. rebuildSolrIndex MUST be run 
> AT LEAST once after installation and also following any data load operation that loads new products using the Ofbiz 
> "install" starting mode (run-install, load-demo, etc.).
> To do this, one can simply use the Webtools backend to invoke the service manually
> (e.g., /webtools/control/setSyncServiceParameters?SERVICE_NAME=rebuildSolrIndex&POOL_NAME=pool&_RUN_SYNC_=Y)
> or summon the Webtools Job Scheduler.
> Once the initial indexing has been performed, one can then use the Job Scheduler to
> invoke rebuildSolrIndex on a regular basis (every hour, every midnight, etc.) to update the Solr index.
> * ECAs/SECAs (addToSolr, for Product data):
> Although the rebuildSolrIndex is always necessary for the initial data import, one may also
> use ECAs and SECAs to import subsequent data changes automatically at every individual data (e.g. Product)
> update instead of running rebuildSolrIndex periodically. This is done by defining ECAs or SECAs that
> trigger the addToSolr service.
> The addToSolr service simply accepts a single "instance" parameter, a GenericValue. At the time of this writing,
> any entity value having a valid "productId" field designating a Product value may be passed; this will trigger reindexing
> for the specific product.
> By default, the addToSolr service implementation (and any service intended for use with ECAs) is disabled globally and
> succeeds silently. It can be enabled by setting the "solr.eca.enabled=true" in "config/solrconfig.properties".
> This property is provided for easy toggling. It can also be specified on the command line or using ant through 
> the true/false "ofbiz.solr.eca.enabled" system property (which has priority over file-based "solr.eca.enabled") in the same
> way described for the solr home system property (see Installation); this allows toggling them per Ant target.
> Some stock/default/example ECAs are provided and are functional once enabled. They can be found in:
> entitydef/eecas.xml
> Custom ECAs may also be added to this file, and custom SECAs may be added to the file:
> servicedef/secas.xml
> Special care may need to be taken to ensure the addToSolr is not triggered during "install" sequences
> (run-install/load-demo) because it requires the solr webapp to be accessible. addToSolr will automatically
> attempt to prevent reindexing in cases where the solr webapp is unavailable. However, it is possible that
> this behavior results in needlessly prevented reindexing (and thus lost updates) in some unknown complex server
> configurations. If this is the case, this behavior can be disabled by setting the option
> "solr.eca.useSolrWebappLoadedCheck=false" in the properties file "config/solrconfig.properties".
>  
> However, if the above useSolrWebappLoadedCheck option is specified, you will have to manually disable or comment the 
> ECA definitions/services to prevent them from executing during data-load operations; the "solr.eca.enabled" property and "ofbiz.solr.eca.enabled" system property can be used for this purpose. One easy way is to set "solr.eca.enabled=true" in 
> the property files and then specify "-Dofbiz.solr.eca.enabled=false" for any relevant 
> Ant build install/run-install/load-demo targets.
> Note that simply commenting the ECAs/SECAs or preventing their inclusion using
> ofbiz-component.xml may eliminate some service call overhead compared to using the property toggling. Ideally,
> a method exploiting ofbiz-component.xml or eecas.xml to prevent ECAs from running in inappropriate contexts
> would be ideal; however, at the time of this writing, such a method could not be established.
> ***
> To reset and clear the solr index completely, it is sufficient to shut down the server and delete the folders:
> solr/data/index
> solr/data/spellchecker
> and any other such folders (must delete the whole folders, not only contents).
> -----------------------------------------------------
> 4. Data Querying
> -----------------------------------------------------
> Solr queries can be done using two methods:
> * Solr Ofbiz services:
> Simply invoke (manually or in code) the query services found in the file:
> servicesdef/solrservices.xml
> These include solrProductsSearch, solrKeywordSearch and others. Note that in general, 
> solr services can only successfully run in contexts where the solr webapp is loaded and accessible.
> * Solr native admin webapp interface:
> One can also perform native Solr queries and diagnostics using the standard
> admin interface, accessible as described under Configuration. Please refer
> to the Apache Solr documentation for usage of this interface.
> -----------------------------------------------------
> 5. Implementation Concerns
> -----------------------------------------------------
> * The structure of the solr/webapp directory closely mirrors the contents of the solr.war
> distribution; however, please note that it contains some Ofbiz-specific modifications such
> as those found in web.xml. The same is true for other folders such as solr/conf. Therefore,
> any library updates must be done with care to preserve these modifications.
> -----------------------------------------------------
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> * In general, solr services can only successfully run in contexts where the solr webapp 
> is loaded and accessible.
> * ECA indexing may be problematic due to data loading and unknown complex server configurations; 
> various toggling options are provided to work around this issue (see Data Indexing for details).
> In the future - and ideally - this could be addressed using ofbiz-component.xml (or eecas.xml) directives.
> * The indexing services often produce the log warning "Problem reading product features" due to their
> use of the stock Ofbiz getProductFeatureSet service.
> ***
> Please report any other issues encountered. Thank you.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (OFBIZ-5042) Apache Solr Implementation

["Paul Piper (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OFBIZ-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Paul Piper updated OFBIZ-5042:
------------------------------

    Description: 
Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community. With my agency, ilscipio, we have done several Apache Solr implementations in the past, and we felt that we could contribute this feature to the community. Currently, Solr is greatly used within syracus.net (our standalone store) and serves to generate the categories & search functionalities. From our team mainly Pascal Proulx, Jonatan Soto and I  were involved in this feature (albeit more having contributed to it. We created a standalone component which can be integrated into any running ofbiz implementation. It was tested against OFBiz 09.04 - 12.04.. 


=====================================================
Solr Component for Apache Ofbiz
=====================================================

This document describes the Ofbiz solr component, an Ofbiz (http://ofbiz.apache.org/) 
implementation of the Apache Solr search platform (http://lucene.apache.org/solr/).
The solr component includes an Ofbiz service-based wrapper layer to the Apache Solr
webapp queries as well as the native Apache Solr web interface itself.

Currently, the solr component focuses on Product data.

Note: This document is a work in progress; information is subject to change.

Contents:

1. Installation
2. Configuration
3. Data Indexing
4. Data Querying
5. Implementation Concerns
6. Known Bugs, Limitations and Issues


-----------------------------------------------------
1. Installation
-----------------------------------------------------

To install solr in an Ofbiz setting, simply extract the solr directory and
and sub-folders to the hot-deploy folder.

Afterward, the solr home system property (solr.solr.home) must be set to the value
hot-deploy/solr manually using one of the following methods:


* solr.solr.home in batch/script file:

Add the parameter "-Dsolr.solr.home=hot-deploy/solr" to the Java command
invocation for ofbiz.jar.
e.g.:
"%JAVA_HOME%\bin\java" -Xms128M -Xmx512M -XX:MaxPermSize=512m -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar


* solr.solr.home in Ant build configuration:

In your root Ofbiz build.xml file, add the element "<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>"
to the "<java jar="ofbiz.jar"...>" invocation of the appropriate Ant target(s) (run, start, run-install, etc.).
e.g.:
<target name="start" description="Start OFBiz">
	<java jar="ofbiz.jar" fork="true">
		<jvmarg value="${memory.initial.param}"/>
		<jvmarg value="${memory.max.param}"/>
		<jvmarg value="${memory.maxpermsize.param}"/>
		<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>
	</java>
</target>

***

It may be possible to specify solr home using other methods (JNDI, web.xml), but at the time
of this writing, this was the most reliable method known.


-----------------------------------------------------
2. Configuration
-----------------------------------------------------

The solr component can run out-of-the-box without configuration, but many
files, settings and interfaces allow custom settings. Some of these include:

* Ofbiz configurations:
** System properties:
*** ofbiz.solr.eca.enabled - Global solr ECA toggling boolean (true/false, see Data Indexing)
** Config files:
*** solr/config/solrconfig.properties - Ofbiz solr service behavior control
*** ofbiz-component.xml - Standard Ofbiz component config

* Apache Solr configurations:
** System properties:
*** solr.solr.home - Solr home (see Installation)
** Config files:
*** solr.xml - Base solr config
*** conf/schema.xml - Solr index schema
*** webapp/WEB-INF/web.xml - Dual Ofbiz/Solr webapp config
** Interfaces:
*** /solr/admin/ - Webapp admin interface (see below)

***

It is possible to set extensive native Solr configuration using
the admin webapp interface noted above. It should be accessible at the address:
http://localhost:8080/solr/admin/
(where 8080 is your server's http port)
Please refer to the Apache Solr documentation for usage of this interface
and other native Solr configuration details.


-----------------------------------------------------
3. Data Indexing
-----------------------------------------------------

The solr component indexes data such as Products into the Apache Solr database
using services defined in the file:
servicesdef/solrservices.xml
The initial indexing may need to be performed or scheduled manually, but subsequent indexing
may be partially or fully automated, though automated methods are disabled by default and must be enabled.
Note that in general, solr services can only successfully run in contexts where the solr webapp is 
loaded and accessible.

There are two methods for indexing data:


* Index rebuilding service (rebuildSolrIndex):

The rebuildSolrIndex is the most important data import service. It reindexes
all Ofbiz Products existing in the system into the solr index. rebuildSolrIndex MUST be run 
AT LEAST once after installation and also following any data load operation that loads new products using the Ofbiz 
"install" starting mode (run-install, load-demo, etc.).

To do this, one can simply use the Webtools backend to invoke the service manually
(e.g., /webtools/control/setSyncServiceParameters?SERVICE_NAME=rebuildSolrIndex&POOL_NAME=pool&_RUN_SYNC_=Y)
or summon the Webtools Job Scheduler.

Once the initial indexing has been performed, one can then use the Job Scheduler to
invoke rebuildSolrIndex on a regular basis (every hour, every midnight, etc.) to update the Solr index.


* ECAs/SECAs (addToSolr, for Product data):

Although the rebuildSolrIndex is always necessary for the initial data import, one may also
use ECAs and SECAs to import subsequent data changes automatically at every individual data (e.g. Product)
update instead of running rebuildSolrIndex periodically. This is done by defining ECAs or SECAs that
trigger the addToSolr service.

The addToSolr service simply accepts a single "instance" parameter, a GenericValue. At the time of this writing,
any entity value having a valid "productId" field designating a Product value may be passed; this will trigger reindexing
for the specific product.

By default, the addToSolr service implementation (and any service intended for use with ECAs) is disabled globally and
succeeds silently. It can be enabled by setting the "solr.eca.enabled=true" in "config/solrconfig.properties".
This property is provided for easy toggling. It can also be specified on the command line or using ant through 
the true/false "ofbiz.solr.eca.enabled" system property (which has priority over file-based "solr.eca.enabled") in the same
way described for the solr home system property (see Installation); this allows toggling them per Ant target.

Some stock/default/example ECAs are provided and are functional once enabled. They can be found in:
entitydef/eecas.xml
Custom ECAs may also be added to this file, and custom SECAs may be added to the file:
servicedef/secas.xml

Special care may need to be taken to ensure the addToSolr is not triggered during "install" sequences
(run-install/load-demo) because it requires the solr webapp to be accessible. addToSolr will automatically
attempt to prevent reindexing in cases where the solr webapp is unavailable. However, it is possible that
this behavior results in needlessly prevented reindexing (and thus lost updates) in some unknown complex server
configurations. If this is the case, this behavior can be disabled by setting the option
"solr.eca.useSolrWebappLoadedCheck=false" in the properties file "config/solrconfig.properties".
 
However, if the above useSolrWebappLoadedCheck option is specified, you will have to manually disable or comment the 
ECA definitions/services to prevent them from executing during data-load operations; the "solr.eca.enabled" property and "ofbiz.solr.eca.enabled" system property can be used for this purpose. One easy way is to set "solr.eca.enabled=true" in 
the property files and then specify "-Dofbiz.solr.eca.enabled=false" for any relevant 
Ant build install/run-install/load-demo targets.

Note that simply commenting the ECAs/SECAs or preventing their inclusion using
ofbiz-component.xml may eliminate some service call overhead compared to using the property toggling. Ideally,
a method exploiting ofbiz-component.xml or eecas.xml to prevent ECAs from running in inappropriate contexts
would be ideal; however, at the time of this writing, such a method could not be established.

***

To reset and clear the solr index completely, it is sufficient to shut down the server and delete the folders:
solr/data/index
solr/data/spellchecker
and any other such folders (must delete the whole folders, not only contents).


-----------------------------------------------------
4. Data Querying
-----------------------------------------------------

Solr queries can be done using two methods:


* Solr Ofbiz services:

Simply invoke (manually or in code) the query services found in the file:
servicesdef/solrservices.xml
These include solrProductsSearch, solrKeywordSearch and others. Note that in general, 
solr services can only successfully run in contexts where the solr webapp is loaded and accessible.


* Solr native admin webapp interface:

One can also perform native Solr queries and diagnostics using the standard
admin interface, accessible as described under Configuration. Please refer
to the Apache Solr documentation for usage of this interface.


-----------------------------------------------------
5. Implementation Concerns
-----------------------------------------------------

* The structure of the solr/webapp directory closely mirrors the contents of the solr.war
distribution; however, please note that it contains some Ofbiz-specific modifications such
as those found in web.xml. The same is true for other folders such as solr/conf. Therefore,
any library updates must be done with care to preserve these modifications.


-----------------------------------------------------
6. Known Bugs, Limitations and Issues
-----------------------------------------------------

* In general, solr services can only successfully run in contexts where the solr webapp 
is loaded and accessible.
* ECA indexing may be problematic due to data loading and unknown complex server configurations; 
various toggling options are provided to work around this issue (see Data Indexing for details).
In the future - and ideally - this could be addressed using ofbiz-component.xml (or eecas.xml) directives.
* The indexing services often produce the log warning "Problem reading product features" due to their
use of the stock Ofbiz getProductFeatureSet service.

***

Please report any other issues encountered. Thank you.




  was:
Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community. With my agency, ilscipio, we have done several Apache Solr implementations in the past, and we felt that we could contribute this feature to the community. Currently, Solr is greatly used within syracus.net (our standalone store) and serves to generate the categories & search functionalities. From our team mainly Pascal Proulx, Jonatan Soto and I were involved in this feature (albeit more having contributed to it. We created a standalone component which can be integrated into any running ofbiz implementation. It was tested against OFBiz 09.04 - 12.04.. 


=====================================================
Solr Component for Apache Ofbiz
=====================================================

This document describes the Ofbiz solr component, an Ofbiz (http://ofbiz.apache.org/) 
implementation of the Apache Solr search platform (http://lucene.apache.org/solr/).
The solr component includes an Ofbiz service-based wrapper layer to the Apache Solr
webapp queries as well as the native Apache Solr web interface itself.

Currently, the solr component focuses on Product data.

Note: This document is a work in progress; information is subject to change.

Contents:

1. Installation
2. Configuration
3. Data Indexing
4. Data Querying
5. Implementation Concerns
6. Known Bugs, Limitations and Issues


-----------------------------------------------------
1. Installation
-----------------------------------------------------

To install solr in an Ofbiz setting, simply extract the solr directory and
and sub-folders to the hot-deploy folder.

Afterward, the solr home system property (solr.solr.home) must be set to the value
hot-deploy/solr manually using one of the following methods:


* solr.solr.home in batch/script file:

Add the parameter "-Dsolr.solr.home=hot-deploy/solr" to the Java command
invocation for ofbiz.jar.
e.g.:
"%JAVA_HOME%\bin\java" -Xms128M -Xmx512M -XX:MaxPermSize=512m -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar


* solr.solr.home in Ant build configuration:

In your root Ofbiz build.xml file, add the element "<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>"
to the "<java jar="ofbiz.jar"...>" invocation of the appropriate Ant target(s) (run, start, run-install, etc.).
e.g.:
<target name="start" description="Start OFBiz">
	<java jar="ofbiz.jar" fork="true">
		<jvmarg value="${memory.initial.param}"/>
		<jvmarg value="${memory.max.param}"/>
		<jvmarg value="${memory.maxpermsize.param}"/>
		<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>
	</java>
</target>

***

It may be possible to specify solr home using other methods (JNDI, web.xml), but at the time
of this writing, this was the most reliable method known.


-----------------------------------------------------
2. Configuration
-----------------------------------------------------

The solr component can run out-of-the-box without configuration, but many
files, settings and interfaces allow custom settings. Some of these include:

* Ofbiz configurations:
** System properties:
*** ofbiz.solr.eca.enabled - Global solr ECA toggling boolean (true/false, see Data Indexing)
** Config files:
*** solr/config/solrconfig.properties - Ofbiz solr service behavior control
*** ofbiz-component.xml - Standard Ofbiz component config

* Apache Solr configurations:
** System properties:
*** solr.solr.home - Solr home (see Installation)
** Config files:
*** solr.xml - Base solr config
*** conf/schema.xml - Solr index schema
*** webapp/WEB-INF/web.xml - Dual Ofbiz/Solr webapp config
** Interfaces:
*** /solr/admin/ - Webapp admin interface (see below)

***

It is possible to set extensive native Solr configuration using
the admin webapp interface noted above. It should be accessible at the address:
http://localhost:8080/solr/admin/
(where 8080 is your server's http port)
Please refer to the Apache Solr documentation for usage of this interface
and other native Solr configuration details.


-----------------------------------------------------
3. Data Indexing
-----------------------------------------------------

The solr component indexes data such as Products into the Apache Solr database
using services defined in the file:
servicesdef/solrservices.xml
The initial indexing may need to be performed or scheduled manually, but subsequent indexing
may be partially or fully automated, though automated methods are disabled by default and must be enabled.
Note that in general, solr services can only successfully run in contexts where the solr webapp is 
loaded and accessible.

There are two methods for indexing data:


* Index rebuilding service (rebuildSolrIndex):

The rebuildSolrIndex is the most important data import service. It reindexes
all Ofbiz Products existing in the system into the solr index. rebuildSolrIndex MUST be run 
AT LEAST once after installation and also following any data load operation that loads new products using the Ofbiz 
"install" starting mode (run-install, load-demo, etc.).

To do this, one can simply use the Webtools backend to invoke the service manually
(e.g., /webtools/control/setSyncServiceParameters?SERVICE_NAME=rebuildSolrIndex&POOL_NAME=pool&_RUN_SYNC_=Y)
or summon the Webtools Job Scheduler.

Once the initial indexing has been performed, one can then use the Job Scheduler to
invoke rebuildSolrIndex on a regular basis (every hour, every midnight, etc.) to update the Solr index.


* ECAs/SECAs (addToSolr, for Product data):

Although the rebuildSolrIndex is always necessary for the initial data import, one may also
use ECAs and SECAs to import subsequent data changes automatically at every individual data (e.g. Product)
update instead of running rebuildSolrIndex periodically. This is done by defining ECAs or SECAs that
trigger the addToSolr service.

The addToSolr service simply accepts a single "instance" parameter, a GenericValue. At the time of this writing,
any entity value having a valid "productId" field designating a Product value may be passed; this will trigger reindexing
for the specific product.

By default, the addToSolr service implementation (and any service intended for use with ECAs) is disabled globally and
succeeds silently. It can be enabled by setting the "solr.eca.enabled=true" in "config/solrconfig.properties".
This property is provided for easy toggling. It can also be specified on the command line or using ant through 
the true/false "ofbiz.solr.eca.enabled" system property (which has priority over file-based "solr.eca.enabled") in the same
way described for the solr home system property (see Installation); this allows toggling them per Ant target.

Some stock/default/example ECAs are provided and are functional once enabled. They can be found in:
entitydef/eecas.xml
Custom ECAs may also be added to this file, and custom SECAs may be added to the file:
servicedef/secas.xml

Special care may need to be taken to ensure the addToSolr is not triggered during "install" sequences
(run-install/load-demo) because it requires the solr webapp to be accessible. addToSolr will automatically
attempt to prevent reindexing in cases where the solr webapp is unavailable. However, it is possible that
this behavior results in needlessly prevented reindexing (and thus lost updates) in some unknown complex server
configurations. If this is the case, this behavior can be disabled by setting the option
"solr.eca.useSolrWebappLoadedCheck=false" in the properties file "config/solrconfig.properties".
 
However, if the above useSolrWebappLoadedCheck option is specified, you will have to manually disable or comment the 
ECA definitions/services to prevent them from executing during data-load operations; the "solr.eca.enabled" property and "ofbiz.solr.eca.enabled" system property can be used for this purpose. One easy way is to set "solr.eca.enabled=true" in 
the property files and then specify "-Dofbiz.solr.eca.enabled=false" for any relevant 
Ant build install/run-install/load-demo targets.

Note that simply commenting the ECAs/SECAs or preventing their inclusion using
ofbiz-component.xml may eliminate some service call overhead compared to using the property toggling. Ideally,
a method exploiting ofbiz-component.xml or eecas.xml to prevent ECAs from running in inappropriate contexts
would be ideal; however, at the time of this writing, such a method could not be established.

***

To reset and clear the solr index completely, it is sufficient to shut down the server and delete the folders:
solr/data/index
solr/data/spellchecker
and any other such folders (must delete the whole folders, not only contents).


-----------------------------------------------------
4. Data Querying
-----------------------------------------------------

Solr queries can be done using two methods:


* Solr Ofbiz services:

Simply invoke (manually or in code) the query services found in the file:
servicesdef/solrservices.xml
These include solrProductsSearch, solrKeywordSearch and others. Note that in general, 
solr services can only successfully run in contexts where the solr webapp is loaded and accessible.


* Solr native admin webapp interface:

One can also perform native Solr queries and diagnostics using the standard
admin interface, accessible as described under Configuration. Please refer
to the Apache Solr documentation for usage of this interface.


-----------------------------------------------------
5. Implementation Concerns
-----------------------------------------------------

* The structure of the solr/webapp directory closely mirrors the contents of the solr.war
distribution; however, please note that it contains some Ofbiz-specific modifications such
as those found in web.xml. The same is true for other folders such as solr/conf. Therefore,
any library updates must be done with care to preserve these modifications.


-----------------------------------------------------
6. Known Bugs, Limitations and Issues
-----------------------------------------------------

* In general, solr services can only successfully run in contexts where the solr webapp 
is loaded and accessible.
* ECA indexing may be problematic due to data loading and unknown complex server configurations; 
various toggling options are provided to work around this issue (see Data Indexing for details).
In the future - and ideally - this could be addressed using ofbiz-component.xml (or eecas.xml) directives.
* The indexing services often produce the log warning "Problem reading product features" due to their
use of the stock Ofbiz getProductFeatureSet service.

***

Please report any other issues encountered. Thank you.




    
> Apache Solr Implementation
> --------------------------
>
>                 Key: OFBIZ-5042
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-5042
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: ALL COMPONENTS
>            Reporter: Paul Piper
>              Labels: component, feature, solr
>         Attachments: solr-v1.00-r4284.zip
>
>
> Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community. With my agency, ilscipio, we have done several Apache Solr implementations in the past, and we felt that we could contribute this feature to the community. Currently, Solr is greatly used within syracus.net (our standalone store) and serves to generate the categories & search functionalities. From our team mainly Pascal Proulx, Jonatan Soto and I  were involved in this feature (albeit more having contributed to it. We created a standalone component which can be integrated into any running ofbiz implementation. It was tested against OFBiz 09.04 - 12.04.. 
> =====================================================
> Solr Component for Apache Ofbiz
> =====================================================
> This document describes the Ofbiz solr component, an Ofbiz (http://ofbiz.apache.org/) 
> implementation of the Apache Solr search platform (http://lucene.apache.org/solr/).
> The solr component includes an Ofbiz service-based wrapper layer to the Apache Solr
> webapp queries as well as the native Apache Solr web interface itself.
> Currently, the solr component focuses on Product data.
> Note: This document is a work in progress; information is subject to change.
> Contents:
> 1. Installation
> 2. Configuration
> 3. Data Indexing
> 4. Data Querying
> 5. Implementation Concerns
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> 1. Installation
> -----------------------------------------------------
> To install solr in an Ofbiz setting, simply extract the solr directory and
> and sub-folders to the hot-deploy folder.
> Afterward, the solr home system property (solr.solr.home) must be set to the value
> hot-deploy/solr manually using one of the following methods:
> * solr.solr.home in batch/script file:
> Add the parameter "-Dsolr.solr.home=hot-deploy/solr" to the Java command
> invocation for ofbiz.jar.
> e.g.:
> "%JAVA_HOME%\bin\java" -Xms128M -Xmx512M -XX:MaxPermSize=512m -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar
> * solr.solr.home in Ant build configuration:
> In your root Ofbiz build.xml file, add the element "<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>"
> to the "<java jar="ofbiz.jar"...>" invocation of the appropriate Ant target(s) (run, start, run-install, etc.).
> e.g.:
> <target name="start" description="Start OFBiz">
> 	<java jar="ofbiz.jar" fork="true">
> 		<jvmarg value="${memory.initial.param}"/>
> 		<jvmarg value="${memory.max.param}"/>
> 		<jvmarg value="${memory.maxpermsize.param}"/>
> 		<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>
> 	</java>
> </target>
> ***
> It may be possible to specify solr home using other methods (JNDI, web.xml), but at the time
> of this writing, this was the most reliable method known.
> -----------------------------------------------------
> 2. Configuration
> -----------------------------------------------------
> The solr component can run out-of-the-box without configuration, but many
> files, settings and interfaces allow custom settings. Some of these include:
> * Ofbiz configurations:
> ** System properties:
> *** ofbiz.solr.eca.enabled - Global solr ECA toggling boolean (true/false, see Data Indexing)
> ** Config files:
> *** solr/config/solrconfig.properties - Ofbiz solr service behavior control
> *** ofbiz-component.xml - Standard Ofbiz component config
> * Apache Solr configurations:
> ** System properties:
> *** solr.solr.home - Solr home (see Installation)
> ** Config files:
> *** solr.xml - Base solr config
> *** conf/schema.xml - Solr index schema
> *** webapp/WEB-INF/web.xml - Dual Ofbiz/Solr webapp config
> ** Interfaces:
> *** /solr/admin/ - Webapp admin interface (see below)
> ***
> It is possible to set extensive native Solr configuration using
> the admin webapp interface noted above. It should be accessible at the address:
> http://localhost:8080/solr/admin/
> (where 8080 is your server's http port)
> Please refer to the Apache Solr documentation for usage of this interface
> and other native Solr configuration details.
> -----------------------------------------------------
> 3. Data Indexing
> -----------------------------------------------------
> The solr component indexes data such as Products into the Apache Solr database
> using services defined in the file:
> servicesdef/solrservices.xml
> The initial indexing may need to be performed or scheduled manually, but subsequent indexing
> may be partially or fully automated, though automated methods are disabled by default and must be enabled.
> Note that in general, solr services can only successfully run in contexts where the solr webapp is 
> loaded and accessible.
> There are two methods for indexing data:
> * Index rebuilding service (rebuildSolrIndex):
> The rebuildSolrIndex is the most important data import service. It reindexes
> all Ofbiz Products existing in the system into the solr index. rebuildSolrIndex MUST be run 
> AT LEAST once after installation and also following any data load operation that loads new products using the Ofbiz 
> "install" starting mode (run-install, load-demo, etc.).
> To do this, one can simply use the Webtools backend to invoke the service manually
> (e.g., /webtools/control/setSyncServiceParameters?SERVICE_NAME=rebuildSolrIndex&POOL_NAME=pool&_RUN_SYNC_=Y)
> or summon the Webtools Job Scheduler.
> Once the initial indexing has been performed, one can then use the Job Scheduler to
> invoke rebuildSolrIndex on a regular basis (every hour, every midnight, etc.) to update the Solr index.
> * ECAs/SECAs (addToSolr, for Product data):
> Although the rebuildSolrIndex is always necessary for the initial data import, one may also
> use ECAs and SECAs to import subsequent data changes automatically at every individual data (e.g. Product)
> update instead of running rebuildSolrIndex periodically. This is done by defining ECAs or SECAs that
> trigger the addToSolr service.
> The addToSolr service simply accepts a single "instance" parameter, a GenericValue. At the time of this writing,
> any entity value having a valid "productId" field designating a Product value may be passed; this will trigger reindexing
> for the specific product.
> By default, the addToSolr service implementation (and any service intended for use with ECAs) is disabled globally and
> succeeds silently. It can be enabled by setting the "solr.eca.enabled=true" in "config/solrconfig.properties".
> This property is provided for easy toggling. It can also be specified on the command line or using ant through 
> the true/false "ofbiz.solr.eca.enabled" system property (which has priority over file-based "solr.eca.enabled") in the same
> way described for the solr home system property (see Installation); this allows toggling them per Ant target.
> Some stock/default/example ECAs are provided and are functional once enabled. They can be found in:
> entitydef/eecas.xml
> Custom ECAs may also be added to this file, and custom SECAs may be added to the file:
> servicedef/secas.xml
> Special care may need to be taken to ensure the addToSolr is not triggered during "install" sequences
> (run-install/load-demo) because it requires the solr webapp to be accessible. addToSolr will automatically
> attempt to prevent reindexing in cases where the solr webapp is unavailable. However, it is possible that
> this behavior results in needlessly prevented reindexing (and thus lost updates) in some unknown complex server
> configurations. If this is the case, this behavior can be disabled by setting the option
> "solr.eca.useSolrWebappLoadedCheck=false" in the properties file "config/solrconfig.properties".
>  
> However, if the above useSolrWebappLoadedCheck option is specified, you will have to manually disable or comment the 
> ECA definitions/services to prevent them from executing during data-load operations; the "solr.eca.enabled" property and "ofbiz.solr.eca.enabled" system property can be used for this purpose. One easy way is to set "solr.eca.enabled=true" in 
> the property files and then specify "-Dofbiz.solr.eca.enabled=false" for any relevant 
> Ant build install/run-install/load-demo targets.
> Note that simply commenting the ECAs/SECAs or preventing their inclusion using
> ofbiz-component.xml may eliminate some service call overhead compared to using the property toggling. Ideally,
> a method exploiting ofbiz-component.xml or eecas.xml to prevent ECAs from running in inappropriate contexts
> would be ideal; however, at the time of this writing, such a method could not be established.
> ***
> To reset and clear the solr index completely, it is sufficient to shut down the server and delete the folders:
> solr/data/index
> solr/data/spellchecker
> and any other such folders (must delete the whole folders, not only contents).
> -----------------------------------------------------
> 4. Data Querying
> -----------------------------------------------------
> Solr queries can be done using two methods:
> * Solr Ofbiz services:
> Simply invoke (manually or in code) the query services found in the file:
> servicesdef/solrservices.xml
> These include solrProductsSearch, solrKeywordSearch and others. Note that in general, 
> solr services can only successfully run in contexts where the solr webapp is loaded and accessible.
> * Solr native admin webapp interface:
> One can also perform native Solr queries and diagnostics using the standard
> admin interface, accessible as described under Configuration. Please refer
> to the Apache Solr documentation for usage of this interface.
> -----------------------------------------------------
> 5. Implementation Concerns
> -----------------------------------------------------
> * The structure of the solr/webapp directory closely mirrors the contents of the solr.war
> distribution; however, please note that it contains some Ofbiz-specific modifications such
> as those found in web.xml. The same is true for other folders such as solr/conf. Therefore,
> any library updates must be done with care to preserve these modifications.
> -----------------------------------------------------
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> * In general, solr services can only successfully run in contexts where the solr webapp 
> is loaded and accessible.
> * ECA indexing may be problematic due to data loading and unknown complex server configurations; 
> various toggling options are provided to work around this issue (see Data Indexing for details).
> In the future - and ideally - this could be addressed using ofbiz-component.xml (or eecas.xml) directives.
> * The indexing services often produce the log warning "Problem reading product features" due to their
> use of the stock Ofbiz getProductFeatureSet service.
> ***
> Please report any other issues encountered. Thank you.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (OFBIZ-5042) Apache Solr Implementation

["Jacques Le Roux (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OFBIZ-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470955#comment-13470955 ] 

Jacques Le Roux edited comment on OFBIZ-5042 at 10/6/12 9:05 AM:
-----------------------------------------------------------------

==== BETTER COMMENT: WE == COMMUNITY ====

Since the community decided to move most specialpurpose components to Apache Extra ([see slim down effort|https://issues.apache.org/jira/browse/OFBIZ/fixforversion/12320551]), the best place for this component is in [Apache OFBiz Extra|http://code.google.com/a/apache-extras.org/hosting/search?q=label%3aOFBiz]. Please read [Apache Extra guidelines|http://community.apache.org/apache-extras/guidelines.html]. 

Note the point 6:
{quote}
Each Apache Extras project is encouraged to publish their releases and complete the project tagging information available on Apache Extras. 
{quote}

As explained in the guidelines, The Apache OFBiz Extra projects will not be official part of the Apache OFBiz project. The goals are pragmatic (at least as I see it):
* Components which are not mandatory (not supposed to be used in each project powered by OFBiz) can be moved to Apache Extra. 
* With limited Apache OFBiz project human resources, it's a way to delegate the code and releases responsabilities to specific communities

But this does not mean that Apache OFBiz Extra projects will be forgotten or neglected by the Apache OFBiz project. Now I feel something is missing in the scheme so far: how to relate an Apache OFBiz Extra project release to an Apache OFBiz release? I think it should part of the Apache OFBiz Extra project community to indicate which OFBiz release/s (trunk included) are compatible with the component. This is kind of part of the guidelines point 6 in my opininon. This is correctly maintained could be used by the Apache OFBiz project when releasing, we could at least have a link to the [Apache OFBiz Extra projects page|http://code.google.com/a/apache-extras.org/hosting/search?q=label%3aOFBiz]. Same from the [Apache OFBiz project site main page|http://ofbiz.apache.org/], a reference should be added. I will do so soon, but will wait that at least a complete usable Apache OFBiz Extra project exists there. The Neogia team made a first effor but nothing was committed yet, the Solr component could be the first to engage the movement. With the ApacheConEu coming it could be also a good way to share about this point.

Also I'd like to have a way to know the value of an Apache OFBiz Extra project, as experienced by its users. For the moment I see only one way to do this: recommend it on Google + (see in each component Project information). Of course this is a very limited way, but at least a mean.

So it would be great if you Paul, Pascal and Jonatan (at least for now, the community will hopefully grow from comments above) could move the Solr component into the Apache OFBiz Extra projects. This would begin the dynamic, completing the pending work initiated by the Neogia team. Hopefully more will come later. For instance I read a lof about Big Fish, why not sharing in Apache OFBiz Extra projects also?
                
      was (Author: jacques.le.roux):
    == BETTER COMMENT: WE == COMMUNITY ==
Since the community decided to move most specialpurpose components to Apache Extra ([see slim down effort|https://issues.apache.org/jira/browse/OFBIZ/fixforversion/12320551]), the best place for this component is in [Apache OFBiz Extra|http://code.google.com/a/apache-extras.org/hosting/search?q=label%3aOFBiz]. Please read [Apache Extra guidelines|http://community.apache.org/apache-extras/guidelines.html]. 

Note the point 6:
{quote}
Each Apache Extras project is encouraged to publish their releases and complete the project tagging information available on Apache Extras. 
{quote}

As explained in the guidelines, The Apache OFBiz Extra projects will not be official part of the Apache OFBiz project. The goals are pragmatic (at least as I see it):
* Components which are not mandatory (not supposed to be used in each project powered by OFBiz) can be moved to Apache Extra. 
* With limited Apache OFBiz project human resources, it's a way to delegate the code and releases responsabilities to specific communities

But this does not mean that Apache OFBiz Extra projects will be forgotten or neglected by the Apache OFBiz project. Now I feel something is missing in the scheme so far: how to relate an Apache OFBiz Extra project release to an Apache OFBiz release? I think it should part of the Apache OFBiz Extra project community to indicate which OFBiz release/s (trunk included) are compatible with the component. This is kind of part of the guidelines point 6 in my opininon. This is correctly maintained could be used by the Apache OFBiz project when releasing, we could at least have a link to the [Apache OFBiz Extra projects page|http://code.google.com/a/apache-extras.org/hosting/search?q=label%3aOFBiz]. Same from the [Apache OFBiz project site main page|http://ofbiz.apache.org/], a reference should be added. I will do so soon, but will wait that at least a complete usable Apache OFBiz Extra project exists there. The Neogia team made a first effor but nothing was committed yet, the Solr component could be the first to engage the movement. With the ApacheConEu coming it could be also a good way to share about this point.

Also I'd like to have a way to know the value of an Apache OFBiz Extra project, as experienced by its users. For the moment I see only one way to do this: recommend it on Google + (see in each component Project information). Of course this is a very limited way, but at least a mean.

So it would be great if you Paul, Pascal and Jonatan (at least for now, the community will hopefully grow from comments above) could move the Solr component into the Apache OFBiz Extra projects. This would begin the dynamic, completing the pending work initiated by the Neogia team. Hopefully more will come later. For instance I read a lof about Big Fish, why not sharing in Apache OFBiz Extra projects also?
                  
> Apache Solr Implementation
> --------------------------
>
>                 Key: OFBIZ-5042
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-5042
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: ALL COMPONENTS
>            Reporter: Paul Piper
>              Labels: component, feature, solr
>         Attachments: solr-v1.00-r4284.zip
>
>
> Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community. With my agency, ilscipio, we have done several Apache Solr implementations in the past, and we felt that we could contribute this feature to the community. Currently, Solr is greatly used within syracus.net (our standalone store) and serves to generate the categories & search functionalities. From our team mainly Pascal Proulx, Jonatan Soto and I  were involved in this feature (albeit more having contributed to it. We created a standalone component which can be integrated into any running ofbiz implementation. It was tested against OFBiz 09.04 - 12.04.. 
> =====================================================
> Solr Component for Apache Ofbiz
> =====================================================
> This document describes the Ofbiz solr component, an Ofbiz (http://ofbiz.apache.org/) 
> implementation of the Apache Solr search platform (http://lucene.apache.org/solr/).
> The solr component includes an Ofbiz service-based wrapper layer to the Apache Solr
> webapp queries as well as the native Apache Solr web interface itself.
> Currently, the solr component focuses on Product data.
> Note: This document is a work in progress; information is subject to change.
> Contents:
> 1. Installation
> 2. Configuration
> 3. Data Indexing
> 4. Data Querying
> 5. Implementation Concerns
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> 1. Installation
> -----------------------------------------------------
> To install solr in an Ofbiz setting, simply extract the solr directory and
> and sub-folders to the hot-deploy folder.
> Afterward, the solr home system property (solr.solr.home) must be set to the value
> hot-deploy/solr manually using one of the following methods:
> * solr.solr.home in batch/script file:
> Add the parameter "-Dsolr.solr.home=hot-deploy/solr" to the Java command
> invocation for ofbiz.jar.
> e.g.:
> "%JAVA_HOME%\bin\java" -Xms128M -Xmx512M -XX:MaxPermSize=512m -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar
> * solr.solr.home in Ant build configuration:
> In your root Ofbiz build.xml file, add the element "<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>"
> to the "<java jar="ofbiz.jar"...>" invocation of the appropriate Ant target(s) (run, start, run-install, etc.).
> e.g.:
> <target name="start" description="Start OFBiz">
> 	<java jar="ofbiz.jar" fork="true">
> 		<jvmarg value="${memory.initial.param}"/>
> 		<jvmarg value="${memory.max.param}"/>
> 		<jvmarg value="${memory.maxpermsize.param}"/>
> 		<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>
> 	</java>
> </target>
> ***
> It may be possible to specify solr home using other methods (JNDI, web.xml), but at the time
> of this writing, this was the most reliable method known.
> -----------------------------------------------------
> 2. Configuration
> -----------------------------------------------------
> The solr component can run out-of-the-box without configuration, but many
> files, settings and interfaces allow custom settings. Some of these include:
> * Ofbiz configurations:
> ** System properties:
> *** ofbiz.solr.eca.enabled - Global solr ECA toggling boolean (true/false, see Data Indexing)
> ** Config files:
> *** solr/config/solrconfig.properties - Ofbiz solr service behavior control
> *** ofbiz-component.xml - Standard Ofbiz component config
> * Apache Solr configurations:
> ** System properties:
> *** solr.solr.home - Solr home (see Installation)
> ** Config files:
> *** solr.xml - Base solr config
> *** conf/schema.xml - Solr index schema
> *** webapp/WEB-INF/web.xml - Dual Ofbiz/Solr webapp config
> ** Interfaces:
> *** /solr/admin/ - Webapp admin interface (see below)
> ***
> It is possible to set extensive native Solr configuration using
> the admin webapp interface noted above. It should be accessible at the address:
> http://localhost:8080/solr/admin/
> (where 8080 is your server's http port)
> Please refer to the Apache Solr documentation for usage of this interface
> and other native Solr configuration details.
> -----------------------------------------------------
> 3. Data Indexing
> -----------------------------------------------------
> The solr component indexes data such as Products into the Apache Solr database
> using services defined in the file:
> servicesdef/solrservices.xml
> The initial indexing may need to be performed or scheduled manually, but subsequent indexing
> may be partially or fully automated, though automated methods are disabled by default and must be enabled.
> Note that in general, solr services can only successfully run in contexts where the solr webapp is 
> loaded and accessible.
> There are two methods for indexing data:
> * Index rebuilding service (rebuildSolrIndex):
> The rebuildSolrIndex is the most important data import service. It reindexes
> all Ofbiz Products existing in the system into the solr index. rebuildSolrIndex MUST be run 
> AT LEAST once after installation and also following any data load operation that loads new products using the Ofbiz 
> "install" starting mode (run-install, load-demo, etc.).
> To do this, one can simply use the Webtools backend to invoke the service manually
> (e.g., /webtools/control/setSyncServiceParameters?SERVICE_NAME=rebuildSolrIndex&POOL_NAME=pool&_RUN_SYNC_=Y)
> or summon the Webtools Job Scheduler.
> Once the initial indexing has been performed, one can then use the Job Scheduler to
> invoke rebuildSolrIndex on a regular basis (every hour, every midnight, etc.) to update the Solr index.
> * ECAs/SECAs (addToSolr, for Product data):
> Although the rebuildSolrIndex is always necessary for the initial data import, one may also
> use ECAs and SECAs to import subsequent data changes automatically at every individual data (e.g. Product)
> update instead of running rebuildSolrIndex periodically. This is done by defining ECAs or SECAs that
> trigger the addToSolr service.
> The addToSolr service simply accepts a single "instance" parameter, a GenericValue. At the time of this writing,
> any entity value having a valid "productId" field designating a Product value may be passed; this will trigger reindexing
> for the specific product.
> By default, the addToSolr service implementation (and any service intended for use with ECAs) is disabled globally and
> succeeds silently. It can be enabled by setting the "solr.eca.enabled=true" in "config/solrconfig.properties".
> This property is provided for easy toggling. It can also be specified on the command line or using ant through 
> the true/false "ofbiz.solr.eca.enabled" system property (which has priority over file-based "solr.eca.enabled") in the same
> way described for the solr home system property (see Installation); this allows toggling them per Ant target.
> Some stock/default/example ECAs are provided and are functional once enabled. They can be found in:
> entitydef/eecas.xml
> Custom ECAs may also be added to this file, and custom SECAs may be added to the file:
> servicedef/secas.xml
> Special care may need to be taken to ensure the addToSolr is not triggered during "install" sequences
> (run-install/load-demo) because it requires the solr webapp to be accessible. addToSolr will automatically
> attempt to prevent reindexing in cases where the solr webapp is unavailable. However, it is possible that
> this behavior results in needlessly prevented reindexing (and thus lost updates) in some unknown complex server
> configurations. If this is the case, this behavior can be disabled by setting the option
> "solr.eca.useSolrWebappLoadedCheck=false" in the properties file "config/solrconfig.properties".
>  
> However, if the above useSolrWebappLoadedCheck option is specified, you will have to manually disable or comment the 
> ECA definitions/services to prevent them from executing during data-load operations; the "solr.eca.enabled" property and "ofbiz.solr.eca.enabled" system property can be used for this purpose. One easy way is to set "solr.eca.enabled=true" in 
> the property files and then specify "-Dofbiz.solr.eca.enabled=false" for any relevant 
> Ant build install/run-install/load-demo targets.
> Note that simply commenting the ECAs/SECAs or preventing their inclusion using
> ofbiz-component.xml may eliminate some service call overhead compared to using the property toggling. Ideally,
> a method exploiting ofbiz-component.xml or eecas.xml to prevent ECAs from running in inappropriate contexts
> would be ideal; however, at the time of this writing, such a method could not be established.
> ***
> To reset and clear the solr index completely, it is sufficient to shut down the server and delete the folders:
> solr/data/index
> solr/data/spellchecker
> and any other such folders (must delete the whole folders, not only contents).
> -----------------------------------------------------
> 4. Data Querying
> -----------------------------------------------------
> Solr queries can be done using two methods:
> * Solr Ofbiz services:
> Simply invoke (manually or in code) the query services found in the file:
> servicesdef/solrservices.xml
> These include solrProductsSearch, solrKeywordSearch and others. Note that in general, 
> solr services can only successfully run in contexts where the solr webapp is loaded and accessible.
> * Solr native admin webapp interface:
> One can also perform native Solr queries and diagnostics using the standard
> admin interface, accessible as described under Configuration. Please refer
> to the Apache Solr documentation for usage of this interface.
> -----------------------------------------------------
> 5. Implementation Concerns
> -----------------------------------------------------
> * The structure of the solr/webapp directory closely mirrors the contents of the solr.war
> distribution; however, please note that it contains some Ofbiz-specific modifications such
> as those found in web.xml. The same is true for other folders such as solr/conf. Therefore,
> any library updates must be done with care to preserve these modifications.
> -----------------------------------------------------
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> * In general, solr services can only successfully run in contexts where the solr webapp 
> is loaded and accessible.
> * ECA indexing may be problematic due to data loading and unknown complex server configurations; 
> various toggling options are provided to work around this issue (see Data Indexing for details).
> In the future - and ideally - this could be addressed using ofbiz-component.xml (or eecas.xml) directives.
> * The indexing services often produce the log warning "Problem reading product features" due to their
> use of the stock Ofbiz getProductFeatureSet service.
> ***
> Please report any other issues encountered. Thank you.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (OFBIZ-5042) Apache Solr Implementation

["Paul Piper (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OFBIZ-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499376#comment-13499376 ] 

Paul Piper commented on OFBIZ-5042:
-----------------------------------

I missed your point on Apache Extra. With the discussion still ongoing I think we should wait before migrating this anywhere.
                
> Apache Solr Implementation
> --------------------------
>
>                 Key: OFBIZ-5042
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-5042
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: ALL COMPONENTS
>            Reporter: Paul Piper
>              Labels: component, feature, solr
>         Attachments: solr-v1.00-r4284.zip
>
>
> Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community. With my agency, ilscipio, we have done several Apache Solr implementations in the past, and we felt that we could contribute this feature to the community. Currently, Solr is greatly used within syracus.net (our standalone store) and serves to generate the categories & search functionalities. From our team mainly Pascal Proulx, Jonatan Soto and I  were involved in this feature (albeit more having contributed to it. We created a standalone component which can be integrated into any running ofbiz implementation. It was tested against OFBiz 09.04 - 12.04.. 
> =====================================================
> Solr Component for Apache Ofbiz
> =====================================================
> This document describes the Ofbiz solr component, an Ofbiz (http://ofbiz.apache.org/) 
> implementation of the Apache Solr search platform (http://lucene.apache.org/solr/).
> The solr component includes an Ofbiz service-based wrapper layer to the Apache Solr
> webapp queries as well as the native Apache Solr web interface itself.
> Currently, the solr component focuses on Product data.
> Note: This document is a work in progress; information is subject to change.
> Contents:
> 1. Installation
> 2. Configuration
> 3. Data Indexing
> 4. Data Querying
> 5. Implementation Concerns
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> 1. Installation
> -----------------------------------------------------
> To install solr in an Ofbiz setting, simply extract the solr directory and
> and sub-folders to the hot-deploy folder.
> Afterward, the solr home system property (solr.solr.home) must be set to the value
> hot-deploy/solr manually using one of the following methods:
> * solr.solr.home in batch/script file:
> Add the parameter "-Dsolr.solr.home=hot-deploy/solr" to the Java command
> invocation for ofbiz.jar.
> e.g.:
> "%JAVA_HOME%\bin\java" -Xms128M -Xmx512M -XX:MaxPermSize=512m -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar
> * solr.solr.home in Ant build configuration:
> In your root Ofbiz build.xml file, add the element "<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>"
> to the "<java jar="ofbiz.jar"...>" invocation of the appropriate Ant target(s) (run, start, run-install, etc.).
> e.g.:
> <target name="start" description="Start OFBiz">
> 	<java jar="ofbiz.jar" fork="true">
> 		<jvmarg value="${memory.initial.param}"/>
> 		<jvmarg value="${memory.max.param}"/>
> 		<jvmarg value="${memory.maxpermsize.param}"/>
> 		<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>
> 	</java>
> </target>
> ***
> It may be possible to specify solr home using other methods (JNDI, web.xml), but at the time
> of this writing, this was the most reliable method known.
> -----------------------------------------------------
> 2. Configuration
> -----------------------------------------------------
> The solr component can run out-of-the-box without configuration, but many
> files, settings and interfaces allow custom settings. Some of these include:
> * Ofbiz configurations:
> ** System properties:
> *** ofbiz.solr.eca.enabled - Global solr ECA toggling boolean (true/false, see Data Indexing)
> ** Config files:
> *** solr/config/solrconfig.properties - Ofbiz solr service behavior control
> *** ofbiz-component.xml - Standard Ofbiz component config
> * Apache Solr configurations:
> ** System properties:
> *** solr.solr.home - Solr home (see Installation)
> ** Config files:
> *** solr.xml - Base solr config
> *** conf/schema.xml - Solr index schema
> *** webapp/WEB-INF/web.xml - Dual Ofbiz/Solr webapp config
> ** Interfaces:
> *** /solr/admin/ - Webapp admin interface (see below)
> ***
> It is possible to set extensive native Solr configuration using
> the admin webapp interface noted above. It should be accessible at the address:
> http://localhost:8080/solr/admin/
> (where 8080 is your server's http port)
> Please refer to the Apache Solr documentation for usage of this interface
> and other native Solr configuration details.
> -----------------------------------------------------
> 3. Data Indexing
> -----------------------------------------------------
> The solr component indexes data such as Products into the Apache Solr database
> using services defined in the file:
> servicesdef/solrservices.xml
> The initial indexing may need to be performed or scheduled manually, but subsequent indexing
> may be partially or fully automated, though automated methods are disabled by default and must be enabled.
> Note that in general, solr services can only successfully run in contexts where the solr webapp is 
> loaded and accessible.
> There are two methods for indexing data:
> * Index rebuilding service (rebuildSolrIndex):
> The rebuildSolrIndex is the most important data import service. It reindexes
> all Ofbiz Products existing in the system into the solr index. rebuildSolrIndex MUST be run 
> AT LEAST once after installation and also following any data load operation that loads new products using the Ofbiz 
> "install" starting mode (run-install, load-demo, etc.).
> To do this, one can simply use the Webtools backend to invoke the service manually
> (e.g., /webtools/control/setSyncServiceParameters?SERVICE_NAME=rebuildSolrIndex&POOL_NAME=pool&_RUN_SYNC_=Y)
> or summon the Webtools Job Scheduler.
> Once the initial indexing has been performed, one can then use the Job Scheduler to
> invoke rebuildSolrIndex on a regular basis (every hour, every midnight, etc.) to update the Solr index.
> * ECAs/SECAs (addToSolr, for Product data):
> Although the rebuildSolrIndex is always necessary for the initial data import, one may also
> use ECAs and SECAs to import subsequent data changes automatically at every individual data (e.g. Product)
> update instead of running rebuildSolrIndex periodically. This is done by defining ECAs or SECAs that
> trigger the addToSolr service.
> The addToSolr service simply accepts a single "instance" parameter, a GenericValue. At the time of this writing,
> any entity value having a valid "productId" field designating a Product value may be passed; this will trigger reindexing
> for the specific product.
> By default, the addToSolr service implementation (and any service intended for use with ECAs) is disabled globally and
> succeeds silently. It can be enabled by setting the "solr.eca.enabled=true" in "config/solrconfig.properties".
> This property is provided for easy toggling. It can also be specified on the command line or using ant through 
> the true/false "ofbiz.solr.eca.enabled" system property (which has priority over file-based "solr.eca.enabled") in the same
> way described for the solr home system property (see Installation); this allows toggling them per Ant target.
> Some stock/default/example ECAs are provided and are functional once enabled. They can be found in:
> entitydef/eecas.xml
> Custom ECAs may also be added to this file, and custom SECAs may be added to the file:
> servicedef/secas.xml
> Special care may need to be taken to ensure the addToSolr is not triggered during "install" sequences
> (run-install/load-demo) because it requires the solr webapp to be accessible. addToSolr will automatically
> attempt to prevent reindexing in cases where the solr webapp is unavailable. However, it is possible that
> this behavior results in needlessly prevented reindexing (and thus lost updates) in some unknown complex server
> configurations. If this is the case, this behavior can be disabled by setting the option
> "solr.eca.useSolrWebappLoadedCheck=false" in the properties file "config/solrconfig.properties".
>  
> However, if the above useSolrWebappLoadedCheck option is specified, you will have to manually disable or comment the 
> ECA definitions/services to prevent them from executing during data-load operations; the "solr.eca.enabled" property and "ofbiz.solr.eca.enabled" system property can be used for this purpose. One easy way is to set "solr.eca.enabled=true" in 
> the property files and then specify "-Dofbiz.solr.eca.enabled=false" for any relevant 
> Ant build install/run-install/load-demo targets.
> Note that simply commenting the ECAs/SECAs or preventing their inclusion using
> ofbiz-component.xml may eliminate some service call overhead compared to using the property toggling. Ideally,
> a method exploiting ofbiz-component.xml or eecas.xml to prevent ECAs from running in inappropriate contexts
> would be ideal; however, at the time of this writing, such a method could not be established.
> ***
> To reset and clear the solr index completely, it is sufficient to shut down the server and delete the folders:
> solr/data/index
> solr/data/spellchecker
> and any other such folders (must delete the whole folders, not only contents).
> -----------------------------------------------------
> 4. Data Querying
> -----------------------------------------------------
> Solr queries can be done using two methods:
> * Solr Ofbiz services:
> Simply invoke (manually or in code) the query services found in the file:
> servicesdef/solrservices.xml
> These include solrProductsSearch, solrKeywordSearch and others. Note that in general, 
> solr services can only successfully run in contexts where the solr webapp is loaded and accessible.
> * Solr native admin webapp interface:
> One can also perform native Solr queries and diagnostics using the standard
> admin interface, accessible as described under Configuration. Please refer
> to the Apache Solr documentation for usage of this interface.
> -----------------------------------------------------
> 5. Implementation Concerns
> -----------------------------------------------------
> * The structure of the solr/webapp directory closely mirrors the contents of the solr.war
> distribution; however, please note that it contains some Ofbiz-specific modifications such
> as those found in web.xml. The same is true for other folders such as solr/conf. Therefore,
> any library updates must be done with care to preserve these modifications.
> -----------------------------------------------------
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> * In general, solr services can only successfully run in contexts where the solr webapp 
> is loaded and accessible.
> * ECA indexing may be problematic due to data loading and unknown complex server configurations; 
> various toggling options are provided to work around this issue (see Data Indexing for details).
> In the future - and ideally - this could be addressed using ofbiz-component.xml (or eecas.xml) directives.
> * The indexing services often produce the log warning "Problem reading product features" due to their
> use of the stock Ofbiz getProductFeatureSet service.
> ***
> Please report any other issues encountered. Thank you.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (OFBIZ-5042) Apache Solr Implementation

["Jacques Le Roux (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OFBIZ-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470955#comment-13470955 ] 

Jacques Le Roux commented on OFBIZ-5042:
----------------------------------------

Since we decided to move most specialpurpose components to Apache Extra ([see slim down effort|https://issues.apache.org/jira/browse/OFBIZ/fixforversion/12320551]), the best place for this component is in [Apache OFBiz Extra|http://code.google.com/a/apache-extras.org/hosting/search?q=label%3aOFBiz]. Please read [Apache Extra guidelines|http://community.apache.org/apache-extras/guidelines.html]. 

Note the point 6:
{quote}
Each Apache Extras project is encouraged to publish their releases and complete the project tagging information available on Apache Extras. 
{quote}

As explained in the guidelines, The Apache OFBiz Extra projects will not be official part of the Apache OFBiz project. The goals are pragmatic (at least as I see it):
* Components which are not mandatory (not supposed to be used in each project powered by OFBiz) can be moved to Apache Extra. 
* With limited Apache OFBiz project human resources, it's a way to delegate the code and releases responsabilities to specific communities

But this does not mean that Apache OFBiz Extra projects will be forgotten or neglected by the Apache OFBiz project. Now I feel something is missing in the scheme so far: how to relate an Apache OFBiz Extra project release to an Apache OFBiz release? I think it should part of the Apache OFBiz Extra project community to indicate which OFBiz release/s (trunk included) are compatible with the component. This is kind of part of the guidelines point 6 in my opininon. This is correctly maintained could be used by the Apache OFBiz project when releasing, we could at least have a link to the [Apache OFBiz Extra projects page|http://code.google.com/a/apache-extras.org/hosting/search?q=label%3aOFBiz]. Same from the [Apache OFBiz project site main page|http://ofbiz.apache.org/], a reference should be added. I will do so soon, but will wait that at least a complete usable Apache OFBiz Extra project exists there. The Neogia team made a first effor but nothing was committed yet, the Solr component could be the first to engage the movement. With the ApacheConEu coming it could be also a good way to share about this point.

Also I'd like to have a way to know the value of an Apache OFBiz Extra project, as experienced by its users. For the moment I see only one way to do this: recommend it on Google + (see in each component Project information). Of course this is a very limited way, but at least a mean.

So it would be great if you Paul, Pascal and Jonatan (at least for now, the community will hopefully grow from comments above) could move the Solr component into the Apache OFBiz Extra projects. This would begin the dynamic, completing the pending work initiated by the Neogia team. Hopefully more will come later. For instance I read a lof about Big Fish, why not sharing in Apache OFBiz Extra projects also?
                
> Apache Solr Implementation
> --------------------------
>
>                 Key: OFBIZ-5042
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-5042
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: ALL COMPONENTS
>            Reporter: Paul Piper
>              Labels: component, feature, solr
>         Attachments: solr-v1.00-r4284.zip
>
>
> Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community. With my agency, ilscipio, we have done several Apache Solr implementations in the past, and we felt that we could contribute this feature to the community. Currently, Solr is greatly used within syracus.net (our standalone store) and serves to generate the categories & search functionalities. From our team mainly Pascal Proulx, Jonatan Soto and I  were involved in this feature (albeit more having contributed to it. We created a standalone component which can be integrated into any running ofbiz implementation. It was tested against OFBiz 09.04 - 12.04.. 
> =====================================================
> Solr Component for Apache Ofbiz
> =====================================================
> This document describes the Ofbiz solr component, an Ofbiz (http://ofbiz.apache.org/) 
> implementation of the Apache Solr search platform (http://lucene.apache.org/solr/).
> The solr component includes an Ofbiz service-based wrapper layer to the Apache Solr
> webapp queries as well as the native Apache Solr web interface itself.
> Currently, the solr component focuses on Product data.
> Note: This document is a work in progress; information is subject to change.
> Contents:
> 1. Installation
> 2. Configuration
> 3. Data Indexing
> 4. Data Querying
> 5. Implementation Concerns
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> 1. Installation
> -----------------------------------------------------
> To install solr in an Ofbiz setting, simply extract the solr directory and
> and sub-folders to the hot-deploy folder.
> Afterward, the solr home system property (solr.solr.home) must be set to the value
> hot-deploy/solr manually using one of the following methods:
> * solr.solr.home in batch/script file:
> Add the parameter "-Dsolr.solr.home=hot-deploy/solr" to the Java command
> invocation for ofbiz.jar.
> e.g.:
> "%JAVA_HOME%\bin\java" -Xms128M -Xmx512M -XX:MaxPermSize=512m -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar
> * solr.solr.home in Ant build configuration:
> In your root Ofbiz build.xml file, add the element "<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>"
> to the "<java jar="ofbiz.jar"...>" invocation of the appropriate Ant target(s) (run, start, run-install, etc.).
> e.g.:
> <target name="start" description="Start OFBiz">
> 	<java jar="ofbiz.jar" fork="true">
> 		<jvmarg value="${memory.initial.param}"/>
> 		<jvmarg value="${memory.max.param}"/>
> 		<jvmarg value="${memory.maxpermsize.param}"/>
> 		<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>
> 	</java>
> </target>
> ***
> It may be possible to specify solr home using other methods (JNDI, web.xml), but at the time
> of this writing, this was the most reliable method known.
> -----------------------------------------------------
> 2. Configuration
> -----------------------------------------------------
> The solr component can run out-of-the-box without configuration, but many
> files, settings and interfaces allow custom settings. Some of these include:
> * Ofbiz configurations:
> ** System properties:
> *** ofbiz.solr.eca.enabled - Global solr ECA toggling boolean (true/false, see Data Indexing)
> ** Config files:
> *** solr/config/solrconfig.properties - Ofbiz solr service behavior control
> *** ofbiz-component.xml - Standard Ofbiz component config
> * Apache Solr configurations:
> ** System properties:
> *** solr.solr.home - Solr home (see Installation)
> ** Config files:
> *** solr.xml - Base solr config
> *** conf/schema.xml - Solr index schema
> *** webapp/WEB-INF/web.xml - Dual Ofbiz/Solr webapp config
> ** Interfaces:
> *** /solr/admin/ - Webapp admin interface (see below)
> ***
> It is possible to set extensive native Solr configuration using
> the admin webapp interface noted above. It should be accessible at the address:
> http://localhost:8080/solr/admin/
> (where 8080 is your server's http port)
> Please refer to the Apache Solr documentation for usage of this interface
> and other native Solr configuration details.
> -----------------------------------------------------
> 3. Data Indexing
> -----------------------------------------------------
> The solr component indexes data such as Products into the Apache Solr database
> using services defined in the file:
> servicesdef/solrservices.xml
> The initial indexing may need to be performed or scheduled manually, but subsequent indexing
> may be partially or fully automated, though automated methods are disabled by default and must be enabled.
> Note that in general, solr services can only successfully run in contexts where the solr webapp is 
> loaded and accessible.
> There are two methods for indexing data:
> * Index rebuilding service (rebuildSolrIndex):
> The rebuildSolrIndex is the most important data import service. It reindexes
> all Ofbiz Products existing in the system into the solr index. rebuildSolrIndex MUST be run 
> AT LEAST once after installation and also following any data load operation that loads new products using the Ofbiz 
> "install" starting mode (run-install, load-demo, etc.).
> To do this, one can simply use the Webtools backend to invoke the service manually
> (e.g., /webtools/control/setSyncServiceParameters?SERVICE_NAME=rebuildSolrIndex&POOL_NAME=pool&_RUN_SYNC_=Y)
> or summon the Webtools Job Scheduler.
> Once the initial indexing has been performed, one can then use the Job Scheduler to
> invoke rebuildSolrIndex on a regular basis (every hour, every midnight, etc.) to update the Solr index.
> * ECAs/SECAs (addToSolr, for Product data):
> Although the rebuildSolrIndex is always necessary for the initial data import, one may also
> use ECAs and SECAs to import subsequent data changes automatically at every individual data (e.g. Product)
> update instead of running rebuildSolrIndex periodically. This is done by defining ECAs or SECAs that
> trigger the addToSolr service.
> The addToSolr service simply accepts a single "instance" parameter, a GenericValue. At the time of this writing,
> any entity value having a valid "productId" field designating a Product value may be passed; this will trigger reindexing
> for the specific product.
> By default, the addToSolr service implementation (and any service intended for use with ECAs) is disabled globally and
> succeeds silently. It can be enabled by setting the "solr.eca.enabled=true" in "config/solrconfig.properties".
> This property is provided for easy toggling. It can also be specified on the command line or using ant through 
> the true/false "ofbiz.solr.eca.enabled" system property (which has priority over file-based "solr.eca.enabled") in the same
> way described for the solr home system property (see Installation); this allows toggling them per Ant target.
> Some stock/default/example ECAs are provided and are functional once enabled. They can be found in:
> entitydef/eecas.xml
> Custom ECAs may also be added to this file, and custom SECAs may be added to the file:
> servicedef/secas.xml
> Special care may need to be taken to ensure the addToSolr is not triggered during "install" sequences
> (run-install/load-demo) because it requires the solr webapp to be accessible. addToSolr will automatically
> attempt to prevent reindexing in cases where the solr webapp is unavailable. However, it is possible that
> this behavior results in needlessly prevented reindexing (and thus lost updates) in some unknown complex server
> configurations. If this is the case, this behavior can be disabled by setting the option
> "solr.eca.useSolrWebappLoadedCheck=false" in the properties file "config/solrconfig.properties".
>  
> However, if the above useSolrWebappLoadedCheck option is specified, you will have to manually disable or comment the 
> ECA definitions/services to prevent them from executing during data-load operations; the "solr.eca.enabled" property and "ofbiz.solr.eca.enabled" system property can be used for this purpose. One easy way is to set "solr.eca.enabled=true" in 
> the property files and then specify "-Dofbiz.solr.eca.enabled=false" for any relevant 
> Ant build install/run-install/load-demo targets.
> Note that simply commenting the ECAs/SECAs or preventing their inclusion using
> ofbiz-component.xml may eliminate some service call overhead compared to using the property toggling. Ideally,
> a method exploiting ofbiz-component.xml or eecas.xml to prevent ECAs from running in inappropriate contexts
> would be ideal; however, at the time of this writing, such a method could not be established.
> ***
> To reset and clear the solr index completely, it is sufficient to shut down the server and delete the folders:
> solr/data/index
> solr/data/spellchecker
> and any other such folders (must delete the whole folders, not only contents).
> -----------------------------------------------------
> 4. Data Querying
> -----------------------------------------------------
> Solr queries can be done using two methods:
> * Solr Ofbiz services:
> Simply invoke (manually or in code) the query services found in the file:
> servicesdef/solrservices.xml
> These include solrProductsSearch, solrKeywordSearch and others. Note that in general, 
> solr services can only successfully run in contexts where the solr webapp is loaded and accessible.
> * Solr native admin webapp interface:
> One can also perform native Solr queries and diagnostics using the standard
> admin interface, accessible as described under Configuration. Please refer
> to the Apache Solr documentation for usage of this interface.
> -----------------------------------------------------
> 5. Implementation Concerns
> -----------------------------------------------------
> * The structure of the solr/webapp directory closely mirrors the contents of the solr.war
> distribution; however, please note that it contains some Ofbiz-specific modifications such
> as those found in web.xml. The same is true for other folders such as solr/conf. Therefore,
> any library updates must be done with care to preserve these modifications.
> -----------------------------------------------------
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> * In general, solr services can only successfully run in contexts where the solr webapp 
> is loaded and accessible.
> * ECA indexing may be problematic due to data loading and unknown complex server configurations; 
> various toggling options are provided to work around this issue (see Data Indexing for details).
> In the future - and ideally - this could be addressed using ofbiz-component.xml (or eecas.xml) directives.
> * The indexing services often produce the log warning "Problem reading product features" due to their
> use of the stock Ofbiz getProductFeatureSet service.
> ***
> Please report any other issues encountered. Thank you.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (OFBIZ-5042) Apache Solr Implementation

["Tom Burns (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OFBIZ-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467069#comment-13467069 ] 

Tom Burns commented on OFBIZ-5042:
----------------------------------

It did help, working now.
Also I would like to second Olivier's comment on your documentation, it is a model for others.

Thanks,

Tom
                
> Apache Solr Implementation
> --------------------------
>
>                 Key: OFBIZ-5042
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-5042
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: ALL COMPONENTS
>            Reporter: Paul Piper
>              Labels: component, feature, solr
>         Attachments: solr-v1.00-r4284.zip
>
>
> Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community. With my agency, ilscipio, we have done several Apache Solr implementations in the past, and we felt that we could contribute this feature to the community. Currently, Solr is greatly used within syracus.net (our standalone store) and serves to generate the categories & search functionalities. From our team mainly Pascal Proulx, Jonatan Soto and I  were involved in this feature (albeit more having contributed to it. We created a standalone component which can be integrated into any running ofbiz implementation. It was tested against OFBiz 09.04 - 12.04.. 
> =====================================================
> Solr Component for Apache Ofbiz
> =====================================================
> This document describes the Ofbiz solr component, an Ofbiz (http://ofbiz.apache.org/) 
> implementation of the Apache Solr search platform (http://lucene.apache.org/solr/).
> The solr component includes an Ofbiz service-based wrapper layer to the Apache Solr
> webapp queries as well as the native Apache Solr web interface itself.
> Currently, the solr component focuses on Product data.
> Note: This document is a work in progress; information is subject to change.
> Contents:
> 1. Installation
> 2. Configuration
> 3. Data Indexing
> 4. Data Querying
> 5. Implementation Concerns
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> 1. Installation
> -----------------------------------------------------
> To install solr in an Ofbiz setting, simply extract the solr directory and
> and sub-folders to the hot-deploy folder.
> Afterward, the solr home system property (solr.solr.home) must be set to the value
> hot-deploy/solr manually using one of the following methods:
> * solr.solr.home in batch/script file:
> Add the parameter "-Dsolr.solr.home=hot-deploy/solr" to the Java command
> invocation for ofbiz.jar.
> e.g.:
> "%JAVA_HOME%\bin\java" -Xms128M -Xmx512M -XX:MaxPermSize=512m -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar
> * solr.solr.home in Ant build configuration:
> In your root Ofbiz build.xml file, add the element "<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>"
> to the "<java jar="ofbiz.jar"...>" invocation of the appropriate Ant target(s) (run, start, run-install, etc.).
> e.g.:
> <target name="start" description="Start OFBiz">
> 	<java jar="ofbiz.jar" fork="true">
> 		<jvmarg value="${memory.initial.param}"/>
> 		<jvmarg value="${memory.max.param}"/>
> 		<jvmarg value="${memory.maxpermsize.param}"/>
> 		<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>
> 	</java>
> </target>
> ***
> It may be possible to specify solr home using other methods (JNDI, web.xml), but at the time
> of this writing, this was the most reliable method known.
> -----------------------------------------------------
> 2. Configuration
> -----------------------------------------------------
> The solr component can run out-of-the-box without configuration, but many
> files, settings and interfaces allow custom settings. Some of these include:
> * Ofbiz configurations:
> ** System properties:
> *** ofbiz.solr.eca.enabled - Global solr ECA toggling boolean (true/false, see Data Indexing)
> ** Config files:
> *** solr/config/solrconfig.properties - Ofbiz solr service behavior control
> *** ofbiz-component.xml - Standard Ofbiz component config
> * Apache Solr configurations:
> ** System properties:
> *** solr.solr.home - Solr home (see Installation)
> ** Config files:
> *** solr.xml - Base solr config
> *** conf/schema.xml - Solr index schema
> *** webapp/WEB-INF/web.xml - Dual Ofbiz/Solr webapp config
> ** Interfaces:
> *** /solr/admin/ - Webapp admin interface (see below)
> ***
> It is possible to set extensive native Solr configuration using
> the admin webapp interface noted above. It should be accessible at the address:
> http://localhost:8080/solr/admin/
> (where 8080 is your server's http port)
> Please refer to the Apache Solr documentation for usage of this interface
> and other native Solr configuration details.
> -----------------------------------------------------
> 3. Data Indexing
> -----------------------------------------------------
> The solr component indexes data such as Products into the Apache Solr database
> using services defined in the file:
> servicesdef/solrservices.xml
> The initial indexing may need to be performed or scheduled manually, but subsequent indexing
> may be partially or fully automated, though automated methods are disabled by default and must be enabled.
> Note that in general, solr services can only successfully run in contexts where the solr webapp is 
> loaded and accessible.
> There are two methods for indexing data:
> * Index rebuilding service (rebuildSolrIndex):
> The rebuildSolrIndex is the most important data import service. It reindexes
> all Ofbiz Products existing in the system into the solr index. rebuildSolrIndex MUST be run 
> AT LEAST once after installation and also following any data load operation that loads new products using the Ofbiz 
> "install" starting mode (run-install, load-demo, etc.).
> To do this, one can simply use the Webtools backend to invoke the service manually
> (e.g., /webtools/control/setSyncServiceParameters?SERVICE_NAME=rebuildSolrIndex&POOL_NAME=pool&_RUN_SYNC_=Y)
> or summon the Webtools Job Scheduler.
> Once the initial indexing has been performed, one can then use the Job Scheduler to
> invoke rebuildSolrIndex on a regular basis (every hour, every midnight, etc.) to update the Solr index.
> * ECAs/SECAs (addToSolr, for Product data):
> Although the rebuildSolrIndex is always necessary for the initial data import, one may also
> use ECAs and SECAs to import subsequent data changes automatically at every individual data (e.g. Product)
> update instead of running rebuildSolrIndex periodically. This is done by defining ECAs or SECAs that
> trigger the addToSolr service.
> The addToSolr service simply accepts a single "instance" parameter, a GenericValue. At the time of this writing,
> any entity value having a valid "productId" field designating a Product value may be passed; this will trigger reindexing
> for the specific product.
> By default, the addToSolr service implementation (and any service intended for use with ECAs) is disabled globally and
> succeeds silently. It can be enabled by setting the "solr.eca.enabled=true" in "config/solrconfig.properties".
> This property is provided for easy toggling. It can also be specified on the command line or using ant through 
> the true/false "ofbiz.solr.eca.enabled" system property (which has priority over file-based "solr.eca.enabled") in the same
> way described for the solr home system property (see Installation); this allows toggling them per Ant target.
> Some stock/default/example ECAs are provided and are functional once enabled. They can be found in:
> entitydef/eecas.xml
> Custom ECAs may also be added to this file, and custom SECAs may be added to the file:
> servicedef/secas.xml
> Special care may need to be taken to ensure the addToSolr is not triggered during "install" sequences
> (run-install/load-demo) because it requires the solr webapp to be accessible. addToSolr will automatically
> attempt to prevent reindexing in cases where the solr webapp is unavailable. However, it is possible that
> this behavior results in needlessly prevented reindexing (and thus lost updates) in some unknown complex server
> configurations. If this is the case, this behavior can be disabled by setting the option
> "solr.eca.useSolrWebappLoadedCheck=false" in the properties file "config/solrconfig.properties".
>  
> However, if the above useSolrWebappLoadedCheck option is specified, you will have to manually disable or comment the 
> ECA definitions/services to prevent them from executing during data-load operations; the "solr.eca.enabled" property and "ofbiz.solr.eca.enabled" system property can be used for this purpose. One easy way is to set "solr.eca.enabled=true" in 
> the property files and then specify "-Dofbiz.solr.eca.enabled=false" for any relevant 
> Ant build install/run-install/load-demo targets.
> Note that simply commenting the ECAs/SECAs or preventing their inclusion using
> ofbiz-component.xml may eliminate some service call overhead compared to using the property toggling. Ideally,
> a method exploiting ofbiz-component.xml or eecas.xml to prevent ECAs from running in inappropriate contexts
> would be ideal; however, at the time of this writing, such a method could not be established.
> ***
> To reset and clear the solr index completely, it is sufficient to shut down the server and delete the folders:
> solr/data/index
> solr/data/spellchecker
> and any other such folders (must delete the whole folders, not only contents).
> -----------------------------------------------------
> 4. Data Querying
> -----------------------------------------------------
> Solr queries can be done using two methods:
> * Solr Ofbiz services:
> Simply invoke (manually or in code) the query services found in the file:
> servicesdef/solrservices.xml
> These include solrProductsSearch, solrKeywordSearch and others. Note that in general, 
> solr services can only successfully run in contexts where the solr webapp is loaded and accessible.
> * Solr native admin webapp interface:
> One can also perform native Solr queries and diagnostics using the standard
> admin interface, accessible as described under Configuration. Please refer
> to the Apache Solr documentation for usage of this interface.
> -----------------------------------------------------
> 5. Implementation Concerns
> -----------------------------------------------------
> * The structure of the solr/webapp directory closely mirrors the contents of the solr.war
> distribution; however, please note that it contains some Ofbiz-specific modifications such
> as those found in web.xml. The same is true for other folders such as solr/conf. Therefore,
> any library updates must be done with care to preserve these modifications.
> -----------------------------------------------------
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> * In general, solr services can only successfully run in contexts where the solr webapp 
> is loaded and accessible.
> * ECA indexing may be problematic due to data loading and unknown complex server configurations; 
> various toggling options are provided to work around this issue (see Data Indexing for details).
> In the future - and ideally - this could be addressed using ofbiz-component.xml (or eecas.xml) directives.
> * The indexing services often produce the log warning "Problem reading product features" due to their
> use of the stock Ofbiz getProductFeatureSet service.
> ***
> Please report any other issues encountered. Thank you.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (OFBIZ-5042) Apache Solr Implementation

["Tri Duc Vo (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OFBIZ-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466448#comment-13466448 ] 

Tri Duc Vo commented on OFBIZ-5042:
-----------------------------------

Nice feature
Cheers
                
> Apache Solr Implementation
> --------------------------
>
>                 Key: OFBIZ-5042
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-5042
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: ALL COMPONENTS
>            Reporter: Paul Piper
>              Labels: component, feature, solr
>         Attachments: solr-v1.00-r4284.zip
>
>
> Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community. With my agency, ilscipio, we have done several Apache Solr implementations in the past, and we felt that we could contribute this feature to the community. Currently, Solr is greatly used within syracus.net (our standalone store) and serves to generate the categories & search functionalities. From our team mainly Pascal Proulx, Jonatan Soto and I  were involved in this feature (albeit more having contributed to it. We created a standalone component which can be integrated into any running ofbiz implementation. It was tested against OFBiz 09.04 - 12.04.. 
> =====================================================
> Solr Component for Apache Ofbiz
> =====================================================
> This document describes the Ofbiz solr component, an Ofbiz (http://ofbiz.apache.org/) 
> implementation of the Apache Solr search platform (http://lucene.apache.org/solr/).
> The solr component includes an Ofbiz service-based wrapper layer to the Apache Solr
> webapp queries as well as the native Apache Solr web interface itself.
> Currently, the solr component focuses on Product data.
> Note: This document is a work in progress; information is subject to change.
> Contents:
> 1. Installation
> 2. Configuration
> 3. Data Indexing
> 4. Data Querying
> 5. Implementation Concerns
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> 1. Installation
> -----------------------------------------------------
> To install solr in an Ofbiz setting, simply extract the solr directory and
> and sub-folders to the hot-deploy folder.
> Afterward, the solr home system property (solr.solr.home) must be set to the value
> hot-deploy/solr manually using one of the following methods:
> * solr.solr.home in batch/script file:
> Add the parameter "-Dsolr.solr.home=hot-deploy/solr" to the Java command
> invocation for ofbiz.jar.
> e.g.:
> "%JAVA_HOME%\bin\java" -Xms128M -Xmx512M -XX:MaxPermSize=512m -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar
> * solr.solr.home in Ant build configuration:
> In your root Ofbiz build.xml file, add the element "<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>"
> to the "<java jar="ofbiz.jar"...>" invocation of the appropriate Ant target(s) (run, start, run-install, etc.).
> e.g.:
> <target name="start" description="Start OFBiz">
> 	<java jar="ofbiz.jar" fork="true">
> 		<jvmarg value="${memory.initial.param}"/>
> 		<jvmarg value="${memory.max.param}"/>
> 		<jvmarg value="${memory.maxpermsize.param}"/>
> 		<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>
> 	</java>
> </target>
> ***
> It may be possible to specify solr home using other methods (JNDI, web.xml), but at the time
> of this writing, this was the most reliable method known.
> -----------------------------------------------------
> 2. Configuration
> -----------------------------------------------------
> The solr component can run out-of-the-box without configuration, but many
> files, settings and interfaces allow custom settings. Some of these include:
> * Ofbiz configurations:
> ** System properties:
> *** ofbiz.solr.eca.enabled - Global solr ECA toggling boolean (true/false, see Data Indexing)
> ** Config files:
> *** solr/config/solrconfig.properties - Ofbiz solr service behavior control
> *** ofbiz-component.xml - Standard Ofbiz component config
> * Apache Solr configurations:
> ** System properties:
> *** solr.solr.home - Solr home (see Installation)
> ** Config files:
> *** solr.xml - Base solr config
> *** conf/schema.xml - Solr index schema
> *** webapp/WEB-INF/web.xml - Dual Ofbiz/Solr webapp config
> ** Interfaces:
> *** /solr/admin/ - Webapp admin interface (see below)
> ***
> It is possible to set extensive native Solr configuration using
> the admin webapp interface noted above. It should be accessible at the address:
> http://localhost:8080/solr/admin/
> (where 8080 is your server's http port)
> Please refer to the Apache Solr documentation for usage of this interface
> and other native Solr configuration details.
> -----------------------------------------------------
> 3. Data Indexing
> -----------------------------------------------------
> The solr component indexes data such as Products into the Apache Solr database
> using services defined in the file:
> servicesdef/solrservices.xml
> The initial indexing may need to be performed or scheduled manually, but subsequent indexing
> may be partially or fully automated, though automated methods are disabled by default and must be enabled.
> Note that in general, solr services can only successfully run in contexts where the solr webapp is 
> loaded and accessible.
> There are two methods for indexing data:
> * Index rebuilding service (rebuildSolrIndex):
> The rebuildSolrIndex is the most important data import service. It reindexes
> all Ofbiz Products existing in the system into the solr index. rebuildSolrIndex MUST be run 
> AT LEAST once after installation and also following any data load operation that loads new products using the Ofbiz 
> "install" starting mode (run-install, load-demo, etc.).
> To do this, one can simply use the Webtools backend to invoke the service manually
> (e.g., /webtools/control/setSyncServiceParameters?SERVICE_NAME=rebuildSolrIndex&POOL_NAME=pool&_RUN_SYNC_=Y)
> or summon the Webtools Job Scheduler.
> Once the initial indexing has been performed, one can then use the Job Scheduler to
> invoke rebuildSolrIndex on a regular basis (every hour, every midnight, etc.) to update the Solr index.
> * ECAs/SECAs (addToSolr, for Product data):
> Although the rebuildSolrIndex is always necessary for the initial data import, one may also
> use ECAs and SECAs to import subsequent data changes automatically at every individual data (e.g. Product)
> update instead of running rebuildSolrIndex periodically. This is done by defining ECAs or SECAs that
> trigger the addToSolr service.
> The addToSolr service simply accepts a single "instance" parameter, a GenericValue. At the time of this writing,
> any entity value having a valid "productId" field designating a Product value may be passed; this will trigger reindexing
> for the specific product.
> By default, the addToSolr service implementation (and any service intended for use with ECAs) is disabled globally and
> succeeds silently. It can be enabled by setting the "solr.eca.enabled=true" in "config/solrconfig.properties".
> This property is provided for easy toggling. It can also be specified on the command line or using ant through 
> the true/false "ofbiz.solr.eca.enabled" system property (which has priority over file-based "solr.eca.enabled") in the same
> way described for the solr home system property (see Installation); this allows toggling them per Ant target.
> Some stock/default/example ECAs are provided and are functional once enabled. They can be found in:
> entitydef/eecas.xml
> Custom ECAs may also be added to this file, and custom SECAs may be added to the file:
> servicedef/secas.xml
> Special care may need to be taken to ensure the addToSolr is not triggered during "install" sequences
> (run-install/load-demo) because it requires the solr webapp to be accessible. addToSolr will automatically
> attempt to prevent reindexing in cases where the solr webapp is unavailable. However, it is possible that
> this behavior results in needlessly prevented reindexing (and thus lost updates) in some unknown complex server
> configurations. If this is the case, this behavior can be disabled by setting the option
> "solr.eca.useSolrWebappLoadedCheck=false" in the properties file "config/solrconfig.properties".
>  
> However, if the above useSolrWebappLoadedCheck option is specified, you will have to manually disable or comment the 
> ECA definitions/services to prevent them from executing during data-load operations; the "solr.eca.enabled" property and "ofbiz.solr.eca.enabled" system property can be used for this purpose. One easy way is to set "solr.eca.enabled=true" in 
> the property files and then specify "-Dofbiz.solr.eca.enabled=false" for any relevant 
> Ant build install/run-install/load-demo targets.
> Note that simply commenting the ECAs/SECAs or preventing their inclusion using
> ofbiz-component.xml may eliminate some service call overhead compared to using the property toggling. Ideally,
> a method exploiting ofbiz-component.xml or eecas.xml to prevent ECAs from running in inappropriate contexts
> would be ideal; however, at the time of this writing, such a method could not be established.
> ***
> To reset and clear the solr index completely, it is sufficient to shut down the server and delete the folders:
> solr/data/index
> solr/data/spellchecker
> and any other such folders (must delete the whole folders, not only contents).
> -----------------------------------------------------
> 4. Data Querying
> -----------------------------------------------------
> Solr queries can be done using two methods:
> * Solr Ofbiz services:
> Simply invoke (manually or in code) the query services found in the file:
> servicesdef/solrservices.xml
> These include solrProductsSearch, solrKeywordSearch and others. Note that in general, 
> solr services can only successfully run in contexts where the solr webapp is loaded and accessible.
> * Solr native admin webapp interface:
> One can also perform native Solr queries and diagnostics using the standard
> admin interface, accessible as described under Configuration. Please refer
> to the Apache Solr documentation for usage of this interface.
> -----------------------------------------------------
> 5. Implementation Concerns
> -----------------------------------------------------
> * The structure of the solr/webapp directory closely mirrors the contents of the solr.war
> distribution; however, please note that it contains some Ofbiz-specific modifications such
> as those found in web.xml. The same is true for other folders such as solr/conf. Therefore,
> any library updates must be done with care to preserve these modifications.
> -----------------------------------------------------
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> * In general, solr services can only successfully run in contexts where the solr webapp 
> is loaded and accessible.
> * ECA indexing may be problematic due to data loading and unknown complex server configurations; 
> various toggling options are provided to work around this issue (see Data Indexing for details).
> In the future - and ideally - this could be addressed using ofbiz-component.xml (or eecas.xml) directives.
> * The indexing services often produce the log warning "Problem reading product features" due to their
> use of the stock Ofbiz getProductFeatureSet service.
> ***
> Please report any other issues encountered. Thank you.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (OFBIZ-5042) Apache Solr Implementation

["Pascal Proulx (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OFBIZ-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466908#comment-13466908 ] 

Pascal Proulx commented on OFBIZ-5042:
--------------------------------------

@Tom Burns:

We got Eclipse debugging to work based on the exact instructions detailed here:
https://cwiki.apache.org/OFBIZ/running-and-debugging-ofbiz-in-eclipse.html#RunningandDebuggingOFBizinEclipse-debuggingInEclipse

Afterward, we simply add the line "-Dsolr.solr.home=hot-deploy/solr" to the VM arguments line under the Arguments tab (following -Xms...), but without the "-jar ofbiz.jar" argument (since you already add it to the classpath under User Entries and then specify main as org.ofbiz.base.start.Start). The -jar is only for startofbiz.sh/bat.

So this command worked when attempted:
-Xms128M -Xmx512M -XX:MaxPermSize=512M
-Dcom.sun.management.jmxremote.port=3333 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dsolr.solr.home=hot-deploy/solr

Hope this helps 
                
> Apache Solr Implementation
> --------------------------
>
>                 Key: OFBIZ-5042
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-5042
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: ALL COMPONENTS
>            Reporter: Paul Piper
>              Labels: component, feature, solr
>         Attachments: solr-v1.00-r4284.zip
>
>
> Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community. With my agency, ilscipio, we have done several Apache Solr implementations in the past, and we felt that we could contribute this feature to the community. Currently, Solr is greatly used within syracus.net (our standalone store) and serves to generate the categories & search functionalities. From our team mainly Pascal Proulx, Jonatan Soto and I  were involved in this feature (albeit more having contributed to it. We created a standalone component which can be integrated into any running ofbiz implementation. It was tested against OFBiz 09.04 - 12.04.. 
> =====================================================
> Solr Component for Apache Ofbiz
> =====================================================
> This document describes the Ofbiz solr component, an Ofbiz (http://ofbiz.apache.org/) 
> implementation of the Apache Solr search platform (http://lucene.apache.org/solr/).
> The solr component includes an Ofbiz service-based wrapper layer to the Apache Solr
> webapp queries as well as the native Apache Solr web interface itself.
> Currently, the solr component focuses on Product data.
> Note: This document is a work in progress; information is subject to change.
> Contents:
> 1. Installation
> 2. Configuration
> 3. Data Indexing
> 4. Data Querying
> 5. Implementation Concerns
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> 1. Installation
> -----------------------------------------------------
> To install solr in an Ofbiz setting, simply extract the solr directory and
> and sub-folders to the hot-deploy folder.
> Afterward, the solr home system property (solr.solr.home) must be set to the value
> hot-deploy/solr manually using one of the following methods:
> * solr.solr.home in batch/script file:
> Add the parameter "-Dsolr.solr.home=hot-deploy/solr" to the Java command
> invocation for ofbiz.jar.
> e.g.:
> "%JAVA_HOME%\bin\java" -Xms128M -Xmx512M -XX:MaxPermSize=512m -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar
> * solr.solr.home in Ant build configuration:
> In your root Ofbiz build.xml file, add the element "<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>"
> to the "<java jar="ofbiz.jar"...>" invocation of the appropriate Ant target(s) (run, start, run-install, etc.).
> e.g.:
> <target name="start" description="Start OFBiz">
> 	<java jar="ofbiz.jar" fork="true">
> 		<jvmarg value="${memory.initial.param}"/>
> 		<jvmarg value="${memory.max.param}"/>
> 		<jvmarg value="${memory.maxpermsize.param}"/>
> 		<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>
> 	</java>
> </target>
> ***
> It may be possible to specify solr home using other methods (JNDI, web.xml), but at the time
> of this writing, this was the most reliable method known.
> -----------------------------------------------------
> 2. Configuration
> -----------------------------------------------------
> The solr component can run out-of-the-box without configuration, but many
> files, settings and interfaces allow custom settings. Some of these include:
> * Ofbiz configurations:
> ** System properties:
> *** ofbiz.solr.eca.enabled - Global solr ECA toggling boolean (true/false, see Data Indexing)
> ** Config files:
> *** solr/config/solrconfig.properties - Ofbiz solr service behavior control
> *** ofbiz-component.xml - Standard Ofbiz component config
> * Apache Solr configurations:
> ** System properties:
> *** solr.solr.home - Solr home (see Installation)
> ** Config files:
> *** solr.xml - Base solr config
> *** conf/schema.xml - Solr index schema
> *** webapp/WEB-INF/web.xml - Dual Ofbiz/Solr webapp config
> ** Interfaces:
> *** /solr/admin/ - Webapp admin interface (see below)
> ***
> It is possible to set extensive native Solr configuration using
> the admin webapp interface noted above. It should be accessible at the address:
> http://localhost:8080/solr/admin/
> (where 8080 is your server's http port)
> Please refer to the Apache Solr documentation for usage of this interface
> and other native Solr configuration details.
> -----------------------------------------------------
> 3. Data Indexing
> -----------------------------------------------------
> The solr component indexes data such as Products into the Apache Solr database
> using services defined in the file:
> servicesdef/solrservices.xml
> The initial indexing may need to be performed or scheduled manually, but subsequent indexing
> may be partially or fully automated, though automated methods are disabled by default and must be enabled.
> Note that in general, solr services can only successfully run in contexts where the solr webapp is 
> loaded and accessible.
> There are two methods for indexing data:
> * Index rebuilding service (rebuildSolrIndex):
> The rebuildSolrIndex is the most important data import service. It reindexes
> all Ofbiz Products existing in the system into the solr index. rebuildSolrIndex MUST be run 
> AT LEAST once after installation and also following any data load operation that loads new products using the Ofbiz 
> "install" starting mode (run-install, load-demo, etc.).
> To do this, one can simply use the Webtools backend to invoke the service manually
> (e.g., /webtools/control/setSyncServiceParameters?SERVICE_NAME=rebuildSolrIndex&POOL_NAME=pool&_RUN_SYNC_=Y)
> or summon the Webtools Job Scheduler.
> Once the initial indexing has been performed, one can then use the Job Scheduler to
> invoke rebuildSolrIndex on a regular basis (every hour, every midnight, etc.) to update the Solr index.
> * ECAs/SECAs (addToSolr, for Product data):
> Although the rebuildSolrIndex is always necessary for the initial data import, one may also
> use ECAs and SECAs to import subsequent data changes automatically at every individual data (e.g. Product)
> update instead of running rebuildSolrIndex periodically. This is done by defining ECAs or SECAs that
> trigger the addToSolr service.
> The addToSolr service simply accepts a single "instance" parameter, a GenericValue. At the time of this writing,
> any entity value having a valid "productId" field designating a Product value may be passed; this will trigger reindexing
> for the specific product.
> By default, the addToSolr service implementation (and any service intended for use with ECAs) is disabled globally and
> succeeds silently. It can be enabled by setting the "solr.eca.enabled=true" in "config/solrconfig.properties".
> This property is provided for easy toggling. It can also be specified on the command line or using ant through 
> the true/false "ofbiz.solr.eca.enabled" system property (which has priority over file-based "solr.eca.enabled") in the same
> way described for the solr home system property (see Installation); this allows toggling them per Ant target.
> Some stock/default/example ECAs are provided and are functional once enabled. They can be found in:
> entitydef/eecas.xml
> Custom ECAs may also be added to this file, and custom SECAs may be added to the file:
> servicedef/secas.xml
> Special care may need to be taken to ensure the addToSolr is not triggered during "install" sequences
> (run-install/load-demo) because it requires the solr webapp to be accessible. addToSolr will automatically
> attempt to prevent reindexing in cases where the solr webapp is unavailable. However, it is possible that
> this behavior results in needlessly prevented reindexing (and thus lost updates) in some unknown complex server
> configurations. If this is the case, this behavior can be disabled by setting the option
> "solr.eca.useSolrWebappLoadedCheck=false" in the properties file "config/solrconfig.properties".
>  
> However, if the above useSolrWebappLoadedCheck option is specified, you will have to manually disable or comment the 
> ECA definitions/services to prevent them from executing during data-load operations; the "solr.eca.enabled" property and "ofbiz.solr.eca.enabled" system property can be used for this purpose. One easy way is to set "solr.eca.enabled=true" in 
> the property files and then specify "-Dofbiz.solr.eca.enabled=false" for any relevant 
> Ant build install/run-install/load-demo targets.
> Note that simply commenting the ECAs/SECAs or preventing their inclusion using
> ofbiz-component.xml may eliminate some service call overhead compared to using the property toggling. Ideally,
> a method exploiting ofbiz-component.xml or eecas.xml to prevent ECAs from running in inappropriate contexts
> would be ideal; however, at the time of this writing, such a method could not be established.
> ***
> To reset and clear the solr index completely, it is sufficient to shut down the server and delete the folders:
> solr/data/index
> solr/data/spellchecker
> and any other such folders (must delete the whole folders, not only contents).
> -----------------------------------------------------
> 4. Data Querying
> -----------------------------------------------------
> Solr queries can be done using two methods:
> * Solr Ofbiz services:
> Simply invoke (manually or in code) the query services found in the file:
> servicesdef/solrservices.xml
> These include solrProductsSearch, solrKeywordSearch and others. Note that in general, 
> solr services can only successfully run in contexts where the solr webapp is loaded and accessible.
> * Solr native admin webapp interface:
> One can also perform native Solr queries and diagnostics using the standard
> admin interface, accessible as described under Configuration. Please refer
> to the Apache Solr documentation for usage of this interface.
> -----------------------------------------------------
> 5. Implementation Concerns
> -----------------------------------------------------
> * The structure of the solr/webapp directory closely mirrors the contents of the solr.war
> distribution; however, please note that it contains some Ofbiz-specific modifications such
> as those found in web.xml. The same is true for other folders such as solr/conf. Therefore,
> any library updates must be done with care to preserve these modifications.
> -----------------------------------------------------
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> * In general, solr services can only successfully run in contexts where the solr webapp 
> is loaded and accessible.
> * ECA indexing may be problematic due to data loading and unknown complex server configurations; 
> various toggling options are provided to work around this issue (see Data Indexing for details).
> In the future - and ideally - this could be addressed using ofbiz-component.xml (or eecas.xml) directives.
> * The indexing services often produce the log warning "Problem reading product features" due to their
> use of the stock Ofbiz getProductFeatureSet service.
> ***
> Please report any other issues encountered. Thank you.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (OFBIZ-5042) Apache Solr Implementation

["Jacques Le Roux (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OFBIZ-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471050#comment-13471050 ] 

Jacques Le Roux commented on OFBIZ-5042:
----------------------------------------

Few notes after 1st quick review:
* The Java classes use 
** packages "package com.ilscipio.solr". I think it's not a pb, since it wil not be official part-of/embedded-in OFBiz. To confirm packages names with PMC. In case of issue they could be something like org.apacheextras.ofbiz. or org.apache-extras.ofbiz. (for more [read this thread|http://mail-archives.apache.org/mod_mbox/community-dev/201209.mbox/%3CCALvzYd_p+Z0Bs_LyDtw7DvXEQpUhJp3mJuxT6L3-aMR1FVWaKQ@mail.gmail.com%3E]). For me personally, it's ok. Like also <vendor>Ilscipio GmbH</vendor> in solrservices.xml (which would not be tolerated in OFBiz OOTB) 
** miss ASL2 headers (I guess you want to make it ASL2 licensed right?)
* solrconfig.properties miss ASL2 header
* <project name="OFBiz - Ecommerce Component" default="jar" basedir="."> 
should be rather 
<project name="OFBiz - Solr Component" default="jar" basedir="."> 
* jquery-1.4.3.min.js in solr\webapp\admin\ is part of Solr, right  (all the webapp I guess?)? Else it should not be part of the component except if it supposed to work alone (solr standalone), jquery is already in OFBIz OOTB (different version by releases...)
* solr\data\ dir could be removed (are we supposed to have OFBiz data proper to OFBiz?), right?
* bin dir should be removed

Well done guys :)
                
> Apache Solr Implementation
> --------------------------
>
>                 Key: OFBIZ-5042
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-5042
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: ALL COMPONENTS
>            Reporter: Paul Piper
>              Labels: component, feature, solr
>         Attachments: solr-v1.00-r4284.zip
>
>
> Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community. With my agency, ilscipio, we have done several Apache Solr implementations in the past, and we felt that we could contribute this feature to the community. Currently, Solr is greatly used within syracus.net (our standalone store) and serves to generate the categories & search functionalities. From our team mainly Pascal Proulx, Jonatan Soto and I  were involved in this feature (albeit more having contributed to it. We created a standalone component which can be integrated into any running ofbiz implementation. It was tested against OFBiz 09.04 - 12.04.. 
> =====================================================
> Solr Component for Apache Ofbiz
> =====================================================
> This document describes the Ofbiz solr component, an Ofbiz (http://ofbiz.apache.org/) 
> implementation of the Apache Solr search platform (http://lucene.apache.org/solr/).
> The solr component includes an Ofbiz service-based wrapper layer to the Apache Solr
> webapp queries as well as the native Apache Solr web interface itself.
> Currently, the solr component focuses on Product data.
> Note: This document is a work in progress; information is subject to change.
> Contents:
> 1. Installation
> 2. Configuration
> 3. Data Indexing
> 4. Data Querying
> 5. Implementation Concerns
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> 1. Installation
> -----------------------------------------------------
> To install solr in an Ofbiz setting, simply extract the solr directory and
> and sub-folders to the hot-deploy folder.
> Afterward, the solr home system property (solr.solr.home) must be set to the value
> hot-deploy/solr manually using one of the following methods:
> * solr.solr.home in batch/script file:
> Add the parameter "-Dsolr.solr.home=hot-deploy/solr" to the Java command
> invocation for ofbiz.jar.
> e.g.:
> "%JAVA_HOME%\bin\java" -Xms128M -Xmx512M -XX:MaxPermSize=512m -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar
> * solr.solr.home in Ant build configuration:
> In your root Ofbiz build.xml file, add the element "<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>"
> to the "<java jar="ofbiz.jar"...>" invocation of the appropriate Ant target(s) (run, start, run-install, etc.).
> e.g.:
> <target name="start" description="Start OFBiz">
> 	<java jar="ofbiz.jar" fork="true">
> 		<jvmarg value="${memory.initial.param}"/>
> 		<jvmarg value="${memory.max.param}"/>
> 		<jvmarg value="${memory.maxpermsize.param}"/>
> 		<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>
> 	</java>
> </target>
> ***
> It may be possible to specify solr home using other methods (JNDI, web.xml), but at the time
> of this writing, this was the most reliable method known.
> -----------------------------------------------------
> 2. Configuration
> -----------------------------------------------------
> The solr component can run out-of-the-box without configuration, but many
> files, settings and interfaces allow custom settings. Some of these include:
> * Ofbiz configurations:
> ** System properties:
> *** ofbiz.solr.eca.enabled - Global solr ECA toggling boolean (true/false, see Data Indexing)
> ** Config files:
> *** solr/config/solrconfig.properties - Ofbiz solr service behavior control
> *** ofbiz-component.xml - Standard Ofbiz component config
> * Apache Solr configurations:
> ** System properties:
> *** solr.solr.home - Solr home (see Installation)
> ** Config files:
> *** solr.xml - Base solr config
> *** conf/schema.xml - Solr index schema
> *** webapp/WEB-INF/web.xml - Dual Ofbiz/Solr webapp config
> ** Interfaces:
> *** /solr/admin/ - Webapp admin interface (see below)
> ***
> It is possible to set extensive native Solr configuration using
> the admin webapp interface noted above. It should be accessible at the address:
> http://localhost:8080/solr/admin/
> (where 8080 is your server's http port)
> Please refer to the Apache Solr documentation for usage of this interface
> and other native Solr configuration details.
> -----------------------------------------------------
> 3. Data Indexing
> -----------------------------------------------------
> The solr component indexes data such as Products into the Apache Solr database
> using services defined in the file:
> servicesdef/solrservices.xml
> The initial indexing may need to be performed or scheduled manually, but subsequent indexing
> may be partially or fully automated, though automated methods are disabled by default and must be enabled.
> Note that in general, solr services can only successfully run in contexts where the solr webapp is 
> loaded and accessible.
> There are two methods for indexing data:
> * Index rebuilding service (rebuildSolrIndex):
> The rebuildSolrIndex is the most important data import service. It reindexes
> all Ofbiz Products existing in the system into the solr index. rebuildSolrIndex MUST be run 
> AT LEAST once after installation and also following any data load operation that loads new products using the Ofbiz 
> "install" starting mode (run-install, load-demo, etc.).
> To do this, one can simply use the Webtools backend to invoke the service manually
> (e.g., /webtools/control/setSyncServiceParameters?SERVICE_NAME=rebuildSolrIndex&POOL_NAME=pool&_RUN_SYNC_=Y)
> or summon the Webtools Job Scheduler.
> Once the initial indexing has been performed, one can then use the Job Scheduler to
> invoke rebuildSolrIndex on a regular basis (every hour, every midnight, etc.) to update the Solr index.
> * ECAs/SECAs (addToSolr, for Product data):
> Although the rebuildSolrIndex is always necessary for the initial data import, one may also
> use ECAs and SECAs to import subsequent data changes automatically at every individual data (e.g. Product)
> update instead of running rebuildSolrIndex periodically. This is done by defining ECAs or SECAs that
> trigger the addToSolr service.
> The addToSolr service simply accepts a single "instance" parameter, a GenericValue. At the time of this writing,
> any entity value having a valid "productId" field designating a Product value may be passed; this will trigger reindexing
> for the specific product.
> By default, the addToSolr service implementation (and any service intended for use with ECAs) is disabled globally and
> succeeds silently. It can be enabled by setting the "solr.eca.enabled=true" in "config/solrconfig.properties".
> This property is provided for easy toggling. It can also be specified on the command line or using ant through 
> the true/false "ofbiz.solr.eca.enabled" system property (which has priority over file-based "solr.eca.enabled") in the same
> way described for the solr home system property (see Installation); this allows toggling them per Ant target.
> Some stock/default/example ECAs are provided and are functional once enabled. They can be found in:
> entitydef/eecas.xml
> Custom ECAs may also be added to this file, and custom SECAs may be added to the file:
> servicedef/secas.xml
> Special care may need to be taken to ensure the addToSolr is not triggered during "install" sequences
> (run-install/load-demo) because it requires the solr webapp to be accessible. addToSolr will automatically
> attempt to prevent reindexing in cases where the solr webapp is unavailable. However, it is possible that
> this behavior results in needlessly prevented reindexing (and thus lost updates) in some unknown complex server
> configurations. If this is the case, this behavior can be disabled by setting the option
> "solr.eca.useSolrWebappLoadedCheck=false" in the properties file "config/solrconfig.properties".
>  
> However, if the above useSolrWebappLoadedCheck option is specified, you will have to manually disable or comment the 
> ECA definitions/services to prevent them from executing during data-load operations; the "solr.eca.enabled" property and "ofbiz.solr.eca.enabled" system property can be used for this purpose. One easy way is to set "solr.eca.enabled=true" in 
> the property files and then specify "-Dofbiz.solr.eca.enabled=false" for any relevant 
> Ant build install/run-install/load-demo targets.
> Note that simply commenting the ECAs/SECAs or preventing their inclusion using
> ofbiz-component.xml may eliminate some service call overhead compared to using the property toggling. Ideally,
> a method exploiting ofbiz-component.xml or eecas.xml to prevent ECAs from running in inappropriate contexts
> would be ideal; however, at the time of this writing, such a method could not be established.
> ***
> To reset and clear the solr index completely, it is sufficient to shut down the server and delete the folders:
> solr/data/index
> solr/data/spellchecker
> and any other such folders (must delete the whole folders, not only contents).
> -----------------------------------------------------
> 4. Data Querying
> -----------------------------------------------------
> Solr queries can be done using two methods:
> * Solr Ofbiz services:
> Simply invoke (manually or in code) the query services found in the file:
> servicesdef/solrservices.xml
> These include solrProductsSearch, solrKeywordSearch and others. Note that in general, 
> solr services can only successfully run in contexts where the solr webapp is loaded and accessible.
> * Solr native admin webapp interface:
> One can also perform native Solr queries and diagnostics using the standard
> admin interface, accessible as described under Configuration. Please refer
> to the Apache Solr documentation for usage of this interface.
> -----------------------------------------------------
> 5. Implementation Concerns
> -----------------------------------------------------
> * The structure of the solr/webapp directory closely mirrors the contents of the solr.war
> distribution; however, please note that it contains some Ofbiz-specific modifications such
> as those found in web.xml. The same is true for other folders such as solr/conf. Therefore,
> any library updates must be done with care to preserve these modifications.
> -----------------------------------------------------
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> * In general, solr services can only successfully run in contexts where the solr webapp 
> is loaded and accessible.
> * ECA indexing may be problematic due to data loading and unknown complex server configurations; 
> various toggling options are provided to work around this issue (see Data Indexing for details).
> In the future - and ideally - this could be addressed using ofbiz-component.xml (or eecas.xml) directives.
> * The indexing services often produce the log warning "Problem reading product features" due to their
> use of the stock Ofbiz getProductFeatureSet service.
> ***
> Please report any other issues encountered. Thank you.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (OFBIZ-5042) Apache Solr Implementation

["Jacques Le Roux (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OFBIZ-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470955#comment-13470955 ] 

Jacques Le Roux edited comment on OFBIZ-5042 at 10/6/12 9:08 AM:
-----------------------------------------------------------------

==== BETTER COMMENT: WE == COMMUNITY ====

Since the community decided to move most specialpurpose components to Apache Extra ([see slim down effort|https://issues.apache.org/jira/browse/OFBIZ/fixforversion/12320551]), the best place for this component is in [Apache OFBiz Extra|http://code.google.com/a/apache-extras.org/hosting/search?q=label%3aOFBiz]. Please read [Apache Extra guidelines|http://community.apache.org/apache-extras/guidelines.html]. 

Note the point 6:
{quote}
Each Apache Extras project is encouraged to publish their releases and complete the project tagging information available on Apache Extras. 
{quote}

As explained in the guidelines, The Apache OFBiz Extra projects will not be official part of the Apache OFBiz project. The goals are pragmatic (at least as I see it):
* Components which are not mandatory (not supposed to be used in each project powered by OFBiz) can be moved to Apache Extra. 
* With limited Apache OFBiz project human resources, it's a way to delegate the code and releases responsabilities to specific communities

But this does not mean that Apache OFBiz Extra projects will be forgotten or neglected by the Apache OFBiz project. Now I feel something is missing in the scheme so far: how to relate an Apache OFBiz Extra project release to an Apache OFBiz release? I think it should part of the Apache OFBiz Extra project community to indicate which OFBiz release/s (trunk included) are compatible with the component. This is kind of part of the guidelines point 6 in my opininon. If this is correctly maintained it could be used by the Apache OFBiz project when releasing, at least we could  have a link to the [Apache OFBiz Extra projects page|http://code.google.com/a/apache-extras.org/hosting/search?q=label%3aOFBiz]. Same from the [Apache OFBiz project site main page|http://ofbiz.apache.org/], a reference should be added. I will do so soon, but will wait that at least a complete usable Apache OFBiz Extra project exists there. The Neogia team made a first effor but nothing was committed yet, the Solr component could be the first to engage the movement. With the ApacheConEu coming it could be also a good way to share about this point.

Also I'd like to have a way to know the value of an Apache OFBiz Extra project, as experienced by its users. For the moment I see only one way to do this: recommend it on Google + (see in each component Project information). Of course this is a very limited way, but at least a mean.

So it would be great if you Paul, Pascal and Jonatan (at least for now, the community will hopefully grow from comments above) could move the Solr component into the Apache OFBiz Extra projects. This would begin the dynamic, completing the pending work initiated by the Neogia team. Hopefully more will come later. For instance I read a lof about Big Fish, why not sharing in Apache OFBiz Extra projects also?
                
      was (Author: jacques.le.roux):
    ==== BETTER COMMENT: WE == COMMUNITY ====

Since the community decided to move most specialpurpose components to Apache Extra ([see slim down effort|https://issues.apache.org/jira/browse/OFBIZ/fixforversion/12320551]), the best place for this component is in [Apache OFBiz Extra|http://code.google.com/a/apache-extras.org/hosting/search?q=label%3aOFBiz]. Please read [Apache Extra guidelines|http://community.apache.org/apache-extras/guidelines.html]. 

Note the point 6:
{quote}
Each Apache Extras project is encouraged to publish their releases and complete the project tagging information available on Apache Extras. 
{quote}

As explained in the guidelines, The Apache OFBiz Extra projects will not be official part of the Apache OFBiz project. The goals are pragmatic (at least as I see it):
* Components which are not mandatory (not supposed to be used in each project powered by OFBiz) can be moved to Apache Extra. 
* With limited Apache OFBiz project human resources, it's a way to delegate the code and releases responsabilities to specific communities

But this does not mean that Apache OFBiz Extra projects will be forgotten or neglected by the Apache OFBiz project. Now I feel something is missing in the scheme so far: how to relate an Apache OFBiz Extra project release to an Apache OFBiz release? I think it should part of the Apache OFBiz Extra project community to indicate which OFBiz release/s (trunk included) are compatible with the component. This is kind of part of the guidelines point 6 in my opininon. This is correctly maintained could be used by the Apache OFBiz project when releasing, we could at least have a link to the [Apache OFBiz Extra projects page|http://code.google.com/a/apache-extras.org/hosting/search?q=label%3aOFBiz]. Same from the [Apache OFBiz project site main page|http://ofbiz.apache.org/], a reference should be added. I will do so soon, but will wait that at least a complete usable Apache OFBiz Extra project exists there. The Neogia team made a first effor but nothing was committed yet, the Solr component could be the first to engage the movement. With the ApacheConEu coming it could be also a good way to share about this point.

Also I'd like to have a way to know the value of an Apache OFBiz Extra project, as experienced by its users. For the moment I see only one way to do this: recommend it on Google + (see in each component Project information). Of course this is a very limited way, but at least a mean.

So it would be great if you Paul, Pascal and Jonatan (at least for now, the community will hopefully grow from comments above) could move the Solr component into the Apache OFBiz Extra projects. This would begin the dynamic, completing the pending work initiated by the Neogia team. Hopefully more will come later. For instance I read a lof about Big Fish, why not sharing in Apache OFBiz Extra projects also?
                  
> Apache Solr Implementation
> --------------------------
>
>                 Key: OFBIZ-5042
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-5042
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: ALL COMPONENTS
>            Reporter: Paul Piper
>              Labels: component, feature, solr
>         Attachments: solr-v1.00-r4284.zip
>
>
> Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community. With my agency, ilscipio, we have done several Apache Solr implementations in the past, and we felt that we could contribute this feature to the community. Currently, Solr is greatly used within syracus.net (our standalone store) and serves to generate the categories & search functionalities. From our team mainly Pascal Proulx, Jonatan Soto and I  were involved in this feature (albeit more having contributed to it. We created a standalone component which can be integrated into any running ofbiz implementation. It was tested against OFBiz 09.04 - 12.04.. 
> =====================================================
> Solr Component for Apache Ofbiz
> =====================================================
> This document describes the Ofbiz solr component, an Ofbiz (http://ofbiz.apache.org/) 
> implementation of the Apache Solr search platform (http://lucene.apache.org/solr/).
> The solr component includes an Ofbiz service-based wrapper layer to the Apache Solr
> webapp queries as well as the native Apache Solr web interface itself.
> Currently, the solr component focuses on Product data.
> Note: This document is a work in progress; information is subject to change.
> Contents:
> 1. Installation
> 2. Configuration
> 3. Data Indexing
> 4. Data Querying
> 5. Implementation Concerns
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> 1. Installation
> -----------------------------------------------------
> To install solr in an Ofbiz setting, simply extract the solr directory and
> and sub-folders to the hot-deploy folder.
> Afterward, the solr home system property (solr.solr.home) must be set to the value
> hot-deploy/solr manually using one of the following methods:
> * solr.solr.home in batch/script file:
> Add the parameter "-Dsolr.solr.home=hot-deploy/solr" to the Java command
> invocation for ofbiz.jar.
> e.g.:
> "%JAVA_HOME%\bin\java" -Xms128M -Xmx512M -XX:MaxPermSize=512m -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar
> * solr.solr.home in Ant build configuration:
> In your root Ofbiz build.xml file, add the element "<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>"
> to the "<java jar="ofbiz.jar"...>" invocation of the appropriate Ant target(s) (run, start, run-install, etc.).
> e.g.:
> <target name="start" description="Start OFBiz">
> 	<java jar="ofbiz.jar" fork="true">
> 		<jvmarg value="${memory.initial.param}"/>
> 		<jvmarg value="${memory.max.param}"/>
> 		<jvmarg value="${memory.maxpermsize.param}"/>
> 		<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>
> 	</java>
> </target>
> ***
> It may be possible to specify solr home using other methods (JNDI, web.xml), but at the time
> of this writing, this was the most reliable method known.
> -----------------------------------------------------
> 2. Configuration
> -----------------------------------------------------
> The solr component can run out-of-the-box without configuration, but many
> files, settings and interfaces allow custom settings. Some of these include:
> * Ofbiz configurations:
> ** System properties:
> *** ofbiz.solr.eca.enabled - Global solr ECA toggling boolean (true/false, see Data Indexing)
> ** Config files:
> *** solr/config/solrconfig.properties - Ofbiz solr service behavior control
> *** ofbiz-component.xml - Standard Ofbiz component config
> * Apache Solr configurations:
> ** System properties:
> *** solr.solr.home - Solr home (see Installation)
> ** Config files:
> *** solr.xml - Base solr config
> *** conf/schema.xml - Solr index schema
> *** webapp/WEB-INF/web.xml - Dual Ofbiz/Solr webapp config
> ** Interfaces:
> *** /solr/admin/ - Webapp admin interface (see below)
> ***
> It is possible to set extensive native Solr configuration using
> the admin webapp interface noted above. It should be accessible at the address:
> http://localhost:8080/solr/admin/
> (where 8080 is your server's http port)
> Please refer to the Apache Solr documentation for usage of this interface
> and other native Solr configuration details.
> -----------------------------------------------------
> 3. Data Indexing
> -----------------------------------------------------
> The solr component indexes data such as Products into the Apache Solr database
> using services defined in the file:
> servicesdef/solrservices.xml
> The initial indexing may need to be performed or scheduled manually, but subsequent indexing
> may be partially or fully automated, though automated methods are disabled by default and must be enabled.
> Note that in general, solr services can only successfully run in contexts where the solr webapp is 
> loaded and accessible.
> There are two methods for indexing data:
> * Index rebuilding service (rebuildSolrIndex):
> The rebuildSolrIndex is the most important data import service. It reindexes
> all Ofbiz Products existing in the system into the solr index. rebuildSolrIndex MUST be run 
> AT LEAST once after installation and also following any data load operation that loads new products using the Ofbiz 
> "install" starting mode (run-install, load-demo, etc.).
> To do this, one can simply use the Webtools backend to invoke the service manually
> (e.g., /webtools/control/setSyncServiceParameters?SERVICE_NAME=rebuildSolrIndex&POOL_NAME=pool&_RUN_SYNC_=Y)
> or summon the Webtools Job Scheduler.
> Once the initial indexing has been performed, one can then use the Job Scheduler to
> invoke rebuildSolrIndex on a regular basis (every hour, every midnight, etc.) to update the Solr index.
> * ECAs/SECAs (addToSolr, for Product data):
> Although the rebuildSolrIndex is always necessary for the initial data import, one may also
> use ECAs and SECAs to import subsequent data changes automatically at every individual data (e.g. Product)
> update instead of running rebuildSolrIndex periodically. This is done by defining ECAs or SECAs that
> trigger the addToSolr service.
> The addToSolr service simply accepts a single "instance" parameter, a GenericValue. At the time of this writing,
> any entity value having a valid "productId" field designating a Product value may be passed; this will trigger reindexing
> for the specific product.
> By default, the addToSolr service implementation (and any service intended for use with ECAs) is disabled globally and
> succeeds silently. It can be enabled by setting the "solr.eca.enabled=true" in "config/solrconfig.properties".
> This property is provided for easy toggling. It can also be specified on the command line or using ant through 
> the true/false "ofbiz.solr.eca.enabled" system property (which has priority over file-based "solr.eca.enabled") in the same
> way described for the solr home system property (see Installation); this allows toggling them per Ant target.
> Some stock/default/example ECAs are provided and are functional once enabled. They can be found in:
> entitydef/eecas.xml
> Custom ECAs may also be added to this file, and custom SECAs may be added to the file:
> servicedef/secas.xml
> Special care may need to be taken to ensure the addToSolr is not triggered during "install" sequences
> (run-install/load-demo) because it requires the solr webapp to be accessible. addToSolr will automatically
> attempt to prevent reindexing in cases where the solr webapp is unavailable. However, it is possible that
> this behavior results in needlessly prevented reindexing (and thus lost updates) in some unknown complex server
> configurations. If this is the case, this behavior can be disabled by setting the option
> "solr.eca.useSolrWebappLoadedCheck=false" in the properties file "config/solrconfig.properties".
>  
> However, if the above useSolrWebappLoadedCheck option is specified, you will have to manually disable or comment the 
> ECA definitions/services to prevent them from executing during data-load operations; the "solr.eca.enabled" property and "ofbiz.solr.eca.enabled" system property can be used for this purpose. One easy way is to set "solr.eca.enabled=true" in 
> the property files and then specify "-Dofbiz.solr.eca.enabled=false" for any relevant 
> Ant build install/run-install/load-demo targets.
> Note that simply commenting the ECAs/SECAs or preventing their inclusion using
> ofbiz-component.xml may eliminate some service call overhead compared to using the property toggling. Ideally,
> a method exploiting ofbiz-component.xml or eecas.xml to prevent ECAs from running in inappropriate contexts
> would be ideal; however, at the time of this writing, such a method could not be established.
> ***
> To reset and clear the solr index completely, it is sufficient to shut down the server and delete the folders:
> solr/data/index
> solr/data/spellchecker
> and any other such folders (must delete the whole folders, not only contents).
> -----------------------------------------------------
> 4. Data Querying
> -----------------------------------------------------
> Solr queries can be done using two methods:
> * Solr Ofbiz services:
> Simply invoke (manually or in code) the query services found in the file:
> servicesdef/solrservices.xml
> These include solrProductsSearch, solrKeywordSearch and others. Note that in general, 
> solr services can only successfully run in contexts where the solr webapp is loaded and accessible.
> * Solr native admin webapp interface:
> One can also perform native Solr queries and diagnostics using the standard
> admin interface, accessible as described under Configuration. Please refer
> to the Apache Solr documentation for usage of this interface.
> -----------------------------------------------------
> 5. Implementation Concerns
> -----------------------------------------------------
> * The structure of the solr/webapp directory closely mirrors the contents of the solr.war
> distribution; however, please note that it contains some Ofbiz-specific modifications such
> as those found in web.xml. The same is true for other folders such as solr/conf. Therefore,
> any library updates must be done with care to preserve these modifications.
> -----------------------------------------------------
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> * In general, solr services can only successfully run in contexts where the solr webapp 
> is loaded and accessible.
> * ECA indexing may be problematic due to data loading and unknown complex server configurations; 
> various toggling options are provided to work around this issue (see Data Indexing for details).
> In the future - and ideally - this could be addressed using ofbiz-component.xml (or eecas.xml) directives.
> * The indexing services often produce the log warning "Problem reading product features" due to their
> use of the stock Ofbiz getProductFeatureSet service.
> ***
> Please report any other issues encountered. Thank you.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (OFBIZ-5042) Apache Solr Implementation

["Olivier Heintz (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OFBIZ-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466702#comment-13466702 ] 

Olivier Heintz commented on OFBIZ-5042:
---------------------------------------

Thank you very much for the very clear documentation and process to test.
One more things in my todo.
Cheers.
                
> Apache Solr Implementation
> --------------------------
>
>                 Key: OFBIZ-5042
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-5042
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: ALL COMPONENTS
>            Reporter: Paul Piper
>              Labels: component, feature, solr
>         Attachments: solr-v1.00-r4284.zip
>
>
> Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community. With my agency, ilscipio, we have done several Apache Solr implementations in the past, and we felt that we could contribute this feature to the community. Currently, Solr is greatly used within syracus.net (our standalone store) and serves to generate the categories & search functionalities. From our team mainly Pascal Proulx, Jonatan Soto and I  were involved in this feature (albeit more having contributed to it. We created a standalone component which can be integrated into any running ofbiz implementation. It was tested against OFBiz 09.04 - 12.04.. 
> =====================================================
> Solr Component for Apache Ofbiz
> =====================================================
> This document describes the Ofbiz solr component, an Ofbiz (http://ofbiz.apache.org/) 
> implementation of the Apache Solr search platform (http://lucene.apache.org/solr/).
> The solr component includes an Ofbiz service-based wrapper layer to the Apache Solr
> webapp queries as well as the native Apache Solr web interface itself.
> Currently, the solr component focuses on Product data.
> Note: This document is a work in progress; information is subject to change.
> Contents:
> 1. Installation
> 2. Configuration
> 3. Data Indexing
> 4. Data Querying
> 5. Implementation Concerns
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> 1. Installation
> -----------------------------------------------------
> To install solr in an Ofbiz setting, simply extract the solr directory and
> and sub-folders to the hot-deploy folder.
> Afterward, the solr home system property (solr.solr.home) must be set to the value
> hot-deploy/solr manually using one of the following methods:
> * solr.solr.home in batch/script file:
> Add the parameter "-Dsolr.solr.home=hot-deploy/solr" to the Java command
> invocation for ofbiz.jar.
> e.g.:
> "%JAVA_HOME%\bin\java" -Xms128M -Xmx512M -XX:MaxPermSize=512m -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar
> * solr.solr.home in Ant build configuration:
> In your root Ofbiz build.xml file, add the element "<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>"
> to the "<java jar="ofbiz.jar"...>" invocation of the appropriate Ant target(s) (run, start, run-install, etc.).
> e.g.:
> <target name="start" description="Start OFBiz">
> 	<java jar="ofbiz.jar" fork="true">
> 		<jvmarg value="${memory.initial.param}"/>
> 		<jvmarg value="${memory.max.param}"/>
> 		<jvmarg value="${memory.maxpermsize.param}"/>
> 		<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>
> 	</java>
> </target>
> ***
> It may be possible to specify solr home using other methods (JNDI, web.xml), but at the time
> of this writing, this was the most reliable method known.
> -----------------------------------------------------
> 2. Configuration
> -----------------------------------------------------
> The solr component can run out-of-the-box without configuration, but many
> files, settings and interfaces allow custom settings. Some of these include:
> * Ofbiz configurations:
> ** System properties:
> *** ofbiz.solr.eca.enabled - Global solr ECA toggling boolean (true/false, see Data Indexing)
> ** Config files:
> *** solr/config/solrconfig.properties - Ofbiz solr service behavior control
> *** ofbiz-component.xml - Standard Ofbiz component config
> * Apache Solr configurations:
> ** System properties:
> *** solr.solr.home - Solr home (see Installation)
> ** Config files:
> *** solr.xml - Base solr config
> *** conf/schema.xml - Solr index schema
> *** webapp/WEB-INF/web.xml - Dual Ofbiz/Solr webapp config
> ** Interfaces:
> *** /solr/admin/ - Webapp admin interface (see below)
> ***
> It is possible to set extensive native Solr configuration using
> the admin webapp interface noted above. It should be accessible at the address:
> http://localhost:8080/solr/admin/
> (where 8080 is your server's http port)
> Please refer to the Apache Solr documentation for usage of this interface
> and other native Solr configuration details.
> -----------------------------------------------------
> 3. Data Indexing
> -----------------------------------------------------
> The solr component indexes data such as Products into the Apache Solr database
> using services defined in the file:
> servicesdef/solrservices.xml
> The initial indexing may need to be performed or scheduled manually, but subsequent indexing
> may be partially or fully automated, though automated methods are disabled by default and must be enabled.
> Note that in general, solr services can only successfully run in contexts where the solr webapp is 
> loaded and accessible.
> There are two methods for indexing data:
> * Index rebuilding service (rebuildSolrIndex):
> The rebuildSolrIndex is the most important data import service. It reindexes
> all Ofbiz Products existing in the system into the solr index. rebuildSolrIndex MUST be run 
> AT LEAST once after installation and also following any data load operation that loads new products using the Ofbiz 
> "install" starting mode (run-install, load-demo, etc.).
> To do this, one can simply use the Webtools backend to invoke the service manually
> (e.g., /webtools/control/setSyncServiceParameters?SERVICE_NAME=rebuildSolrIndex&POOL_NAME=pool&_RUN_SYNC_=Y)
> or summon the Webtools Job Scheduler.
> Once the initial indexing has been performed, one can then use the Job Scheduler to
> invoke rebuildSolrIndex on a regular basis (every hour, every midnight, etc.) to update the Solr index.
> * ECAs/SECAs (addToSolr, for Product data):
> Although the rebuildSolrIndex is always necessary for the initial data import, one may also
> use ECAs and SECAs to import subsequent data changes automatically at every individual data (e.g. Product)
> update instead of running rebuildSolrIndex periodically. This is done by defining ECAs or SECAs that
> trigger the addToSolr service.
> The addToSolr service simply accepts a single "instance" parameter, a GenericValue. At the time of this writing,
> any entity value having a valid "productId" field designating a Product value may be passed; this will trigger reindexing
> for the specific product.
> By default, the addToSolr service implementation (and any service intended for use with ECAs) is disabled globally and
> succeeds silently. It can be enabled by setting the "solr.eca.enabled=true" in "config/solrconfig.properties".
> This property is provided for easy toggling. It can also be specified on the command line or using ant through 
> the true/false "ofbiz.solr.eca.enabled" system property (which has priority over file-based "solr.eca.enabled") in the same
> way described for the solr home system property (see Installation); this allows toggling them per Ant target.
> Some stock/default/example ECAs are provided and are functional once enabled. They can be found in:
> entitydef/eecas.xml
> Custom ECAs may also be added to this file, and custom SECAs may be added to the file:
> servicedef/secas.xml
> Special care may need to be taken to ensure the addToSolr is not triggered during "install" sequences
> (run-install/load-demo) because it requires the solr webapp to be accessible. addToSolr will automatically
> attempt to prevent reindexing in cases where the solr webapp is unavailable. However, it is possible that
> this behavior results in needlessly prevented reindexing (and thus lost updates) in some unknown complex server
> configurations. If this is the case, this behavior can be disabled by setting the option
> "solr.eca.useSolrWebappLoadedCheck=false" in the properties file "config/solrconfig.properties".
>  
> However, if the above useSolrWebappLoadedCheck option is specified, you will have to manually disable or comment the 
> ECA definitions/services to prevent them from executing during data-load operations; the "solr.eca.enabled" property and "ofbiz.solr.eca.enabled" system property can be used for this purpose. One easy way is to set "solr.eca.enabled=true" in 
> the property files and then specify "-Dofbiz.solr.eca.enabled=false" for any relevant 
> Ant build install/run-install/load-demo targets.
> Note that simply commenting the ECAs/SECAs or preventing their inclusion using
> ofbiz-component.xml may eliminate some service call overhead compared to using the property toggling. Ideally,
> a method exploiting ofbiz-component.xml or eecas.xml to prevent ECAs from running in inappropriate contexts
> would be ideal; however, at the time of this writing, such a method could not be established.
> ***
> To reset and clear the solr index completely, it is sufficient to shut down the server and delete the folders:
> solr/data/index
> solr/data/spellchecker
> and any other such folders (must delete the whole folders, not only contents).
> -----------------------------------------------------
> 4. Data Querying
> -----------------------------------------------------
> Solr queries can be done using two methods:
> * Solr Ofbiz services:
> Simply invoke (manually or in code) the query services found in the file:
> servicesdef/solrservices.xml
> These include solrProductsSearch, solrKeywordSearch and others. Note that in general, 
> solr services can only successfully run in contexts where the solr webapp is loaded and accessible.
> * Solr native admin webapp interface:
> One can also perform native Solr queries and diagnostics using the standard
> admin interface, accessible as described under Configuration. Please refer
> to the Apache Solr documentation for usage of this interface.
> -----------------------------------------------------
> 5. Implementation Concerns
> -----------------------------------------------------
> * The structure of the solr/webapp directory closely mirrors the contents of the solr.war
> distribution; however, please note that it contains some Ofbiz-specific modifications such
> as those found in web.xml. The same is true for other folders such as solr/conf. Therefore,
> any library updates must be done with care to preserve these modifications.
> -----------------------------------------------------
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> * In general, solr services can only successfully run in contexts where the solr webapp 
> is loaded and accessible.
> * ECA indexing may be problematic due to data loading and unknown complex server configurations; 
> various toggling options are provided to work around this issue (see Data Indexing for details).
> In the future - and ideally - this could be addressed using ofbiz-component.xml (or eecas.xml) directives.
> * The indexing services often produce the log warning "Problem reading product features" due to their
> use of the stock Ofbiz getProductFeatureSet service.
> ***
> Please report any other issues encountered. Thank you.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (OFBIZ-5042) Apache Solr Implementation

["Paul Piper (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OFBIZ-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Paul Piper updated OFBIZ-5042:
------------------------------

    Attachment: solr-v1.00-r4284.zip

Attached you can find the standalone solr implementation as created by Pascal Proulx last night. 
                
> Apache Solr Implementation
> --------------------------
>
>                 Key: OFBIZ-5042
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-5042
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: ALL COMPONENTS
>            Reporter: Paul Piper
>              Labels: component, feature, solr
>         Attachments: solr-v1.00-r4284.zip
>
>
> Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community. With my agency, ilscipio, we have done several Apache Solr implementations in the past, and we felt that we could contribute this feature to the community. Currently, Solr is greatly used within syracus.net (our standalone store) and serves to generate the categories & search functionalities. From our team mainly Pascal Proulx, Jonatan Soto and I were involved in this feature (albeit more having contributed to it. We created a standalone component which can be integrated into any running ofbiz implementation. It was tested against OFBiz 09.04 - 12.04.. 
> =====================================================
> Solr Component for Apache Ofbiz
> =====================================================
> This document describes the Ofbiz solr component, an Ofbiz (http://ofbiz.apache.org/) 
> implementation of the Apache Solr search platform (http://lucene.apache.org/solr/).
> The solr component includes an Ofbiz service-based wrapper layer to the Apache Solr
> webapp queries as well as the native Apache Solr web interface itself.
> Currently, the solr component focuses on Product data.
> Note: This document is a work in progress; information is subject to change.
> Contents:
> 1. Installation
> 2. Configuration
> 3. Data Indexing
> 4. Data Querying
> 5. Implementation Concerns
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> 1. Installation
> -----------------------------------------------------
> To install solr in an Ofbiz setting, simply extract the solr directory and
> and sub-folders to the hot-deploy folder.
> Afterward, the solr home system property (solr.solr.home) must be set to the value
> hot-deploy/solr manually using one of the following methods:
> * solr.solr.home in batch/script file:
> Add the parameter "-Dsolr.solr.home=hot-deploy/solr" to the Java command
> invocation for ofbiz.jar.
> e.g.:
> "%JAVA_HOME%\bin\java" -Xms128M -Xmx512M -XX:MaxPermSize=512m -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar
> * solr.solr.home in Ant build configuration:
> In your root Ofbiz build.xml file, add the element "<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>"
> to the "<java jar="ofbiz.jar"...>" invocation of the appropriate Ant target(s) (run, start, run-install, etc.).
> e.g.:
> <target name="start" description="Start OFBiz">
> 	<java jar="ofbiz.jar" fork="true">
> 		<jvmarg value="${memory.initial.param}"/>
> 		<jvmarg value="${memory.max.param}"/>
> 		<jvmarg value="${memory.maxpermsize.param}"/>
> 		<jvmarg value="-Dsolr.solr.home=hot-deploy/solr"/>
> 	</java>
> </target>
> ***
> It may be possible to specify solr home using other methods (JNDI, web.xml), but at the time
> of this writing, this was the most reliable method known.
> -----------------------------------------------------
> 2. Configuration
> -----------------------------------------------------
> The solr component can run out-of-the-box without configuration, but many
> files, settings and interfaces allow custom settings. Some of these include:
> * Ofbiz configurations:
> ** System properties:
> *** ofbiz.solr.eca.enabled - Global solr ECA toggling boolean (true/false, see Data Indexing)
> ** Config files:
> *** solr/config/solrconfig.properties - Ofbiz solr service behavior control
> *** ofbiz-component.xml - Standard Ofbiz component config
> * Apache Solr configurations:
> ** System properties:
> *** solr.solr.home - Solr home (see Installation)
> ** Config files:
> *** solr.xml - Base solr config
> *** conf/schema.xml - Solr index schema
> *** webapp/WEB-INF/web.xml - Dual Ofbiz/Solr webapp config
> ** Interfaces:
> *** /solr/admin/ - Webapp admin interface (see below)
> ***
> It is possible to set extensive native Solr configuration using
> the admin webapp interface noted above. It should be accessible at the address:
> http://localhost:8080/solr/admin/
> (where 8080 is your server's http port)
> Please refer to the Apache Solr documentation for usage of this interface
> and other native Solr configuration details.
> -----------------------------------------------------
> 3. Data Indexing
> -----------------------------------------------------
> The solr component indexes data such as Products into the Apache Solr database
> using services defined in the file:
> servicesdef/solrservices.xml
> The initial indexing may need to be performed or scheduled manually, but subsequent indexing
> may be partially or fully automated, though automated methods are disabled by default and must be enabled.
> Note that in general, solr services can only successfully run in contexts where the solr webapp is 
> loaded and accessible.
> There are two methods for indexing data:
> * Index rebuilding service (rebuildSolrIndex):
> The rebuildSolrIndex is the most important data import service. It reindexes
> all Ofbiz Products existing in the system into the solr index. rebuildSolrIndex MUST be run 
> AT LEAST once after installation and also following any data load operation that loads new products using the Ofbiz 
> "install" starting mode (run-install, load-demo, etc.).
> To do this, one can simply use the Webtools backend to invoke the service manually
> (e.g., /webtools/control/setSyncServiceParameters?SERVICE_NAME=rebuildSolrIndex&POOL_NAME=pool&_RUN_SYNC_=Y)
> or summon the Webtools Job Scheduler.
> Once the initial indexing has been performed, one can then use the Job Scheduler to
> invoke rebuildSolrIndex on a regular basis (every hour, every midnight, etc.) to update the Solr index.
> * ECAs/SECAs (addToSolr, for Product data):
> Although the rebuildSolrIndex is always necessary for the initial data import, one may also
> use ECAs and SECAs to import subsequent data changes automatically at every individual data (e.g. Product)
> update instead of running rebuildSolrIndex periodically. This is done by defining ECAs or SECAs that
> trigger the addToSolr service.
> The addToSolr service simply accepts a single "instance" parameter, a GenericValue. At the time of this writing,
> any entity value having a valid "productId" field designating a Product value may be passed; this will trigger reindexing
> for the specific product.
> By default, the addToSolr service implementation (and any service intended for use with ECAs) is disabled globally and
> succeeds silently. It can be enabled by setting the "solr.eca.enabled=true" in "config/solrconfig.properties".
> This property is provided for easy toggling. It can also be specified on the command line or using ant through 
> the true/false "ofbiz.solr.eca.enabled" system property (which has priority over file-based "solr.eca.enabled") in the same
> way described for the solr home system property (see Installation); this allows toggling them per Ant target.
> Some stock/default/example ECAs are provided and are functional once enabled. They can be found in:
> entitydef/eecas.xml
> Custom ECAs may also be added to this file, and custom SECAs may be added to the file:
> servicedef/secas.xml
> Special care may need to be taken to ensure the addToSolr is not triggered during "install" sequences
> (run-install/load-demo) because it requires the solr webapp to be accessible. addToSolr will automatically
> attempt to prevent reindexing in cases where the solr webapp is unavailable. However, it is possible that
> this behavior results in needlessly prevented reindexing (and thus lost updates) in some unknown complex server
> configurations. If this is the case, this behavior can be disabled by setting the option
> "solr.eca.useSolrWebappLoadedCheck=false" in the properties file "config/solrconfig.properties".
>  
> However, if the above useSolrWebappLoadedCheck option is specified, you will have to manually disable or comment the 
> ECA definitions/services to prevent them from executing during data-load operations; the "solr.eca.enabled" property and "ofbiz.solr.eca.enabled" system property can be used for this purpose. One easy way is to set "solr.eca.enabled=true" in 
> the property files and then specify "-Dofbiz.solr.eca.enabled=false" for any relevant 
> Ant build install/run-install/load-demo targets.
> Note that simply commenting the ECAs/SECAs or preventing their inclusion using
> ofbiz-component.xml may eliminate some service call overhead compared to using the property toggling. Ideally,
> a method exploiting ofbiz-component.xml or eecas.xml to prevent ECAs from running in inappropriate contexts
> would be ideal; however, at the time of this writing, such a method could not be established.
> ***
> To reset and clear the solr index completely, it is sufficient to shut down the server and delete the folders:
> solr/data/index
> solr/data/spellchecker
> and any other such folders (must delete the whole folders, not only contents).
> -----------------------------------------------------
> 4. Data Querying
> -----------------------------------------------------
> Solr queries can be done using two methods:
> * Solr Ofbiz services:
> Simply invoke (manually or in code) the query services found in the file:
> servicesdef/solrservices.xml
> These include solrProductsSearch, solrKeywordSearch and others. Note that in general, 
> solr services can only successfully run in contexts where the solr webapp is loaded and accessible.
> * Solr native admin webapp interface:
> One can also perform native Solr queries and diagnostics using the standard
> admin interface, accessible as described under Configuration. Please refer
> to the Apache Solr documentation for usage of this interface.
> -----------------------------------------------------
> 5. Implementation Concerns
> -----------------------------------------------------
> * The structure of the solr/webapp directory closely mirrors the contents of the solr.war
> distribution; however, please note that it contains some Ofbiz-specific modifications such
> as those found in web.xml. The same is true for other folders such as solr/conf. Therefore,
> any library updates must be done with care to preserve these modifications.
> -----------------------------------------------------
> 6. Known Bugs, Limitations and Issues
> -----------------------------------------------------
> * In general, solr services can only successfully run in contexts where the solr webapp 
> is loaded and accessible.
> * ECA indexing may be problematic due to data loading and unknown complex server configurations; 
> various toggling options are provided to work around this issue (see Data Indexing for details).
> In the future - and ideally - this could be addressed using ofbiz-component.xml (or eecas.xml) directives.
> * The indexing services often produce the log warning "Problem reading product features" due to their
> use of the stock Ofbiz getProductFeatureSet service.
> ***
> Please report any other issues encountered. Thank you.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira