You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@allura.apache.org by gc...@apache.org on 2024/08/23 15:58:42 UTC
(allura) branch master updated (2cb75a4d1 -> 2b6eec65d)
This is an automated email from the ASF dual-hosted git repository.
gcruz pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/allura.git
from 2cb75a4d1 bump WebOb 1.8.7 -> 1.8.8
new f4f060311 set solr.use_new_types=true for tests
new 760dede19 clean up unused solr fields
new c8638b5e3 enable autoSoftCommit in config, since commitWithin from code doesn't seem to be working
new 0d510374f be explicit about docValues for faceting used on tickets
new 2b6eec65d add storeOffsetsWithPositions in case "unified" highligher is used in future, it gives better performance
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
Allura/allura/lib/solr.py | 7 ++
Allura/test.ini | 3 -
ForgeTracker/forgetracker/search.py | 1 +
.../forgetracker/tests/unit/test_globals_model.py | 7 +-
.../tests/unit/test_root_controller.py | 2 +-
solr_config/allura/conf/schema.xml | 77 ++--------------------
solr_config/allura/conf/solrconfig.xml | 2 +-
7 files changed, 21 insertions(+), 78 deletions(-)
(allura) 01/05: set solr.use_new_types=true for tests
Posted by gc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gcruz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/allura.git
commit f4f060311dbf645a9a3bb8ebb95549674683f298
Author: Dave Brondsema <db...@slashdotmedia.com>
AuthorDate: Tue Jul 30 15:17:00 2024 -0400
set solr.use_new_types=true for tests
---
Allura/test.ini | 3 ---
ForgeTracker/forgetracker/tests/unit/test_globals_model.py | 7 ++++---
ForgeTracker/forgetracker/tests/unit/test_root_controller.py | 2 +-
3 files changed, 5 insertions(+), 7 deletions(-)
diff --git a/Allura/test.ini b/Allura/test.ini
index d7a870deb..8c3604c11 100644
--- a/Allura/test.ini
+++ b/Allura/test.ini
@@ -81,9 +81,6 @@ support_tool_choices = wiki tickets discussion
; tests expect max length of 40000
markdown_render_max_length = 40000
-; TODO: update tests and let this be true
-solr.use_new_types = false
-
; TODO: update tests and remove this setting override
auth.require_email_addr = false
diff --git a/ForgeTracker/forgetracker/tests/unit/test_globals_model.py b/ForgeTracker/forgetracker/tests/unit/test_globals_model.py
index 9353d6b9f..99b6336a6 100644
--- a/ForgeTracker/forgetracker/tests/unit/test_globals_model.py
+++ b/ForgeTracker/forgetracker/tests/unit/test_globals_model.py
@@ -151,15 +151,16 @@ class TestCustomFields(TrackerTestWithModel):
tracker_globals = globals_with_custom_fields(
[dict(label='Iteration Number',
name='_iteration_number',
+ type='number',
show_in_search=False),
dict(label='Point Estimate',
name='_point_estimate',
+ type='number',
show_in_search=True)])
- expected = [dict(sortable_name='_point_estimate_s',
+ expected = [dict(sortable_name='_point_estimate_d',
name='_point_estimate',
label='Point Estimate')]
- assert tracker_globals.sortable_custom_fields_shown_in_search(
- ) == expected
+ assert tracker_globals.sortable_custom_fields_shown_in_search() == expected
def globals_with_custom_fields(custom_fields):
diff --git a/ForgeTracker/forgetracker/tests/unit/test_root_controller.py b/ForgeTracker/forgetracker/tests/unit/test_root_controller.py
index 43414cb8c..d0af09fcc 100644
--- a/ForgeTracker/forgetracker/tests/unit/test_root_controller.py
+++ b/ForgeTracker/forgetracker/tests/unit/test_root_controller.py
@@ -45,7 +45,7 @@ class TestWhenSearchingWithCustomFields(WithUserAndBugsApp):
self.response = tracker_main.RootController().search(q='friends')
def test_that_sortable_custom_fields_are_present(self):
- expected = [dict(sortable_name='_iteration_number_s',
+ expected = [dict(sortable_name='_iteration_number_d',
name='_iteration_number',
label='Iteration Number')]
assert self.response['sortable_custom_fields'] == expected
(allura) 03/05: enable autoSoftCommit in config, since commitWithin from code doesn't seem to be working
Posted by gc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gcruz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/allura.git
commit c8638b5e37914996a362b81e80f13234e19ffe75
Author: Dave Brondsema <db...@slashdotmedia.com>
AuthorDate: Wed Aug 14 09:57:57 2024 -0400
enable autoSoftCommit in config, since commitWithin from code doesn't seem to be working
---
solr_config/allura/conf/solrconfig.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/solr_config/allura/conf/solrconfig.xml b/solr_config/allura/conf/solrconfig.xml
index 6e3d4e9c6..06e7fc1e5 100644
--- a/solr_config/allura/conf/solrconfig.xml
+++ b/solr_config/allura/conf/solrconfig.xml
@@ -192,7 +192,7 @@
faster and more near-realtime friendly than a hard commit.
-->
<autoSoftCommit>
- <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
+ <maxTime>${solr.autoSoftCommit.maxTime:2000}</maxTime>
</autoSoftCommit>
</updateHandler>
(allura) 04/05: be explicit about docValues for faceting used on tickets
Posted by gc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gcruz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/allura.git
commit 0d510374fec0ed61c31f5e24abea49e1fdaf5b67
Author: Dave Brondsema <db...@slashdotmedia.com>
AuthorDate: Wed Aug 21 15:55:11 2024 -0400
be explicit about docValues for faceting used on tickets
---
ForgeTracker/forgetracker/search.py | 1 +
solr_config/allura/conf/schema.xml | 6 ++++++
2 files changed, 7 insertions(+)
diff --git a/ForgeTracker/forgetracker/search.py b/ForgeTracker/forgetracker/search.py
index aaec822ea..10cd3c083 100644
--- a/ForgeTracker/forgetracker/search.py
+++ b/ForgeTracker/forgetracker/search.py
@@ -22,6 +22,7 @@ import six
FACET_PARAMS = {
'facet': 'true',
+ # fields used in faceting should have docValues="true" in solr, for faster performance
'facet.field': ['_milestone_s', 'status_s', 'assigned_to_s', 'reported_by_s'],
'facet.limit': -1,
'facet.sort': 'index',
diff --git a/solr_config/allura/conf/schema.xml b/solr_config/allura/conf/schema.xml
index fcb1ced3b..8c0059b51 100644
--- a/solr_config/allura/conf/schema.xml
+++ b/solr_config/allura/conf/schema.xml
@@ -112,6 +112,12 @@
<!-- END of required field values -->
<field name="title" type="text_general" indexed="true" stored="true" multiValued="true"/>
+ <!-- these are dynamic *_s but we add specific records to set docValues="true" and ensure faster facet performance
+ (although *_s seems to have docValues by default, but lets be explicit) -->
+ <field name="_milestone_s" type="string" indexed="true" stored="true" docValues="true"/>
+ <field name="status_s" type="string" indexed="true" stored="true" docValues="true"/>
+ <field name="assigned_to_s" type="string" indexed="true" stored="true" docValues="true"/>
+ <field name="reported_by_s" type="string" indexed="true" stored="true" docValues="true"/>
<!-- Dynamic field definitions allow using convention over configuration
for fields via the specification of patterns to match field names.
(allura) 02/05: clean up unused solr fields
Posted by gc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gcruz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/allura.git
commit 760dede19609f818410b6f6c9a37b754337f3ac9
Author: Dave Brondsema <db...@slashdotmedia.com>
AuthorDate: Wed Aug 14 09:57:25 2024 -0400
clean up unused solr fields
---
Allura/allura/lib/solr.py | 7 ++++
solr_config/allura/conf/schema.xml | 69 --------------------------------------
2 files changed, 7 insertions(+), 69 deletions(-)
diff --git a/Allura/allura/lib/solr.py b/Allura/allura/lib/solr.py
index 49c76afb3..7635b36d6 100644
--- a/Allura/allura/lib/solr.py
+++ b/Allura/allura/lib/solr.py
@@ -154,6 +154,13 @@ class MockSOLR:
for o in objects:
o['text'] = ''.join(o['text'])
json.dumps(o) # ensure no errors (since pysolr 3.9+ uses json API to solr)
+ for k in o.keys():
+ if k.endswith(('_i', '_s', '_l', '_t', '_b', '_f', '_d', '_dt', '_ws')):
+ continue
+ elif k in ('id', 'text', 'title'):
+ continue
+ else:
+ raise ValueError(f'Unexpected solr field {k!r}, probably not in schema.xml')
self.db[o['id']] = o
def commit(self):
diff --git a/solr_config/allura/conf/schema.xml b/solr_config/allura/conf/schema.xml
index afca49639..fcb1ced3b 100644
--- a/solr_config/allura/conf/schema.xml
+++ b/solr_config/allura/conf/schema.xml
@@ -111,72 +111,7 @@
<field name="text" type="text_general" indexed="true" stored="true" multiValued="true"/>
<!-- END of required field values -->
- <field name="description" type="text_general" indexed="true" stored="true" />
- <field name="group_id" type="tint" indexed="true" stored="true"/>
- <field name="group_ranking" type="tint" indexed="true" stored="true"/>
- <field name="has_file" type="tint" indexed="true" stored="true"/>
- <field name="help_wanted" type="boolean" indexed="true" stored="true" />
- <field name="latest_file_date" type="date" indexed="true" stored="true" />
- <field name="license" type="string" indexed="true" stored="true" />
- <field name="license_other" type="string" indexed="true" stored="true" />
- <field name="name" type="text_general" indexed="true" stored="true"/>
- <field name="num_developers" type="tint" indexed="true" stored="true"/>
- <field name="num_downloads" type="tint" indexed="true" stored="true"/>
- <field name="num_downloads_week" type="tint" indexed="true" stored="true"/>
- <field name="num_services" type="tint" indexed="true" stored="true"/>
- <field name="percentile" type="float" indexed="true" stored="true"/>
- <field name="project_type" type="tint" indexed="true" stored="true" />
- <field name="project_doc_id" type="string" indexed="true" stored="true" />
- <field name="registration_date" type="date" indexed="true" stored="true" />
- <field name="screenshot_url" type="string" indexed="true" stored="true" />
- <field name="trove" type="text_general" indexed="true" stored="true" />
- <field name="unix_group_name" type="string" indexed="true" stored="true" />
- <field name="source" type="string" indexed="true" stored="true" />
- <field name="rating" type="float" indexed="true" stored="true" />
- <field name="review_count" type="tint" indexed="true" stored="true" />
-
- <!-- Common metadata fields, named specifically to match up with
- SolrCell metadata when parsing rich documents such as Word, PDF.
- Some fields are multiValued only because Tika currently may return
- multiple values for them. Some metadata is parsed from the documents,
- but there are some which come from the client context:
- "content_type": From the HTTP headers of incoming stream
- "resourcename": From SolrCell request param resource.name
- -->
<field name="title" type="text_general" indexed="true" stored="true" multiValued="true"/>
- <field name="subject" type="text_general" indexed="true" stored="true"/>
- <field name="comments" type="text_general" indexed="true" stored="true"/>
- <field name="author" type="text_general" indexed="true" stored="true"/>
- <field name="keywords" type="text_general" indexed="true" stored="true"/>
- <field name="category" type="text_general" indexed="true" stored="true"/>
- <field name="content_type" type="string" indexed="true" stored="true" multiValued="true"/>
- <field name="last_modified" type="date" indexed="true" stored="true"/>
- <field name="links" type="string" indexed="true" stored="true" multiValued="true"/>
-
- <!-- Main body of document extracted by SolrCell.
- NOTE: This field is not indexed by default, since it is also copied to "text"
- using copyField below. This is to save space. Use this field for returning and
- highlighting document content. Use the "text" field to search the content. -->
-
-
- <!-- catchall field, containing all other searchable text fields (implemented
- via copyField further on in this schema -->
-
- <!-- catchall text field that indexes tokens both normally and in reverse for efficient
- leading wildcard queries. -->
-
- <!-- non-tokenized version of manufacturer to make it easier to sort or group
- results by manufacturer. copied from "manu" via copyField -->
- <!--
- Some fields such as popularity and manu_exact could be modified to
- leverage doc values:
- <field name="popularity" type="int" indexed="true" stored="true" docValues="true" default="0" />
- <field name="manu_exact" type="string" indexed="false" stored="false" docValues="true" default="" />
-
- Although it would make indexing slightly slower and the index bigger, it
- would also make the index faster to load, more memory-efficient and more
- NRT-friendly.
- -->
<!-- Dynamic field definitions allow using convention over configuration
for fields via the specification of patterns to match field names.
@@ -184,10 +119,6 @@
RESTRICTION: the glob-like pattern in the name attribute must have
a "*" only at the start or the end. -->
- <!-- Type used to index the lat and lon components for the "location" FieldType -->
-
- <!-- some trie-coded dynamic fields for faster range queries -->
-
<!-- uncomment the following to ignore any fields that don't already match an existing
field name or dynamic field, rather than reporting them as an error.
alternately, change the type="ignored" to some other type e.g. "text" if you want
(allura) 05/05: add storeOffsetsWithPositions in case "unified" highligher is used in future, it gives better performance
Posted by gc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gcruz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/allura.git
commit 2b6eec65dd5603de7688b8249ca99ce596a75ea3
Author: Dave Brondsema <db...@slashdotmedia.com>
AuthorDate: Wed Aug 21 16:56:10 2024 -0400
add storeOffsetsWithPositions in case "unified" highligher is used in future, it gives better performance
---
solr_config/allura/conf/schema.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/solr_config/allura/conf/schema.xml b/solr_config/allura/conf/schema.xml
index 8c0059b51..db77da282 100644
--- a/solr_config/allura/conf/schema.xml
+++ b/solr_config/allura/conf/schema.xml
@@ -108,7 +108,7 @@
<!-- Required field values for SOLR, DO NOT REMOVE -->
<field name="_version_" type="long" indexed="true" stored="true" multiValued="false" />
<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
- <field name="text" type="text_general" indexed="true" stored="true" multiValued="true"/>
+ <field name="text" type="text_general" indexed="true" stored="true" multiValued="true" storeOffsetsWithPositions="true"/>
<!-- END of required field values -->
<field name="title" type="text_general" indexed="true" stored="true" multiValued="true"/>