You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by GitBox <gi...@apache.org> on 2021/03/23 00:22:24 UTC
[GitHub] [solr] thelabdude opened a new pull request #42: SOLR-15277: Schema designer UI and supporting backend
thelabdude opened a new pull request #42:
URL: https://github.com/apache/solr/pull/42
# Description
See https://issues.apache.org/jira/browse/SOLR-15277
# Solution
WIP ~ still needs some hardening, more unit tests, and so on ... but close enough to my original vision to start getting feedback.
# Tests
Some happy path unit tests in the PR but more needed ...
# Checklist
Please review the following and check all that apply:
- [ ] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability.
- [ ] I have created a Jira issue and added the issue ID to my pull request title.
- [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
- [ ] I have developed this patch against the `main` branch.
- [ ] I have run `./gradlew check`.
- [ ] I have added tests for my changes.
- [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [solr] HoustonPutman edited a comment on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
HoustonPutman edited a comment on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-845376159
Some thoughts while playing around:
- [ ] I would like to be able to populate the example document with a document that exists. Maybe a button in the query results.
- [x] docValues should be enabled by default when creating a fieldType/field
- [ ] Do we recommend the use of `useDocValuesAsStored` instead of `stored`?
- [ ] When you analyze the fields it sometimes brings up a question about language and some options. Not clear that this is talking about the entire schema. Would be better to highlight the configName in the menu.
- [ ] Maybe guess long strings as text fields, not string fields?
- [ ] Would be nice to be able to filter on a field not having a feature (docValues)
- [ ] When filtering fields, it would be nice to filter the dynamic fields as well.
- [ ] In the future it would be nice to have information on each of the options. Not sure what the best way to go about this is though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] MarcusSorealheis commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-853507516
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
thelabdude commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-848845362
Thanks for taking a look @HoustonPutman ... I addressed most of your concerns but there's some nuance around a few of your requests:
> I would like to be able to populate the example document with a document that exists. Maybe a button in the query results.
I haven't implemented this yet. The query results only include sample documents already, so maybe you want to revise the sample documents in the paste text area? I think that's a valid thing to do and will think about this some more. My only concern would be adding another option that may complicate the experience. Can you elaborate on the user experience you want here?
> Do we recommend the use of useDocValuesAsStored instead of stored?
I'm not sure about whether that is a recommendation? Seems reasonable to me that if you're already using DocValues, then `useDocValuesAsStored` seems more efficient from a storage perspective (if you store and use doc values, then seems like you're doing double storage), but maybe there is some performance hit for doing this? I can certainly turn that off if we don't like defaulting to stored=false and useDocValuesAsStored=true.
> When you analyze the fields it sometimes brings up a question about language and some options. Not clear that this is talking about the entire schema. Would be better to highlight the configName in the menu.
There are some gremlins in that jstree we're using where the selected node doesn't always show as selected in the tree. I've tried to improve this with my latest commit and seems to be rendering correctly. When the root node is selected, you see the panel with languages and options. Those apply to the entire schema. I added the schema name to that panel to help make this clear.
> In the future it would be nice to have information on each of the options. Not sure what the best way to go about this is though.
I think some of the options are pretty confusing and need supporting documentation, so have deferred to the ref guide via the help icon links vs. adding more docs directly in the Schema Designer UI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman edited a comment on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
HoustonPutman edited a comment on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-845376159
Some thoughts while playing around:
- [ ] I would like to be able to populate the example document with a document that exists. Maybe a button in the query results.
- [x] docValues should be enabled by default when creating a fieldType/field
- [ ] Do we recommend the use of `useDocValuesAsStored` instead of `stored`?
- [x] When you analyze the fields it sometimes brings up a question about language and some options. Not clear that this is talking about the entire schema. Would be better to highlight the configName in the menu.
- [x] Maybe guess long strings as text fields, not string fields?
- [x] Would be nice to be able to filter on a field not having a feature (docValues)
- [x] When filtering fields, it would be nice to filter the dynamic fields as well.
- [ ] In the future it would be nice to have information on each of the options. Not sure what the best way to go about this is though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman edited a comment on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
HoustonPutman edited a comment on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-845376159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] epugh commented on a change in pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
epugh commented on a change in pull request #42:
URL: https://github.com/apache/solr/pull/42#discussion_r637090384
##########
File path: solr/webapp/web/partials/schema-designer.html
##########
@@ -0,0 +1,919 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<div id="designer">
+ <div class="clearfix">
+ <div id="designer-top">
+ <div id="actions" class="actions">
+
+ <div id="schema-selector">
+ <div class="left">
+ <select id="select-schema" placeholder-text-single="'Schema Selector'"
+ chosen
+ ng-model="currentSchema"
+ ng-change="loadSchema()"
+ ng-options="schema for schema in schemas"></select>
+ <button id="add" class="action" ng-click="showNewSchemaDialog()"><span>New Schema</span></button>
+ </div>
+ <div class="middle">
+ <div id="schema-actions" class="schema-actions clearfix">
+ <button id="addField" class="action" ng-click="toggleAddField('field')" ng-show="showSchemaActions"><span>Add Field</span></button>
+ <button id="addFieldType" class="action" ng-click="toggleAddField('type')" ng-show="showSchemaActions"><span>Add Field Type</span></button>
+ <div id="updateStatusMessage" ng-class="{working: updateWorking}" ng-show="updateStatusMessage"><span>{{updateStatusMessage}}</span></div>
+
+ <div class="action add" data-rel="add" ng-show="showAddField" escape-pressed="hideAll()">
+
+ <p class="clearfix"><label for="add_name">name:</label>
+ <input type="text" id="add_name" ng-model="newField.name" focus-when="showAddField" placeholder="enter a name"></p>
+
+ <p class="clearfix" ng-show="adding=='field'"><label for="add_type">field type:</label>
+ <select chosen type="text" id="add_type" ng-model="newField.type" ng-options="type for type in types"></select>
+ </p>
+ <p class="clearfix" ng-show="adding=='type'"><label for="add_class">class:</label>
+ <input type="text" id="add_class" ng-model="newField.class" placeholder="class name"></p>
+
+ <p class="clearfix" ng-show="adding=='field'"><label for="add_default">default:</label>
+ <input type="text" id="add_default" ng-model="newField.default" placeholder="enter a default value if needed"></p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_stored">
+ <input type="checkbox" ng-model="newField.stored" id="add_stored" title="Full field should be stored in index." ng-true-value="'true'" ng-false-value="'false'">
+ stored
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_indexed">
+ <input type="checkbox" ng-model="newField.indexed" id="add_indexed" title="Field should be indexed." ng-true-value="'true'" ng-false-value="'false'">
+ indexed
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_uninvertible">
+ <input type="checkbox" ng-model="newField.uninvertible" id="add_uninvertible" title="Field should be uninvertible, it is generally recomended to use docValues instead."
+ ng-true-value="'true'" ng-false-value="'false'">
+ uninvertible
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_docValues">
+ <input type="checkbox" ng-model="newField.docValues" id="add_docValues" title="DocValues should be stored for the field." ng-true-value="'true'" ng-false-value="'false'">
+ docValues
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_multiValued">
+ <input type="checkbox" ng-model="newField.multiValued" id="add_multiValued" title="Multiple values are allowed for this field." ng-true-value="'true'" ng-false-value="'false'">
+ multiValued
+ </label>
+ </p>
+
+ <p class="clearfix" ng-show="adding=='field'">
+ <label class="checkbox" for="add_required">
+ <input type="checkbox" ng-model="newField.required" id="add_required" title="Field must be provided for all documents." ng-true-value="'true'" ng-false-value="'false'">
+ required
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <a ng-click="showOmit=!showOmit">
+ <span class="add_showhide" ng-hide="showOmit">Show omit options</span>
+ <span class="add_showhide open" ng-show="showOmit">Hide omit options</span>
+ </a>
+ </p>
+
+ <div ng-show="showOmit">
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_omitNorms">
+ <input type="checkbox" ng-model="newField.omitNorms" id="add_omitNorms" title="Full field should be omitNorms in index." ng-true-value="'true'" ng-false-value="'false'">
+ omitNorms
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_omitTermFreqAndPositions">
+ <input type="checkbox" ng-model="newField.omitTermFreqAndPositions" id="add_omitTermFreqAndPositions" title="Full field should be omitTermFreqAndPositions in index."
+ ng-true-value="'true'" ng-false-value="'false'">
+ omitTermFreqAndPositions
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_omitPositions">
+ <input type="checkbox" ng-model="newField.omitPositions" id="add_omitPositions" title="Full field should be omitPositions in index." ng-true-value="'true'" ng-false-value="'false'">
+ omitPositions
+ </label>
+ </p>
+ </div>
+
+ <p class="clearfix">
+ <a ng-click="showTermVectors=!showTermVectors">
+ <span class="add_showhide" ng-hide="showTermVectors">Show term vector options</span>
+ <span class="add_showhide open" ng-show="showTermVectors">Hide term vector options</span>
+ </a>
+ </p>
+ <div ng-show="showTermVectors">
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_termVectors">
+ <input type="checkbox" ng-model="newField.termVectors" id="add_termVectors" title="Full field should be termVectors in index." ng-true-value="'true'" ng-false-value="'false'">
+ termVectors
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_termPositions">
+ <input type="checkbox" ng-model="newField.termPositions" id="add_termPositions" title="Full field should be termPositions in index." ng-true-value="'true'" ng-false-value="'false'">
+ termPositions
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_termOffsets">
+ <input type="checkbox" ng-model="newField.termOffsets" id="add_termOffsets" title="Full field should be termOffsets in index." ng-true-value="'true'" ng-false-value="'false'">
+ termOffsets
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_termPayloads">
+ <input type="checkbox" ng-model="newField.termPayloads" id="add_termPayloads" title="Full field should be termPayloads in index." ng-true-value="'true'" ng-false-value="'false'">
+ termPayloads
+ </label>
+ </p>
+
+ </div>
+
+ <p class="clearfix">
+ <a ng-click="showSort=!showSort">
+ <span class="add_showhide" ng-hide="showSort">Show sort options</span>
+ <span class="add_showhide open" ng-show="showSort">Show sort options</span>
+ </a>
+ </p>
+ <div ng-show="showSort">
+ <p class="clearfix">
+ <label class="checkbox" for="add_sortMissingFirst">
+ <input type="checkbox" ng-model="newField.sortMissingFirst" id="add_sortMissingFirst" title="Full field should be sortMissingFirst in index." ng-true-value="'true'"
+ ng-false-value="'false'">
+ sortMissingFirst
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_sortMissingLast">
+ <input type="checkbox" ng-model="newField.sortMissingLast" id="add_sortMissingLast" title="Full field should be sortMissingLast in index." ng-true-value="'true'"
+ ng-false-value="'false'">
+ sortMissingLast
+ </label>
+ </p>
+
+ </div>
+
+ <p class="clearfix" ng-show="adding=='type'">
+ <a ng-click="showAnalysisJson=!showAnalysisJson">
+ <span class="add_showhide" ng-hide="showAnalysisJson">Show text analysis JSON</span>
+ <span class="add_showhide open" ng-show="showAnalysisJson">Hide text analysis JSON</span>
+ </a>
+ </p>
+ <div ng-show="showAnalysisJson">
+ <textarea ng-model="textAnalysisJson" name="add_analysis_json" id="add_analysis_json" title="Text Analysis JSON" rows="8" cols="40" placeholder=""></textarea>
+ </div>
+
+ <p class="clearfix buttons">
+ <button type="submit" class="submit" ng-class="{success: added}" ng-click="addField()"><span>Add</span></button>
+ <button type="reset" class="reset" ng-click="toggleAddField()"><span>Cancel</span></button>
+ </p>
+
+ <div id="add-errors">
+ <div ng-repeat="error in addErrors" ng-show="addErrors" class="error"><span>{{error}}</span></div>
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="right">
+ <div ng-show="showSchemaActions">
+ <button class="action publish-button" ng-click="togglePublish($event)"><span>Publish</span></button>
+ <button id="download-button" class="action" ng-click="downloadConfig()"><span>Download Config</span></button>
+ </div>
+ </div>
+ </div>
+
+ <div id="error-dialog" class="error-dialog" ng-show="designerAPIError">
+ <div id="error-dialog-note"><p class="clearfix"><img src="img/ico/prohibition.png"/> {{designerAPIError}}</p></div>
+ <div id="error-dialog-details"><div ng-show="designerAPIErrorDetails"><textarea rows="10" cols="55">{{designerAPIErrorDetails}}</textarea></div></div>
+ <div id="error-dialog-buttons" class="clearfix">
+ <button type="reset" class="reload-error-button" ng-click="closeErrorDialog()" ng-show="isVersionMismatch"><span>Reload Schema</span></button>
+ <button type="reset" class="error-button" ng-click="closeErrorDialog()" ng-show="!isVersionMismatch"><span>OK</span></button>
+ </div>
+ </div>
+
+ <div id="show-diff-dialog" class="diff" ng-show="showDiff && showPublish">
+ <div ng-hide="schemaDiffExists" ng-show="!schemaDiffExists">
+ <p class="clearfix diff-text">No differences found.</p>
+ </div>
+ {{fieldsDiff=schemaDiff.fieldsDiff;""}}
+ <div id="fields-diff" ng-hide="fieldsDiff == null" ng-show="Object.keys(fieldsDiff).length > 0">
+ <div id="fields-updated" ng-hide="fieldsDiff.updated == null" ng-show="Object.keys(fieldsDiff.updated).length > 0">
+ <p class="clearfix diff-text">Updated Fields</p>
+ <table class="diff">
+ <tr>
+ <th>Name</th>
+ <th>Old</th>
+ <th>New</th>
+ </tr>
+ <tr ng-repeat="(field, fieldDiff) in fieldsDiff.updated">
+ <td>{{ field }}</td>
+ <td><p class="old-value">{{fieldDiff[0] | json}}</p></td>
+ <td><p class="new-value">{{fieldDiff[1] | json}}</p></td>
+ </tr>
+ </table>
+ </div>
+ <div id="fields-added" ng-show="schemaDiff.addedFields.length > 0">
+ <p class="clearfix diff-text">New Fields</p>
+ <ul>
+ <li class="clearfix element" ng-repeat="f in schemaDiff.addedFields">
+ <p class="new-value"> {{f}} </p>
+ </li>
+ </ul>
+ </div>
+ <div id="fields-removed" ng-show="schemaDiff.removedFields.length > 0">
+ <p class="clearfix diff-text">Removed Fields</p>
+ <ul>
+ <li class="clearfix element" ng-repeat="f in schemaDiff.removedFields">
+ <p class="rem-value"> {{f}} </p>
+ </li>
+ </ul>
+ </div>
+ </div>
+ {{fieldTypesDiff=schemaDiff.fieldTypesDiff;""}}
+ <div id="fieldtypes-diff" ng-hide="fieldTypesDiff == null" ng-show="Object.keys(fieldTypesDiff).length > 0">
+ <div id="fieldtypes-updated" ng-hide="fieldTypesDiff.updated == null" ng-show="Object.keys(fieldTypesDiff.updated).length > 0">
+ <p class="clearfix diff-text">Updated Field Types</p>
+ <table class="diff" ng-show="Object.keys(fieldTypesDiff.updated).length > 0">
+ <tr>
+ <th>Name</th>
+ <th>Old</th>
+ <th>New</th>
+ </tr>
+ <tr ng-repeat="(fieldType, fieldTypeDiff) in fieldTypesDiff.updated">
+ <td>{{ fieldType }}</td>
+ <td><p class="old-value">{{fieldTypeDiff[0] | json}}</p></td>
+ <td><p class="new-value">{{fieldTypeDiff[1] | json}}</p></td>
+ </tr>
+ </table>
+ </div>
+ <div id="fieldtypes-added" ng-hide="fieldTypesDiff.added == null" ng-show="Object.keys(fieldTypesDiff.added).length > 0">
+ <p class="clearfix diff-text">New Field Types</p>
+ <ul>
+ <li class="clearfix element" ng-repeat="fieldtype in fieldTypesDiff.added">
+ <p class="new-value"> {{fieldtype | json}} </p>
+ </li>
+ </ul>
+ </div>
+ <div id="fieldtypes-removed" ng-show="schemaDiff.removedTypes.length > 0">
+ <p class="clearfix diff-text">Removed Field Types</p>
+ <ul>
+ <li class="clearfix element" ng-repeat="fieldtype in schemaDiff.removedTypes">
+ <p class="rem-value"> {{fieldtype}} </p>
+ </li>
+ </ul>
+ </div>
+ </div>
+ {{dynamicFieldsDiff=schemaDiff.dynamicFieldsDiff;""}}
+ <div id="dynamicfields-diff" ng-hide="dynamicFieldsDiff == null" ng-show="Object.keys(dynamicFieldsDiff).length > 0">
+ <div id="dynamicfields-updated" ng-hide="dynamicFieldsDiff.updated == null" ng-show="Object.keys(dynamicFieldsDiff.updated).length > 0">
+ <p class="clearfix diff-text">Updated Dynamic Fields</p>
+ <table class="diff">
+ <tr>
+ <th>Name</th>
+ <th>Old</th>
+ <th>New</th>
+ </tr>
+ <tr ng-repeat="(dfield, dfieldDiff) in dynamicFieldsDiff.updated">
+ <td>{{ dfield }}</td>
+ <td><p class="old-value">{{dfieldDiff[0] | json}}</p></td>
+ <td><p class="new-value">{{dfieldDiff[1] | json}}</p></td>
+ </tr>
+ </table>
+ </div>
+ <div id="dynamicfields-added" ng-hide="dynamicFieldsDiff.added == null" ng-show="Object.keys(dynamicFieldsDiff.added).length > 0">
+ <p class="clearfix diff-text">New Dynamic Fields</p>
+ <ul>
+ <li class="clearfix" ng-repeat="dfield in dynamicFieldsDiff.added">
+ <p class="new-value"> {{dfield | json}} </p>
+ </li>
+ </ul>
+ </div>
+ <div id="dynamicfields-removed" ng-hide="dynamicFieldsDiff.removed == null" ng-show="Object.keys(dynamicFieldsDiff.removed).length > 0">
+ <p class="clearfix diff-text">Removed Dynamic Fields</p>
+ <ul>
+ <li class="clearfix" ng-repeat="dfield in dynamicFieldsDiff.removed">
+ <p class="rem-value"> {{dfield.name}} </p>
+ </li>
+ </ul>
+ </div>
+ </div>
+ {{copyFieldsDiff=schemaDiff.copyFieldsDiff;""}}
+ <div id="copyfields-diff" ng-hide="copyFieldsDiff == null" ng-show="Object.keys(copyFieldsDiff).length > 0">
+ <p class="clearfix diff-text">Copy Fields</p>
+ <table class="diff" ng-show="copyFieldsDiff.old.length > 0 || copyFieldsDiff.new.length > 0">
+ <tr>
+ <th>Source</th>
+ <th>Destination</th>
+ </tr>
+ <tr ng-show="copyFieldsDiff.old.length > 0" ng-repeat="cfield in copyFieldsDiff.old">
+ <td><p class="old-value">{{cfield.source}}</p></td>
+ <td><p class="old-value">{{cfield.dest}}</p></td>
+ </tr>
+ <tr ng-show="copyFieldsDiff.new.length > 0" ng-repeat="cfield in copyFieldsDiff.new">
+ <td><p class="new-value">{{cfield.source}}</p></td>
+ <td><p class="new-value">{{cfield.dest}}</p></td>
+ </tr>
+ </table>
+ </div>
+
+ <div class="clearfix diff-buttons">
+ <button type="reset" class="reset" ng-click="toggleDiff()"><span>Close</span></button>
+ </div>
+ </div>
+
+ <div id="publish-dialog" class="publish" ng-show="showPublish" escape-pressed="hideAll()">
+ <div id="publish-dialog-note"><p class="clearfix">Publish schema and associated configs to Zookeeper as a Solr ConfigSet.</p></div>
+ <button class="action diff-button" ng-click="toggleDiff($event)"><span>Schema Diff</span></button>
+ <div id="publish-affected">
+ <p class="clearfix">Affected collections:</p>
+ <ul>
+ <li class="clearfix" ng-repeat="coll in collectionsForConfig">{{coll}}</li>
+ </ul>
+ <div ng-show="collectionsForConfig.length == 0" class="clearfix">No existing collections</div>
+ </div>
+ <div id="reload-form">
+ <div class="field-form" ng-show="collectionsForConfig.length > 0">
+ <input type="checkbox" ng-model="reloadOnPublish" id="reload_on_publish" title="reload affected collections" ng-true-value="'true'" ng-false-value="'false'"/>
+ <label for="reload_on_publish" class="checkbox" title="Reload affected collections">Reload affected collections?</label>
+ </div>
+ </div>
+
+ <div id="publish-new-coll">
+ <p>Add new collection with published config?</p>
+ <form>
+ <p class="clearfix"><label for="add_coll_name">name:</label>
+ <input type="text" name="name" id="add_coll_name" ng-model="newCollection.name" placeholder="new collection"></p>
+
+ <p class="clearfix"><label for="add_coll_numShards">numShards:</label>
+ <input type="text" name="numShards" id="add_coll_numShards" ng-model="newCollection.numShards"></p>
+
+ <p class="clearfix"><label for="add_coll_replicationFactor">replicationFactor:</label>
+ <input type="text" name="replicationFactor" id="add_coll_replicationFactor" ng-model="newCollection.replicationFactor"></p>
+ </form>
+ <div class="field-form">
+ <input type="checkbox" ng-model="newCollection.indexToCollection" id="add_coll_index_sample" title="index sample docs" ng-true-value="'true'" ng-false-value="'false'"/>
+ <label for="add_coll_index_sample" class="checkbox" title="Index sample docs">Index sample docs in new collection?</label>
+ </div>
+ </div>
+ <div class="field-form">
+ <input type="checkbox" ng-model="disableDesigner" id="disable_designer" title="Disable future changes in schema designer" ng-true-value="'true'" ng-false-value="'false'"/>
+ <label for="disable_designer" class="checkbox" title="Disable future changes in schema designer">Disable future changes by the schema designer?</label>
+ </div>
+ <div ng-repeat="error in publishErrors" ng-show="publishErrors" class="clearfix note error"><span>{{error}}</span></div>
+ <div class="clearfix publish-buttons">
+ <button type="submit" class="action publish-button" ng-click="doPublish()"><span>Publish</span></button>
+ <button type="reset" class="reset" ng-click="togglePublish()"><span>Cancel</span></button>
+ </div>
+ </div>
+
+ <div id="confirm-dialog" class="action add" data-rel="add" ng-show="showConfirmEditSchema" style="display:block;">
+ <p class="clearfix warn"><span>Warning: You've chosen to load an existing schema that is already being used by active collections. Making changes to the '{{confirmSchema}}' schema will impact existing documents in these collections. Please proceed with caution.<br/><br/></span></p>
Review comment:
"these collections" or should it be "those collections". nitpick I know!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
thelabdude commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-854135714
Schema Designer only runs in cloud-mode, from your screenshot, you're running in standalone. See screenshots of the Schema Designer UI in the JIRA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman edited a comment on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
HoustonPutman edited a comment on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-845376159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] MarcusSorealheis commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-854932531
> This doesn't sound like it is Schema Designer UI specific. I believe I'm referencing icons the same way all the other screens do. If you think this is a Schema Designer specific issue, please provide a link to the offending code. Otherwise, please open another JIRA to address that problem independently of this issue.
I added it here [#164](https://github.com/apache/solr/pull/164)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude edited a comment on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
thelabdude edited a comment on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-853963136
> Schema Designer should not simply throw generic errors that do no help the user. For instance, when a user tries to create a field that already exists, there is just an error processing banner. Ideally, you could grab the important detail in the response body and show it to the user. Consider `data["error"]["details"][0]["errorMessages"][0]` as seen below to populate the banner with something helpful to the user:
>
> ```
> Possibly unhandled rejection: {"data":{"responseHeader":{"status":400,"QTime":1},"error":{"metadata":["error-class","org.apache.solr.api.ApiBag$ExceptionWithErrObject","root-error-class","org.apache.solr.api.ApiBag$ExceptionWithErrObject"],"details":[{"add-field":{"stored":"true","indexed":"true","uninvertible":"true","name":"name","type":"string"},"errorMessages":["Field 'name' already exists.\n"]}],"msg":"error processing commands","code":400}},"status":400,"config":{"method":"POST","transformRequest":[null],"transformResponse":[null],"jsonpCallbackParam":"callback","data":{"add-field":{"stored":"true","indexed":"true","uninvertible":"true","name":"name","type":"string"}},"url":"techproducts/schema","params":{"wt":"json","_":1622686070279},"headers":{"Accept":"application/json, text/plain, */*","X-Requested-With":"XMLHttpRequest","Content-Type":"application/json;charset=utf-8"},"timeout":10000},"statusText":"Bad Request","xhrStatus":"complete","resource":{"add-field":"..."}}``
`
> ```
Can you please provide reproduction steps for how you got this error (including browser details)? Schema Designer UI should be displaying user friendly errors in a dialog. Moreover, I don't even know how you got this error since the JS code validates the field doesn't already exist before submitting the request. See screenshot of what I see in my env. <img width="357" alt="Screen Shot 2021-06-03 at 9 32 19 AM" src="https://user-images.githubusercontent.com/417074/120671957-1c3aad00-c44f-11eb-8c62-ff6051ba9316.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] epugh commented on a change in pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
epugh commented on a change in pull request #42:
URL: https://github.com/apache/solr/pull/42#discussion_r637090855
##########
File path: solr/webapp/web/partials/schema-designer.html
##########
@@ -0,0 +1,919 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<div id="designer">
+ <div class="clearfix">
+ <div id="designer-top">
+ <div id="actions" class="actions">
+
+ <div id="schema-selector">
+ <div class="left">
+ <select id="select-schema" placeholder-text-single="'Schema Selector'"
+ chosen
+ ng-model="currentSchema"
+ ng-change="loadSchema()"
+ ng-options="schema for schema in schemas"></select>
+ <button id="add" class="action" ng-click="showNewSchemaDialog()"><span>New Schema</span></button>
+ </div>
+ <div class="middle">
+ <div id="schema-actions" class="schema-actions clearfix">
+ <button id="addField" class="action" ng-click="toggleAddField('field')" ng-show="showSchemaActions"><span>Add Field</span></button>
+ <button id="addFieldType" class="action" ng-click="toggleAddField('type')" ng-show="showSchemaActions"><span>Add Field Type</span></button>
+ <div id="updateStatusMessage" ng-class="{working: updateWorking}" ng-show="updateStatusMessage"><span>{{updateStatusMessage}}</span></div>
+
+ <div class="action add" data-rel="add" ng-show="showAddField" escape-pressed="hideAll()">
+
+ <p class="clearfix"><label for="add_name">name:</label>
+ <input type="text" id="add_name" ng-model="newField.name" focus-when="showAddField" placeholder="enter a name"></p>
+
+ <p class="clearfix" ng-show="adding=='field'"><label for="add_type">field type:</label>
+ <select chosen type="text" id="add_type" ng-model="newField.type" ng-options="type for type in types"></select>
+ </p>
+ <p class="clearfix" ng-show="adding=='type'"><label for="add_class">class:</label>
+ <input type="text" id="add_class" ng-model="newField.class" placeholder="class name"></p>
+
+ <p class="clearfix" ng-show="adding=='field'"><label for="add_default">default:</label>
+ <input type="text" id="add_default" ng-model="newField.default" placeholder="enter a default value if needed"></p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_stored">
+ <input type="checkbox" ng-model="newField.stored" id="add_stored" title="Full field should be stored in index." ng-true-value="'true'" ng-false-value="'false'">
+ stored
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_indexed">
+ <input type="checkbox" ng-model="newField.indexed" id="add_indexed" title="Field should be indexed." ng-true-value="'true'" ng-false-value="'false'">
+ indexed
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_uninvertible">
+ <input type="checkbox" ng-model="newField.uninvertible" id="add_uninvertible" title="Field should be uninvertible, it is generally recomended to use docValues instead."
+ ng-true-value="'true'" ng-false-value="'false'">
+ uninvertible
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_docValues">
+ <input type="checkbox" ng-model="newField.docValues" id="add_docValues" title="DocValues should be stored for the field." ng-true-value="'true'" ng-false-value="'false'">
+ docValues
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_multiValued">
+ <input type="checkbox" ng-model="newField.multiValued" id="add_multiValued" title="Multiple values are allowed for this field." ng-true-value="'true'" ng-false-value="'false'">
+ multiValued
+ </label>
+ </p>
+
+ <p class="clearfix" ng-show="adding=='field'">
+ <label class="checkbox" for="add_required">
+ <input type="checkbox" ng-model="newField.required" id="add_required" title="Field must be provided for all documents." ng-true-value="'true'" ng-false-value="'false'">
+ required
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <a ng-click="showOmit=!showOmit">
+ <span class="add_showhide" ng-hide="showOmit">Show omit options</span>
+ <span class="add_showhide open" ng-show="showOmit">Hide omit options</span>
+ </a>
+ </p>
+
+ <div ng-show="showOmit">
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_omitNorms">
+ <input type="checkbox" ng-model="newField.omitNorms" id="add_omitNorms" title="Full field should be omitNorms in index." ng-true-value="'true'" ng-false-value="'false'">
+ omitNorms
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_omitTermFreqAndPositions">
+ <input type="checkbox" ng-model="newField.omitTermFreqAndPositions" id="add_omitTermFreqAndPositions" title="Full field should be omitTermFreqAndPositions in index."
+ ng-true-value="'true'" ng-false-value="'false'">
+ omitTermFreqAndPositions
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_omitPositions">
+ <input type="checkbox" ng-model="newField.omitPositions" id="add_omitPositions" title="Full field should be omitPositions in index." ng-true-value="'true'" ng-false-value="'false'">
+ omitPositions
+ </label>
+ </p>
+ </div>
+
+ <p class="clearfix">
+ <a ng-click="showTermVectors=!showTermVectors">
+ <span class="add_showhide" ng-hide="showTermVectors">Show term vector options</span>
+ <span class="add_showhide open" ng-show="showTermVectors">Hide term vector options</span>
+ </a>
+ </p>
+ <div ng-show="showTermVectors">
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_termVectors">
+ <input type="checkbox" ng-model="newField.termVectors" id="add_termVectors" title="Full field should be termVectors in index." ng-true-value="'true'" ng-false-value="'false'">
+ termVectors
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_termPositions">
+ <input type="checkbox" ng-model="newField.termPositions" id="add_termPositions" title="Full field should be termPositions in index." ng-true-value="'true'" ng-false-value="'false'">
+ termPositions
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_termOffsets">
+ <input type="checkbox" ng-model="newField.termOffsets" id="add_termOffsets" title="Full field should be termOffsets in index." ng-true-value="'true'" ng-false-value="'false'">
+ termOffsets
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_termPayloads">
+ <input type="checkbox" ng-model="newField.termPayloads" id="add_termPayloads" title="Full field should be termPayloads in index." ng-true-value="'true'" ng-false-value="'false'">
+ termPayloads
+ </label>
+ </p>
+
+ </div>
+
+ <p class="clearfix">
+ <a ng-click="showSort=!showSort">
+ <span class="add_showhide" ng-hide="showSort">Show sort options</span>
+ <span class="add_showhide open" ng-show="showSort">Show sort options</span>
+ </a>
+ </p>
+ <div ng-show="showSort">
+ <p class="clearfix">
+ <label class="checkbox" for="add_sortMissingFirst">
+ <input type="checkbox" ng-model="newField.sortMissingFirst" id="add_sortMissingFirst" title="Full field should be sortMissingFirst in index." ng-true-value="'true'"
+ ng-false-value="'false'">
+ sortMissingFirst
+ </label>
+ </p>
+
+ <p class="clearfix">
+ <label class="checkbox" for="add_sortMissingLast">
+ <input type="checkbox" ng-model="newField.sortMissingLast" id="add_sortMissingLast" title="Full field should be sortMissingLast in index." ng-true-value="'true'"
+ ng-false-value="'false'">
+ sortMissingLast
+ </label>
+ </p>
+
+ </div>
+
+ <p class="clearfix" ng-show="adding=='type'">
+ <a ng-click="showAnalysisJson=!showAnalysisJson">
+ <span class="add_showhide" ng-hide="showAnalysisJson">Show text analysis JSON</span>
+ <span class="add_showhide open" ng-show="showAnalysisJson">Hide text analysis JSON</span>
+ </a>
+ </p>
+ <div ng-show="showAnalysisJson">
+ <textarea ng-model="textAnalysisJson" name="add_analysis_json" id="add_analysis_json" title="Text Analysis JSON" rows="8" cols="40" placeholder=""></textarea>
+ </div>
+
+ <p class="clearfix buttons">
+ <button type="submit" class="submit" ng-class="{success: added}" ng-click="addField()"><span>Add</span></button>
+ <button type="reset" class="reset" ng-click="toggleAddField()"><span>Cancel</span></button>
+ </p>
+
+ <div id="add-errors">
+ <div ng-repeat="error in addErrors" ng-show="addErrors" class="error"><span>{{error}}</span></div>
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="right">
+ <div ng-show="showSchemaActions">
+ <button class="action publish-button" ng-click="togglePublish($event)"><span>Publish</span></button>
+ <button id="download-button" class="action" ng-click="downloadConfig()"><span>Download Config</span></button>
+ </div>
+ </div>
+ </div>
+
+ <div id="error-dialog" class="error-dialog" ng-show="designerAPIError">
+ <div id="error-dialog-note"><p class="clearfix"><img src="img/ico/prohibition.png"/> {{designerAPIError}}</p></div>
+ <div id="error-dialog-details"><div ng-show="designerAPIErrorDetails"><textarea rows="10" cols="55">{{designerAPIErrorDetails}}</textarea></div></div>
+ <div id="error-dialog-buttons" class="clearfix">
+ <button type="reset" class="reload-error-button" ng-click="closeErrorDialog()" ng-show="isVersionMismatch"><span>Reload Schema</span></button>
+ <button type="reset" class="error-button" ng-click="closeErrorDialog()" ng-show="!isVersionMismatch"><span>OK</span></button>
+ </div>
+ </div>
+
+ <div id="show-diff-dialog" class="diff" ng-show="showDiff && showPublish">
+ <div ng-hide="schemaDiffExists" ng-show="!schemaDiffExists">
+ <p class="clearfix diff-text">No differences found.</p>
+ </div>
+ {{fieldsDiff=schemaDiff.fieldsDiff;""}}
+ <div id="fields-diff" ng-hide="fieldsDiff == null" ng-show="Object.keys(fieldsDiff).length > 0">
+ <div id="fields-updated" ng-hide="fieldsDiff.updated == null" ng-show="Object.keys(fieldsDiff.updated).length > 0">
+ <p class="clearfix diff-text">Updated Fields</p>
+ <table class="diff">
+ <tr>
+ <th>Name</th>
+ <th>Old</th>
+ <th>New</th>
+ </tr>
+ <tr ng-repeat="(field, fieldDiff) in fieldsDiff.updated">
+ <td>{{ field }}</td>
+ <td><p class="old-value">{{fieldDiff[0] | json}}</p></td>
+ <td><p class="new-value">{{fieldDiff[1] | json}}</p></td>
+ </tr>
+ </table>
+ </div>
+ <div id="fields-added" ng-show="schemaDiff.addedFields.length > 0">
+ <p class="clearfix diff-text">New Fields</p>
+ <ul>
+ <li class="clearfix element" ng-repeat="f in schemaDiff.addedFields">
+ <p class="new-value"> {{f}} </p>
+ </li>
+ </ul>
+ </div>
+ <div id="fields-removed" ng-show="schemaDiff.removedFields.length > 0">
+ <p class="clearfix diff-text">Removed Fields</p>
+ <ul>
+ <li class="clearfix element" ng-repeat="f in schemaDiff.removedFields">
+ <p class="rem-value"> {{f}} </p>
+ </li>
+ </ul>
+ </div>
+ </div>
+ {{fieldTypesDiff=schemaDiff.fieldTypesDiff;""}}
+ <div id="fieldtypes-diff" ng-hide="fieldTypesDiff == null" ng-show="Object.keys(fieldTypesDiff).length > 0">
+ <div id="fieldtypes-updated" ng-hide="fieldTypesDiff.updated == null" ng-show="Object.keys(fieldTypesDiff.updated).length > 0">
+ <p class="clearfix diff-text">Updated Field Types</p>
+ <table class="diff" ng-show="Object.keys(fieldTypesDiff.updated).length > 0">
+ <tr>
+ <th>Name</th>
+ <th>Old</th>
+ <th>New</th>
+ </tr>
+ <tr ng-repeat="(fieldType, fieldTypeDiff) in fieldTypesDiff.updated">
+ <td>{{ fieldType }}</td>
+ <td><p class="old-value">{{fieldTypeDiff[0] | json}}</p></td>
+ <td><p class="new-value">{{fieldTypeDiff[1] | json}}</p></td>
+ </tr>
+ </table>
+ </div>
+ <div id="fieldtypes-added" ng-hide="fieldTypesDiff.added == null" ng-show="Object.keys(fieldTypesDiff.added).length > 0">
+ <p class="clearfix diff-text">New Field Types</p>
+ <ul>
+ <li class="clearfix element" ng-repeat="fieldtype in fieldTypesDiff.added">
+ <p class="new-value"> {{fieldtype | json}} </p>
+ </li>
+ </ul>
+ </div>
+ <div id="fieldtypes-removed" ng-show="schemaDiff.removedTypes.length > 0">
+ <p class="clearfix diff-text">Removed Field Types</p>
+ <ul>
+ <li class="clearfix element" ng-repeat="fieldtype in schemaDiff.removedTypes">
+ <p class="rem-value"> {{fieldtype}} </p>
+ </li>
+ </ul>
+ </div>
+ </div>
+ {{dynamicFieldsDiff=schemaDiff.dynamicFieldsDiff;""}}
+ <div id="dynamicfields-diff" ng-hide="dynamicFieldsDiff == null" ng-show="Object.keys(dynamicFieldsDiff).length > 0">
+ <div id="dynamicfields-updated" ng-hide="dynamicFieldsDiff.updated == null" ng-show="Object.keys(dynamicFieldsDiff.updated).length > 0">
+ <p class="clearfix diff-text">Updated Dynamic Fields</p>
+ <table class="diff">
+ <tr>
+ <th>Name</th>
+ <th>Old</th>
+ <th>New</th>
+ </tr>
+ <tr ng-repeat="(dfield, dfieldDiff) in dynamicFieldsDiff.updated">
+ <td>{{ dfield }}</td>
+ <td><p class="old-value">{{dfieldDiff[0] | json}}</p></td>
+ <td><p class="new-value">{{dfieldDiff[1] | json}}</p></td>
+ </tr>
+ </table>
+ </div>
+ <div id="dynamicfields-added" ng-hide="dynamicFieldsDiff.added == null" ng-show="Object.keys(dynamicFieldsDiff.added).length > 0">
+ <p class="clearfix diff-text">New Dynamic Fields</p>
+ <ul>
+ <li class="clearfix" ng-repeat="dfield in dynamicFieldsDiff.added">
+ <p class="new-value"> {{dfield | json}} </p>
+ </li>
+ </ul>
+ </div>
+ <div id="dynamicfields-removed" ng-hide="dynamicFieldsDiff.removed == null" ng-show="Object.keys(dynamicFieldsDiff.removed).length > 0">
+ <p class="clearfix diff-text">Removed Dynamic Fields</p>
+ <ul>
+ <li class="clearfix" ng-repeat="dfield in dynamicFieldsDiff.removed">
+ <p class="rem-value"> {{dfield.name}} </p>
+ </li>
+ </ul>
+ </div>
+ </div>
+ {{copyFieldsDiff=schemaDiff.copyFieldsDiff;""}}
+ <div id="copyfields-diff" ng-hide="copyFieldsDiff == null" ng-show="Object.keys(copyFieldsDiff).length > 0">
+ <p class="clearfix diff-text">Copy Fields</p>
+ <table class="diff" ng-show="copyFieldsDiff.old.length > 0 || copyFieldsDiff.new.length > 0">
+ <tr>
+ <th>Source</th>
+ <th>Destination</th>
+ </tr>
+ <tr ng-show="copyFieldsDiff.old.length > 0" ng-repeat="cfield in copyFieldsDiff.old">
+ <td><p class="old-value">{{cfield.source}}</p></td>
+ <td><p class="old-value">{{cfield.dest}}</p></td>
+ </tr>
+ <tr ng-show="copyFieldsDiff.new.length > 0" ng-repeat="cfield in copyFieldsDiff.new">
+ <td><p class="new-value">{{cfield.source}}</p></td>
+ <td><p class="new-value">{{cfield.dest}}</p></td>
+ </tr>
+ </table>
+ </div>
+
+ <div class="clearfix diff-buttons">
+ <button type="reset" class="reset" ng-click="toggleDiff()"><span>Close</span></button>
+ </div>
+ </div>
+
+ <div id="publish-dialog" class="publish" ng-show="showPublish" escape-pressed="hideAll()">
+ <div id="publish-dialog-note"><p class="clearfix">Publish schema and associated configs to Zookeeper as a Solr ConfigSet.</p></div>
+ <button class="action diff-button" ng-click="toggleDiff($event)"><span>Schema Diff</span></button>
+ <div id="publish-affected">
+ <p class="clearfix">Affected collections:</p>
+ <ul>
+ <li class="clearfix" ng-repeat="coll in collectionsForConfig">{{coll}}</li>
+ </ul>
+ <div ng-show="collectionsForConfig.length == 0" class="clearfix">No existing collections</div>
+ </div>
+ <div id="reload-form">
+ <div class="field-form" ng-show="collectionsForConfig.length > 0">
+ <input type="checkbox" ng-model="reloadOnPublish" id="reload_on_publish" title="reload affected collections" ng-true-value="'true'" ng-false-value="'false'"/>
+ <label for="reload_on_publish" class="checkbox" title="Reload affected collections">Reload affected collections?</label>
+ </div>
+ </div>
+
+ <div id="publish-new-coll">
+ <p>Add new collection with published config?</p>
+ <form>
+ <p class="clearfix"><label for="add_coll_name">name:</label>
+ <input type="text" name="name" id="add_coll_name" ng-model="newCollection.name" placeholder="new collection"></p>
+
+ <p class="clearfix"><label for="add_coll_numShards">numShards:</label>
+ <input type="text" name="numShards" id="add_coll_numShards" ng-model="newCollection.numShards"></p>
+
+ <p class="clearfix"><label for="add_coll_replicationFactor">replicationFactor:</label>
+ <input type="text" name="replicationFactor" id="add_coll_replicationFactor" ng-model="newCollection.replicationFactor"></p>
+ </form>
+ <div class="field-form">
+ <input type="checkbox" ng-model="newCollection.indexToCollection" id="add_coll_index_sample" title="index sample docs" ng-true-value="'true'" ng-false-value="'false'"/>
+ <label for="add_coll_index_sample" class="checkbox" title="Index sample docs">Index sample docs in new collection?</label>
+ </div>
+ </div>
+ <div class="field-form">
+ <input type="checkbox" ng-model="disableDesigner" id="disable_designer" title="Disable future changes in schema designer" ng-true-value="'true'" ng-false-value="'false'"/>
+ <label for="disable_designer" class="checkbox" title="Disable future changes in schema designer">Disable future changes by the schema designer?</label>
+ </div>
+ <div ng-repeat="error in publishErrors" ng-show="publishErrors" class="clearfix note error"><span>{{error}}</span></div>
+ <div class="clearfix publish-buttons">
+ <button type="submit" class="action publish-button" ng-click="doPublish()"><span>Publish</span></button>
+ <button type="reset" class="reset" ng-click="togglePublish()"><span>Cancel</span></button>
+ </div>
+ </div>
+
+ <div id="confirm-dialog" class="action add" data-rel="add" ng-show="showConfirmEditSchema" style="display:block;">
+ <p class="clearfix warn"><span>Warning: You've chosen to load an existing schema that is already being used by active collections. Making changes to the '{{confirmSchema}}' schema will impact existing documents in these collections. Please proceed with caution.<br/><br/></span></p>
Review comment:
actually, since you do list the collections, maybe these is right word!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] MarcusSorealheis commented on a change in pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis commented on a change in pull request #42:
URL: https://github.com/apache/solr/pull/42#discussion_r626131777
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -805,6 +806,12 @@ public void load() {
containerHandlers.getApiBag().registerObject(clusterAPI);
containerHandlers.getApiBag().registerObject(clusterAPI.commands);
containerHandlers.getApiBag().registerObject(clusterAPI.configSetCommands);
+
+ if (this.isZooKeeperAware()) {
+ SchemaDesignerAPI schemaDesignerAPI = new SchemaDesignerAPI(this);
+ containerHandlers.getApiBag().registerObject(schemaDesignerAPI);
+ } // Schema Designer not available in standalone (non-cloud) mode
+
Review comment:
ahh. That makes sense. I wasn't sure if there would be two different personas interfacing, e.g, someone posting to a shared instance a schema they have ben toying with locally and another person reviewing the same schema in the UI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude commented on a change in pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
thelabdude commented on a change in pull request #42:
URL: https://github.com/apache/solr/pull/42#discussion_r626109518
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -805,6 +806,12 @@ public void load() {
containerHandlers.getApiBag().registerObject(clusterAPI);
containerHandlers.getApiBag().registerObject(clusterAPI.commands);
containerHandlers.getApiBag().registerObject(clusterAPI.configSetCommands);
+
+ if (this.isZooKeeperAware()) {
+ SchemaDesignerAPI schemaDesignerAPI = new SchemaDesignerAPI(this);
+ containerHandlers.getApiBag().registerObject(schemaDesignerAPI);
+ } // Schema Designer not available in standalone (non-cloud) mode
+
Review comment:
Not sure? The API is mostly intended to support the UI ... this feature needs UI and I don't foresee too many people wanting to interact with the Schema Designer API directly w/o using the Web Admin UI. The Schema Designer link in the Admin UI should not even appear in standalone mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
thelabdude commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-853963136
> Schema Designer should not simply throw generic errors that do no help the user. For instance, when a user tries to create a field that already exists, there is just an error processing banner. Ideally, you could grab the important detail in the response body and show it to the user. Consider `data["error"]["details"][0]["errorMessages"][0]` as seen below to populate the banner with something helpful to the user:
>
> ```
> Possibly unhandled rejection: {"data":{"responseHeader":{"status":400,"QTime":1},"error":{"metadata":["error-class","org.apache.solr.api.ApiBag$ExceptionWithErrObject","root-error-class","org.apache.solr.api.ApiBag$ExceptionWithErrObject"],"details":[{"add-field":{"stored":"true","indexed":"true","uninvertible":"true","name":"name","type":"string"},"errorMessages":["Field 'name' already exists.\n"]}],"msg":"error processing commands","code":400}},"status":400,"config":{"method":"POST","transformRequest":[null],"transformResponse":[null],"jsonpCallbackParam":"callback","data":{"add-field":{"stored":"true","indexed":"true","uninvertible":"true","name":"name","type":"string"}},"url":"techproducts/schema","params":{"wt":"json","_":1622686070279},"headers":{"Accept":"application/json, text/plain, */*","X-Requested-With":"XMLHttpRequest","Content-Type":"application/json;charset=utf-8"},"timeout":10000},"statusText":"Bad Request","xhrStatus":"complete","resource":{"add-field":"..."}}``
`
> ```
Can you please provide reproduction steps for how you got this error (including browser details)
<img width="357" alt="Screen Shot 2021-06-03 at 9 32 19 AM" src="https://user-images.githubusercontent.com/417074/120671957-1c3aad00-c44f-11eb-8c62-ff6051ba9316.png">
? Schema Designer UI should be displaying user friendly errors in a dialog. Moreover, I don't even know how you got this error since the JS code validates the field doesn't already exist before submitting the request. See screenshot of what I see in my env.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude commented on a change in pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
thelabdude commented on a change in pull request #42:
URL: https://github.com/apache/solr/pull/42#discussion_r644903516
##########
File path: solr/webapp/web/index.html
##########
@@ -26,6 +26,7 @@
<link rel="stylesheet" type="text/css" href="css/angular/angular-csp.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/common.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/analysis.css?_=${version}">
+ <link rel="stylesheet" type="text/css" href="css/angular/schema-designer.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/cloud.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/cores.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/collections.css?_=${version}">
Review comment:
hmmm ... This too seems unrelated to the Schema Designer UI. Please raise another JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] MarcusSorealheis commented on a change in pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis commented on a change in pull request #42:
URL: https://github.com/apache/solr/pull/42#discussion_r644427250
##########
File path: solr/webapp/web/js/angular/app.js
##########
@@ -430,7 +436,11 @@ solrAdminApp.config([
$location.path('/login');
}
} else {
- $rootScope.exceptions[rejection.config.url] = rejection.data.error;
+ // schema designer prefers to handle errors itselft
Review comment:
nit: drop the trailing `t` on this line.. reading closely. :)
##########
File path: solr/webapp/web/index.html
##########
@@ -26,6 +26,7 @@
<link rel="stylesheet" type="text/css" href="css/angular/angular-csp.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/common.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/analysis.css?_=${version}">
+ <link rel="stylesheet" type="text/css" href="css/angular/schema-designer.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/cloud.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/cores.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/collections.css?_=${version}">
Review comment:
another nit: the doctype should be changed to be HTML 5. It's as simple as: `<!DOCTYPE html>` at the top of this file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] muse-dev[bot] commented on a change in pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
muse-dev[bot] commented on a change in pull request #42:
URL: https://github.com/apache/solr/pull/42#discussion_r619501904
##########
File path: solr/core/src/java/org/apache/solr/handler/SchemaDesignerConfigSetHelper.java
##########
@@ -0,0 +1,954 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.handler;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.invoke.MethodHandles;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipOutputStream;
+
+import com.google.common.collect.Sets;
+import org.apache.commons.io.FileUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpResponse;
+import org.apache.http.HttpStatus;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ByteArrayEntity;
+import org.apache.http.util.EntityUtils;
+import org.apache.solr.client.solrj.SolrResponse;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.schema.FieldTypeDefinition;
+import org.apache.solr.client.solrj.request.schema.SchemaRequest;
+import org.apache.solr.client.solrj.response.schema.SchemaResponse;
+import org.apache.solr.cloud.ZkConfigSetService;
+import org.apache.solr.cloud.ZkSolrResourceLoader;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.cloud.ClusterState;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.common.cloud.UrlScheme;
+import org.apache.solr.common.cloud.ZkMaintenanceUtils;
+import org.apache.solr.common.cloud.ZkStateReader;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.Utils;
+import org.apache.solr.core.CoreContainer;
+import org.apache.solr.core.SolrConfig;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.handler.admin.CollectionsHandler;
+import org.apache.solr.handler.loader.DefaultSampleDocumentsLoader;
+import org.apache.solr.schema.CopyField;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.ManagedIndexSchema;
+import org.apache.solr.schema.ManagedIndexSchemaFactory;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.schema.SchemaSuggester;
+import org.apache.solr.schema.TextField;
+import org.apache.solr.util.RTimer;
+import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.data.Stat;
+import org.noggit.JSONParser;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.solr.common.util.Utils.fromJSONString;
+import static org.apache.solr.common.util.Utils.makeMap;
+import static org.apache.solr.handler.SchemaDesignerAPI.DESIGNER_PREFIX;
+import static org.apache.solr.handler.SchemaDesignerAPI.SOLR_CONFIG_XML;
+import static org.apache.solr.handler.SchemaDesignerAPI.getConfigSetZkPath;
+import static org.apache.solr.handler.SchemaDesignerAPI.getMutableId;
+import static org.apache.solr.handler.admin.ConfigSetsHandler.DEFAULT_CONFIGSET_NAME;
+import static org.apache.solr.schema.ManagedIndexSchemaFactory.DEFAULT_MANAGED_SCHEMA_RESOURCE_NAME;
+
+public class SchemaDesignerConfigSetHelper {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final String BLOB_STORE_ID = ".system";
+
+ private static final Set<String> excludeConfigSetNames = new HashSet<>(Arrays.asList(DEFAULT_CONFIGSET_NAME, BLOB_STORE_ID));
+ private static final Set<String> removeFieldProps = new HashSet<>(Arrays.asList("href", "id", "copyDest"));
+
+ private final CoreContainer cc;
+ private final SchemaSuggester schemaSuggester;
+ private final SchemaDesignerSettingsDAO settingsDAO;
+
+ SchemaDesignerConfigSetHelper(CoreContainer cc, SchemaSuggester schemaSuggester, SchemaDesignerSettingsDAO settingsDAO) {
+ this.cc = cc;
+ this.schemaSuggester = schemaSuggester;
+ this.settingsDAO = settingsDAO;
+ }
+
+ @SuppressWarnings("unchecked")
+ Map<String, Object> analyzeField(String mutableId, String fieldName, String fieldText) throws IOException {
+ String baseUrl = getBaseUrl(mutableId);
+ String fieldNameEnc = URLEncoder.encode(fieldName, StandardCharsets.UTF_8);
+ String url = baseUrl + "/" + mutableId + "/analysis/field?wt=json&analysis.showmatch=true&analysis.fieldname=" + fieldNameEnc + "&analysis.fieldvalue=POST";
+ HttpEntity entity;
+ Map<String, Object> analysis = Collections.emptyMap();
+ HttpPost httpPost = new HttpPost(url);
+ try {
+ httpPost.setHeader("Content-Type", "text/plain");
+ httpPost.setEntity(new ByteArrayEntity(fieldText.getBytes(StandardCharsets.UTF_8)));
+ entity = cloudClient().getHttpClient().execute(httpPost).getEntity();
+ Map<String, Object> response = (Map<String, Object>) fromJSONString(EntityUtils.toString(entity, StandardCharsets.UTF_8));
+ if (response != null) {
+ analysis = (Map<String, Object>) response.get("analysis");
+ }
+ } finally {
+ httpPost.releaseConnection();
+ }
+ return analysis;
+ }
+
+ List<String> listCollectionsForConfig(String configSet) {
+ final List<String> collections = new LinkedList<>();
+ Map<String, ClusterState.CollectionRef> states = zkStateReader().getClusterState().getCollectionStates();
+ for (Map.Entry<String, ClusterState.CollectionRef> e : states.entrySet()) {
+ final String coll = e.getKey();
+ if (coll.startsWith(DESIGNER_PREFIX)) {
+ continue; // ignore temp
+ }
+
+ try {
+ if (configSet.equals(zkStateReader().readConfigName(coll)) && e.getValue().get() != null) {
+ collections.add(coll);
+ }
+ } catch (Exception exc) {
+ log.warn("Failed to get config name for {}", coll, exc);
+ }
+ }
+ return collections;
+ }
+
+ Map<String, Boolean> listEnabledConfigs() throws IOException {
+ List<String> configsInZk = listConfigsInZk();
+ final Map<String, Boolean> configs = configsInZk.stream()
+ .filter(c -> !excludeConfigSetNames.contains(c) && !c.startsWith(DESIGNER_PREFIX))
+ .collect(Collectors.toMap(c -> c, c -> !settingsDAO.isDesignerDisabled(c)));
+
+ // add the in-progress but drop the _designer prefix
+ configsInZk.stream()
+ .filter(c -> c.startsWith(DESIGNER_PREFIX))
+ .map(c -> c.substring(DESIGNER_PREFIX.length()))
+ .forEach(c -> configs.putIfAbsent(c, true));
+
+ return configs;
+ }
+
+ @SuppressWarnings("unchecked")
+ public String addSchemaObject(String mutableId, Map<String, Object> addJson) throws Exception {
+ SchemaRequest.Update addAction;
+ String action;
+ String objectName = null;
+ if (addJson.containsKey("add-field")) {
+ action = "add-field";
+ Map<String, Object> fieldAttrs = (Map<String, Object>) addJson.get(action);
+ objectName = (String) fieldAttrs.get("name");
+ addAction = new SchemaRequest.AddField(fieldAttrs);
+ } else if (addJson.containsKey("add-dynamic-field")) {
+ action = "add-dynamic-field";
+ Map<String, Object> fieldAttrs = (Map<String, Object>) addJson.get(action);
+ objectName = (String) fieldAttrs.get("name");
+ addAction = new SchemaRequest.AddDynamicField(fieldAttrs);
+ } else if (addJson.containsKey("add-copy-field")) {
+ action = "add-copy-field";
+ Map<String, Object> map = (Map<String, Object>) addJson.get(action);
+ Object dest = map.get("dest");
+ List<String> destFields = null;
+ if (dest instanceof String) {
+ destFields = Collections.singletonList((String) dest);
+ } else if (dest instanceof List) {
+ destFields = (List<String>) dest;
+ } else if (dest instanceof Collection) {
+ Collection<String> destColl = (Collection<String>) dest;
+ destFields = new ArrayList<>(destColl);
+ }
+ addAction = new SchemaRequest.AddCopyField((String) map.get("source"), destFields);
+ } else if (addJson.containsKey("add-field-type")) {
+ action = "add-field-type";
+ Map<String, Object> fieldAttrs = (Map<String, Object>) addJson.get(action);
+ objectName = (String) fieldAttrs.get("name");
+ FieldTypeDefinition ftDef = new FieldTypeDefinition();
+ ftDef.setAttributes(fieldAttrs);
+ addAction = new SchemaRequest.AddFieldType(ftDef);
+ } else {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Unsupported action in request body! " + addJson);
+ }
+
+ SchemaResponse.UpdateResponse schemaResponse = addAction.process(cloudClient(), mutableId);
+ if (schemaResponse.getStatus() != 0) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, schemaResponse.getException());
+ }
+
+ return objectName;
+ }
+
+ void reloadTempCollection(String mutableId, boolean delete) throws Exception {
+ if (delete) {
+ log.debug("Deleting and re-creating existing collection {} after schema update", mutableId);
+ CollectionAdminRequest.deleteCollection(mutableId).process(cloudClient());
+ zkStateReader().waitForState(mutableId, 30, TimeUnit.SECONDS, Objects::isNull);
+ createCollection(mutableId, mutableId);
+ log.debug("Deleted and re-created existing collection: {}", mutableId);
+ } else {
+ CollectionAdminRequest.reloadCollection(mutableId).process(cloudClient());
+ log.debug("Reloaded existing collection: {}", mutableId);
+ }
+ }
+
+ Map<String, Object> updateSchemaObject(String configSet, Map<String, Object> updateJson, ManagedIndexSchema schemaBeforeUpdate) throws Exception {
+ String name = (String) updateJson.get("name");
+ String mutableId = getMutableId(configSet);
+
+ SolrException solrExc = null;
+ boolean needsRebuild = false;
+ String updateType = "field";
+ String updateError = null;
+ if (updateJson.get("type") != null) {
+ try {
+ needsRebuild = updateField(configSet, updateJson, schemaBeforeUpdate);
+ } catch (SolrException exc) {
+ if (exc.code() != 400) {
+ throw exc;
+ }
+ solrExc = exc;
+ updateError = solrExc.getMessage() + " Previous settings will be restored.";
+ }
+ } else {
+ updateType = "type";
+
+ Map<String, Object> typeAttrs = updateJson.entrySet().stream()
+ .filter(e -> !removeFieldProps.contains(e.getKey()))
+ .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+
+ FieldType fieldType = schemaBeforeUpdate.getFieldTypeByName(name);
+
+ // this is a field type
+ Object multiValued = typeAttrs.get("multiValued");
+ if (multiValued == null || (Boolean.TRUE.equals(multiValued) && !fieldType.isMultiValued()) || (Boolean.FALSE.equals(multiValued) && fieldType.isMultiValued())) {
+ needsRebuild = true;
+ log.warn("Re-building the temp collection for {} after type {} updated to multi-valued {}", configSet, name, multiValued);
+ }
+
+ // nice, the json for this field looks like
+ // "synonymQueryStyle": "org.apache.solr.parser.SolrQueryParserBase$SynonymQueryStyle:AS_SAME_TERM"
+ if (typeAttrs.get("synonymQueryStyle") instanceof String) {
+ String synonymQueryStyle = (String) typeAttrs.get("synonymQueryStyle");
+ if (synonymQueryStyle.lastIndexOf(':') != -1) {
+ typeAttrs.put("synonymQueryStyle", synonymQueryStyle.substring(synonymQueryStyle.lastIndexOf(':') + 1));
+ }
+ }
+
+ ManagedIndexSchema updatedSchema = schemaBeforeUpdate.replaceFieldType(fieldType.getTypeName(), (String) typeAttrs.get("class"), typeAttrs);
+ if (!updatedSchema.persistManagedSchema(false)) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Failed to persist schema: " + mutableId);
+ }
+ }
+
+ // the update may have required a full rebuild of the index, otherwise, it's just a reload / re-index sample
+ reloadTempCollection(mutableId, needsRebuild);
+
+ return makeMap("rebuild", needsRebuild, "updateType", updateType, "updateError", updateError, "solrExc", solrExc);
+ }
+
+ boolean updateField(String configSet, Map<String, Object> updateField, ManagedIndexSchema schemaBeforeUpdate) throws IOException, SolrServerException {
+ String mutableId = getMutableId(configSet);
+
+ String name = (String) updateField.get("name");
+ String type = (String) updateField.get("type");
+ String copyDest = (String) updateField.get("copyDest");
+ Map<String, Object> fieldAttributes = updateField.entrySet().stream()
+ .filter(e -> !removeFieldProps.contains(e.getKey()))
+ .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+
+ boolean needsRebuild = false;
+
+ SchemaField schemaField = schemaBeforeUpdate.getField(name);
+ String currentType = schemaField.getType().getTypeName();
+
+ SimpleOrderedMap<Object> fromTypeProps;
+ if (type.equals(currentType)) {
+ // no type change, so just pull the current type's props (with defaults) as we'll use these
+ // to determine which props get explicitly overridden on the field
+ fromTypeProps = schemaBeforeUpdate.getFieldTypeByName(currentType).getNamedPropertyValues(true);
+ } else {
+ // validate type change
+ FieldType newType = schemaBeforeUpdate.getFieldTypeByName(type);
+ if (newType == null) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ "Invalid update request for field " + name + "! Field type " + type + " doesn't exist!");
+ }
+ validateTypeChange(configSet, schemaField, newType);
+
+ // type change looks valid
+ fromTypeProps = newType.getNamedPropertyValues(true);
+ }
+
+ // the diff holds all the explicit properties not inherited from the type
+ Map<String, Object> diff = new HashMap<>();
+ for (Map.Entry<String, Object> e : fieldAttributes.entrySet()) {
+ String attr = e.getKey();
+ Object attrValue = e.getValue();
+ if ("name".equals(attr) || "type".equals(attr)) {
+ continue; // we don't want these in the diff map
+ }
+
+ if ("required".equals(attr)) {
+ diff.put(attr, attrValue != null ? attrValue : false);
+ } else {
+ Object fromType = fromTypeProps.get(attr);
+ if (fromType == null || !fromType.equals(attrValue)) {
+ diff.put(attr, attrValue);
+ }
+ }
+ }
+
+ // detect if they're trying to copy multi-valued fields into a single-valued field
+ Object multiValued = diff.get("multiValued");
+ if (multiValued == null) {
+ // mv not overridden explicitly, but we need the actual value, which will come from the new type (if that changed) or the current field
+ multiValued = type.equals(currentType) ? schemaField.multiValued() : schemaBeforeUpdate.getFieldTypeByName(type).isMultiValued();
+ }
+
+ if (Boolean.FALSE.equals(multiValued)) {
+ // make sure there are no mv source fields if this is a copy dest
+ for (String src : schemaBeforeUpdate.getCopySources(name)) {
+ SchemaField srcField = schemaBeforeUpdate.getField(src);
+ if (srcField.multiValued()) {
+ log.warn("Cannot change multi-valued field {} to single-valued because it is a copy field destination for multi-valued field {}", name, src);
+ multiValued = Boolean.TRUE;
+ diff.put("multiValued", multiValued);
+ break;
+ }
+ }
+ }
+
+ if (Boolean.FALSE.equals(multiValued) && schemaField.multiValued()) {
+ // changing from multi- to single value ... verify the data agrees ...
+ validateMultiValuedChange(configSet, schemaField, Boolean.FALSE);
+ }
+
+ // switch from single-valued to multi-valued requires a full rebuild
+ // See SOLR-12185 ... if we're switching from single to multi-valued, then it's a big operation
+ if (hasMultivalueChange(multiValued, schemaField)) {
+ needsRebuild = true;
+ log.warn("Need to rebuild the temp collection for {} after field {} updated to multi-valued {}", configSet, name, multiValued);
+ }
+
+ if (!needsRebuild) {
+ // check term vectors too
+ Boolean storeTermVector = (Boolean) fieldAttributes.getOrDefault("termVectors", Boolean.FALSE);
+ if (schemaField.storeTermVector() != storeTermVector) {
+ // cannot change termVectors w/o a full-rebuild
+ needsRebuild = true;
+ }
+ }
+
+ log.info("For {}, replacing field {} with attributes: {}", configSet, name, diff);
+ ManagedIndexSchema updatedSchema = schemaBeforeUpdate.replaceField(name, schemaBeforeUpdate.getFieldTypeByName(type), diff);
+
+ // persist the change before applying the copy-field updates
+ if (!updatedSchema.persistManagedSchema(false)) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Failed to persist schema: " + mutableId);
+ }
+
+ return applyCopyFieldUpdates(mutableId, copyDest, name, updatedSchema) || needsRebuild;
+ }
+
+ protected void validateMultiValuedChange(String configSet, SchemaField field, Boolean multiValued) throws IOException {
+ List<SolrInputDocument> docs = loadSampleDocsFromBlobStore(configSet);
+ if (!docs.isEmpty()) {
+ boolean isMV = schemaSuggester.isMultiValued(field.getName(), docs);
+ if (isMV && !multiValued) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ "Cannot change field " + field.getName() + " to single-valued as some sample docs have multiple values!");
+ }
+ }
+ }
+
+ protected void validateTypeChange(String configSet, SchemaField field, FieldType toType) throws IOException {
+ if ("_version_".equals(field.getName())) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ "Cannot change type of the _version_ field; it must be a plong.");
+ }
+ List<SolrInputDocument> docs = loadSampleDocsFromBlobStore(configSet);
+ if (!docs.isEmpty()) {
+ schemaSuggester.validateTypeChange(field, toType, docs);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ List<SolrInputDocument> loadSampleDocsFromBlobStore(final String configSet) throws IOException {
+ List<SolrInputDocument> docs = null;
+ String baseUrl = getBaseUrl(BLOB_STORE_ID);
+ String url = baseUrl + "/" + BLOB_STORE_ID + "/blob/" + configSet + "_sample?wt=filestream";
+ HttpGet httpGet = new HttpGet(url);
Review comment:
*HTTP_PARAMETER_POLLUTION:* Concatenating user-controlled input into a URL [(details)](https://find-sec-bugs.github.io/bugs.htm#HTTP_PARAMETER_POLLUTION)
(at-me [in a reply](https://docs.muse.dev/docs/talk-to-muse/) with `help` or `ignore`)
##########
File path: solr/core/src/java/org/apache/solr/handler/SchemaDesignerConfigSetHelper.java
##########
@@ -0,0 +1,954 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.handler;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.invoke.MethodHandles;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipOutputStream;
+
+import com.google.common.collect.Sets;
+import org.apache.commons.io.FileUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpResponse;
+import org.apache.http.HttpStatus;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ByteArrayEntity;
+import org.apache.http.util.EntityUtils;
+import org.apache.solr.client.solrj.SolrResponse;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.schema.FieldTypeDefinition;
+import org.apache.solr.client.solrj.request.schema.SchemaRequest;
+import org.apache.solr.client.solrj.response.schema.SchemaResponse;
+import org.apache.solr.cloud.ZkConfigSetService;
+import org.apache.solr.cloud.ZkSolrResourceLoader;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.cloud.ClusterState;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.common.cloud.UrlScheme;
+import org.apache.solr.common.cloud.ZkMaintenanceUtils;
+import org.apache.solr.common.cloud.ZkStateReader;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.Utils;
+import org.apache.solr.core.CoreContainer;
+import org.apache.solr.core.SolrConfig;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.handler.admin.CollectionsHandler;
+import org.apache.solr.handler.loader.DefaultSampleDocumentsLoader;
+import org.apache.solr.schema.CopyField;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.ManagedIndexSchema;
+import org.apache.solr.schema.ManagedIndexSchemaFactory;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.schema.SchemaSuggester;
+import org.apache.solr.schema.TextField;
+import org.apache.solr.util.RTimer;
+import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.data.Stat;
+import org.noggit.JSONParser;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.solr.common.util.Utils.fromJSONString;
+import static org.apache.solr.common.util.Utils.makeMap;
+import static org.apache.solr.handler.SchemaDesignerAPI.DESIGNER_PREFIX;
+import static org.apache.solr.handler.SchemaDesignerAPI.SOLR_CONFIG_XML;
+import static org.apache.solr.handler.SchemaDesignerAPI.getConfigSetZkPath;
+import static org.apache.solr.handler.SchemaDesignerAPI.getMutableId;
+import static org.apache.solr.handler.admin.ConfigSetsHandler.DEFAULT_CONFIGSET_NAME;
+import static org.apache.solr.schema.ManagedIndexSchemaFactory.DEFAULT_MANAGED_SCHEMA_RESOURCE_NAME;
+
+public class SchemaDesignerConfigSetHelper {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final String BLOB_STORE_ID = ".system";
+
+ private static final Set<String> excludeConfigSetNames = new HashSet<>(Arrays.asList(DEFAULT_CONFIGSET_NAME, BLOB_STORE_ID));
+ private static final Set<String> removeFieldProps = new HashSet<>(Arrays.asList("href", "id", "copyDest"));
+
+ private final CoreContainer cc;
+ private final SchemaSuggester schemaSuggester;
+ private final SchemaDesignerSettingsDAO settingsDAO;
+
+ SchemaDesignerConfigSetHelper(CoreContainer cc, SchemaSuggester schemaSuggester, SchemaDesignerSettingsDAO settingsDAO) {
+ this.cc = cc;
+ this.schemaSuggester = schemaSuggester;
+ this.settingsDAO = settingsDAO;
+ }
+
+ @SuppressWarnings("unchecked")
+ Map<String, Object> analyzeField(String mutableId, String fieldName, String fieldText) throws IOException {
+ String baseUrl = getBaseUrl(mutableId);
+ String fieldNameEnc = URLEncoder.encode(fieldName, StandardCharsets.UTF_8);
+ String url = baseUrl + "/" + mutableId + "/analysis/field?wt=json&analysis.showmatch=true&analysis.fieldname=" + fieldNameEnc + "&analysis.fieldvalue=POST";
+ HttpEntity entity;
+ Map<String, Object> analysis = Collections.emptyMap();
+ HttpPost httpPost = new HttpPost(url);
+ try {
+ httpPost.setHeader("Content-Type", "text/plain");
+ httpPost.setEntity(new ByteArrayEntity(fieldText.getBytes(StandardCharsets.UTF_8)));
+ entity = cloudClient().getHttpClient().execute(httpPost).getEntity();
+ Map<String, Object> response = (Map<String, Object>) fromJSONString(EntityUtils.toString(entity, StandardCharsets.UTF_8));
+ if (response != null) {
+ analysis = (Map<String, Object>) response.get("analysis");
+ }
+ } finally {
+ httpPost.releaseConnection();
+ }
+ return analysis;
+ }
+
+ List<String> listCollectionsForConfig(String configSet) {
+ final List<String> collections = new LinkedList<>();
+ Map<String, ClusterState.CollectionRef> states = zkStateReader().getClusterState().getCollectionStates();
+ for (Map.Entry<String, ClusterState.CollectionRef> e : states.entrySet()) {
+ final String coll = e.getKey();
+ if (coll.startsWith(DESIGNER_PREFIX)) {
+ continue; // ignore temp
+ }
+
+ try {
+ if (configSet.equals(zkStateReader().readConfigName(coll)) && e.getValue().get() != null) {
+ collections.add(coll);
+ }
+ } catch (Exception exc) {
+ log.warn("Failed to get config name for {}", coll, exc);
+ }
+ }
+ return collections;
+ }
+
+ Map<String, Boolean> listEnabledConfigs() throws IOException {
+ List<String> configsInZk = listConfigsInZk();
+ final Map<String, Boolean> configs = configsInZk.stream()
+ .filter(c -> !excludeConfigSetNames.contains(c) && !c.startsWith(DESIGNER_PREFIX))
+ .collect(Collectors.toMap(c -> c, c -> !settingsDAO.isDesignerDisabled(c)));
+
+ // add the in-progress but drop the _designer prefix
+ configsInZk.stream()
+ .filter(c -> c.startsWith(DESIGNER_PREFIX))
+ .map(c -> c.substring(DESIGNER_PREFIX.length()))
+ .forEach(c -> configs.putIfAbsent(c, true));
+
+ return configs;
+ }
+
+ @SuppressWarnings("unchecked")
+ public String addSchemaObject(String mutableId, Map<String, Object> addJson) throws Exception {
+ SchemaRequest.Update addAction;
+ String action;
+ String objectName = null;
+ if (addJson.containsKey("add-field")) {
+ action = "add-field";
+ Map<String, Object> fieldAttrs = (Map<String, Object>) addJson.get(action);
+ objectName = (String) fieldAttrs.get("name");
+ addAction = new SchemaRequest.AddField(fieldAttrs);
+ } else if (addJson.containsKey("add-dynamic-field")) {
+ action = "add-dynamic-field";
+ Map<String, Object> fieldAttrs = (Map<String, Object>) addJson.get(action);
+ objectName = (String) fieldAttrs.get("name");
+ addAction = new SchemaRequest.AddDynamicField(fieldAttrs);
+ } else if (addJson.containsKey("add-copy-field")) {
+ action = "add-copy-field";
+ Map<String, Object> map = (Map<String, Object>) addJson.get(action);
+ Object dest = map.get("dest");
+ List<String> destFields = null;
+ if (dest instanceof String) {
+ destFields = Collections.singletonList((String) dest);
+ } else if (dest instanceof List) {
+ destFields = (List<String>) dest;
+ } else if (dest instanceof Collection) {
+ Collection<String> destColl = (Collection<String>) dest;
+ destFields = new ArrayList<>(destColl);
+ }
+ addAction = new SchemaRequest.AddCopyField((String) map.get("source"), destFields);
+ } else if (addJson.containsKey("add-field-type")) {
+ action = "add-field-type";
+ Map<String, Object> fieldAttrs = (Map<String, Object>) addJson.get(action);
+ objectName = (String) fieldAttrs.get("name");
+ FieldTypeDefinition ftDef = new FieldTypeDefinition();
+ ftDef.setAttributes(fieldAttrs);
+ addAction = new SchemaRequest.AddFieldType(ftDef);
+ } else {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Unsupported action in request body! " + addJson);
+ }
+
+ SchemaResponse.UpdateResponse schemaResponse = addAction.process(cloudClient(), mutableId);
+ if (schemaResponse.getStatus() != 0) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, schemaResponse.getException());
+ }
+
+ return objectName;
+ }
+
+ void reloadTempCollection(String mutableId, boolean delete) throws Exception {
+ if (delete) {
+ log.debug("Deleting and re-creating existing collection {} after schema update", mutableId);
+ CollectionAdminRequest.deleteCollection(mutableId).process(cloudClient());
+ zkStateReader().waitForState(mutableId, 30, TimeUnit.SECONDS, Objects::isNull);
+ createCollection(mutableId, mutableId);
+ log.debug("Deleted and re-created existing collection: {}", mutableId);
+ } else {
+ CollectionAdminRequest.reloadCollection(mutableId).process(cloudClient());
+ log.debug("Reloaded existing collection: {}", mutableId);
+ }
+ }
+
+ Map<String, Object> updateSchemaObject(String configSet, Map<String, Object> updateJson, ManagedIndexSchema schemaBeforeUpdate) throws Exception {
+ String name = (String) updateJson.get("name");
+ String mutableId = getMutableId(configSet);
+
+ SolrException solrExc = null;
+ boolean needsRebuild = false;
+ String updateType = "field";
+ String updateError = null;
+ if (updateJson.get("type") != null) {
+ try {
+ needsRebuild = updateField(configSet, updateJson, schemaBeforeUpdate);
+ } catch (SolrException exc) {
+ if (exc.code() != 400) {
+ throw exc;
+ }
+ solrExc = exc;
+ updateError = solrExc.getMessage() + " Previous settings will be restored.";
+ }
+ } else {
+ updateType = "type";
+
+ Map<String, Object> typeAttrs = updateJson.entrySet().stream()
+ .filter(e -> !removeFieldProps.contains(e.getKey()))
+ .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+
+ FieldType fieldType = schemaBeforeUpdate.getFieldTypeByName(name);
+
+ // this is a field type
+ Object multiValued = typeAttrs.get("multiValued");
+ if (multiValued == null || (Boolean.TRUE.equals(multiValued) && !fieldType.isMultiValued()) || (Boolean.FALSE.equals(multiValued) && fieldType.isMultiValued())) {
+ needsRebuild = true;
+ log.warn("Re-building the temp collection for {} after type {} updated to multi-valued {}", configSet, name, multiValued);
+ }
+
+ // nice, the json for this field looks like
+ // "synonymQueryStyle": "org.apache.solr.parser.SolrQueryParserBase$SynonymQueryStyle:AS_SAME_TERM"
+ if (typeAttrs.get("synonymQueryStyle") instanceof String) {
+ String synonymQueryStyle = (String) typeAttrs.get("synonymQueryStyle");
+ if (synonymQueryStyle.lastIndexOf(':') != -1) {
+ typeAttrs.put("synonymQueryStyle", synonymQueryStyle.substring(synonymQueryStyle.lastIndexOf(':') + 1));
+ }
+ }
+
+ ManagedIndexSchema updatedSchema = schemaBeforeUpdate.replaceFieldType(fieldType.getTypeName(), (String) typeAttrs.get("class"), typeAttrs);
+ if (!updatedSchema.persistManagedSchema(false)) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Failed to persist schema: " + mutableId);
+ }
+ }
+
+ // the update may have required a full rebuild of the index, otherwise, it's just a reload / re-index sample
+ reloadTempCollection(mutableId, needsRebuild);
+
+ return makeMap("rebuild", needsRebuild, "updateType", updateType, "updateError", updateError, "solrExc", solrExc);
+ }
+
+ boolean updateField(String configSet, Map<String, Object> updateField, ManagedIndexSchema schemaBeforeUpdate) throws IOException, SolrServerException {
+ String mutableId = getMutableId(configSet);
+
+ String name = (String) updateField.get("name");
+ String type = (String) updateField.get("type");
+ String copyDest = (String) updateField.get("copyDest");
+ Map<String, Object> fieldAttributes = updateField.entrySet().stream()
+ .filter(e -> !removeFieldProps.contains(e.getKey()))
+ .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+
+ boolean needsRebuild = false;
+
+ SchemaField schemaField = schemaBeforeUpdate.getField(name);
+ String currentType = schemaField.getType().getTypeName();
+
+ SimpleOrderedMap<Object> fromTypeProps;
+ if (type.equals(currentType)) {
+ // no type change, so just pull the current type's props (with defaults) as we'll use these
+ // to determine which props get explicitly overridden on the field
+ fromTypeProps = schemaBeforeUpdate.getFieldTypeByName(currentType).getNamedPropertyValues(true);
+ } else {
+ // validate type change
+ FieldType newType = schemaBeforeUpdate.getFieldTypeByName(type);
+ if (newType == null) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ "Invalid update request for field " + name + "! Field type " + type + " doesn't exist!");
+ }
+ validateTypeChange(configSet, schemaField, newType);
+
+ // type change looks valid
+ fromTypeProps = newType.getNamedPropertyValues(true);
+ }
+
+ // the diff holds all the explicit properties not inherited from the type
+ Map<String, Object> diff = new HashMap<>();
+ for (Map.Entry<String, Object> e : fieldAttributes.entrySet()) {
+ String attr = e.getKey();
+ Object attrValue = e.getValue();
+ if ("name".equals(attr) || "type".equals(attr)) {
+ continue; // we don't want these in the diff map
+ }
+
+ if ("required".equals(attr)) {
+ diff.put(attr, attrValue != null ? attrValue : false);
+ } else {
+ Object fromType = fromTypeProps.get(attr);
+ if (fromType == null || !fromType.equals(attrValue)) {
+ diff.put(attr, attrValue);
+ }
+ }
+ }
+
+ // detect if they're trying to copy multi-valued fields into a single-valued field
+ Object multiValued = diff.get("multiValued");
+ if (multiValued == null) {
+ // mv not overridden explicitly, but we need the actual value, which will come from the new type (if that changed) or the current field
+ multiValued = type.equals(currentType) ? schemaField.multiValued() : schemaBeforeUpdate.getFieldTypeByName(type).isMultiValued();
+ }
+
+ if (Boolean.FALSE.equals(multiValued)) {
+ // make sure there are no mv source fields if this is a copy dest
+ for (String src : schemaBeforeUpdate.getCopySources(name)) {
+ SchemaField srcField = schemaBeforeUpdate.getField(src);
+ if (srcField.multiValued()) {
+ log.warn("Cannot change multi-valued field {} to single-valued because it is a copy field destination for multi-valued field {}", name, src);
+ multiValued = Boolean.TRUE;
+ diff.put("multiValued", multiValued);
+ break;
+ }
+ }
+ }
+
+ if (Boolean.FALSE.equals(multiValued) && schemaField.multiValued()) {
+ // changing from multi- to single value ... verify the data agrees ...
+ validateMultiValuedChange(configSet, schemaField, Boolean.FALSE);
+ }
+
+ // switch from single-valued to multi-valued requires a full rebuild
+ // See SOLR-12185 ... if we're switching from single to multi-valued, then it's a big operation
+ if (hasMultivalueChange(multiValued, schemaField)) {
+ needsRebuild = true;
+ log.warn("Need to rebuild the temp collection for {} after field {} updated to multi-valued {}", configSet, name, multiValued);
+ }
+
+ if (!needsRebuild) {
+ // check term vectors too
+ Boolean storeTermVector = (Boolean) fieldAttributes.getOrDefault("termVectors", Boolean.FALSE);
+ if (schemaField.storeTermVector() != storeTermVector) {
+ // cannot change termVectors w/o a full-rebuild
+ needsRebuild = true;
+ }
+ }
+
+ log.info("For {}, replacing field {} with attributes: {}", configSet, name, diff);
+ ManagedIndexSchema updatedSchema = schemaBeforeUpdate.replaceField(name, schemaBeforeUpdate.getFieldTypeByName(type), diff);
+
+ // persist the change before applying the copy-field updates
+ if (!updatedSchema.persistManagedSchema(false)) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Failed to persist schema: " + mutableId);
+ }
+
+ return applyCopyFieldUpdates(mutableId, copyDest, name, updatedSchema) || needsRebuild;
+ }
+
+ protected void validateMultiValuedChange(String configSet, SchemaField field, Boolean multiValued) throws IOException {
+ List<SolrInputDocument> docs = loadSampleDocsFromBlobStore(configSet);
+ if (!docs.isEmpty()) {
+ boolean isMV = schemaSuggester.isMultiValued(field.getName(), docs);
+ if (isMV && !multiValued) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ "Cannot change field " + field.getName() + " to single-valued as some sample docs have multiple values!");
+ }
+ }
+ }
+
+ protected void validateTypeChange(String configSet, SchemaField field, FieldType toType) throws IOException {
+ if ("_version_".equals(field.getName())) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ "Cannot change type of the _version_ field; it must be a plong.");
+ }
+ List<SolrInputDocument> docs = loadSampleDocsFromBlobStore(configSet);
+ if (!docs.isEmpty()) {
+ schemaSuggester.validateTypeChange(field, toType, docs);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ List<SolrInputDocument> loadSampleDocsFromBlobStore(final String configSet) throws IOException {
+ List<SolrInputDocument> docs = null;
+ String baseUrl = getBaseUrl(BLOB_STORE_ID);
+ String url = baseUrl + "/" + BLOB_STORE_ID + "/blob/" + configSet + "_sample?wt=filestream";
+ HttpGet httpGet = new HttpGet(url);
+ try {
+ HttpResponse entity = cloudClient().getHttpClient().execute(httpGet);
+ int statusCode = entity.getStatusLine().getStatusCode();
+ if (statusCode == HttpStatus.SC_OK) {
+ byte[] bytes = streamAsBytes(entity.getEntity().getContent());
+ if (bytes.length > 0) {
+ docs = (List<SolrInputDocument>) Utils.fromJavabin(bytes);
+ }
+ } else if (statusCode != HttpStatus.SC_NOT_FOUND) {
+ byte[] bytes = streamAsBytes(entity.getEntity().getContent());
+ throw new IOException("Failed to lookup stored docs for " + configSet + " due to: " + new String(bytes, StandardCharsets.UTF_8));
+ } // else not found is ok
+ } finally {
+ httpGet.releaseConnection();
+ }
+ return docs != null ? docs : Collections.emptyList();
+ }
+
+ @SuppressWarnings({"rawtypes"})
+ protected Map postDataToBlobStore(CloudSolrClient cloudClient, String blobName, byte[] bytes) throws IOException {
+ Map m = null;
+ HttpEntity entity;
+ String response = null;
+ String baseUrl = getBaseUrl(BLOB_STORE_ID);
+ HttpPost httpPost = new HttpPost(baseUrl + "/" + BLOB_STORE_ID + "/blob/" + blobName);
+ try {
+ httpPost.setHeader("Content-Type", "application/octet-stream");
+ httpPost.setEntity(new ByteArrayEntity(bytes));
+ entity = cloudClient.getHttpClient().execute(httpPost).getEntity();
+ try {
+ response = EntityUtils.toString(entity, StandardCharsets.UTF_8);
+ m = (Map) fromJSONString(response);
+ } catch (JSONParser.ParseException e) {
+ log.error("$ERROR$: {}", response, e);
+ }
+ } finally {
+ httpPost.releaseConnection();
+ }
+
+ return m;
+ }
+
+ private byte[] streamAsBytes(final InputStream in) throws IOException {
+ return DefaultSampleDocumentsLoader.streamAsBytes(in);
+ }
+
+ String getBaseUrl(final String collection) {
+ String baseUrl = null;
+ try {
+ Set<String> liveNodes = zkStateReader().getClusterState().getLiveNodes();
+ DocCollection docColl = zkStateReader().getCollection(collection);
+ if (docColl != null && !liveNodes.isEmpty()) {
+ Optional<Replica> maybeActive = docColl.getReplicas().stream().filter(r -> r.isActive(liveNodes)).findAny();
+ if (maybeActive.isPresent()) {
+ baseUrl = maybeActive.get().getBaseUrl();
+ }
+ }
+ } catch (Exception exc) {
+ log.warn("Failed to lookup base URL for collection {}", collection, exc);
+ }
+
+ if (baseUrl == null) {
+ baseUrl = UrlScheme.INSTANCE.getBaseUrlForNodeName(cc.getZkController().getNodeName());
+ }
+
+ return baseUrl;
+ }
+
+ protected String getManagedSchemaZkPath(final String configSet) {
+ return getConfigSetZkPath(configSet, DEFAULT_MANAGED_SCHEMA_RESOURCE_NAME);
+ }
+
+ void toggleNestedDocsFields(String mutableId, ManagedIndexSchema schema, boolean enabled) throws IOException, SolrServerException {
+ if (enabled) {
+ enableNestedDocsFields(schema, mutableId);
+ } else {
+ deleteNestedDocsFieldsIfNeeded(schema, mutableId, true);
+ }
+ }
+
+ protected void enableNestedDocsFields(ManagedIndexSchema schema, String mutableId) throws IOException, SolrServerException {
+ if (!schema.hasExplicitField("_root_")) {
+ Map<String, Object> fieldAttrs = new HashMap<>();
+ fieldAttrs.put("name", "_root_");
+ fieldAttrs.put("type", "string");
+ fieldAttrs.put("docValues", false);
+ fieldAttrs.put("indexed", true);
+ fieldAttrs.put("stored", false);
+ SchemaRequest.AddField addAction = new SchemaRequest.AddField(fieldAttrs);
+ SchemaResponse.UpdateResponse schemaResponse = addAction.process(cloudClient(), mutableId);
+ if (schemaResponse.getStatus() != 0) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Failed to add _root_ field due to: " + schemaResponse.getException());
+ }
+ }
+
+ if (!schema.hasExplicitField("_nest_path_")) {
+ Map<String, Object> fieldAttrs = new HashMap<>();
+ fieldAttrs.put("name", "_nest_path_");
+ fieldAttrs.put("type", "_nest_path_");
+ SchemaRequest.AddField addAction = new SchemaRequest.AddField(fieldAttrs);
+ SchemaResponse.UpdateResponse schemaResponse = addAction.process(cloudClient(), mutableId);
+ if (schemaResponse.getStatus() != 0) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Failed to add _nest_path_ field due to: " + schemaResponse.getException());
+ }
+ }
+ }
+
+ protected ManagedIndexSchema deleteNestedDocsFieldsIfNeeded(ManagedIndexSchema schema, String mutableId, boolean persist) {
+ List<String> toDelete = new LinkedList<>();
+ if (schema.hasExplicitField("_root_")) {
+ toDelete.add("_root_");
+ }
+ if (schema.hasExplicitField("_nest_path_")) {
+ toDelete.add("_nest_path_");
+ }
+ if (!toDelete.isEmpty()) {
+ schema = schema.deleteFields(toDelete);
+ if (persist && !schema.persistManagedSchema(false)) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Failed to persist schema: " + mutableId);
+ }
+ }
+ return schema;
+ }
+
+ SolrConfig loadSolrConfig(String configSet) {
+ SolrResourceLoader resourceLoader = cc.getResourceLoader();
+ ZkSolrResourceLoader zkLoader =
+ new ZkSolrResourceLoader(resourceLoader.getInstancePath(), configSet, resourceLoader.getClassLoader(), cc.getZkController());
+ return SolrConfig.readFromResourceLoader(zkLoader, SOLR_CONFIG_XML, true, null);
+ }
+
+ ManagedIndexSchema loadLatestSchema(String configSet) {
+ return loadLatestSchema(loadSolrConfig(configSet));
+ }
+
+ ManagedIndexSchema loadLatestSchema(SolrConfig solrConfig) {
+ ManagedIndexSchemaFactory factory = new ManagedIndexSchemaFactory();
+ factory.init(new NamedList<>());
+ return factory.create(DEFAULT_MANAGED_SCHEMA_RESOURCE_NAME, solrConfig, null);
+ }
+
+ int getCurrentSchemaVersion(final String configSet) throws KeeperException, InterruptedException {
+ int currentVersion = -1;
+ final String path = getManagedSchemaZkPath(configSet);
+ try {
+ Stat stat = cc.getZkController().getZkClient().exists(path, null, true);
+ if (stat != null) {
+ currentVersion = stat.getVersion();
+ }
+ } catch (KeeperException.NoNodeException notExists) {
+ // safe to ignore
+ }
+ return currentVersion;
+ }
+
+ void createCollection(final String collection, final String configSet) throws Exception {
+ RTimer timer = new RTimer();
+ SolrResponse rsp = CollectionAdminRequest.createCollection(collection, configSet, 1, 1).process(cloudClient());
+ CollectionsHandler.waitForActiveCollection(collection, cc, rsp);
+ double tookMs = timer.getTime();
+ log.debug("Took {} ms to create new collection {} with configSet {}", tookMs, collection, configSet);
+ }
+
+ protected CloudSolrClient cloudClient() {
+ return cc.getSolrClientCache().getCloudSolrClient(cc.getZkController().getZkServerAddress());
+ }
+
+ protected ZkStateReader zkStateReader() {
+ return cc.getZkController().getZkStateReader();
+ }
+
+ boolean applyCopyFieldUpdates(String mutableId, String copyDest, String fieldName, ManagedIndexSchema schema) throws IOException, SolrServerException {
+ boolean updated = false;
+
+ if (copyDest == null || copyDest.trim().isEmpty()) {
+ // delete all the copy field directives for this field
+ List<CopyField> copyFieldsList = schema.getCopyFieldsList(fieldName);
+ if (!copyFieldsList.isEmpty()) {
+ List<String> dests = copyFieldsList.stream().map(cf -> cf.getDestination().getName()).collect(Collectors.toList());
+ SchemaRequest.DeleteCopyField delAction = new SchemaRequest.DeleteCopyField(fieldName, dests);
+ SchemaResponse.UpdateResponse schemaResponse = delAction.process(cloudClient(), mutableId);
+ if (schemaResponse.getStatus() != 0) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, schemaResponse.getException());
+ }
+ updated = true;
+ }
+ } else {
+ SchemaField field = schema.getField(fieldName);
+ Set<String> desired = new HashSet<>();
+ for (String dest : copyDest.trim().split(",")) {
+ String toAdd = dest.trim();
+ if (toAdd.equals(fieldName)) {
+ continue; // cannot copy to self
+ }
+
+ // make sure the field exists and is multi-valued if this field is
+ SchemaField toAddField = schema.getFieldOrNull(toAdd);
+ if (toAddField != null) {
+ if (!field.multiValued() || toAddField.multiValued()) {
+ desired.add(toAdd);
+ } else {
+ log.warn("Skipping copy-field dest {} for {} because it is not multi-valued!", toAdd, fieldName);
+ }
+ } else {
+ log.warn("Skipping copy-field dest {} for {} because it doesn't exist!", toAdd, fieldName);
+ }
+ }
+ Set<String> existing = schema.getCopyFieldsList(fieldName).stream().map(cf -> cf.getDestination().getName()).collect(Collectors.toSet());
+ Set<String> add = Sets.difference(desired, existing);
+ if (!add.isEmpty()) {
+ SchemaRequest.AddCopyField addAction = new SchemaRequest.AddCopyField(fieldName, new ArrayList<>(add));
+ SchemaResponse.UpdateResponse schemaResponse = addAction.process(cloudClient(), mutableId);
+ if (schemaResponse.getStatus() != 0) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, schemaResponse.getException());
+ }
+ updated = true;
+ } // no additions ...
+
+ Set<String> del = Sets.difference(existing, desired);
+ if (!del.isEmpty()) {
+ SchemaRequest.DeleteCopyField delAction = new SchemaRequest.DeleteCopyField(fieldName, new ArrayList<>(del));
+ SchemaResponse.UpdateResponse schemaResponse = delAction.process(cloudClient(), mutableId);
+ if (schemaResponse.getStatus() != 0) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, schemaResponse.getException());
+ }
+ updated = true;
+ } // no deletions ...
+ }
+
+ return updated;
+ }
+
+ protected boolean hasMultivalueChange(Object multiValued, SchemaField schemaField) {
+ return (multiValued == null ||
+ (Boolean.TRUE.equals(multiValued) && !schemaField.multiValued()) ||
+ (Boolean.FALSE.equals(multiValued) && schemaField.multiValued()));
+ }
+
+ ManagedIndexSchema syncLanguageSpecificObjectsAndFiles(String configSet, ManagedIndexSchema schema, List<String> langs, boolean dynamicEnabled, String copyFrom) throws KeeperException, InterruptedException {
+ if (!langs.isEmpty()) {
+ // there's a subset of languages applied, so remove all the other langs
+ schema = removeLanguageSpecificObjectsAndFiles(configSet, schema, langs);
+ }
+
+ // now restore any missing types / files for the languages we need, optionally adding back dynamic fields too
+ schema = restoreLanguageSpecificObjectsAndFiles(configSet, schema, langs, dynamicEnabled, copyFrom);
+
+ if (!schema.persistManagedSchema(false)) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Failed to persist schema: " + configSet);
+ }
+ return schema;
+ }
+
+ protected ManagedIndexSchema removeLanguageSpecificObjectsAndFiles(String configSet, ManagedIndexSchema schema, List<String> langs) throws KeeperException, InterruptedException {
+ final Set<String> languages = new HashSet<>(Arrays.asList("ws", "general", "rev", "sort"));
+ languages.addAll(langs);
+
+ final Set<String> usedTypes = schema.getFields().values().stream().map(f -> f.getType().getTypeName()).collect(Collectors.toSet());
+ Map<String, FieldType> types = schema.getFieldTypes();
+ final Set<String> toRemove = types.values().stream()
+ .filter(t -> t.getTypeName().startsWith("text_") && TextField.class.equals(t.getClass()))
+ .filter(t -> !languages.contains(t.getTypeName().substring("text_".length())))
+ .map(FieldType::getTypeName)
+ .filter(t -> !usedTypes.contains(t)) // not explicitly used by a field
+ .collect(Collectors.toSet());
+
+ // find dynamic fields that refer to the types we're removing ...
+ List<String> toRemoveDF = Arrays.stream(schema.getDynamicFields())
+ .filter(df -> toRemove.contains(df.getPrototype().getType().getTypeName()))
+ .map(df -> df.getPrototype().getName())
+ .collect(Collectors.toList());
+
+ schema = schema.deleteDynamicFields(toRemoveDF);
+ schema = schema.deleteFieldTypes(toRemove);
+
+ SolrZkClient zkClient = cc.getZkController().getZkClient();
+ final String configPathInZk = ZkConfigSetService.CONFIGS_ZKNODE + "/" + configSet;
+ final Set<String> toRemoveFiles = new HashSet<>();
+ final Set<String> langExt = languages.stream().map(l -> "_" + l).collect(Collectors.toSet());
+ try {
+ ZkMaintenanceUtils.traverseZkTree(zkClient, configPathInZk, ZkMaintenanceUtils.VISIT_ORDER.VISIT_POST, path -> {
+ if (path.endsWith(".txt")) {
+ int slashAt = path.lastIndexOf('/');
+ String fileName = slashAt != -1 ? path.substring(slashAt + 1) : "";
+ if (!fileName.contains("_")) return; // not a match
+
+ final String pathWoExt = fileName.substring(0, fileName.length() - 4);
+ boolean matchesLang = false;
+ for (String lang : langExt) {
+ if (pathWoExt.endsWith(lang)) {
+ matchesLang = true;
+ break;
+ }
+ }
+ if (!matchesLang) {
+ toRemoveFiles.add(path);
+ }
+ }
+ });
+ } catch (KeeperException.NoNodeException nne) {
+ // no-op
+ }
+
+ for (String path : toRemoveFiles) {
+ try {
+ zkClient.delete(path, -1, false);
+ } catch (KeeperException.NoNodeException nne) {
+ // no-op
+ }
+ }
+
+ return schema;
+ }
+
+ protected ManagedIndexSchema restoreLanguageSpecificObjectsAndFiles(String configSet, ManagedIndexSchema schema, List<String> langs, boolean dynamicEnabled, String copyFrom) throws KeeperException, InterruptedException {
+ // pull the dynamic fields from the copyFrom schema
+ ManagedIndexSchema copyFromSchema = loadLatestSchema(copyFrom);
+
+ final Set<String> langSet = new HashSet<>(Arrays.asList("ws", "general", "rev", "sort"));
+ langSet.addAll(langs);
+
+ boolean restoreAllLangs = langs.isEmpty();
+
+ final Set<String> langFilesToRestore = new HashSet<>();
+
+ // Restore missing files
+ SolrZkClient zkClient = zkStateReader().getZkClient();
+ String configPathInZk = ZkConfigSetService.CONFIGS_ZKNODE + "/" + copyFrom;
+ final Set<String> langExt = langSet.stream().map(l -> "_" + l).collect(Collectors.toSet());
+ try {
+ ZkMaintenanceUtils.traverseZkTree(zkClient, configPathInZk, ZkMaintenanceUtils.VISIT_ORDER.VISIT_POST, path -> {
+ if (path.endsWith(".txt")) {
+ if (restoreAllLangs) {
+ langFilesToRestore.add(path);
+ return;
+ }
+
+ final String pathWoExt = path.substring(0, path.length() - 4);
+ for (String lang : langExt) {
+ if (pathWoExt.endsWith(lang)) {
+ langFilesToRestore.add(path);
+ break;
+ }
+ }
+ }
+ });
+ } catch (KeeperException.NoNodeException nne) {
+ // no-op
+ }
+
+ if (!langFilesToRestore.isEmpty()) {
+ final String replacePathDir = "/" + configSet;
+ final String origPathDir = "/" + copyFrom;
+ for (String path : langFilesToRestore) {
+ String copyToPath = path.replace(origPathDir, replacePathDir);
+ if (!zkClient.exists(copyToPath, true)) {
+ zkClient.makePath(copyToPath, false, true);
+ zkClient.setData(copyToPath, zkClient.getData(path, null, null, true), true);
+ }
+ }
+ }
+
+ // Restore field types
+ final Map<String, FieldType> existingTypes = schema.getFieldTypes();
+ Map<String, FieldType> srcTypes = copyFromSchema.getFieldTypes();
+ List<FieldType> addTypes = srcTypes.values().stream()
+ .filter(t -> t.getTypeName().startsWith("text_") && TextField.class.equals(t.getClass()) && (restoreAllLangs || langSet.contains(t.getTypeName().substring("text_".length()))))
+ .filter(t -> !existingTypes.containsKey(t.getTypeName()))
+ .collect(Collectors.toList());
+ if (!addTypes.isEmpty()) {
+ schema = schema.addFieldTypes(addTypes, false);
+
+ if (dynamicEnabled) {
+ // restore language specific dynamic fields
+ final Set<String> existingDynFields =
+ Arrays.stream(schema.getDynamicFieldPrototypes()).map(SchemaField::getName).collect(Collectors.toSet());
+ final Set<String> retoredTypeNames = addTypes.stream().map(FieldType::getTypeName).collect(Collectors.toSet());
+ IndexSchema.DynamicField[] srcDynamicFields = copyFromSchema.getDynamicFields();
+ List<SchemaField> addDynFields = Arrays.stream(srcDynamicFields)
+ .filter(df -> retoredTypeNames.contains(df.getPrototype().getType().getTypeName()))
+ .filter(df -> !existingDynFields.contains(df.getPrototype().getName()))
+ .map(IndexSchema.DynamicField::getPrototype)
+ .collect(Collectors.toList());
+ if (!addDynFields.isEmpty()) {
+ schema = schema.addDynamicFields(addDynFields, null, false);
+ }
+ }
+ }
+
+ return schema;
+ }
+
+ protected ManagedIndexSchema removeDynamicFields(ManagedIndexSchema schema) {
+ List<String> dynamicFieldNames =
+ Arrays.stream(schema.getDynamicFields()).map(f -> f.getPrototype().getName()).collect(Collectors.toList());
+ if (!dynamicFieldNames.isEmpty()) {
+ schema = schema.deleteDynamicFields(dynamicFieldNames);
+ }
+ return schema;
+ }
+
+ protected ManagedIndexSchema restoreDynamicFields(ManagedIndexSchema schema, List<String> langs, String copyFrom) {
+ // pull the dynamic fields from the copyFrom schema
+ ManagedIndexSchema copyFromSchema = loadLatestSchema(copyFrom);
+ IndexSchema.DynamicField[] dynamicFields = copyFromSchema.getDynamicFields();
+ if (dynamicFields.length == 0 && !DEFAULT_CONFIGSET_NAME.equals(copyFrom)) {
+ copyFromSchema = loadLatestSchema(DEFAULT_CONFIGSET_NAME);
+ dynamicFields = copyFromSchema.getDynamicFields();
+ }
+
+ if (dynamicFields.length == 0) {
+ return schema;
+ }
+
+ final Set<String> existingDFNames =
+ Arrays.stream(schema.getDynamicFields()).map(df -> df.getPrototype().getName()).collect(Collectors.toSet());
+ List<SchemaField> toAdd = Arrays.stream(dynamicFields)
+ .filter(df -> !existingDFNames.contains(df.getPrototype().getName()))
+ .map(IndexSchema.DynamicField::getPrototype)
+ .collect(Collectors.toList());
+
+ // only restore language specific dynamic fields that match our langSet
+ if (!langs.isEmpty()) {
+ final Set<String> langSet = new HashSet<>(Arrays.asList("ws", "general", "rev", "sort"));
+ langSet.addAll(langs);
+ toAdd = toAdd.stream()
+ .filter(df -> !df.getName().startsWith("*_txt_") || langSet.contains(df.getName().substring("*_txt_".length())))
+ .collect(Collectors.toList());
+ }
+
+ if (!toAdd.isEmpty()) {
+ // grab any field types that need to be re-added
+ final Map<String, FieldType> fieldTypes = schema.getFieldTypes();
+ List<FieldType> addTypes = toAdd.stream()
+ .map(SchemaField::getType)
+ .filter(t -> !fieldTypes.containsKey(t.getTypeName()))
+ .collect(Collectors.toList());
+ if (!addTypes.isEmpty()) {
+ schema = schema.addFieldTypes(addTypes, false);
+ }
+
+ schema = schema.addDynamicFields(toAdd, null, true);
+ }
+
+ return schema;
+ }
+
+ void checkSchemaVersion(String configSet, final int versionInRequest, int currentVersion) throws KeeperException, InterruptedException {
+ if (versionInRequest < 0) {
+ return; // don't enforce the version check
+ }
+
+ if (currentVersion == -1) {
+ currentVersion = getCurrentSchemaVersion(configSet);
+ }
+
+ if (currentVersion != versionInRequest) {
+ if (configSet.startsWith(DESIGNER_PREFIX)) {
+ configSet = configSet.substring(DESIGNER_PREFIX.length());
+ }
+ throw new SolrException(SolrException.ErrorCode.CONFLICT,
+ "Your schema version " + versionInRequest + " for " + configSet + " is out-of-date; current version is: " + currentVersion +
+ ". Perhaps another user also updated the schema while you were editing it? You'll need to retry your update after the schema is refreshed.");
+ }
+ }
+
+ private List<String> listConfigsInZk() throws IOException {
+ return cc.getConfigSetService().listConfigs();
+ }
+
+ byte[] downloadAndZipConfigSet(String configId) throws IOException {
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ Path tmpDirectory = Files.createTempDirectory("schema-designer-" + configId);
Review comment:
*PATH_TRAVERSAL_IN:* This API (java/nio/file/Files.createTempDirectory(Ljava/lang/String;[Ljava/nio/file/attribute/FileAttribute;)Ljava/nio/file/Path;) reads a file whose location might be specified by user input [(details)](https://find-sec-bugs.github.io/bugs.htm#PATH_TRAVERSAL_IN)
(at-me [in a reply](https://docs.muse.dev/docs/talk-to-muse/) with `help` or `ignore`)
##########
File path: solr/core/src/java/org/apache/solr/handler/loader/DefaultSampleDocumentsLoader.java
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.handler.loader;
+
+import javax.xml.stream.XMLInputFactory;
+import javax.xml.stream.XMLStreamConstants;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.XMLStreamReader;
+import java.io.BufferedReader;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.Reader;
+import java.io.StringReader;
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.SolrInputField;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.ContentStream;
+import org.apache.solr.common.util.ContentStreamBase;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.request.SolrQueryRequestBase;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.update.processor.UpdateRequestProcessor;
+import org.apache.solr.util.SafeXMLParsing;
+import org.noggit.JSONParser;
+import org.noggit.ObjectBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+import org.xml.sax.SAXException;
+
+import static org.apache.solr.common.params.CommonParams.JSON_MIME;
+import static org.apache.solr.handler.loader.CSVLoaderBase.SEPARATOR;
+
+public class DefaultSampleDocumentsLoader implements SampleDocumentsLoader {
+ public static final String CSV_MULTI_VALUE_DELIM_PARAM = "csvMultiValueDelimiter";
+ private static final int MAX_STREAM_SIZE = (5 * 1024 * 1024);
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static byte[] streamAsBytes(final InputStream in) throws IOException {
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ byte[] buf = new byte[1024];
+ int r;
+ try (in) {
+ while ((r = in.read(buf)) != -1) baos.write(buf, 0, r);
+ }
+ return baos.toByteArray();
+ }
+
+ @Override
+ public SampleDocuments load(SolrParams params, ContentStream stream, final int maxDocsToLoad) throws IOException {
+ final String contentType = stream.getContentType();
+ if (contentType == null) {
+ return SampleDocuments.NONE;
+ }
+
+ if (params == null) {
+ params = new ModifiableSolrParams();
+ }
+
+ Long streamSize = stream.getSize();
+ if (streamSize != null && streamSize > MAX_STREAM_SIZE) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ "Sample is too big! " + MAX_STREAM_SIZE + " bytes is the max upload size for sample documents.");
+ }
+
+ String fileSource = "paste";
+ if ("file".equals(stream.getName())) {
+ fileSource = stream.getSourceInfo() != null ? stream.getSourceInfo() : "file";
+ }
+
+ byte[] uploadedBytes = streamAsBytes(stream.getStream());
+ // recheck the upload size in case the stream returned null for getSize
+ if (uploadedBytes.length > MAX_STREAM_SIZE) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ "Sample is too big! " + MAX_STREAM_SIZE + " bytes is the max upload size for sample documents.");
+ }
+ // use a byte stream for the parsers in case they need to re-parse using a different strategy
+ // e.g. JSON vs. JSON lines or different CSV strategies ...
+ ContentStreamBase.ByteArrayStream byteStream = new ContentStreamBase.ByteArrayStream(uploadedBytes, fileSource, contentType);
+ String charset = ContentStreamBase.getCharsetFromContentType(stream.getContentType());
+ if (charset == null) {
+ charset = ContentStreamBase.DEFAULT_CHARSET;
+ }
+
+ List<SolrInputDocument> docs = null;
+ if (stream.getSize() > 0) {
+ if (contentType.contains(JSON_MIME)) {
+ docs = loadJsonDocs(params, byteStream, maxDocsToLoad);
+ } else if (contentType.contains("text/xml") || contentType.contains("application/xml")) {
+ docs = loadXmlDocs(params, byteStream, maxDocsToLoad);
+ } else if (contentType.contains("text/csv") || contentType.contains("application/csv")) {
+ docs = loadCsvDocs(params, fileSource, uploadedBytes, charset, maxDocsToLoad);
+ } else if (contentType.contains("text/plain") || contentType.contains("application/octet-stream")) {
+ docs = loadJsonLines(params, byteStream, maxDocsToLoad);
+ } else {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, contentType + " not supported yet!");
+ }
+
+ if (docs != null && maxDocsToLoad > 0 && docs.size() > maxDocsToLoad) {
+ docs = docs.subList(0, maxDocsToLoad);
+ }
+ }
+
+ return new SampleDocuments(docs, contentType, fileSource);
+ }
+
+ protected List<SolrInputDocument> loadCsvDocs(SolrParams params, String source, byte[] streamBytes, String charset, final int maxDocsToLoad) throws IOException {
+ ContentStream stream;
+ if (params.get(SEPARATOR) == null) {
+ String csvStr = new String(streamBytes, charset);
+ char sep = detectTSV(csvStr);
+ ModifiableSolrParams modifiableSolrParams = new ModifiableSolrParams(params);
+ modifiableSolrParams.set(SEPARATOR, String.valueOf(sep));
+ params = modifiableSolrParams;
+ stream = new ContentStreamBase.StringStream(csvStr, "text/csv");
+ } else {
+ stream = new ContentStreamBase.ByteArrayStream(streamBytes, source, "text/csv");
+ }
+ return (new SampleCSVLoader(new CSVRequest(params), maxDocsToLoad)).loadDocs(stream);
+ }
+
+ @SuppressWarnings("unchecked")
+ protected List<SolrInputDocument> loadJsonLines(SolrParams params, ContentStreamBase.ByteArrayStream stream, final int maxDocsToLoad) throws IOException {
+ List<Map<String, Object>> docs = new LinkedList<>();
+ try (Reader r = stream.getReader()) {
+ BufferedReader br = new BufferedReader(r);
+ String line;
+ while ((line = br.readLine()) != null) {
+ line = line.trim();
+ if (!line.isEmpty() && line.startsWith("{") && line.endsWith("}")) {
+ Object jsonLine = ObjectBuilder.getVal(new JSONParser(line));
+ if (jsonLine instanceof Map) {
+ docs.add((Map<String, Object>) jsonLine);
+ }
+ }
+ if (maxDocsToLoad > 0 && docs.size() == maxDocsToLoad) {
+ break;
+ }
+ }
+ }
+
+ return docs.stream().map(JsonLoader::buildDoc).collect(Collectors.toList());
+ }
+
+ @SuppressWarnings("unchecked")
+ protected List<SolrInputDocument> loadJsonDocs(SolrParams params, ContentStreamBase.ByteArrayStream stream, final int maxDocsToLoad) throws IOException {
+ Object json;
+ try (Reader r = stream.getReader()) {
+ json = ObjectBuilder.getVal(new JSONParser(r));
+ }
+ if (json == null) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Expected at least 1 JSON doc in the request body!");
+ }
+
+ List<Map<String, Object>> docs;
+ if (json instanceof List) {
+ // list of docs
+ docs = (List<Map<String, Object>>) json;
+ } else if (json instanceof Map) {
+ // single doc ... see if this is a json lines file
+ boolean isJsonLines = false;
+ String charset = ContentStreamBase.getCharsetFromContentType(stream.getContentType());
+ String jsonStr = new String(streamAsBytes(stream.getStream()), charset != null ? charset : ContentStreamBase.DEFAULT_CHARSET);
+ String[] lines = jsonStr.split("\n");
+ if (lines.length > 1) {
+ for (String line : lines) {
+ line = line.trim();
+ if (!line.isEmpty() && line.startsWith("{") && line.endsWith("}")) {
+ isJsonLines = true;
+ break;
+ }
+ }
+ }
+ if (isJsonLines) {
+ docs = loadJsonLines(lines);
+ } else {
+ docs = Collections.singletonList((Map<String, Object>) json);
+ }
+ } else {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Expected one or more JSON docs in the request body!");
+ }
+ if (maxDocsToLoad > 0 && docs.size() > maxDocsToLoad) {
+ docs = docs.subList(0, maxDocsToLoad);
+ }
+ return docs.stream().map(JsonLoader::buildDoc).collect(Collectors.toList());
+ }
+
+ protected List<SolrInputDocument> loadXmlDocs(SolrParams params, ContentStreamBase.ByteArrayStream stream, final int maxDocsToLoad) throws IOException {
+ String xmlString = readInputAsString(stream.getStream()).trim();
+ List<SolrInputDocument> docs;
+ if (xmlString.contains("<add>") && xmlString.contains("<doc>")) {
+ XMLInputFactory inputFactory = XMLInputFactory.newInstance();
+ XMLStreamReader parser = null;
+ try {
+ parser = inputFactory.createXMLStreamReader(new StringReader(xmlString));
Review comment:
*XXE_XMLSTREAMREADER:* The XML parsing is vulnerable to XML External Entity attacks [(details)](https://find-sec-bugs.github.io/bugs.htm#XXE_XMLSTREAMREADER)
(at-me [in a reply](https://docs.muse.dev/docs/talk-to-muse/) with `help` or `ignore`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
HoustonPutman commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-848873024
> > I would like to be able to populate the example document with a document that exists. Maybe a button in the query results.
> I haven't implemented this yet. The query results only include sample documents already, so maybe you want to revise the sample documents in the paste text area? I think that's a valid thing to do and will think about this some more. My only concern would be adding another option that may complicate the experience. Can you elaborate on the user experience you want here?
For sure. revising the sample documents is exactly what I want. If I manually input a complex document, I don't want to have to do it again by scratch. This could be in the next iteration, it's definitely not a deal breaker, just an inconvenience. Maybe there's just an option to see the JSON in the sample results screen, so I could copy and paste.
> > Do we recommend the use of useDocValuesAsStored instead of stored?
> I'm not sure about whether that is a recommendation? Seems reasonable to me that if you're already using DocValues, then useDocValuesAsStored seems more efficient from a storage perspective (if you store and use doc values, then seems like you're doing double storage), but maybe there is some performance hit for doing this? I can certainly turn that off if we don't like defaulting to stored=false and useDocValuesAsStored=true.
I think I'm in the wrong here. I remember hearing rumblings of certain features that didn't support fields with `useDocValuesAsStored` only. But it seems like it's the default option in schemas with version 1.6. Anyway, if its an issue we can certainly fix it, or the missing `useDocValuesAsStored` features, in the future.
> > In the future it would be nice to have information on each of the options. Not sure what the best way to go about this is though.
> I think some of the options are pretty confusing and need supporting documentation, so have deferred to the ref guide via the help icon links vs. adding more docs directly in the Schema Designer UI.
👍
Great work as always!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
HoustonPutman commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-845376159
Some thoughts while playing around:
- [ ] I would like to be able to populate the example document with a document that exists. Maybe a button in the query results.
- [ ] docValues should be enabled by default when creating a fieldType/field
- [ ] Do we recommend the use of `useDocValuesAsStored` instead of `stored`?
- [ ] When you analyze the fields it sometimes brings up a question about language and some options. Not clear that this is talking about the entire schema. Would be better to highlight the configName in the menu.
- [ ] Maybe guess long strings as text fields, not string fields?
- [ ] Would be nice to be able to filter on a field not having a feature (docValues)
- [ ] When filtering fields, it would be nice to filter the dynamic fields as well.
- [ ] In the future it would be nice to have information on each of the options. Not sure what the best way to go about this is though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude merged pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
thelabdude merged pull request #42:
URL: https://github.com/apache/solr/pull/42
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] MarcusSorealheis commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-854119635
> > there's also a few security exceptions that if thrown in the wrong corporate org could start problematic conversations running SOLR. Everything is under scrutiny right now given the recent events.
> > To fix those errors referencing image resources being blocked by the CSP, we need to add this clause to `jetty.xml`: `img-src 'self' data:`. The change will still restrict img resources to the host but it will allow SVG and IMG. We cannot be more specific because of the port. :) It is a replacement of the existing `img-srv 'self'`.
>
> This doesn't sound like it is Schema Designer UI specific. I believe I'm referencing icons the same way all the other screens do. If you think this is a Schema Designer specific issue, please provide a link to the offending code. Otherwise, please open another JIRA to address that problem independently of this issue.
It's not Schema designer specific. It's Solr. I'll open a JIRA, thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] muse-dev[bot] commented on a change in pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
muse-dev[bot] commented on a change in pull request #42:
URL: https://github.com/apache/solr/pull/42#discussion_r626099636
##########
File path: solr/core/src/java/org/apache/solr/handler/designer/SchemaDesignerConfigSetHelper.java
##########
@@ -0,0 +1,1029 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.handler.designer;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipOutputStream;
+
+import com.google.common.collect.Sets;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.http.HttpResponse;
+import org.apache.http.HttpStatus;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.http.client.utils.URLEncodedUtils;
+import org.apache.http.entity.ByteArrayEntity;
+import org.apache.http.util.EntityUtils;
+import org.apache.solr.client.solrj.SolrResponse;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.schema.FieldTypeDefinition;
+import org.apache.solr.client.solrj.request.schema.SchemaRequest;
+import org.apache.solr.client.solrj.response.schema.SchemaResponse;
+import org.apache.solr.cloud.ZkConfigSetService;
+import org.apache.solr.cloud.ZkSolrResourceLoader;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.cloud.ClusterState;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.common.cloud.UrlScheme;
+import org.apache.solr.common.cloud.ZkMaintenanceUtils;
+import org.apache.solr.common.cloud.ZkStateReader;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.Utils;
+import org.apache.solr.core.CoreContainer;
+import org.apache.solr.core.SolrConfig;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.handler.admin.CollectionsHandler;
+import org.apache.solr.handler.loader.DefaultSampleDocumentsLoader;
+import org.apache.solr.schema.CopyField;
+import org.apache.solr.schema.FieldType;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.ManagedIndexSchema;
+import org.apache.solr.schema.ManagedIndexSchemaFactory;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.schema.SchemaSuggester;
+import org.apache.solr.schema.TextField;
+import org.apache.solr.util.RTimer;
+import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.data.Stat;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.solr.common.util.Utils.fromJSONString;
+import static org.apache.solr.common.util.Utils.toJavabin;
+import static org.apache.solr.handler.designer.SchemaDesignerAPI.getConfigSetZkPath;
+import static org.apache.solr.handler.designer.SchemaDesignerAPI.getMutableId;
+import static org.apache.solr.handler.admin.ConfigSetsHandler.DEFAULT_CONFIGSET_NAME;
+import static org.apache.solr.schema.IndexSchema.NEST_PATH_FIELD_NAME;
+import static org.apache.solr.schema.IndexSchema.ROOT_FIELD_NAME;
+import static org.apache.solr.schema.ManagedIndexSchemaFactory.DEFAULT_MANAGED_SCHEMA_RESOURCE_NAME;
+
+public class SchemaDesignerConfigSetHelper implements SchemaDesignerConstants {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final Set<String> removeFieldProps = new HashSet<>(Arrays.asList("href", "id", "copyDest"));
+ private static final List<String> includeLangIds = Arrays.asList("ws", "general", "rev", "sort");
+ private static final String ZNODE_PATH_DELIM = "/";
+ private static final String MULTIVALUED = "multiValued";
+ private static final int TEXT_PREFIX_LEN = "text_".length();
+
+
+ private final CoreContainer cc;
+ private final SchemaSuggester schemaSuggester;
+
+ SchemaDesignerConfigSetHelper(CoreContainer cc, SchemaSuggester schemaSuggester) {
+ this.cc = cc;
+ this.schemaSuggester = schemaSuggester;
+ }
+
+ @SuppressWarnings("unchecked")
+ Map<String, Object> analyzeField(String configSet, String fieldName, String fieldText) throws IOException {
+ final String mutableId = getMutableId(configSet);
+ final URI uri;
+ try {
+ uri = collectionApiEndpoint(mutableId, "analysis", "field")
+ .setParameter(CommonParams.WT, CommonParams.JSON)
+ .setParameter("analysis.showmatch", "true")
+ .setParameter("analysis.fieldname", fieldName)
+ .setParameter("analysis.fieldvalue", "POST")
+ .build();
+ } catch (URISyntaxException e) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, e);
+ }
+
+ Map<String, Object> analysis = Collections.emptyMap();
+ HttpPost httpPost = new HttpPost(uri);
+ httpPost.setHeader("Content-Type", "text/plain");
+ httpPost.setEntity(new ByteArrayEntity(fieldText.getBytes(StandardCharsets.UTF_8)));
+ try {
+ HttpResponse resp = cloudClient().getHttpClient().execute(httpPost);
+ int statusCode = resp.getStatusLine().getStatusCode();
+ if (statusCode != HttpStatus.SC_OK) {
+ throw new SolrException(SolrException.ErrorCode.getErrorCode(statusCode),
+ EntityUtils.toString(resp.getEntity(), StandardCharsets.UTF_8));
+ }
+
+ Map<String, Object> response = (Map<String, Object>) fromJSONString(EntityUtils.toString(resp.getEntity(), StandardCharsets.UTF_8));
+ if (response != null) {
+ analysis = (Map<String, Object>) response.get("analysis");
+ }
+ } finally {
+ httpPost.releaseConnection();
+ }
+
+ return analysis;
+ }
+
+ List<String> listCollectionsForConfig(String configSet) {
+ final List<String> collections = new LinkedList<>();
+ Map<String, ClusterState.CollectionRef> states = zkStateReader().getClusterState().getCollectionStates();
+ for (Map.Entry<String, ClusterState.CollectionRef> e : states.entrySet()) {
+ final String coll = e.getKey();
+ if (coll.startsWith(DESIGNER_PREFIX)) {
+ continue; // ignore temp
+ }
+
+ try {
+ if (configSet.equals(zkStateReader().readConfigName(coll)) && e.getValue().get() != null) {
+ collections.add(coll);
+ }
+ } catch (Exception exc) {
+ log.warn("Failed to get config name for {}", coll, exc);
+ }
+ }
+ return collections;
+ }
+
+ @SuppressWarnings("unchecked")
+ public String addSchemaObject(String configSet, Map<String, Object> addJson) throws Exception {
+ String mutableId = getMutableId(configSet);
+ SchemaRequest.Update addAction;
+ String action;
+ String objectName = null;
+ if (addJson.containsKey("add-field")) {
+ action = "add-field";
+ Map<String, Object> fieldAttrs = (Map<String, Object>) addJson.get(action);
+ objectName = (String) fieldAttrs.get("name");
+ addAction = new SchemaRequest.AddField(fieldAttrs);
+ } else if (addJson.containsKey("add-dynamic-field")) {
+ action = "add-dynamic-field";
+ Map<String, Object> fieldAttrs = (Map<String, Object>) addJson.get(action);
+ objectName = (String) fieldAttrs.get("name");
+ addAction = new SchemaRequest.AddDynamicField(fieldAttrs);
+ } else if (addJson.containsKey("add-copy-field")) {
+ action = "add-copy-field";
+ Map<String, Object> map = (Map<String, Object>) addJson.get(action);
+ Object dest = map.get("dest");
+ List<String> destFields = null;
+ if (dest instanceof String) {
+ destFields = Collections.singletonList((String) dest);
+ } else if (dest instanceof List) {
+ destFields = (List<String>) dest;
+ } else if (dest instanceof Collection) {
+ Collection<String> destColl = (Collection<String>) dest;
+ destFields = new ArrayList<>(destColl);
+ }
+ addAction = new SchemaRequest.AddCopyField((String) map.get("source"), destFields);
+ } else if (addJson.containsKey("add-field-type")) {
+ action = "add-field-type";
+ Map<String, Object> fieldAttrs = (Map<String, Object>) addJson.get(action);
+ objectName = (String) fieldAttrs.get("name");
+ FieldTypeDefinition ftDef = new FieldTypeDefinition();
+ ftDef.setAttributes(fieldAttrs);
+ addAction = new SchemaRequest.AddFieldType(ftDef);
+ } else {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Unsupported action in request body! " + addJson);
+ }
+
+ // Using the SchemaAPI vs. working on the schema directly because SchemaField.create methods are package protected
+ SchemaResponse.UpdateResponse schemaResponse = addAction.process(cloudClient(), mutableId);
+ if (schemaResponse.getStatus() != 0) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, schemaResponse.getException());
+ }
+
+ return objectName;
+ }
+
+ void reloadTempCollection(String mutableId, boolean delete) throws Exception {
+ if (delete) {
+ log.debug("Deleting and re-creating existing collection {} after schema update", mutableId);
+ CollectionAdminRequest.deleteCollection(mutableId).process(cloudClient());
+ zkStateReader().waitForState(mutableId, 30, TimeUnit.SECONDS, Objects::isNull);
+ createCollection(mutableId, mutableId);
+ log.debug("Deleted and re-created existing collection: {}", mutableId);
+ } else {
+ CollectionAdminRequest.reloadCollection(mutableId).process(cloudClient());
+ log.debug("Reloaded existing collection: {}", mutableId);
+ }
+ }
+
+ Map<String, Object> updateSchemaObject(String configSet, Map<String, Object> updateJson, ManagedIndexSchema schemaBeforeUpdate) throws Exception {
+ String name = (String) updateJson.get("name");
+ String mutableId = getMutableId(configSet);
+
+ boolean needsRebuild = false;
+
+ SolrException solrExc = null;
+ String updateType;
+ String updateError = null;
+ if (updateJson.get("type") != null) {
+ updateType = schemaBeforeUpdate.isDynamicField(name) ? "dynamicField" : "field";
+ try {
+ needsRebuild = updateField(configSet, updateJson, schemaBeforeUpdate);
+ } catch (SolrException exc) {
+ if (exc.code() != 400) {
+ throw exc;
+ }
+ solrExc = exc;
+ updateError = solrExc.getMessage() + " Previous settings will be restored.";
+ }
+ } else {
+ updateType = "type";
+ needsRebuild = updateFieldType(configSet, name, updateJson, schemaBeforeUpdate);
+ }
+
+ // the update may have required a full rebuild of the index, otherwise, it's just a reload / re-index sample
+ reloadTempCollection(mutableId, needsRebuild);
+
+ Map<String, Object> results = new HashMap<>();
+ results.put("rebuild", needsRebuild);
+ results.put("updateType", updateType);
+ if (updateError != null) {
+ results.put("updateError", updateError);
+ }
+ if (solrExc != null) {
+ results.put("solrExc", solrExc);
+ }
+ return results;
+ }
+
+ protected boolean updateFieldType(String configSet, String typeName, Map<String, Object> updateJson, ManagedIndexSchema schemaBeforeUpdate) {
+ boolean needsRebuild = false;
+
+ Map<String, Object> typeAttrs = updateJson.entrySet().stream()
+ .filter(e -> !removeFieldProps.contains(e.getKey()))
+ .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+
+ FieldType fieldType = schemaBeforeUpdate.getFieldTypeByName(typeName);
+
+ // this is a field type
+ Object multiValued = typeAttrs.get(MULTIVALUED);
+ if (typeHasMultiValuedChange(multiValued, fieldType)) {
+ needsRebuild = true;
+ log.warn("Re-building the temp collection for {} after type {} updated to multi-valued {}", configSet, typeName, multiValued);
+ }
+
+ // nice, the json for this field looks like
+ // "synonymQueryStyle": "org.apache.solr.parser.SolrQueryParserBase$SynonymQueryStyle:AS_SAME_TERM"
+ if (typeAttrs.get("synonymQueryStyle") instanceof String) {
+ String synonymQueryStyle = (String) typeAttrs.get("synonymQueryStyle");
+ if (synonymQueryStyle.lastIndexOf(':') != -1) {
+ typeAttrs.put("synonymQueryStyle", synonymQueryStyle.substring(synonymQueryStyle.lastIndexOf(':') + 1));
+ }
+ }
+
+ ManagedIndexSchema updatedSchema =
+ schemaBeforeUpdate.replaceFieldType(fieldType.getTypeName(), (String) typeAttrs.get("class"), typeAttrs);
+ updatedSchema.persistManagedSchema(false);
+
+ return needsRebuild;
+ }
+
+ boolean updateField(String configSet, Map<String, Object> updateField, ManagedIndexSchema schemaBeforeUpdate) throws IOException, SolrServerException {
+ String mutableId = getMutableId(configSet);
+
+ String name = (String) updateField.get("name");
+ String type = (String) updateField.get("type");
+ String copyDest = (String) updateField.get("copyDest");
+ Map<String, Object> fieldAttributes = updateField.entrySet().stream()
+ .filter(e -> !removeFieldProps.contains(e.getKey()))
+ .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+
+ boolean needsRebuild = false;
+
+ SchemaField schemaField = schemaBeforeUpdate.getField(name);
+ boolean isDynamic = schemaBeforeUpdate.isDynamicField(name);
+ String currentType = schemaField.getType().getTypeName();
+
+ SimpleOrderedMap<Object> fromTypeProps;
+ if (type.equals(currentType)) {
+ // no type change, so just pull the current type's props (with defaults) as we'll use these
+ // to determine which props get explicitly overridden on the field
+ fromTypeProps = schemaBeforeUpdate.getFieldTypeByName(currentType).getNamedPropertyValues(true);
+ } else {
+ // validate type change
+ FieldType newType = schemaBeforeUpdate.getFieldTypeByName(type);
+ if (newType == null) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ "Invalid update request for field " + name + "! Field type " + type + " doesn't exist!");
+ }
+ validateTypeChange(configSet, schemaField, newType);
+
+ // type change looks valid
+ fromTypeProps = newType.getNamedPropertyValues(true);
+ }
+
+ // the diff holds all the explicit properties not inherited from the type
+ Map<String, Object> diff = new HashMap<>();
+ for (Map.Entry<String, Object> e : fieldAttributes.entrySet()) {
+ String attr = e.getKey();
+ Object attrValue = e.getValue();
+ if ("name".equals(attr) || "type".equals(attr)) {
+ continue; // we don't want these in the diff map
+ }
+
+ if ("required".equals(attr)) {
+ diff.put(attr, attrValue != null ? attrValue : false);
+ } else {
+ Object fromType = fromTypeProps.get(attr);
+ if (fromType == null || !fromType.equals(attrValue)) {
+ diff.put(attr, attrValue);
+ }
+ }
+ }
+
+ // detect if they're trying to copy multi-valued fields into a single-valued field
+ Object multiValued = diff.get(MULTIVALUED);
+ if (multiValued == null) {
+ // mv not overridden explicitly, but we need the actual value, which will come from the new type (if that changed) or the current field
+ multiValued = type.equals(currentType) ? schemaField.multiValued() : schemaBeforeUpdate.getFieldTypeByName(type).isMultiValued();
+ }
+
+ if (!isDynamic && Boolean.FALSE.equals(multiValued)) {
+ // make sure there are no mv source fields if this is a copy dest
+ for (String src : schemaBeforeUpdate.getCopySources(name)) {
+ SchemaField srcField = schemaBeforeUpdate.getField(src);
+ if (srcField.multiValued()) {
+ log.warn("Cannot change multi-valued field {} to single-valued because it is a copy field destination for multi-valued field {}", name, src);
+ multiValued = Boolean.TRUE;
+ diff.put(MULTIVALUED, multiValued);
+ break;
+ }
+ }
+ }
+
+ if (Boolean.FALSE.equals(multiValued) && schemaField.multiValued()) {
+ // changing from multi- to single value ... verify the data agrees ...
+ validateMultiValuedChange(configSet, schemaField, Boolean.FALSE);
+ }
+
+ // switch from single-valued to multi-valued requires a full rebuild
+ // See SOLR-12185 ... if we're switching from single to multi-valued, then it's a big operation
+ if (fieldHasMultiValuedChange(multiValued, schemaField)) {
+ needsRebuild = true;
+ log.warn("Need to rebuild the temp collection for {} after field {} updated to multi-valued {}", configSet, name, multiValued);
+ }
+
+ if (!needsRebuild) {
+ // check term vectors too
+ Boolean storeTermVector = (Boolean) fieldAttributes.getOrDefault("termVectors", Boolean.FALSE);
+ if (schemaField.storeTermVector() != storeTermVector) {
+ // cannot change termVectors w/o a full-rebuild
+ needsRebuild = true;
+ }
+ }
+
+ log.info("For {}, replacing field {} with attributes: {}", configSet, name, diff);
+ final FieldType fieldType = schemaBeforeUpdate.getFieldTypeByName(type);
+ ManagedIndexSchema updatedSchema = isDynamic ? schemaBeforeUpdate.replaceDynamicField(name, fieldType, diff)
+ : schemaBeforeUpdate.replaceField(name, fieldType, diff);
+
+ // persist the change before applying the copy-field updates
+ updatedSchema.persistManagedSchema(false);
+
+ if (!isDynamic) {
+ applyCopyFieldUpdates(mutableId, copyDest, name, updatedSchema);
+ }
+
+ return needsRebuild;
+ }
+
+ protected void validateMultiValuedChange(String configSet, SchemaField field, Boolean multiValued) throws IOException {
+ List<SolrInputDocument> docs = getStoredSampleDocs(configSet);
+ if (!docs.isEmpty()) {
+ boolean isMV = schemaSuggester.isMultiValued(field.getName(), docs);
+ if (isMV && !multiValued) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ "Cannot change field " + field.getName() + " to single-valued as some sample docs have multiple values!");
+ }
+ }
+ }
+
+ protected void validateTypeChange(String configSet, SchemaField field, FieldType toType) throws IOException {
+ if ("_version_".equals(field.getName())) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ "Cannot change type of the _version_ field; it must be a plong.");
+ }
+ List<SolrInputDocument> docs = getStoredSampleDocs(configSet);
+ if (!docs.isEmpty()) {
+ schemaSuggester.validateTypeChange(field, toType, docs);
+ }
+ }
+
+ void deleteStoredSampleDocs(String configSet) {
+ try {
+ cloudClient().deleteByQuery(BLOB_STORE_ID, "id:" + configSet + "_sample/*", 10);
+ } catch (IOException | SolrServerException exc) {
+ log.warn("Failed to delete sample docs from blob store for {}", configSet, exc);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ List<SolrInputDocument> getStoredSampleDocs(final String configSet) throws IOException {
+ List<SolrInputDocument> docs = null;
+
+ final URI uri;
+ try {
+ uri = collectionApiEndpoint(BLOB_STORE_ID, "blob", configSet + "_sample")
+ .setParameter(CommonParams.WT, "filestream")
+ .build();
+ } catch (URISyntaxException e) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, e);
+ }
+
+ HttpGet httpGet = new HttpGet(uri);
+ try {
+ HttpResponse entity = cloudClient().getHttpClient().execute(httpGet);
+ int statusCode = entity.getStatusLine().getStatusCode();
+ if (statusCode == HttpStatus.SC_OK) {
+ byte[] bytes = DefaultSampleDocumentsLoader.streamAsBytes(entity.getEntity().getContent());
+ if (bytes.length > 0) {
+ docs = (List<SolrInputDocument>) Utils.fromJavabin(bytes);
+ }
+ } else if (statusCode != HttpStatus.SC_NOT_FOUND) {
+ byte[] bytes = DefaultSampleDocumentsLoader.streamAsBytes(entity.getEntity().getContent());
+ throw new IOException("Failed to lookup stored docs for " + configSet + " due to: " + new String(bytes, StandardCharsets.UTF_8));
+ } // else not found is ok
+ } finally {
+ httpGet.releaseConnection();
+ }
+ return docs != null ? docs : Collections.emptyList();
+ }
+
+ void storeSampleDocs(final String configSet, List<SolrInputDocument> docs) throws IOException {
+ postDataToBlobStore(cloudClient(), configSet + "_sample",
+ DefaultSampleDocumentsLoader.streamAsBytes(toJavabin(docs)));
+ }
+
+ protected void postDataToBlobStore(CloudSolrClient cloudClient, String blobName, byte[] bytes) throws IOException {
+ final URI uri;
+ try {
+ uri = collectionApiEndpoint(BLOB_STORE_ID, "blob", blobName).build();
+ } catch (URISyntaxException e) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, e);
+ }
+
+ HttpPost httpPost = new HttpPost(uri);
+ try {
+ httpPost.setHeader("Content-Type", "application/octet-stream");
+ httpPost.setEntity(new ByteArrayEntity(bytes));
+ HttpResponse resp = cloudClient.getHttpClient().execute(httpPost);
+ int statusCode = resp.getStatusLine().getStatusCode();
+ if (statusCode != HttpStatus.SC_OK) {
+ throw new SolrException(SolrException.ErrorCode.getErrorCode(statusCode),
+ EntityUtils.toString(resp.getEntity(), StandardCharsets.UTF_8));
+ }
+ } finally {
+ httpPost.releaseConnection();
+ }
+ }
+
+ private String getBaseUrl(final String collection) {
+ String baseUrl = null;
+ try {
+ Set<String> liveNodes = zkStateReader().getClusterState().getLiveNodes();
+ DocCollection docColl = zkStateReader().getCollection(collection);
+ if (docColl != null && !liveNodes.isEmpty()) {
+ Optional<Replica> maybeActive = docColl.getReplicas().stream().filter(r -> r.isActive(liveNodes)).findAny();
+ if (maybeActive.isPresent()) {
+ baseUrl = maybeActive.get().getBaseUrl();
+ }
+ }
+ } catch (Exception exc) {
+ log.warn("Failed to lookup base URL for collection {}", collection, exc);
+ }
+
+ if (baseUrl == null) {
+ baseUrl = UrlScheme.INSTANCE.getBaseUrlForNodeName(cc.getZkController().getNodeName());
+ }
+
+ return baseUrl;
+ }
+
+ private URIBuilder collectionApiEndpoint(final String collection, final String... morePathSegments) throws URISyntaxException {
+ URI baseUrl = new URI(getBaseUrl(collection));
+ // build up a list of path segments including any path in the base URL, collection, and additional segments provided by caller
+ List<String> path = new ArrayList<>(URLEncodedUtils.parsePathSegments(baseUrl.getPath()));
+ path.add(collection);
+ if (morePathSegments != null && morePathSegments.length > 0) {
+ path.addAll(Arrays.asList(morePathSegments));
+ }
+ return new URIBuilder(baseUrl).setPathSegments(path);
+ }
+
+ protected String getManagedSchemaZkPath(final String configSet) {
+ return getConfigSetZkPath(configSet, DEFAULT_MANAGED_SCHEMA_RESOURCE_NAME);
+ }
+
+ ManagedIndexSchema toggleNestedDocsFields(ManagedIndexSchema schema, boolean enabled) {
+ return enabled ? enableNestedDocsFields(schema, true) : deleteNestedDocsFieldsIfNeeded(schema, true);
+ }
+
+ ManagedIndexSchema enableNestedDocsFields(ManagedIndexSchema schema, boolean persist) {
+ boolean madeChanges = false;
+
+ if (!schema.hasExplicitField(ROOT_FIELD_NAME)) {
+ Map<String, Object> fieldAttrs = Map.of("docValues", false, "indexed", true, "stored", false);
+ schema = (ManagedIndexSchema) schema.addField(schema.newField(ROOT_FIELD_NAME, "string", fieldAttrs), false);
+ madeChanges = true;
+ }
+
+ if (!schema.hasExplicitField(NEST_PATH_FIELD_NAME)) {
+ schema = (ManagedIndexSchema) schema.addField(schema.newField(NEST_PATH_FIELD_NAME, NEST_PATH_FIELD_NAME, Collections.emptyMap()), false);
+ madeChanges = true;
+ }
+
+ if (madeChanges && persist) {
+ schema.persistManagedSchema(false);
+ }
+
+ return schema;
+ }
+
+ ManagedIndexSchema deleteNestedDocsFieldsIfNeeded(ManagedIndexSchema schema, boolean persist) {
+ List<String> toDelete = new LinkedList<>();
+ if (schema.hasExplicitField(ROOT_FIELD_NAME)) {
+ toDelete.add(ROOT_FIELD_NAME);
+ }
+ if (schema.hasExplicitField(NEST_PATH_FIELD_NAME)) {
+ toDelete.add(NEST_PATH_FIELD_NAME);
+ }
+ if (!toDelete.isEmpty()) {
+ schema = schema.deleteFields(toDelete);
+ if (persist) {
+ schema.persistManagedSchema(false);
+ }
+ }
+ return schema;
+ }
+
+ SolrConfig loadSolrConfig(String configSet) {
+ SolrResourceLoader resourceLoader = cc.getResourceLoader();
+ ZkSolrResourceLoader zkLoader =
+ new ZkSolrResourceLoader(resourceLoader.getInstancePath(), configSet, resourceLoader.getClassLoader(), cc.getZkController());
+ return SolrConfig.readFromResourceLoader(zkLoader, SOLR_CONFIG_XML, true, null);
+ }
+
+ ManagedIndexSchema loadLatestSchema(String configSet) {
+ return loadLatestSchema(loadSolrConfig(configSet));
+ }
+
+ ManagedIndexSchema loadLatestSchema(SolrConfig solrConfig) {
+ ManagedIndexSchemaFactory factory = new ManagedIndexSchemaFactory();
+ factory.init(new NamedList<>());
+ return factory.create(DEFAULT_MANAGED_SCHEMA_RESOURCE_NAME, solrConfig, null);
+ }
+
+ int getCurrentSchemaVersion(final String configSet) throws KeeperException, InterruptedException {
+ int currentVersion = -1;
+ final String path = getManagedSchemaZkPath(configSet);
+ try {
+ Stat stat = cc.getZkController().getZkClient().exists(path, null, true);
+ if (stat != null) {
+ currentVersion = stat.getVersion();
+ }
+ } catch (KeeperException.NoNodeException notExists) {
+ // safe to ignore
+ }
+ return currentVersion;
+ }
+
+ void createCollection(final String collection, final String configSet) throws KeeperException, InterruptedException, IOException, SolrServerException {
+ RTimer timer = new RTimer();
+ SolrResponse rsp = CollectionAdminRequest.createCollection(collection, configSet, 1, 1).process(cloudClient());
+ CollectionsHandler.waitForActiveCollection(collection, cc, rsp);
+ double tookMs = timer.getTime();
+ log.debug("Took {} ms to create new collection {} with configSet {}", tookMs, collection, configSet);
+ }
+
+ protected CloudSolrClient cloudClient() {
+ return cc.getSolrClientCache().getCloudSolrClient(cc.getZkController().getZkServerAddress());
+ }
+
+ protected ZkStateReader zkStateReader() {
+ return cc.getZkController().getZkStateReader();
+ }
+
+ boolean applyCopyFieldUpdates(String mutableId, String copyDest, String fieldName, ManagedIndexSchema schema) throws IOException, SolrServerException {
+ boolean updated = false;
+
+ if (copyDest == null || copyDest.trim().isEmpty()) {
+ // delete all the copy field directives for this field
+ List<CopyField> copyFieldsList = schema.getCopyFieldsList(fieldName);
+ if (!copyFieldsList.isEmpty()) {
+ List<String> dests = copyFieldsList.stream().map(cf -> cf.getDestination().getName()).collect(Collectors.toList());
+ SchemaRequest.DeleteCopyField delAction = new SchemaRequest.DeleteCopyField(fieldName, dests);
+ SchemaResponse.UpdateResponse schemaResponse = delAction.process(cloudClient(), mutableId);
+ if (schemaResponse.getStatus() != 0) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, schemaResponse.getException());
+ }
+ updated = true;
+ }
+ } else {
+ SchemaField field = schema.getField(fieldName);
+ Set<String> desired = new HashSet<>();
+ for (String dest : copyDest.trim().split(",")) {
+ String toAdd = dest.trim();
+ if (toAdd.equals(fieldName)) {
+ continue; // cannot copy to self
+ }
+
+ // make sure the field exists and is multi-valued if this field is
+ SchemaField toAddField = schema.getFieldOrNull(toAdd);
+ if (toAddField != null) {
+ if (!field.multiValued() || toAddField.multiValued()) {
+ desired.add(toAdd);
+ } else {
+ log.warn("Skipping copy-field dest {} for {} because it is not multi-valued!", toAdd, fieldName);
+ }
+ } else {
+ log.warn("Skipping copy-field dest {} for {} because it doesn't exist!", toAdd, fieldName);
+ }
+ }
+ Set<String> existing = schema.getCopyFieldsList(fieldName).stream().map(cf -> cf.getDestination().getName()).collect(Collectors.toSet());
+ Set<String> add = Sets.difference(desired, existing);
+ if (!add.isEmpty()) {
+ SchemaRequest.AddCopyField addAction = new SchemaRequest.AddCopyField(fieldName, new ArrayList<>(add));
+ SchemaResponse.UpdateResponse schemaResponse = addAction.process(cloudClient(), mutableId);
+ if (schemaResponse.getStatus() != 0) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, schemaResponse.getException());
+ }
+ updated = true;
+ } // no additions ...
+
+ Set<String> del = Sets.difference(existing, desired);
+ if (!del.isEmpty()) {
+ SchemaRequest.DeleteCopyField delAction = new SchemaRequest.DeleteCopyField(fieldName, new ArrayList<>(del));
+ SchemaResponse.UpdateResponse schemaResponse = delAction.process(cloudClient(), mutableId);
+ if (schemaResponse.getStatus() != 0) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, schemaResponse.getException());
+ }
+ updated = true;
+ } // no deletions ...
+ }
+
+ return updated;
+ }
+
+ protected boolean fieldHasMultiValuedChange(Object multiValued, SchemaField schemaField) {
+ return (multiValued == null ||
+ (Boolean.TRUE.equals(multiValued) && !schemaField.multiValued()) ||
+ (Boolean.FALSE.equals(multiValued) && schemaField.multiValued()));
+ }
+
+ protected boolean typeHasMultiValuedChange(Object multiValued, FieldType fieldType) {
+ return (multiValued == null ||
+ (Boolean.TRUE.equals(multiValued) && !fieldType.isMultiValued()) ||
+ (Boolean.FALSE.equals(multiValued) && fieldType.isMultiValued()));
+ }
+
+ ManagedIndexSchema syncLanguageSpecificObjectsAndFiles(String configSet, ManagedIndexSchema schema, List<String> langs, boolean dynamicEnabled, String copyFrom) throws KeeperException, InterruptedException {
+ if (!langs.isEmpty()) {
+ // there's a subset of languages applied, so remove all the other langs
+ schema = removeLanguageSpecificObjectsAndFiles(configSet, schema, langs);
+ }
+
+ // now restore any missing types / files for the languages we need, optionally adding back dynamic fields too
+ schema = restoreLanguageSpecificObjectsAndFiles(configSet, schema, langs, dynamicEnabled, copyFrom);
+
+ schema.persistManagedSchema(false);
+ return schema;
+ }
+
+ protected ManagedIndexSchema removeLanguageSpecificObjectsAndFiles(String configSet, ManagedIndexSchema schema, List<String> langs) throws KeeperException, InterruptedException {
+ final Set<String> languages = new HashSet<>(includeLangIds);
+ languages.addAll(langs);
+
+ final Set<String> usedTypes =
+ schema.getFields().values().stream().map(f -> f.getType().getTypeName()).collect(Collectors.toSet());
+
+ final Set<String> usedLangs =
+ schema.getFields().values().stream().filter(f -> isTextType(f.getType()))
+ .map(f -> f.getType().getTypeName().substring(TEXT_PREFIX_LEN)).collect(Collectors.toSet());
+
+ // don't remove types / files for langs that are explicitly being used by a field
+ languages.addAll(usedLangs);
+
+ Map<String, FieldType> types = schema.getFieldTypes();
+ final Set<String> toRemove = types.values().stream()
+ .filter(this::isTextType)
+ .filter(t -> !languages.contains(t.getTypeName().substring(TEXT_PREFIX_LEN)))
+ .map(FieldType::getTypeName)
+ .filter(t -> !usedTypes.contains(t)) // not explicitly used by a field
+ .collect(Collectors.toSet());
+
+ // find dynamic fields that refer to the types we're removing ...
+ List<String> toRemoveDF = Arrays.stream(schema.getDynamicFields())
+ .filter(df -> toRemove.contains(df.getPrototype().getType().getTypeName()))
+ .map(df -> df.getPrototype().getName())
+ .collect(Collectors.toList());
+
+ schema = schema.deleteDynamicFields(toRemoveDF);
+ schema = schema.deleteFieldTypes(toRemove);
+
+ SolrZkClient zkClient = cc.getZkController().getZkClient();
+ final String configPathInZk = ZkConfigSetService.CONFIGS_ZKNODE + ZNODE_PATH_DELIM + configSet;
+ final Set<String> toRemoveFiles = new HashSet<>();
+ final Set<String> langExt = languages.stream().map(l -> "_" + l).collect(Collectors.toSet());
+ try {
+ ZkMaintenanceUtils.traverseZkTree(zkClient, configPathInZk, ZkMaintenanceUtils.VISIT_ORDER.VISIT_POST, path -> {
+ if (!isMatchingLangOrNonLangFile(path, langExt)) toRemoveFiles.add(path);
+ });
+ } catch (KeeperException.NoNodeException nne) {
+ // no-op
+ }
+
+ for (String path : toRemoveFiles) {
+ try {
+ zkClient.delete(path, -1, false);
+ } catch (KeeperException.NoNodeException nne) {
+ // no-op
+ }
+ }
+
+ return schema;
+ }
+
+ protected ManagedIndexSchema restoreLanguageSpecificObjectsAndFiles(String configSet,
+ ManagedIndexSchema schema,
+ List<String> langs,
+ boolean dynamicEnabled,
+ String copyFrom) throws KeeperException, InterruptedException {
+ // pull the dynamic fields from the copyFrom schema
+ ManagedIndexSchema copyFromSchema = loadLatestSchema(copyFrom);
+
+ final Set<String> langSet = new HashSet<>(includeLangIds);
+ langSet.addAll(langs);
+
+ boolean restoreAllLangs = langs.isEmpty();
+
+ final Set<String> langFilesToRestore = new HashSet<>();
+
+ // Restore missing files
+ SolrZkClient zkClient = zkStateReader().getZkClient();
+ String configPathInZk = ZkConfigSetService.CONFIGS_ZKNODE + ZNODE_PATH_DELIM + copyFrom;
+ final Set<String> langExt = langSet.stream().map(l -> "_" + l).collect(Collectors.toSet());
+ try {
+ ZkMaintenanceUtils.traverseZkTree(zkClient, configPathInZk, ZkMaintenanceUtils.VISIT_ORDER.VISIT_POST, path -> {
+ if (path.endsWith(".txt")) {
+ if (restoreAllLangs) {
+ langFilesToRestore.add(path);
+ return;
+ }
+
+ final String pathWoExt = path.substring(0, path.length() - 4);
+ for (String lang : langExt) {
+ if (pathWoExt.endsWith(lang)) {
+ langFilesToRestore.add(path);
+ break;
+ }
+ }
+ }
+ });
+ } catch (KeeperException.NoNodeException nne) {
+ // no-op
+ }
+
+ if (!langFilesToRestore.isEmpty()) {
+ final String replacePathDir = "/" + configSet;
+ final String origPathDir = "/" + copyFrom;
+ for (String path : langFilesToRestore) {
+ String copyToPath = path.replace(origPathDir, replacePathDir);
+ if (!zkClient.exists(copyToPath, true)) {
+ zkClient.makePath(copyToPath, false, true);
+ zkClient.setData(copyToPath, zkClient.getData(path, null, null, true), true);
+ }
+ }
+ }
+
+ // Restore field types
+ final Map<String, FieldType> existingTypes = schema.getFieldTypes();
+ List<FieldType> addTypes = copyFromSchema.getFieldTypes().values().stream()
+ .filter(t -> isLangTextType(t, restoreAllLangs ? null : langSet) && !existingTypes.containsKey(t.getTypeName()))
+ .collect(Collectors.toList());
+ if (!addTypes.isEmpty()) {
+ schema = schema.addFieldTypes(addTypes, false);
+ }
+
+ if (dynamicEnabled) {
+ // restore language specific dynamic fields
+ final Set<String> existingDynFields = Arrays.stream(schema.getDynamicFieldPrototypes())
+ .map(SchemaField::getName)
+ .collect(Collectors.toSet());
+
+ final Set<String> langFieldTypeNames = schema.getFieldTypes().values().stream()
+ .filter(t -> isLangTextType(t, restoreAllLangs ? null : langSet))
+ .map(FieldType::getTypeName)
+ .collect(Collectors.toSet());
+
+ List<SchemaField> addDynFields = Arrays.stream(copyFromSchema.getDynamicFields())
+ .filter(df -> langFieldTypeNames.contains(df.getPrototype().getType().getTypeName()))
+ .filter(df -> !existingDynFields.contains(df.getPrototype().getName()))
+ .map(IndexSchema.DynamicField::getPrototype)
+ .collect(Collectors.toList());
+ if (!addDynFields.isEmpty()) {
+ schema = schema.addDynamicFields(addDynFields, null, false);
+ }
+ } else {
+ schema = removeDynamicFields(schema);
+ }
+
+ return schema;
+ }
+
+ private boolean isMatchingLangOrNonLangFile(final String path, final Set<String> langs) {
+ if (!path.endsWith(".txt"))
+ return true; // not a .txt file, always include
+
+ int slashAt = path.lastIndexOf('/');
+ String fileName = slashAt != -1 ? path.substring(slashAt + 1) : "";
+ if (!fileName.contains("_"))
+ return true; // looking for file names like stopwords_en.txt, not a match, so skip it
+
+ // remove the .txt extension
+ final String pathWoExt = fileName.substring(0, fileName.length() - 4);
+ for (String lang : langs) {
+ if (pathWoExt.endsWith(lang)) {
+ return true;
+ }
+ }
+
+ // if we fall thru to here, then the file should be excluded
+ return false;
+ }
+
+ private boolean isTextType(final FieldType t) {
+ return t.getTypeName().startsWith("text_") && TextField.class.equals(t.getClass());
+ }
+
+ private boolean isLangTextType(final FieldType t, final Set<String> langSet) {
+ return isTextType(t) && (langSet == null || langSet.contains(t.getTypeName().substring("text_".length())));
+ }
+
+ protected ManagedIndexSchema removeDynamicFields(ManagedIndexSchema schema) {
+ List<String> dynamicFieldNames =
+ Arrays.stream(schema.getDynamicFields()).map(f -> f.getPrototype().getName()).collect(Collectors.toList());
+ if (!dynamicFieldNames.isEmpty()) {
+ schema = schema.deleteDynamicFields(dynamicFieldNames);
+ }
+ return schema;
+ }
+
+ protected ManagedIndexSchema restoreDynamicFields(ManagedIndexSchema schema, List<String> langs, String copyFrom) {
+ // pull the dynamic fields from the copyFrom schema
+ ManagedIndexSchema copyFromSchema = loadLatestSchema(copyFrom);
+ IndexSchema.DynamicField[] dynamicFields = copyFromSchema.getDynamicFields();
+ if (dynamicFields.length == 0 && !DEFAULT_CONFIGSET_NAME.equals(copyFrom)) {
+ copyFromSchema = loadLatestSchema(DEFAULT_CONFIGSET_NAME);
+ dynamicFields = copyFromSchema.getDynamicFields();
+ }
+
+ if (dynamicFields.length == 0) {
+ return schema;
+ }
+
+ final Set<String> existingDFNames =
+ Arrays.stream(schema.getDynamicFields()).map(df -> df.getPrototype().getName()).collect(Collectors.toSet());
+ List<SchemaField> toAdd = Arrays.stream(dynamicFields)
+ .filter(df -> !existingDFNames.contains(df.getPrototype().getName()))
+ .map(IndexSchema.DynamicField::getPrototype)
+ .collect(Collectors.toList());
+
+ // only restore language specific dynamic fields that match our langSet
+ if (!langs.isEmpty()) {
+ final Set<String> langSet = new HashSet<>(includeLangIds);
+ langSet.addAll(langs);
+ toAdd = toAdd.stream()
+ .filter(df -> !df.getName().startsWith("*_txt_") || langSet.contains(df.getName().substring("*_txt_".length())))
+ .collect(Collectors.toList());
+ }
+
+ if (!toAdd.isEmpty()) {
+ // grab any field types that need to be re-added
+ final Map<String, FieldType> fieldTypes = schema.getFieldTypes();
+ List<FieldType> addTypes = toAdd.stream()
+ .map(SchemaField::getType)
+ .filter(t -> !fieldTypes.containsKey(t.getTypeName()))
+ .collect(Collectors.toList());
+ if (!addTypes.isEmpty()) {
+ schema = schema.addFieldTypes(addTypes, false);
+ }
+
+ schema = schema.addDynamicFields(toAdd, null, true);
+ }
+
+ return schema;
+ }
+
+ void checkSchemaVersion(String configSet, final int versionInRequest, int currentVersion) throws KeeperException, InterruptedException {
+ if (versionInRequest < 0) {
+ return; // don't enforce the version check
+ }
+
+ if (currentVersion == -1) {
+ currentVersion = getCurrentSchemaVersion(configSet);
+ }
+
+ if (currentVersion != versionInRequest) {
+ if (configSet.startsWith(DESIGNER_PREFIX)) {
+ configSet = configSet.substring(DESIGNER_PREFIX.length());
+ }
+ throw new SolrException(SolrException.ErrorCode.CONFLICT,
+ "Your schema version " + versionInRequest + " for " + configSet + " is out-of-date; current version is: " + currentVersion +
+ ". Perhaps another user also updated the schema while you were editing it? You'll need to retry your update after the schema is refreshed.");
+ }
+ }
+
+ List<String> listConfigsInZk() throws IOException {
+ return cc.getConfigSetService().listConfigs();
+ }
+
+ byte[] downloadAndZipConfigSet(String configId) throws IOException {
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ Path tmpDirectory = Files.createTempDirectory("schema-designer-" + FilenameUtils.getName(configId));
Review comment:
*PATH_TRAVERSAL_IN:* This API (java/nio/file/Files.createTempDirectory(Ljava/lang/String;[Ljava/nio/file/attribute/FileAttribute;)Ljava/nio/file/Path;) reads a file whose location might be specified by user input [(details)](https://find-sec-bugs.github.io/bugs.htm#PATH_TRAVERSAL_IN)
(at-me [in a reply](https://docs.muse.dev/docs/talk-to-muse/) with `help` or `ignore`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] MarcusSorealheis commented on a change in pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis commented on a change in pull request #42:
URL: https://github.com/apache/solr/pull/42#discussion_r644427250
##########
File path: solr/webapp/web/js/angular/app.js
##########
@@ -430,7 +436,11 @@ solrAdminApp.config([
$location.path('/login');
}
} else {
- $rootScope.exceptions[rejection.config.url] = rejection.data.error;
+ // schema designer prefers to handle errors itselft
Review comment:
nit: drop the trailing `t` on this line.. reading closely. :)
##########
File path: solr/webapp/web/index.html
##########
@@ -26,6 +26,7 @@
<link rel="stylesheet" type="text/css" href="css/angular/angular-csp.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/common.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/analysis.css?_=${version}">
+ <link rel="stylesheet" type="text/css" href="css/angular/schema-designer.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/cloud.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/cores.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/collections.css?_=${version}">
Review comment:
another nit: the doctype should be changed to be HTML 5. It's as simple as: `<!DOCTYPE html>` at the top of this file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] MarcusSorealheis commented on a change in pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis commented on a change in pull request #42:
URL: https://github.com/apache/solr/pull/42#discussion_r604457028
##########
File path: solr/webapp/web/js/angular/services.js
##########
@@ -254,6 +254,17 @@ solrAdminServices.factory('System',
get: {method: "GET"}
})
}])
+.factory('SchemaDesigner',
+ ['$resource', function($resource) {
+ return $resource('/api/schema-designer/:path', {wt: 'json', path: '@path', _:Date.now()}, {
+ get: {method: "GET"},
+ post: {method: "POST"},
+ put: {method: "PUT"},
+ postXml: {headers: {'Content-type': 'text/xml'}, method: "POST"},
+ postCsv: {headers: {'Content-type': 'application/csv'}, method: "POST"},
+ upload: {method: "POST", transformRequest: angular.identity, headers: {'Content-Type': undefined} }
Review comment:
That makes sense. I am still testing out the tool and profiling the network requests. I had not gotten to this request in the real world, though it was on my list. This was my reaction when I was simply reading the code.
Awesome job. Important feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] MarcusSorealheis commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-854126530
I just rebuilt from nothing and got the same error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] epugh commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
epugh commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-846247471
1. when "Analyzing your sample data, schema will load momentarily" text comes up, could we swap the gear for a moving animated gear icon?
1. download products a file called "download", not a "download.zip", or even better... "techproducts_configset.zip"
1. Is the config a config? Or a ConfigSet????
1. I wish the folder icon when clicked on would open it. Versus the small > arrow. This I find annoying in the SolrCloud ZK tree as well, though there you might argue you want to look at metadata.. I wish there it did both showing metadata and opened, and on schema designer it opened!
1. I wonder if you could get more screen space by having any kind of paning close/open options?
1. in the facet listing, when you shift click you get all between the first and the second click, versus shift click letting you pick them individually, which I think is what you wanted!
1. are there any features that you expose that NO ONE should ever use and could be removed???
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] MarcusSorealheis commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-853527455
there's also a few security exceptions that if thrown in the wrong corporate org could start problematic conversations running SOLR. Everything is under scrutiny right now given the recent events.
To fix those errors referencing image resources being blocked by the CSP, we need to add this clause to `jetty.xml`: `img-src 'self' data:`. The change will still restrict img resources to the host but it will allow SVG and IMG. We cannot be more specific because of the port. :) It is a replacement of the existing `img-srv 'self' data`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] MarcusSorealheis edited a comment on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis edited a comment on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-853527455
there's also a few security exceptions that if thrown in the wrong corporate org could start problematic conversations running SOLR. Everything is under scrutiny right now given the recent events.
To fix those errors referencing image resources being blocked by the CSP, we need to add this clause to `jetty.xml`: `img-src 'self' data:`. The change will still restrict img resources to the host but it will allow SVG and IMG. We cannot be more specific because of the port. :) It is a replacement of the existing `img-srv 'self'`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude edited a comment on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
thelabdude edited a comment on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-848885276
> Maybe there's just an option to see the JSON in the sample results screen, so I could copy and paste.
Ha! Great idea ;-) I've been wanting to add that anyway, so this is another reason to do so. Thanks for the suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] MarcusSorealheis commented on a change in pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis commented on a change in pull request #42:
URL: https://github.com/apache/solr/pull/42#discussion_r625368513
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -805,6 +806,12 @@ public void load() {
containerHandlers.getApiBag().registerObject(clusterAPI);
containerHandlers.getApiBag().registerObject(clusterAPI.commands);
containerHandlers.getApiBag().registerObject(clusterAPI.configSetCommands);
+
+ if (this.isZooKeeperAware()) {
+ SchemaDesignerAPI schemaDesignerAPI = new SchemaDesignerAPI(this);
+ containerHandlers.getApiBag().registerObject(schemaDesignerAPI);
+ } // Schema Designer not available in standalone (non-cloud) mode
+
Review comment:
Do you think the ideal programmatic experience would be for users in Standalone mode to receive this information (`not available in Standalone mode`) via API response?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] MarcusSorealheis commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-854136815
Ignore this.. Looking into the client side errors now. First two nits and SOLR-wide issue persists. New JIRA incoming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude edited a comment on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
thelabdude edited a comment on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-853963136
> Schema Designer should not simply throw generic errors that do no help the user. For instance, when a user tries to create a field that already exists, there is just an error processing banner. Ideally, you could grab the important detail in the response body and show it to the user. Consider `data["error"]["details"][0]["errorMessages"][0]` as seen below to populate the banner with something helpful to the user:
>
> ```
> Possibly unhandled rejection: {"data":{"responseHeader":{"status":400,"QTime":1},"error":{"metadata":["error-class","org.apache.solr.api.ApiBag$ExceptionWithErrObject","root-error-class","org.apache.solr.api.ApiBag$ExceptionWithErrObject"],"details":[{"add-field":{"stored":"true","indexed":"true","uninvertible":"true","name":"name","type":"string"},"errorMessages":["Field 'name' already exists.\n"]}],"msg":"error processing commands","code":400}},"status":400,"config":{"method":"POST","transformRequest":[null],"transformResponse":[null],"jsonpCallbackParam":"callback","data":{"add-field":{"stored":"true","indexed":"true","uninvertible":"true","name":"name","type":"string"}},"url":"techproducts/schema","params":{"wt":"json","_":1622686070279},"headers":{"Accept":"application/json, text/plain, */*","X-Requested-With":"XMLHttpRequest","Content-Type":"application/json;charset=utf-8"},"timeout":10000},"statusText":"Bad Request","xhrStatus":"complete","resource":{"add-field":"..."}}``
`
> ```
Can you please provide reproduction steps for how you got this error (including browser details)? Schema Designer UI should be displaying user friendly errors in a dialog. Moreover, I don't even know how you got this error since the JS code validates the field doesn't already exist before submitting the request. See screenshot of what I see in my env. <img width="357" alt="Screen Shot 2021-06-03 at 9 32 19 AM" src="https://user-images.githubusercontent.com/417074/120671957-1c3aad00-c44f-11eb-8c62-ff6051ba9316.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] MarcusSorealheis edited a comment on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis edited a comment on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-853527455
there's also a few security exceptions that if thrown in the wrong corporate org could start problematic conversations running SOLR. Everything is under scrutiny right now given the recent events.
To fix those errors referencing image resources being blocked by the CSP, we need to add this clause to `jetty.xml`: `img-src 'self' data:`. The change will still restrict img resources to the host but it will allow SVG and IMG. We cannot be more specific because of the port. :) It is a replacement of the existing `img-srv 'self'`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
thelabdude commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-848853798
Thanks @epugh ... my latest commit should address most of your concerns ...
> when "Analyzing your sample data, schema will load momentarily" text comes up, could we swap the gear for a moving animated gear icon?
done.
> download products a file called "download", not a "download.zip", or even better... "techproducts_configset.zip"
done.
> Is the config a config? Or a ConfigSet????
Where in the UI are you seeing mention of config? I've tried to refer to `Configset` but that's not really a term new users would really understand anyway, so I've taken some liberties on just focusing on `schema` in most places.
> I wish the folder icon when clicked on would open it. Versus the small > arrow. This I find annoying in the SolrCloud ZK tree as well, though there you might argue you want to look at metadata.. I wish there it did both showing metadata and opened, and on schema designer it opened!
Double-click works, but I couldn't figure out how to get it to open on single-click. May not be supported by the underlying `jstree` UI component we're using.
> I wonder if you could get more screen space by having any kind of paning close/open options?
We could probably allow closing the Sample documents area once the data is loaded, to give more room for the Schema Editor (middle section). There's also some wasted space on the right for Text Analysis if you're not looking at text fields. I'm curious what the minimum resolution / screen size we should design this for though? Having a view of your sample docs, schema, text analysis, and query tester form / results all in once place is crucial to the design.
> in the facet listing, when you shift click you get all between the first and the second click, versus shift click letting you pick them individually, which I think is what you wanted!
Ok, I removed the tip about using shift as I'm not sure how it works on Windows. On Mac, it's cmd + click to select non-adjacent items. I guess we can just assume people will figure it out on their OS.
> are there any features that you expose that NO ONE should ever use and could be removed???
Not sure? You tell me ;-) Solr schema design is a complicated process and I've designed a solution to help with that, but the fact remains it's still pretty complicated but at least we can give users quick feedback and let them iterate in a safe manner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] MarcusSorealheis commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-853507516
Schema Designer should not simply throw generic errors that do no help the user. For instance, when a user tries to create a field that already exists, there is just an error processing banner. Ideally, you could grab the important detail in the response body and show it to the user. Consider `data["error"]["details"][0]["errorMessages"][0]` as seen below to populate the banner with something helpful to the user:
```
Possibly unhandled rejection: {"data":{"responseHeader":{"status":400,"QTime":1},"error":{"metadata":["error-class","org.apache.solr.api.ApiBag$ExceptionWithErrObject","root-error-class","org.apache.solr.api.ApiBag$ExceptionWithErrObject"],"details":[{"add-field":{"stored":"true","indexed":"true","uninvertible":"true","name":"name","type":"string"},"errorMessages":["Field 'name' already exists.\n"]}],"msg":"error processing commands","code":400}},"status":400,"config":{"method":"POST","transformRequest":[null],"transformResponse":[null],"jsonpCallbackParam":"callback","data":{"add-field":{"stored":"true","indexed":"true","uninvertible":"true","name":"name","type":"string"}},"url":"techproducts/schema","params":{"wt":"json","_":1622686070279},"headers":{"Accept":"application/json, text/plain, */*","X-Requested-With":"XMLHttpRequest","Content-Type":"application/json;charset=utf-8"},"timeout":10000},"statusText":"Bad Request","xhrStatus":"complete","resource":{"add-field":"..."}}```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
thelabdude commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-853964812
> there's also a few security exceptions that if thrown in the wrong corporate org could start problematic conversations running SOLR. Everything is under scrutiny right now given the recent events.
>
> To fix those errors referencing image resources being blocked by the CSP, we need to add this clause to `jetty.xml`: `img-src 'self' data:`. The change will still restrict img resources to the host but it will allow SVG and IMG. We cannot be more specific because of the port. :) It is a replacement of the existing `img-srv 'self'`.
This doesn't sound like it is Schema Designer UI specific. I believe I'm referencing icons the same way all the other screens do. If you think this is a Schema Designer specific issue, please provide a link to the offending code. Otherwise, please open another JIRA to address that problem independently of this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude commented on a change in pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
thelabdude commented on a change in pull request #42:
URL: https://github.com/apache/solr/pull/42#discussion_r604452376
##########
File path: solr/webapp/web/js/angular/services.js
##########
@@ -254,6 +254,17 @@ solrAdminServices.factory('System',
get: {method: "GET"}
})
}])
+.factory('SchemaDesigner',
+ ['$resource', function($resource) {
+ return $resource('/api/schema-designer/:path', {wt: 'json', path: '@path', _:Date.now()}, {
+ get: {method: "GET"},
+ post: {method: "POST"},
+ put: {method: "PUT"},
+ postXml: {headers: {'Content-type': 'text/xml'}, method: "POST"},
+ postCsv: {headers: {'Content-type': 'application/csv'}, method: "POST"},
+ upload: {method: "POST", transformRequest: angular.identity, headers: {'Content-Type': undefined} }
Review comment:
Hi @MarcusSorealheis, thanks for taking a look ...
With jQuery, the default content-type is always `application/json` except where I've declared it explicitly. Almost all of the API calls the designer makes to the backend send and expect JSON.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
thelabdude commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-853963136
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude commented on a change in pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
thelabdude commented on a change in pull request #42:
URL: https://github.com/apache/solr/pull/42#discussion_r644903516
##########
File path: solr/webapp/web/index.html
##########
@@ -26,6 +26,7 @@
<link rel="stylesheet" type="text/css" href="css/angular/angular-csp.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/common.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/analysis.css?_=${version}">
+ <link rel="stylesheet" type="text/css" href="css/angular/schema-designer.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/cloud.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/cores.css?_=${version}">
<link rel="stylesheet" type="text/css" href="css/angular/collections.css?_=${version}">
Review comment:
hmmm ... This too seems unrelated to the Schema Designer UI. Please raise another JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] MarcusSorealheis commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-854122801
> > Schema Designer should not simply throw generic errors that do no help the user. For instance, when a user tries to create a field that already exists, there is just an error processing banner. Ideally, you could grab the important detail in the response body and show it to the user. Consider `data["error"]["details"][0]["errorMessages"][0]` as seen below to populate the banner with something helpful to the user:
> > ```
> > Possibly unhandled rejection: {"data":{"responseHeader":{"status":400,"QTime":1},"error":{"metadata":["error-class","org.apache.solr.api.ApiBag$ExceptionWithErrObject","root-error-class","org.apache.solr.api.ApiBag$ExceptionWithErrObject"],"details":[{"add-field":{"stored":"true","indexed":"true","uninvertible":"true","name":"name","type":"string"},"errorMessages":["Field 'name' already exists.\n"]}],"msg":"error processing commands","code":400}},"status":400,"config":{"method":"POST","transformRequest":[null],"transformResponse":[null],"jsonpCallbackParam":"callback","data":{"add-field":{"stored":"true","indexed":"true","uninvertible":"true","name":"name","type":"string"}},"url":"techproducts/schema","params":{"wt":"json","_":1622686070279},"headers":{"Accept":"application/json, text/plain, */*","X-Requested-With":"XMLHttpRequest","Content-Type":"application/json;charset=utf-8"},"timeout":10000},"statusText":"Bad Request","xhrStatus":"complete","resource":{"add-field":"..."}}
```
> > ```
>
> Can you please provide reproduction steps for how you got this error (including browser details)? Schema Designer UI should be displaying user friendly errors in a dialog. Moreover, I don't even know how you got this error since the JS code validates the field doesn't already exist before submitting the request. See screenshot of what I see in my env. <img alt="Screen Shot 2021-06-03 at 9 32 19 AM" width="357" src="https://user-images.githubusercontent.com/417074/120671957-1c3aad00-c44f-11eb-8c62-ff6051ba9316.png">
1. pull down the repo at this point
2. `gh checkout pr 42`
3. ./gradlew assemble
4. cd into the solr snapshot
5. bin/solr start -e techproducts
6. open Chrome Version 91.0.4472.77 (Official Build) (x86_64)
7. click schema
8. add field
9. see screenshot - type name in the field and set it to the string type
10. see screenshot - click add field
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] MarcusSorealheis commented on a change in pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis commented on a change in pull request #42:
URL: https://github.com/apache/solr/pull/42#discussion_r626131777
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -805,6 +806,12 @@ public void load() {
containerHandlers.getApiBag().registerObject(clusterAPI);
containerHandlers.getApiBag().registerObject(clusterAPI.commands);
containerHandlers.getApiBag().registerObject(clusterAPI.configSetCommands);
+
+ if (this.isZooKeeperAware()) {
+ SchemaDesignerAPI schemaDesignerAPI = new SchemaDesignerAPI(this);
+ containerHandlers.getApiBag().registerObject(schemaDesignerAPI);
+ } // Schema Designer not available in standalone (non-cloud) mode
+
Review comment:
ahh. That makes sense. I wasn't sure if there would be two different personas interfacing, e.g, someone posting to a shared instance a schema they have ben toying with locally and another person reviewing the same schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude commented on pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
thelabdude commented on pull request #42:
URL: https://github.com/apache/solr/pull/42#issuecomment-848885276
> Maybe there's just an option to see the JSON in the sample results screen, so I could copy and paste.
Ha! Great idea ;-) I've been wanting to add that anyway, so this is another reason to do so. Thanks for the suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] MarcusSorealheis commented on a change in pull request #42: SOLR-15277: Schema designer UI and supporting backend
Posted by GitBox <gi...@apache.org>.
MarcusSorealheis commented on a change in pull request #42:
URL: https://github.com/apache/solr/pull/42#discussion_r602623320
##########
File path: solr/webapp/web/js/angular/services.js
##########
@@ -254,6 +254,17 @@ solrAdminServices.factory('System',
get: {method: "GET"}
})
}])
+.factory('SchemaDesigner',
+ ['$resource', function($resource) {
+ return $resource('/api/schema-designer/:path', {wt: 'json', path: '@path', _:Date.now()}, {
+ get: {method: "GET"},
+ post: {method: "POST"},
+ put: {method: "PUT"},
+ postXml: {headers: {'Content-type': 'text/xml'}, method: "POST"},
+ postCsv: {headers: {'Content-type': 'application/csv'}, method: "POST"},
+ upload: {method: "POST", transformRequest: angular.identity, headers: {'Content-Type': undefined} }
Review comment:
Does it make sense to define this action's `Content-Type`? Maybe `multipart/form-data`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org