You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by GitBox <gi...@apache.org> on 2022/05/27 16:14:46 UTC
[GitHub] [solr] NazerkeBS opened a new pull request, #882: SOLR-14699: Update Solr request logs to JSON format
NazerkeBS opened a new pull request, #882:
URL: https://github.com/apache/solr/pull/882
https://issues.apache.org/jira/browse/SOLR-XXXXX
<!--
_(If you are a project committer then you may remove some/all of the following template.)_
Before creating a pull request, please file an issue in the ASF Jira system for Solr:
* https://issues.apache.org/jira/projects/SOLR
You will need to create an account in Jira in order to create an issue.
The title of the PR should reference the Jira issue number in the form:
* SOLR-####: <short description of problem or changes>
SOLR must be fully capitalized. A short description helps people scanning pull requests for items they can work on.
Properly referencing the issue in the title ensures that Jira is correctly updated with code review comments and commits. -->
# Description
Please provide a short description of the changes you're making with this pull request.
# Solution
Please provide a short description of the approach taken to implement your solution.
# Tests
Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem.
# Checklist
Please review the following and check all that apply:
- [ ] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability.
- [ ] I have created a Jira issue and added the issue ID to my pull request title.
- [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
- [ ] I have developed this patch against the `main` branch.
- [ ] I have run `./gradlew check`.
- [ ] I have added tests for my changes.
- [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] NazerkeBS commented on a diff in pull request #882: SOLR-14699: Update Solr request logs to JSON format
Posted by GitBox <gi...@apache.org>.
NazerkeBS commented on code in PR #882:
URL: https://github.com/apache/solr/pull/882#discussion_r893227325
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,71 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
+ private Map<String, Object> extractJsonFormattedMessage(String line) {
+ int startPos = line.indexOf('{'); // '{' starts at
+ int endPos = line.lastIndexOf('}'); // '}' ends at
Review Comment:
We might have trailing spaces
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-14699: Update Solr request logs to JSON format [solr]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #882:
URL: https://github.com/apache/solr/pull/882#issuecomment-1953292445
This PR had no visible activity in the past 60 days, labeling it as stale. Any new activity will remove the stale label. To attract more reviewers, please tag someone or notify the dev@solr.apache.org mailing list. Thank you for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on a diff in pull request #882: SOLR-14699: Update Solr request logs to JSON format
Posted by GitBox <gi...@apache.org>.
dsmiley commented on code in PR #882:
URL: https://github.com/apache/solr/pull/882#discussion_r883947375
##########
solr/core/src/java/org/apache/solr/response/SolrQueryResponse.java:
##########
@@ -217,17 +219,15 @@ public String getToLogAsString() {
/** Returns a string of the form "prefix name1=value1 name2=value2 ..." */
public String getToLogAsString(String prefix) {
- StringBuilder sb = new StringBuilder(prefix);
- for (int i = 0; i < toLog.size(); i++) {
- if (sb.length() > 0) {
- sb.append(' ');
- }
- String name = toLog.getName(i);
- Object val = toLog.getVal(i);
- if (name != null) {
- sb.append(name).append('=');
- }
- sb.append(val);
+ StringBuilder sb = new StringBuilder();
+ if (prefix != null && !prefix.isEmpty()) {
+ // TODO: remove "prefix"
+ Map<String, Object> keyValPairs = new LinkedHashMap<>();
+ keyValPairs.put("prefix", prefix);
+ keyValPairs.putAll(toLog);
+ sb.append(JSONUtil.toJSON(keyValPairs, -1));
+ } else {
+ sb.append(JSONUtil.toJSON(toLog, -1));
Review Comment:
Beautiful. @gus-asf with the move to JSON here, generating JSON is now a one-liner and not custom code. Note the "prefix" thing ought to go away (separate issue).
##########
solr/core/src/java/org/apache/solr/response/SolrQueryResponse.java:
##########
@@ -217,17 +219,15 @@ public String getToLogAsString() {
/** Returns a string of the form "prefix name1=value1 name2=value2 ..." */
public String getToLogAsString(String prefix) {
- StringBuilder sb = new StringBuilder(prefix);
- for (int i = 0; i < toLog.size(); i++) {
- if (sb.length() > 0) {
- sb.append(' ');
- }
- String name = toLog.getName(i);
- Object val = toLog.getVal(i);
- if (name != null) {
- sb.append(name).append('=');
- }
- sb.append(val);
+ StringBuilder sb = new StringBuilder();
Review Comment:
Why do we have a StringBuilder here at all anymore?
##########
solr/core/src/java/org/apache/solr/rest/BaseSolrResource.java:
##########
@@ -147,7 +147,7 @@ protected void handleException(Logger log) {
getSolrResponse().add("error", info);
String message = (String) info.get("msg");
if (null != message && !message.trim().isEmpty()) {
- getSolrResponse().getToLog().add("msg", "{" + message.trim() + "}");
+ getSolrResponse().addToLog("msg", "{" + message.trim() + "}");
Review Comment:
Why curly brackets? This is a symptom of code that tried to make the log parseable but it should not be the job of this class here (the caller of addToLog) to concern itself with that.
##########
solr/core/src/java/org/apache/solr/core/SolrCore.java:
##########
@@ -2922,7 +2919,7 @@ public String[] getParams(String param) { // assume param is in lpSet
} // assume in lpSet
};
- toLog.add("params", "{" + filteredParams + "}");
Review Comment:
Removal of needless curly brackets is good!
##########
solr/core/src/java/org/apache/solr/response/SolrQueryResponse.java:
##########
@@ -217,17 +219,15 @@ public String getToLogAsString() {
/** Returns a string of the form "prefix name1=value1 name2=value2 ..." */
Review Comment:
This Javadoc is now wrong.
##########
solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java:
##########
@@ -893,7 +893,7 @@ private void handleAdminRequest() throws IOException {
handler != null
? MarkerFactory.getMarker(handler.getClass().getName())
: MarkerFactory.getMarker(HttpSolrCall.class.getName()),
- solrResp.getToLogAsString("[admin]"));
+ solrResp.getToLogAsString("admin"));
Review Comment:
Instead, could `addToLog("admin", true)` be called at some early point to ensure it shows up (first)?
##########
solr/core/src/test/org/apache/solr/handler/component/ResponseLogComponentTest.java:
##########
@@ -54,7 +54,7 @@ public void testToLogIds() throws Exception {
"responseLog",
"true");
SolrQueryResponse qr = h.queryAndResponse(handler, req);
- NamedList<Object> entries = qr.getToLog();
+ Map<String, Object> entries = qr.getToLog();
Review Comment:
Could declare as "var" and future-proof :-). This is a test after-all.
##########
solr/core/src/test/org/apache/solr/response/TestSolrQueryResponse.java:
##########
@@ -90,35 +90,30 @@ public void testResponse() throws Exception {
public void testToLog() throws Exception {
final SolrQueryResponse response = new SolrQueryResponse();
assertEquals("toLog initially not empty", 0, response.getToLog().size());
- assertEquals("logid_only", response.getToLogAsString("logid_only"));
+ assertEquals("{\"prefix\":\"prefix_only\"}", response.getToLogAsString("prefix_only"));
// initially empty, then add something
response.addToLog("key1", "value1");
- {
- final Iterator<Map.Entry<String, Object>> it = response.getToLog().iterator();
- assertTrue(it.hasNext());
- final Map.Entry<String, Object> entry1 = it.next();
- assertEquals("key1", entry1.getKey());
- assertEquals("value1", entry1.getValue());
- assertFalse(it.hasNext());
- }
- assertEquals("key1=value1", response.getToLogAsString(""));
- assertEquals("abc123 key1=value1", response.getToLogAsString("abc123"));
+
+ Map<String, Object> toLog = response.getToLog();
Review Comment:
I recommend doing assertEquals with a Map you create yourself like {{Map.of("key1", "value1")}}. Same with further below -- way more concise and easy to read/review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] NazerkeBS commented on pull request #882: SOLR-14699: Update Solr request logs to JSON format
Posted by GitBox <gi...@apache.org>.
NazerkeBS commented on PR #882:
URL: https://github.com/apache/solr/pull/882#issuecomment-1144848113
I updated `SolrLogPostTool` but not the entire TestSolrLogPostTool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] NazerkeBS commented on a diff in pull request #882: SOLR-14699: Update Solr request logs to JSON format
Posted by GitBox <gi...@apache.org>.
NazerkeBS commented on code in PR #882:
URL: https://github.com/apache/solr/pull/882#discussion_r887922668
##########
solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java:
##########
@@ -893,7 +893,7 @@ private void handleAdminRequest() throws IOException {
handler != null
? MarkerFactory.getMarker(handler.getClass().getName())
: MarkerFactory.getMarker(HttpSolrCall.class.getName()),
- solrResp.getToLogAsString("[admin]"));
+ solrResp.getToLogAsString("admin"));
Review Comment:
I see there is another place where #getToLogAsString is called with a different prefix: `rspObject.getToLogAsString("TaskId: " + this.taskId);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on a diff in pull request #882: SOLR-14699: Update Solr request logs to JSON format
Posted by GitBox <gi...@apache.org>.
dsmiley commented on code in PR #882:
URL: https://github.com/apache/solr/pull/882#discussion_r888042243
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,72 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
-
- String path = parsePath(line);
- lineRecord.setField("path_s", path);
-
- if (line.contains("hits=")) {
- lineRecord.setField("hits_l", parseHits(line));
+ private Map<String, Object> extractJSONFormattedMessage(String line) {
+ if (line.contains("o.a.s.c.S.Request")
+ || (line.contains("o.a.s.s.HttpSolrCall") && line.contains("\"prefix\""))) {
+ int startPos = 0; // '{' starts at
+ while (startPos < line.length() && line.charAt(startPos) != '{') {
+ startPos++;
+ }
Review Comment:
See `String.indexOf` :-)
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,72 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
-
- String path = parsePath(line);
- lineRecord.setField("path_s", path);
-
- if (line.contains("hits=")) {
- lineRecord.setField("hits_l", parseHits(line));
+ private Map<String, Object> extractJSONFormattedMessage(String line) {
+ if (line.contains("o.a.s.c.S.Request")
+ || (line.contains("o.a.s.s.HttpSolrCall") && line.contains("\"prefix\""))) {
+ int startPos = 0; // '{' starts at
+ while (startPos < line.length() && line.charAt(startPos) != '{') {
+ startPos++;
+ }
+ int endPos = startPos + 1; // '}' ends at
+ while (endPos < line.length() && line.charAt(endPos) != '}') {
+ endPos++;
+ }
+ String json = line.substring(startPos, endPos + 1);
+ @SuppressWarnings("unchecked")
+ Map<String, Object> fromJSON = (Map<String, Object>) Utils.fromJSONString(json);
+ return fromJSON;
}
+ return null;
+ }
- String params = parseParams(line);
- lineRecord.setField("params_t", params);
- addParams(lineRecord, params);
+ private void parseRecord(SolrInputDocument lineRecord, String line) {
+ Map<String, Object> keyValuePairs = extractJSONFormattedMessage(line);
+ if (keyValuePairs != null) {
+ // query request record
+ if (keyValuePairs.containsKey("QTime")) {
+ lineRecord.setField("qtime_i", keyValuePairs.get("QTime"));
+ lineRecord.setField("status_s", keyValuePairs.get("status"));
+ lineRecord.setField("path_s", keyValuePairs.get("path"));
+ if (keyValuePairs.containsKey("hits")) {
+ lineRecord.setField("hits_l", keyValuePairs.get("hits"));
+ }
+ lineRecord.setField("params_t", keyValuePairs.get("params"));
+ addParams(lineRecord, keyValuePairs.get("params").toString());
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("node_s", parseNode(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
+ lineRecord.setField("node_s", keyValuePairs.get("node_name"));
- if (path != null && path.contains("/admin")) {
- lineRecord.setField("type_s", "admin");
- } else if (path != null && params.contains("/replication")) {
- lineRecord.setField("type_s", "replication");
- } else if (path != null && path.contains("/get")) {
- lineRecord.setField("type_s", "get");
- } else {
- lineRecord.setField("type_s", "query");
+ if (keyValuePairs.containsKey("prefix") && keyValuePairs.get("prefix").equals("admin")) {
+ lineRecord.setField("type_s", "admin");
+ } else if (keyValuePairs.get("params").toString().contains("/replication")) {
+ lineRecord.setField("type_s", "replication");
+ } else if (keyValuePairs.get("path").equals("/get")) {
+ lineRecord.setField("type_s", "get");
+ } else {
+ lineRecord.setField("type_s", "query");
+ }
+ }
+
+ // update request record
+ if (keyValuePairs.get("path").equals("/update")) {
+ if (keyValuePairs.containsKey("deleteByQuery")) {
+ lineRecord.setField("type_s", "deleteByQuery");
+ } else if (keyValuePairs.containsKey("delete")) {
+ lineRecord.setField("type_s", "delete");
+ } else if (keyValuePairs.containsKey("commit") && (Boolean) keyValuePairs.get("commit")) {
+ lineRecord.setField("type_s", "commit");
+ } else {
+ lineRecord.setField("type_s", "update");
+ }
+ }
}
}
- private void parseNewSearch(SolrInputDocument lineRecord, String line) {
+ private void setMDCFields(SolrInputDocument lineRecord, String line) {
+ // x:value format
lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("type_s", "newSearcher");
Review Comment:
why remove?
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,72 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
-
- String path = parsePath(line);
- lineRecord.setField("path_s", path);
-
- if (line.contains("hits=")) {
- lineRecord.setField("hits_l", parseHits(line));
+ private Map<String, Object> extractJSONFormattedMessage(String line) {
+ if (line.contains("o.a.s.c.S.Request")
+ || (line.contains("o.a.s.s.HttpSolrCall") && line.contains("\"prefix\""))) {
+ int startPos = 0; // '{' starts at
+ while (startPos < line.length() && line.charAt(startPos) != '{') {
+ startPos++;
+ }
+ int endPos = startPos + 1; // '}' ends at
+ while (endPos < line.length() && line.charAt(endPos) != '}') {
+ endPos++;
+ }
Review Comment:
Can we assume the message ends with '}'? I think so.
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,72 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
-
- String path = parsePath(line);
- lineRecord.setField("path_s", path);
-
- if (line.contains("hits=")) {
- lineRecord.setField("hits_l", parseHits(line));
+ private Map<String, Object> extractJSONFormattedMessage(String line) {
+ if (line.contains("o.a.s.c.S.Request")
Review Comment:
Maybe we shouldn't even bother looking for the specific classes we know do this; just look for valid JSON? This will future-proof other use of JSON and allow someone to format the logger names more fully and have this tool still work.
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,72 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
-
- String path = parsePath(line);
- lineRecord.setField("path_s", path);
-
- if (line.contains("hits=")) {
- lineRecord.setField("hits_l", parseHits(line));
+ private Map<String, Object> extractJSONFormattedMessage(String line) {
Review Comment:
As a general rule, when creating variable/method/class names, don't use all-caps for acronyms. Yes, you will see many cases where existing code doesn't do this but most developers find it best to just lowercase them. Thus this here use "Json" not "JSON". Same for "Mdc" not "MDC".
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,72 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
-
- String path = parsePath(line);
- lineRecord.setField("path_s", path);
-
- if (line.contains("hits=")) {
- lineRecord.setField("hits_l", parseHits(line));
+ private Map<String, Object> extractJSONFormattedMessage(String line) {
+ if (line.contains("o.a.s.c.S.Request")
+ || (line.contains("o.a.s.s.HttpSolrCall") && line.contains("\"prefix\""))) {
+ int startPos = 0; // '{' starts at
+ while (startPos < line.length() && line.charAt(startPos) != '{') {
+ startPos++;
+ }
+ int endPos = startPos + 1; // '}' ends at
+ while (endPos < line.length() && line.charAt(endPos) != '}') {
+ endPos++;
+ }
Review Comment:
Also, note `String.lastIndexOf` :-)
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,72 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
-
- String path = parsePath(line);
- lineRecord.setField("path_s", path);
-
- if (line.contains("hits=")) {
- lineRecord.setField("hits_l", parseHits(line));
+ private Map<String, Object> extractJSONFormattedMessage(String line) {
+ if (line.contains("o.a.s.c.S.Request")
+ || (line.contains("o.a.s.s.HttpSolrCall") && line.contains("\"prefix\""))) {
+ int startPos = 0; // '{' starts at
+ while (startPos < line.length() && line.charAt(startPos) != '{') {
+ startPos++;
+ }
+ int endPos = startPos + 1; // '}' ends at
+ while (endPos < line.length() && line.charAt(endPos) != '}') {
+ endPos++;
+ }
+ String json = line.substring(startPos, endPos + 1);
+ @SuppressWarnings("unchecked")
+ Map<String, Object> fromJSON = (Map<String, Object>) Utils.fromJSONString(json);
+ return fromJSON;
}
+ return null;
+ }
- String params = parseParams(line);
- lineRecord.setField("params_t", params);
- addParams(lineRecord, params);
+ private void parseRecord(SolrInputDocument lineRecord, String line) {
+ Map<String, Object> keyValuePairs = extractJSONFormattedMessage(line);
+ if (keyValuePairs != null) {
+ // query request record
+ if (keyValuePairs.containsKey("QTime")) {
+ lineRecord.setField("qtime_i", keyValuePairs.get("QTime"));
+ lineRecord.setField("status_s", keyValuePairs.get("status"));
+ lineRecord.setField("path_s", keyValuePairs.get("path"));
+ if (keyValuePairs.containsKey("hits")) {
+ lineRecord.setField("hits_l", keyValuePairs.get("hits"));
+ }
+ lineRecord.setField("params_t", keyValuePairs.get("params"));
+ addParams(lineRecord, keyValuePairs.get("params").toString());
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("node_s", parseNode(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
+ lineRecord.setField("node_s", keyValuePairs.get("node_name"));
- if (path != null && path.contains("/admin")) {
- lineRecord.setField("type_s", "admin");
- } else if (path != null && params.contains("/replication")) {
- lineRecord.setField("type_s", "replication");
- } else if (path != null && path.contains("/get")) {
- lineRecord.setField("type_s", "get");
- } else {
- lineRecord.setField("type_s", "query");
+ if (keyValuePairs.containsKey("prefix") && keyValuePairs.get("prefix").equals("admin")) {
+ lineRecord.setField("type_s", "admin");
+ } else if (keyValuePairs.get("params").toString().contains("/replication")) {
+ lineRecord.setField("type_s", "replication");
+ } else if (keyValuePairs.get("path").equals("/get")) {
+ lineRecord.setField("type_s", "get");
+ } else {
+ lineRecord.setField("type_s", "query");
+ }
+ }
+
+ // update request record
+ if (keyValuePairs.get("path").equals("/update")) {
+ if (keyValuePairs.containsKey("deleteByQuery")) {
+ lineRecord.setField("type_s", "deleteByQuery");
+ } else if (keyValuePairs.containsKey("delete")) {
+ lineRecord.setField("type_s", "delete");
+ } else if (keyValuePairs.containsKey("commit") && (Boolean) keyValuePairs.get("commit")) {
+ lineRecord.setField("type_s", "commit");
+ } else {
Review Comment:
These else-if's surprise me; I think a log line could have all of these? @joel-bernstein I see Nazerke merely updated the same logic so this decision I think was written by you. Should the "type" of the log line be, in effect, multi-valued because there are multiple update payloads (delete (id and/or query), docs, commits)?
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,72 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
-
- String path = parsePath(line);
- lineRecord.setField("path_s", path);
-
- if (line.contains("hits=")) {
- lineRecord.setField("hits_l", parseHits(line));
+ private Map<String, Object> extractJSONFormattedMessage(String line) {
+ if (line.contains("o.a.s.c.S.Request")
+ || (line.contains("o.a.s.s.HttpSolrCall") && line.contains("\"prefix\""))) {
+ int startPos = 0; // '{' starts at
+ while (startPos < line.length() && line.charAt(startPos) != '{') {
+ startPos++;
+ }
+ int endPos = startPos + 1; // '}' ends at
+ while (endPos < line.length() && line.charAt(endPos) != '}') {
+ endPos++;
+ }
+ String json = line.substring(startPos, endPos + 1);
+ @SuppressWarnings("unchecked")
+ Map<String, Object> fromJSON = (Map<String, Object>) Utils.fromJSONString(json);
+ return fromJSON;
}
+ return null;
+ }
- String params = parseParams(line);
- lineRecord.setField("params_t", params);
- addParams(lineRecord, params);
+ private void parseRecord(SolrInputDocument lineRecord, String line) {
+ Map<String, Object> keyValuePairs = extractJSONFormattedMessage(line);
+ if (keyValuePairs != null) {
+ // query request record
+ if (keyValuePairs.containsKey("QTime")) {
+ lineRecord.setField("qtime_i", keyValuePairs.get("QTime"));
+ lineRecord.setField("status_s", keyValuePairs.get("status"));
+ lineRecord.setField("path_s", keyValuePairs.get("path"));
+ if (keyValuePairs.containsKey("hits")) {
+ lineRecord.setField("hits_l", keyValuePairs.get("hits"));
+ }
+ lineRecord.setField("params_t", keyValuePairs.get("params"));
+ addParams(lineRecord, keyValuePairs.get("params").toString());
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("node_s", parseNode(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
+ lineRecord.setField("node_s", keyValuePairs.get("node_name"));
- if (path != null && path.contains("/admin")) {
- lineRecord.setField("type_s", "admin");
- } else if (path != null && params.contains("/replication")) {
- lineRecord.setField("type_s", "replication");
- } else if (path != null && path.contains("/get")) {
- lineRecord.setField("type_s", "get");
- } else {
- lineRecord.setField("type_s", "query");
+ if (keyValuePairs.containsKey("prefix") && keyValuePairs.get("prefix").equals("admin")) {
+ lineRecord.setField("type_s", "admin");
+ } else if (keyValuePairs.get("params").toString().contains("/replication")) {
+ lineRecord.setField("type_s", "replication");
+ } else if (keyValuePairs.get("path").equals("/get")) {
+ lineRecord.setField("type_s", "get");
+ } else {
+ lineRecord.setField("type_s", "query");
+ }
+ }
+
+ // update request record
+ if (keyValuePairs.get("path").equals("/update")) {
Review Comment:
there used to be a method `parseUpdate` and I think it makes sense to continue to have that structure
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,72 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
-
- String path = parsePath(line);
- lineRecord.setField("path_s", path);
-
- if (line.contains("hits=")) {
- lineRecord.setField("hits_l", parseHits(line));
+ private Map<String, Object> extractJSONFormattedMessage(String line) {
+ if (line.contains("o.a.s.c.S.Request")
Review Comment:
If we can assume the pertinent log lines end with a "}", this is a very cheap condition to add as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] NazerkeBS commented on a diff in pull request #882: SOLR-14699: Update Solr request logs to JSON format
Posted by GitBox <gi...@apache.org>.
NazerkeBS commented on code in PR #882:
URL: https://github.com/apache/solr/pull/882#discussion_r892073100
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,72 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
-
- String path = parsePath(line);
- lineRecord.setField("path_s", path);
-
- if (line.contains("hits=")) {
- lineRecord.setField("hits_l", parseHits(line));
+ private Map<String, Object> extractJSONFormattedMessage(String line) {
+ if (line.contains("o.a.s.c.S.Request")
+ || (line.contains("o.a.s.s.HttpSolrCall") && line.contains("\"prefix\""))) {
+ int startPos = 0; // '{' starts at
+ while (startPos < line.length() && line.charAt(startPos) != '{') {
+ startPos++;
+ }
+ int endPos = startPos + 1; // '}' ends at
+ while (endPos < line.length() && line.charAt(endPos) != '}') {
+ endPos++;
+ }
+ String json = line.substring(startPos, endPos + 1);
+ @SuppressWarnings("unchecked")
+ Map<String, Object> fromJSON = (Map<String, Object>) Utils.fromJSONString(json);
+ return fromJSON;
}
+ return null;
+ }
- String params = parseParams(line);
- lineRecord.setField("params_t", params);
- addParams(lineRecord, params);
+ private void parseRecord(SolrInputDocument lineRecord, String line) {
+ Map<String, Object> keyValuePairs = extractJSONFormattedMessage(line);
+ if (keyValuePairs != null) {
+ // query request record
+ if (keyValuePairs.containsKey("QTime")) {
+ lineRecord.setField("qtime_i", keyValuePairs.get("QTime"));
+ lineRecord.setField("status_s", keyValuePairs.get("status"));
+ lineRecord.setField("path_s", keyValuePairs.get("path"));
+ if (keyValuePairs.containsKey("hits")) {
+ lineRecord.setField("hits_l", keyValuePairs.get("hits"));
+ }
+ lineRecord.setField("params_t", keyValuePairs.get("params"));
+ addParams(lineRecord, keyValuePairs.get("params").toString());
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("node_s", parseNode(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
+ lineRecord.setField("node_s", keyValuePairs.get("node_name"));
- if (path != null && path.contains("/admin")) {
- lineRecord.setField("type_s", "admin");
- } else if (path != null && params.contains("/replication")) {
- lineRecord.setField("type_s", "replication");
- } else if (path != null && path.contains("/get")) {
- lineRecord.setField("type_s", "get");
- } else {
- lineRecord.setField("type_s", "query");
+ if (keyValuePairs.containsKey("prefix") && keyValuePairs.get("prefix").equals("admin")) {
+ lineRecord.setField("type_s", "admin");
+ } else if (keyValuePairs.get("params").toString().contains("/replication")) {
+ lineRecord.setField("type_s", "replication");
+ } else if (keyValuePairs.get("path").equals("/get")) {
+ lineRecord.setField("type_s", "get");
+ } else {
+ lineRecord.setField("type_s", "query");
+ }
+ }
+
+ // update request record
+ if (keyValuePairs.get("path").equals("/update")) {
+ if (keyValuePairs.containsKey("deleteByQuery")) {
+ lineRecord.setField("type_s", "deleteByQuery");
+ } else if (keyValuePairs.containsKey("delete")) {
+ lineRecord.setField("type_s", "delete");
+ } else if (keyValuePairs.containsKey("commit") && (Boolean) keyValuePairs.get("commit")) {
+ lineRecord.setField("type_s", "commit");
+ } else {
+ lineRecord.setField("type_s", "update");
+ }
+ }
}
}
- private void parseNewSearch(SolrInputDocument lineRecord, String line) {
+ private void setMDCFields(SolrInputDocument lineRecord, String line) {
+ // x:value format
lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("type_s", "newSearcher");
Review Comment:
Didn't remove, just updated; it's added while checking `"Registered new searcher"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] NazerkeBS commented on a diff in pull request #882: SOLR-14699: Update Solr request logs to JSON format
Posted by GitBox <gi...@apache.org>.
NazerkeBS commented on code in PR #882:
URL: https://github.com/apache/solr/pull/882#discussion_r893226702
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,71 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
+ private Map<String, Object> extractJsonFormattedMessage(String line) {
+ int startPos = line.indexOf('{'); // '{' starts at
+ int endPos = line.lastIndexOf('}'); // '}' ends at
Review Comment:
We might have trailing spaces at the end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] NazerkeBS commented on a diff in pull request #882: SOLR-14699: Update Solr request logs to JSON format
Posted by GitBox <gi...@apache.org>.
NazerkeBS commented on code in PR #882:
URL: https://github.com/apache/solr/pull/882#discussion_r893229503
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,71 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
+ private Map<String, Object> extractJsonFormattedMessage(String line) {
+ int startPos = line.indexOf('{'); // '{' starts at
+ int endPos = line.lastIndexOf('}'); // '}' ends at
+ if (startPos != -1 && endPos != -1) {
+ String json = line.substring(startPos, endPos + 1);
+ @SuppressWarnings("unchecked")
+ Map<String, Object> fromJSON = (Map<String, Object>) Utils.fromJSONString(json);
+ return fromJSON;
+ }
+ return null;
+ }
- String path = parsePath(line);
- lineRecord.setField("path_s", path);
+ private void parseRecord(SolrInputDocument lineRecord, String line) {
+ Map<String, Object> keyValuePairs = extractJsonFormattedMessage(line);
+ if (keyValuePairs != null) {
+ // query request record
+ if (keyValuePairs.containsKey("QTime")) {
+ lineRecord.setField("qtime_i", keyValuePairs.get("QTime"));
+ lineRecord.setField("status_s", keyValuePairs.get("status"));
+ lineRecord.setField("path_s", keyValuePairs.get("path"));
+ if (keyValuePairs.containsKey("hits")) {
+ lineRecord.setField("hits_l", keyValuePairs.get("hits"));
+ }
+ lineRecord.setField("params_t", keyValuePairs.get("params"));
+ addParams(lineRecord, keyValuePairs.get("params").toString());
- if (line.contains("hits=")) {
- lineRecord.setField("hits_l", parseHits(line));
- }
+ lineRecord.setField("node_s", keyValuePairs.get("node_name"));
- String params = parseParams(line);
- lineRecord.setField("params_t", params);
- addParams(lineRecord, params);
+ if (keyValuePairs.containsKey("prefix") && keyValuePairs.get("prefix").equals("admin")) {
+ lineRecord.setField("type_s", "admin");
+ } else if (keyValuePairs.get("params").toString().contains("/replication")) {
+ lineRecord.setField("type_s", "replication");
+ } else if (keyValuePairs.get("path").equals("/get")) {
Review Comment:
I assumed `path=/get`. If I am wrong , then wondering if we can get something like `/get/subpath` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on a diff in pull request #882: SOLR-14699: Update Solr request logs to JSON format
Posted by GitBox <gi...@apache.org>.
dsmiley commented on code in PR #882:
URL: https://github.com/apache/solr/pull/882#discussion_r894843028
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,71 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
+ private Map<String, Object> extractJsonFormattedMessage(String line) {
+ int startPos = line.indexOf('{'); // '{' starts at
+ int endPos = line.lastIndexOf('}'); // '}' ends at
Review Comment:
Might? Why exactly? If a space shows up, we have control in Solr to ensure it doesn't. If you are seeing a space in fact, check the logging format pattern. The message should go exactly last with nothing following. LMK if I'm wrong about this please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on a diff in pull request #882: SOLR-14699: Update Solr request logs to JSON format
Posted by GitBox <gi...@apache.org>.
dsmiley commented on code in PR #882:
URL: https://github.com/apache/solr/pull/882#discussion_r894843621
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,71 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
+ private Map<String, Object> extractJsonFormattedMessage(String line) {
+ int startPos = line.indexOf('{'); // '{' starts at
+ int endPos = line.lastIndexOf('}'); // '}' ends at
+ if (startPos != -1 && endPos != -1) {
+ String json = line.substring(startPos, endPos + 1);
+ @SuppressWarnings("unchecked")
+ Map<String, Object> fromJSON = (Map<String, Object>) Utils.fromJSONString(json);
+ return fromJSON;
+ }
+ return null;
+ }
- String path = parsePath(line);
- lineRecord.setField("path_s", path);
+ private void parseRecord(SolrInputDocument lineRecord, String line) {
+ Map<String, Object> keyValuePairs = extractJsonFormattedMessage(line);
+ if (keyValuePairs != null) {
+ // query request record
+ if (keyValuePairs.containsKey("QTime")) {
+ lineRecord.setField("qtime_i", keyValuePairs.get("QTime"));
+ lineRecord.setField("status_s", keyValuePairs.get("status"));
+ lineRecord.setField("path_s", keyValuePairs.get("path"));
+ if (keyValuePairs.containsKey("hits")) {
+ lineRecord.setField("hits_l", keyValuePairs.get("hits"));
+ }
+ lineRecord.setField("params_t", keyValuePairs.get("params"));
+ addParams(lineRecord, keyValuePairs.get("params").toString());
- if (line.contains("hits=")) {
- lineRecord.setField("hits_l", parseHits(line));
- }
+ lineRecord.setField("node_s", keyValuePairs.get("node_name"));
- String params = parseParams(line);
- lineRecord.setField("params_t", params);
- addParams(lineRecord, params);
+ if (keyValuePairs.containsKey("prefix") && keyValuePairs.get("prefix").equals("admin")) {
+ lineRecord.setField("type_s", "admin");
+ } else if (keyValuePairs.get("params").toString().contains("/replication")) {
+ lineRecord.setField("type_s", "replication");
+ } else if (keyValuePairs.get("path").equals("/get")) {
Review Comment:
No, it will end with /get and (or /replication etc.) and not have anything following. These are the request handler names in solrconfig.xml.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on a diff in pull request #882: SOLR-14699: Update Solr request logs to JSON format
Posted by GitBox <gi...@apache.org>.
dsmiley commented on code in PR #882:
URL: https://github.com/apache/solr/pull/882#discussion_r892805323
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,71 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
+ private Map<String, Object> extractJsonFormattedMessage(String line) {
+ int startPos = line.indexOf('{'); // '{' starts at
+ int endPos = line.lastIndexOf('}'); // '}' ends at
+ if (startPos != -1 && endPos != -1) {
Review Comment:
and check `startPos < endPos` :-)
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,71 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
+ private Map<String, Object> extractJsonFormattedMessage(String line) {
+ int startPos = line.indexOf('{'); // '{' starts at
+ int endPos = line.lastIndexOf('}'); // '}' ends at
+ if (startPos != -1 && endPos != -1) {
+ String json = line.substring(startPos, endPos + 1);
+ @SuppressWarnings("unchecked")
+ Map<String, Object> fromJSON = (Map<String, Object>) Utils.fromJSONString(json);
+ return fromJSON;
+ }
+ return null;
+ }
- String path = parsePath(line);
- lineRecord.setField("path_s", path);
+ private void parseRecord(SolrInputDocument lineRecord, String line) {
+ Map<String, Object> keyValuePairs = extractJsonFormattedMessage(line);
+ if (keyValuePairs != null) {
+ // query request record
+ if (keyValuePairs.containsKey("QTime")) {
+ lineRecord.setField("qtime_i", keyValuePairs.get("QTime"));
+ lineRecord.setField("status_s", keyValuePairs.get("status"));
+ lineRecord.setField("path_s", keyValuePairs.get("path"));
+ if (keyValuePairs.containsKey("hits")) {
+ lineRecord.setField("hits_l", keyValuePairs.get("hits"));
+ }
+ lineRecord.setField("params_t", keyValuePairs.get("params"));
+ addParams(lineRecord, keyValuePairs.get("params").toString());
- if (line.contains("hits=")) {
- lineRecord.setField("hits_l", parseHits(line));
- }
+ lineRecord.setField("node_s", keyValuePairs.get("node_name"));
- String params = parseParams(line);
- lineRecord.setField("params_t", params);
- addParams(lineRecord, params);
+ if (keyValuePairs.containsKey("prefix") && keyValuePairs.get("prefix").equals("admin")) {
+ lineRecord.setField("type_s", "admin");
+ } else if (keyValuePairs.get("params").toString().contains("/replication")) {
Review Comment:
I suspect this logic, which you ported from the existing logic, is kind of a quirk/bug. @joel-bernstein -- shouldn't the check for replication be that the path is `/replication`; don't need/use params to detect this?
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,71 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
+ private Map<String, Object> extractJsonFormattedMessage(String line) {
+ int startPos = line.indexOf('{'); // '{' starts at
+ int endPos = line.lastIndexOf('}'); // '}' ends at
+ if (startPos != -1 && endPos != -1) {
+ String json = line.substring(startPos, endPos + 1);
+ @SuppressWarnings("unchecked")
+ Map<String, Object> fromJSON = (Map<String, Object>) Utils.fromJSONString(json);
+ return fromJSON;
+ }
+ return null;
+ }
- String path = parsePath(line);
- lineRecord.setField("path_s", path);
+ private void parseRecord(SolrInputDocument lineRecord, String line) {
+ Map<String, Object> keyValuePairs = extractJsonFormattedMessage(line);
+ if (keyValuePairs != null) {
+ // query request record
+ if (keyValuePairs.containsKey("QTime")) {
+ lineRecord.setField("qtime_i", keyValuePairs.get("QTime"));
+ lineRecord.setField("status_s", keyValuePairs.get("status"));
+ lineRecord.setField("path_s", keyValuePairs.get("path"));
+ if (keyValuePairs.containsKey("hits")) {
+ lineRecord.setField("hits_l", keyValuePairs.get("hits"));
+ }
+ lineRecord.setField("params_t", keyValuePairs.get("params"));
+ addParams(lineRecord, keyValuePairs.get("params").toString());
- if (line.contains("hits=")) {
- lineRecord.setField("hits_l", parseHits(line));
- }
+ lineRecord.setField("node_s", keyValuePairs.get("node_name"));
- String params = parseParams(line);
- lineRecord.setField("params_t", params);
- addParams(lineRecord, params);
+ if (keyValuePairs.containsKey("prefix") && keyValuePairs.get("prefix").equals("admin")) {
+ lineRecord.setField("type_s", "admin");
+ } else if (keyValuePairs.get("params").toString().contains("/replication")) {
+ lineRecord.setField("type_s", "replication");
+ } else if (keyValuePairs.get("path").equals("/get")) {
Review Comment:
previously, this was `contains` but you made it `equals`. Was this intentional? I could imagine `endsWith` is better since I could imagine the path including stuff before the handler if not now then eventually.
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,71 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
+ private Map<String, Object> extractJsonFormattedMessage(String line) {
+ int startPos = line.indexOf('{'); // '{' starts at
+ int endPos = line.lastIndexOf('}'); // '}' ends at
Review Comment:
We're only doing this when the entire message is JSON, thus the line ends with '}'. Right? It would be a very fast check to check `endsWith` up front.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] NazerkeBS commented on a diff in pull request #882: SOLR-14699: Update Solr request logs to JSON format
Posted by GitBox <gi...@apache.org>.
NazerkeBS commented on code in PR #882:
URL: https://github.com/apache/solr/pull/882#discussion_r892086497
##########
solr/core/src/java/org/apache/solr/util/SolrLogPostTool.java:
##########
@@ -278,48 +279,72 @@ private void parseError(SolrInputDocument lineRecord, String line, String trace)
if (this.cause != null) {
lineRecord.setField("root_cause_t", cause.replace("Caused by:", "").trim());
}
-
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
}
- private void parseQueryRecord(SolrInputDocument lineRecord, String line) {
- lineRecord.setField("qtime_i", parseQTime(line));
- lineRecord.setField("status_s", parseStatus(line));
-
- String path = parsePath(line);
- lineRecord.setField("path_s", path);
-
- if (line.contains("hits=")) {
- lineRecord.setField("hits_l", parseHits(line));
+ private Map<String, Object> extractJSONFormattedMessage(String line) {
+ if (line.contains("o.a.s.c.S.Request")
+ || (line.contains("o.a.s.s.HttpSolrCall") && line.contains("\"prefix\""))) {
+ int startPos = 0; // '{' starts at
+ while (startPos < line.length() && line.charAt(startPos) != '{') {
+ startPos++;
+ }
+ int endPos = startPos + 1; // '}' ends at
+ while (endPos < line.length() && line.charAt(endPos) != '}') {
+ endPos++;
+ }
+ String json = line.substring(startPos, endPos + 1);
+ @SuppressWarnings("unchecked")
+ Map<String, Object> fromJSON = (Map<String, Object>) Utils.fromJSONString(json);
+ return fromJSON;
}
+ return null;
+ }
- String params = parseParams(line);
- lineRecord.setField("params_t", params);
- addParams(lineRecord, params);
+ private void parseRecord(SolrInputDocument lineRecord, String line) {
+ Map<String, Object> keyValuePairs = extractJSONFormattedMessage(line);
+ if (keyValuePairs != null) {
+ // query request record
+ if (keyValuePairs.containsKey("QTime")) {
+ lineRecord.setField("qtime_i", keyValuePairs.get("QTime"));
+ lineRecord.setField("status_s", keyValuePairs.get("status"));
+ lineRecord.setField("path_s", keyValuePairs.get("path"));
+ if (keyValuePairs.containsKey("hits")) {
+ lineRecord.setField("hits_l", keyValuePairs.get("hits"));
+ }
+ lineRecord.setField("params_t", keyValuePairs.get("params"));
+ addParams(lineRecord, keyValuePairs.get("params").toString());
- lineRecord.setField("collection_s", parseCollection(line));
- lineRecord.setField("core_s", parseCore(line));
- lineRecord.setField("node_s", parseNode(line));
- lineRecord.setField("shard_s", parseShard(line));
- lineRecord.setField("replica_s", parseReplica(line));
+ lineRecord.setField("node_s", keyValuePairs.get("node_name"));
- if (path != null && path.contains("/admin")) {
- lineRecord.setField("type_s", "admin");
- } else if (path != null && params.contains("/replication")) {
- lineRecord.setField("type_s", "replication");
- } else if (path != null && path.contains("/get")) {
- lineRecord.setField("type_s", "get");
- } else {
- lineRecord.setField("type_s", "query");
+ if (keyValuePairs.containsKey("prefix") && keyValuePairs.get("prefix").equals("admin")) {
+ lineRecord.setField("type_s", "admin");
+ } else if (keyValuePairs.get("params").toString().contains("/replication")) {
+ lineRecord.setField("type_s", "replication");
+ } else if (keyValuePairs.get("path").equals("/get")) {
+ lineRecord.setField("type_s", "get");
+ } else {
+ lineRecord.setField("type_s", "query");
+ }
+ }
+
+ // update request record
+ if (keyValuePairs.get("path").equals("/update")) {
+ if (keyValuePairs.containsKey("deleteByQuery")) {
+ lineRecord.setField("type_s", "deleteByQuery");
+ } else if (keyValuePairs.containsKey("delete")) {
+ lineRecord.setField("type_s", "delete");
+ } else if (keyValuePairs.containsKey("commit") && (Boolean) keyValuePairs.get("commit")) {
+ lineRecord.setField("type_s", "commit");
+ } else {
Review Comment:
I kept the logic because `type_s` field might have different values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org