You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@solr.apache.org by Joel Bernstein <jo...@gmail.com> on 2023/03/20 14:09:35 UTC

Logging KNN vector queries

I've run into an issue with KNN vector queries and the Solr logs. The issue
is for large vectors you can end up with 20K+ of just vector data in the
logs. This is because each floating point value in the vector can take 20+
characters to express as a string. In our testing the log line is getting
truncated with "..." and the log is unusable. This truncation may be
something configurable in the logger as I'm not seeing code in Solr that
would truncate the log. But if this truncating behavior can't be rectified
I think we need to fix this or KNN vector queries are incompatible with our
logs.

Even if we can configure longer log records I think we need the ability to
specify parameters that we don't want to end up in the log.

Any thoughts? I'll create a jira issue shortly as well.




Joel Bernstein
http://joelsolr.blogspot.com/

Re: Logging KNN vector queries

Posted by Joel Bernstein <jo...@gmail.com>.

It does appear that the logs are being truncated by the logger:

https://logging.apache.org/log4j/log4j-2.8/log4j-core/apidocs/org/apache/logging/log4j/core/pattern/MaxLengthConverter.html
.

But, I've created a ticket to selectively suppress certain parameters in
the logs:
https://issues.apache.org/jira/browse/SOLR-16710



On Mon, Mar 20, 2023 at 10:09 AM Joel Bernstein <jo...@gmail.com> wrote:

> I've run into an issue with KNN vector queries and the Solr logs. The
> issue is for large vectors you can end up with 20K+ of just vector data in
> the logs. This is because each floating point value in the vector can take
> 20+ characters to express as a string. In our testing the log line is
> getting truncated with "..." and the log is unusable. This truncation may
> be something configurable in the logger as I'm not seeing code in Solr that
> would truncate the log. But if this truncating behavior can't be rectified
> I think we need to fix this or KNN vector queries are incompatible with our
> logs.
>
> Even if we can configure longer log records I think we need the ability to
> specify parameters that we don't want to end up in the log.
>
> Any thoughts? I'll create a jira issue shortly as well.
>
>
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>