[GitHub] [accumulo] ctubbsii commented on issue #3334: Accumulo Monitor cannot handle the truth

["ctubbsii (via GitHub)" <gi...@apache.org>]

ctubbsii commented on issue #3334:
URL: https://github.com/apache/accumulo/issues/3334#issuecomment-1520924263

   I believe glassfish is in the stack for Jersey. As the reference implementation for Jersey, the glassfish implementation seemed to be the best supported and easiest to use. That was added to support the REST/AJAX based site that provides a more responsive user experience when running with many servers (no longer required to do full page reloads, and better retaining client-side state, like table sorting). It's definitely a far cry better than the manual servlets we used to maintain before. I don't think we'll want to go backwards and get rid of that, but there might be some trade-offs we're making here where we can afford to improve, or maybe there's an implementation that has a smaller footprint.
   
   If I understand you correctly, this pertains to the optional feature to forward warnings/error messages to the monitor? For some of that, our hands were tied in the implementation, because we had to switch to using an implementation that sent messages using the REST endpoint, in order to avoid severe CVEs in log4j's socket receiver.
   
   I'm not sure exactly how you have logging configured, but by default, our config ships using log4j2's async loggers. There may be some configuration to tweak their send behavior to throttle them, so they don't overwhelm the monitor: https://logging.apache.org/log4j/2.x/manual/async.html
   
   For example, you have the option to configure filters to log4j2 to deduplicate repeated messages or to control bursts (you can also write a custom filter for your specific needs). See https://logging.apache.org/log4j/2.x/manual/filters.html
   
   You can also choose to completely turn off the monitor appender, and rely on some other more robust log collection/analysis that is suitable for a large system. That's probably what I would recommend anyway, because the monitor will only keep a small window of recent warnings/errors, and on a large system, important things can easily scroll out and be missed.
   
   Given that a lot of this is based on the environment, and tweakable by users, I don't think I would characterize this as a bug (it has the flavor of being susceptible to a denial-of-service attack, rather than an explicit bug). There seems to be many options on the log sending side, but I'm not sure what we can do on the log receiving side inside the monitor to avoid being overwhelmed. If there's something specific we can achieve within the monitor to avoid being overwhelmed, I'd be interested, but I'm not sure what options are available to us on the server-side. We definitely can't go back to the socket appender from log4j 1.2, since that's no longer available to us. We could run a separate Thrift service on the monitor, but that comes with its own problems/trade-offs, and I'm not sure it would ultimately fix the problem, since that too could be overwhelmed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org