You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@logging.apache.org by "lu-xiaoshuang (via GitHub)" <gi...@apache.org> on 2025/02/08 05:25:03 UTC
[I] make SocketAppender support server list (logging-log4j2)
lu-xiaoshuang opened a new issue, #3442:
URL: https://github.com/apache/logging-log4j2/issues/3442
Typically, applications generate massive amounts of log data, which need to be sent to the backend systems for storage, analyzing and so on.
At present, a SocketAppender only supports one address port pair. Seems that remote server is very likely the bottleneck (network traffic) and single point failure (without gateway).
Would it be possible that SocketAppender supports a server list?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make SocketAppender support server list (logging-log4j2)
Posted by "ppkarwasz (via GitHub)" <gi...@apache.org>.
ppkarwasz commented on issue #3442:
URL: https://github.com/apache/logging-log4j2/issues/3442#issuecomment-2652808212
> KafkaAppender is not capable of sending logs generated by org.apache.kafka:kafka-clients:jar, is it?
Right, that usually causes problems with recursive logging.
> By the way, If community decides to make SocketAppender support server list, what about creating multiple sockets? When append method is invoked, log4j2 can choose a socket randomly. I think this simple solution is free of connection pool stuff.
Log4j is highly pluggable, so I would recommend to:
1. Create and publish a new appender (let's say `MultiSocketAppender`) in your personal or company repo under Apache-2.0.
2. Test the plugin in your test environment.
3. Announce it on our `dev@logging.apache.org` mailing list. We will add a section for it in our [Appenders page](https://logging.apache.org/log4j/2.x/manual/appenders.html), so users can easily find it.
4. If the appender is well received by users we will consider adding it to the standard distribution.
Note that we are adopting the same strategy for our own plugins, e.g. my [copernik-eu/log4j-tomcat](https://github.com/copernik-eu/log4j-tomcat/) is waiting for some user feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make SocketAppender support server list [logging-log4j2]
Posted by "vy (via GitHub)" <gi...@apache.org>.
vy commented on issue #3442:
URL: https://github.com/apache/logging-log4j2/issues/3442#issuecomment-2676323382
@lu-xiaoshuang, in #3482, I see you randomly choose hosts to write to. Can't you simply define multiple socket appenders and choose them using [scripted routes](https://logging.apache.org/log4j/2.x/manual/appenders/delegating.html#RoutingAppender-example-script):
```xml
<Configuration xmlns="https://logging.apache.org/xml/ns"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
https://logging.apache.org/xml/ns
https://logging.apache.org/xml/ns/log4j-config-2.xsd">
<Appenders>
<Routing name="ROUTING">
<Script language="groovy">
staticVariables.hosts = [
'h1.example.com',
'h2.example.com',
'h3.example.com'
];
staticVariables.random = new Random();
</Script>
<Routes>
<Script language="groovy">
int hostIndex = staticVariables.random.nextInt(staticVariables.hosts.length);
String host = staticVariables.hosts[hostIndex];
return configuration.properties['host'] = host;
</Script>
<Route>
<Socket name="${host}"
protocol="TCP"
host="${host}"
port="500">
<JsonTemplateLayout/>
</Socket>
</Route>
</Routes>
</Routing>
</Appenders>
<Loggers>
<Root level="INFO">
<AppenderRef ref="ROUTING"/>
</Root>
</Loggers>
</Configuration>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make SocketAppender support server list (logging-log4j2)
Posted by "ppkarwasz (via GitHub)" <gi...@apache.org>.
ppkarwasz commented on issue #3442:
URL: https://github.com/apache/logging-log4j2/issues/3442#issuecomment-2646179772
@lu-xiaoshuang,
> At present, a SocketAppender only supports one address-port. Seems that remote server represented by this address-port is very likely the bottleneck (network traffic) and single point failure endpoint (without gateway).
> It would be great if there are no additional prerequisites.
The `Socket` appender is based on the basic Java socket classes. These classes don't provide support for connection pooling, so we would need additional libraries (like Apache MINA or Netty) or a lot of custom code to handle multiple connections.
Since TCP does not guarantee a reliable delivery without message loss, it is probably better to look at [message queue appenders](https://logging.apache.org/log4j/2.x/manual/appenders/message-queue.html). Some of them do support multiple servers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make SocketAppender support server list (logging-log4j2)
Posted by "lu-xiaoshuang (via GitHub)" <gi...@apache.org>.
lu-xiaoshuang commented on issue #3442:
URL: https://github.com/apache/logging-log4j2/issues/3442#issuecomment-2652418157
> [@lu-xiaoshuang](https://github.com/lu-xiaoshuang),
>
> > At present, a SocketAppender only supports one address-port. Seems that remote server represented by this address-port is very likely the bottleneck (network traffic) and single point failure endpoint (without gateway).
>
> > It would be great if there are no additional prerequisites.
>
> The `Socket` appender is based on the basic Java socket classes. These classes don't provide support for connection pooling, so we would need additional libraries (like Apache MINA or Netty) or a lot of custom code to handle multiple connections.
>
> Since TCP does not guarantee a reliable delivery without message loss, it is probably better to look at [message queue appenders](https://logging.apache.org/log4j/2.x/manual/appenders/message-queue.html). Some of them do support multiple servers.
Thanks, @ppkarwasz
KafkaAppender is not the capable of sending logs generated by org.apache.kafka:kafka-clients:jar, is it?
By the way, If community decides to make SocketAppender support server list, what about creating multiple sockets? When append method is invoked, log4j2 can choose a socket randomly. I think this simple solution is free of connection pool stuff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make SocketAppender support server list (logging-log4j2)
Posted by "vy (via GitHub)" <gi...@apache.org>.
vy commented on issue #3442:
URL: https://github.com/apache/logging-log4j2/issues/3442#issuecomment-2649208156
@lu-xiaoshuang, curious: what is the log sink you use where you encounter this problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make SocketAppender support server list [logging-log4j2]
Posted by "vy (via GitHub)" <gi...@apache.org>.
vy closed issue #3442: make SocketAppender support server list
URL: https://github.com/apache/logging-log4j2/issues/3442
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make SocketAppender support server list (logging-log4j2)
Posted by "swebb2066 (via GitHub)" <gi...@apache.org>.
swebb2066 commented on issue #3442:
URL: https://github.com/apache/logging-log4j2/issues/3442#issuecomment-2644524432
Have you considered sending logging events via the [fluent-bit Forward Upstream plugin](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/upstream-servers)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make SocketAppender support server list (logging-log4j2)
Posted by "lu-xiaoshuang (via GitHub)" <gi...@apache.org>.
lu-xiaoshuang commented on issue #3442:
URL: https://github.com/apache/logging-log4j2/issues/3442#issuecomment-2652407866
> [@lu-xiaoshuang](https://github.com/lu-xiaoshuang), curious: what is the log sink you use where you encounter this problem?
logstash
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make SocketAppender support server list (logging-log4j2)
Posted by "lu-xiaoshuang (via GitHub)" <gi...@apache.org>.
lu-xiaoshuang commented on issue #3442:
URL: https://github.com/apache/logging-log4j2/issues/3442#issuecomment-2644538331
> Have you considered sending logging events via the [fluent-bit Forward Upstream plugin](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/upstream-servers)?
Good idea. Thank you for your thoughts. @swebb2066
It would be great if there are no additional prerequisites.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make SocketAppender support server list (logging-log4j2)
Posted by "lu-xiaoshuang (via GitHub)" <gi...@apache.org>.
lu-xiaoshuang commented on issue #3442:
URL: https://github.com/apache/logging-log4j2/issues/3442#issuecomment-2653507527
> 1. in your personal or company repo under Apache-2.0.
Good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make SocketAppender support server list [logging-log4j2]
Posted by "lu-xiaoshuang (via GitHub)" <gi...@apache.org>.
lu-xiaoshuang commented on issue #3442:
URL: https://github.com/apache/logging-log4j2/issues/3442#issuecomment-2699866460
> [@lu-xiaoshuang](https://github.com/lu-xiaoshuang), in [#3482](https://github.com/apache/logging-log4j2/pull/3482), I see you randomly choose hosts to write to. Can't you simply define multiple socket appenders and choose them using [scripted routes](https://logging.apache.org/log4j/2.x/manual/appenders/delegating.html#RoutingAppender-example-script):
>
> <Configuration xmlns="https://logging.apache.org/xml/ns"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="
> https://logging.apache.org/xml/ns
> https://logging.apache.org/xml/ns/log4j-config-2.xsd">
> <Appenders>
> <Routing name="ROUTING">
> <Script language="groovy">
> staticVariables.hosts = [
> 'h1.example.com',
> 'h2.example.com',
> 'h3.example.com'
> ];
> staticVariables.random = new Random();
> </Script>
> <Routes>
> <Script language="groovy">
> int hostIndex = staticVariables.random.nextInt(staticVariables.hosts.length);
> String host = staticVariables.hosts[hostIndex];
> return configuration.properties['host'] = host;
> </Script>
> <Route>
> <Socket name="${host}"
> protocol="TCP"
> host="${host}"
> port="500">
> <JsonTemplateLayout/>
> </Socket>
> </Route>
> </Routes>
> </Routing>
> </Appenders>
> <Loggers>
> <Root level="INFO">
> <AppenderRef ref="ROUTING"/>
> </Root>
> </Loggers>
> </Configuration>
Thanks for your reply, @vy
I'm afraid ```Routing``` is not an elegant way. It seems a little bit ugly instead.
It would be great if the framework could hide the underlying complexity (i.e. transparent to users).
I think offering a new ```SocketAppender``` (assuming ```SocketAppenderV2```) may be a possible solution. Because origin design of ```SocketAppender``` is not easy to support multiple sockets.
When ```SocketAppenderV2``` is ready, users can migrate from old version to new version. At last, ```log4j``` can deprecate ```SocketAppender```.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make SocketAppender support server list [logging-log4j2]
Posted by "vy (via GitHub)" <gi...@apache.org>.
vy commented on issue #3442:
URL: https://github.com/apache/logging-log4j2/issues/3442#issuecomment-2724028069
Since multiple solutions (i.e., using a `Routing` appender or creating a custom one) are provided to the user, closing this one as `Not Planned`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org