You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "prateekjainaa@gmail.com" <pr...@gmail.com> on 2023/03/15 12:26:30 UTC
artemis slave errors | jgroups | k8s
Hi All,
How are you? I am a bit confused about the errors reported by artemis slave
instance (running in clustered mode using jgroups). I might be missing
something here so please bear with me.
Deployment details: I am running 2 pods of artemis in master/slave
configuration in k8s. It supports NFS v4. I am using jgroups configuration
(same file which is supplied in examples of artemis) and I can see that
correct pod names and IPs are updated in share directory
-------- output from jgroups file ----------------
root@jms-01-deployment-6d7dcb8ccd-2t9g2:/opt# cat
/usr/share/artemis-jgroups/active_broadcast_channel/ab62408d-6a7a-4894-9923-2649d8bf3b28.jms-01-deployment-6d7dcb8ccd-2t9g2-29095.list
jms-01-deployment-6d7dcb8ccd-2t9g2-29095
ab62408d-6a7a-4894-9923-2649d8bf3b28 192.168.48.154:7800 T
jms-01-deployment-6d7dcb8ccd-j7pxl-34353
5b3d099b-8e5f-4dd0-ab43-604ad1ae87c3 192.168.199.2:7800 F
--------------------------------------------------------
POD "jms-01-deployment-6d7dcb8ccd-2t9g2" is the master and another is
slave. Correct POD ips are listed there. But in log/artemis.log file of
slave pod, I see these logs
-------------------------------
2023-03-15 11:29:51,519 ERROR [org.apache.activemq.artemis.core.client]
AMQ214016: Failed to create netty connection
java.net.UnknownHostException: jms-01-deployment-6d7dcb8ccd-2t9g2
at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) ~[?:?]
at java.net.InetAddress.getAllByName0(InetAddress.java:1524) ~[?:?]
at java.net.InetAddress.getAllByName(InetAddress.java:1382) ~[?:?]
-----------------------------------
jgroups is correctly creating and updating files. IPs of master/slave pods
are visible from each other. I have tested them using command
"./artemis check node".
Right now I have created deployment objects. Is it necessary to create
stateful sets?
Am I missing something here? Let me know if you need more information on
this.
Regards,
Prateek Jain
--------------------------------------------------------------
EXPECTATION : Causes all troubles......
--------------------------------------------------------------
Re: artemis slave errors | jgroups | k8s
Posted by Justin Bertram <jb...@apache.org>.
Thanks for following up!
Justin
On Wed, Mar 15, 2023 at 12:56 PM prateekjainaa@gmail.com <
prateekjainaa@gmail.com> wrote:
> Hi All,
>
> I am able to fix this issue. Posting my solution here, just in case anyone
> else faces a similar issue. I converted deployment to a statefulset (to get
> predictable names). Then used headless services for each pod. This enabled
> both master and slave pods to lookup each other just by pod name. In case
> anyone is looking for more details, feel free to connect.
>
> Regards,
> Prateek Jain
>
> --------------------------------------------------------------
> EXPECTATION : Causes all troubles......
> --------------------------------------------------------------
>
>
> On Wed, Mar 15, 2023 at 3:54 PM prateekjainaa@gmail.com <
> prateekjainaa@gmail.com> wrote:
>
> >
> > Hi Justin,
> >
> > Thanks for replying, I think artemis is failing to lookup POD by its
> name
> > because we cannot just get POD IP just by its name. Given below is the
> > error which is getting repeated in logs:
> >
> > 2023-03-15 10:04:06,611 INFO [org.apache.activemq.artemis] AMQ241002:
> > Artemis Jolokia REST API available at
> > http://jms-01-deployment-6d7dcb8ccd-j7pxl:8161/console/jolokia
> > 2023-03-15 10:04:06,611 INFO [org.apache.activemq.artemis] AMQ241004:
> > Artemis Console available at
> > http://jms-01-deployment-6d7dcb8ccd-j7pxl:8161/console
> > 2023-03-15 10:04:10,868 ERROR [org.apache.activemq.artemis.core.client]
> > AMQ214016: Failed to create netty connection
> > java.net.UnknownHostException: jms-01-deployment-6d7dcb8ccd-2t9g2
> > at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
> > ~[?:?]
> > at java.net.InetAddress.getAllByName0(InetAddress.java:1524)
> ~[?:?]
> > at java.net.InetAddress.getAllByName(InetAddress.java:1382)
> ~[?:?]
> > at java.net.InetAddress.getAllByName(InetAddress.java:1306)
> ~[?:?]
> > at java.net.InetAddress.getByName(InetAddress.java:1256) ~[?:?]
> > at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:156)
> > ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> > at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:153)
> > ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> > at java.security.AccessController.doPrivileged(Native Method)
> > ~[?:?]
> > at
> > io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:153)
> > ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:41)
> > ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
> > at
> > io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:61)
> > ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
> > at
> > io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:53)
> > ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55)
> > ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:31)
> > ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:106)
> > ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
> > at
> > io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:206)
> > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> > at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:46)
> > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> > at
> > io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:180)
> > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> > at
> > io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:166)
> > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
> > ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
> > ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
> > ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
> > ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
> > ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105)
> > ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
> > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:990)
> > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:516)
> > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429)
> > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486)
> > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
> > ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
> > ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
> > ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> > at
> > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:406)
> > ~[netty-transport-classes-epoll-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> > ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> > at
> > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> > ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> > at
> >
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > ~[artemis-commons-2.28.0.jar:?]
> > 2023-03-15 10:04:10,868 ERROR [org.apache.activemq.artemis.core.client]
> > AMQ214016: Failed to create netty connection
> > java.net.UnknownHostException: jms-01-deployment-6d7dcb8ccd-2t9g2
> > at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
> > ~[?:?]
> > at java.net.InetAddress.getAllByName0(InetAddress.java:1524)
> ~[?:?]
> > at java.net.InetAddress.getAllByName(InetAddress.java:1382)
> ~[?:?]
> > at java.net.InetAddress.getAllByName(InetAddress.java:1306)
> ~[?:?]
> > at java.net.InetAddress.getByName(InetAddress.java:1256) ~[?:?]
> >
> > IMO, I need to find some way in which the pod name resolves to pod ip. If
> > you have any recommendations, let me know.
> >
> > Regards,
> > Prateek Jain
> >
> >
> > --------------------------------------------------------------
> > EXPECTATION : Causes all troubles......
> > --------------------------------------------------------------
> >
> >
> > On Wed, Mar 15, 2023 at 2:23 PM Justin Bertram <jb...@apache.org>
> > wrote:
> >
> >> ActiveMQ Artemis offers JGroups integration for cloud use-cases such as
> >> yours, but, generally speaking, we don't really provide support for
> >> JGroups
> >> in the ActiveMQ community since we don't maintain the JGroups code-base
> >> and
> >> don't know its inner workings.
> >>
> >> That said, I would suggest that the JGroups configuration using
> FILE_PING
> >> shipped in the examples may not be the best for your use-case. We chose
> >> the
> >> FILE_PING configuration because it was the simplest for the use-cases in
> >> the examples (i.e. where all the nodes are running on the same machine
> on
> >> different ports with access to the same disk). However, something like
> >> KUBE_PING [1] would probably be a better fit for your use-case since you
> >> are, in fact, using Kubernetes.
> >>
> >> In any event, it's hard to comment on what may be happening without more
> >> information. You've only provided a snippet of the stack-trace for the
> >> UnknownHostException so it's not clear what class is actually attempting
> >> that operation.
> >>
> >>
> >> Justin
> >>
> >> [1] https://github.com/jgroups-extras/jgroups-kubernetes
> >>
> >> On Wed, Mar 15, 2023 at 7:27 AM prateekjainaa@gmail.com <
> >> prateekjainaa@gmail.com> wrote:
> >>
> >> > Hi All,
> >> >
> >> > How are you? I am a bit confused about the errors reported by artemis
> >> slave
> >> > instance (running in clustered mode using jgroups). I might be missing
> >> > something here so please bear with me.
> >> >
> >> > Deployment details: I am running 2 pods of artemis in master/slave
> >> > configuration in k8s. It supports NFS v4. I am using jgroups
> >> configuration
> >> > (same file which is supplied in examples of artemis) and I can see
> that
> >> > correct pod names and IPs are updated in share directory
> >> >
> >> > -------- output from jgroups file ----------------
> >> > root@jms-01-deployment-6d7dcb8ccd-2t9g2:/opt# cat
> >> >
> >> >
> >>
> /usr/share/artemis-jgroups/active_broadcast_channel/ab62408d-6a7a-4894-9923-2649d8bf3b28.jms-01-deployment-6d7dcb8ccd-2t9g2-29095.list
> >> >
> >> > jms-01-deployment-6d7dcb8ccd-2t9g2-29095
> >> > ab62408d-6a7a-4894-9923-2649d8bf3b28 192.168.48.154:7800 T
> >> > jms-01-deployment-6d7dcb8ccd-j7pxl-34353
> >> > 5b3d099b-8e5f-4dd0-ab43-604ad1ae87c3 192.168.199.2:7800 F
> >> > --------------------------------------------------------
> >> >
> >> > POD "jms-01-deployment-6d7dcb8ccd-2t9g2" is the master and another is
> >> > slave. Correct POD ips are listed there. But in log/artemis.log file
> of
> >> > slave pod, I see these logs
> >> >
> >> > -------------------------------
> >> > 2023-03-15 11:29:51,519 ERROR
> [org.apache.activemq.artemis.core.client]
> >> > AMQ214016: Failed to create netty connection
> >> > java.net.UnknownHostException: jms-01-deployment-6d7dcb8ccd-2t9g2
> >> > at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
> ~[?:?]
> >> > at java.net.InetAddress.getAllByName0(InetAddress.java:1524) ~[?:?]
> >> > at java.net.InetAddress.getAllByName(InetAddress.java:1382) ~[?:?]
> >> > -----------------------------------
> >> >
> >> > jgroups is correctly creating and updating files. IPs of master/slave
> >> pods
> >> > are visible from each other. I have tested them using command
> >> >
> >> > "./artemis check node".
> >> > Right now I have created deployment objects. Is it necessary to create
> >> > stateful sets?
> >> >
> >> > Am I missing something here? Let me know if you need more information
> on
> >> > this.
> >> >
> >> >
> >> > Regards,
> >> > Prateek Jain
> >> >
> >> >
> >> > --------------------------------------------------------------
> >> > EXPECTATION : Causes all troubles......
> >> > --------------------------------------------------------------
> >> >
> >>
> >
>
Re: artemis slave errors | jgroups | k8s
Posted by "prateekjainaa@gmail.com" <pr...@gmail.com>.
Hi All,
I am able to fix this issue. Posting my solution here, just in case anyone
else faces a similar issue. I converted deployment to a statefulset (to get
predictable names). Then used headless services for each pod. This enabled
both master and slave pods to lookup each other just by pod name. In case
anyone is looking for more details, feel free to connect.
Regards,
Prateek Jain
--------------------------------------------------------------
EXPECTATION : Causes all troubles......
--------------------------------------------------------------
On Wed, Mar 15, 2023 at 3:54 PM prateekjainaa@gmail.com <
prateekjainaa@gmail.com> wrote:
>
> Hi Justin,
>
> Thanks for replying, I think artemis is failing to lookup POD by its name
> because we cannot just get POD IP just by its name. Given below is the
> error which is getting repeated in logs:
>
> 2023-03-15 10:04:06,611 INFO [org.apache.activemq.artemis] AMQ241002:
> Artemis Jolokia REST API available at
> http://jms-01-deployment-6d7dcb8ccd-j7pxl:8161/console/jolokia
> 2023-03-15 10:04:06,611 INFO [org.apache.activemq.artemis] AMQ241004:
> Artemis Console available at
> http://jms-01-deployment-6d7dcb8ccd-j7pxl:8161/console
> 2023-03-15 10:04:10,868 ERROR [org.apache.activemq.artemis.core.client]
> AMQ214016: Failed to create netty connection
> java.net.UnknownHostException: jms-01-deployment-6d7dcb8ccd-2t9g2
> at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
> ~[?:?]
> at java.net.InetAddress.getAllByName0(InetAddress.java:1524) ~[?:?]
> at java.net.InetAddress.getAllByName(InetAddress.java:1382) ~[?:?]
> at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]
> at java.net.InetAddress.getByName(InetAddress.java:1256) ~[?:?]
> at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:156)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:153)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:?]
> at
> io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:153)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:41)
> ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:61)
> ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:53)
> ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55)
> ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:31)
> ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:106)
> ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:206)
> ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:46)
> ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:180)
> ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:166)
> ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
> ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:990)
> ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:516)
> ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429)
> ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486)
> ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:406)
> ~[netty-transport-classes-epoll-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> ~[artemis-commons-2.28.0.jar:?]
> 2023-03-15 10:04:10,868 ERROR [org.apache.activemq.artemis.core.client]
> AMQ214016: Failed to create netty connection
> java.net.UnknownHostException: jms-01-deployment-6d7dcb8ccd-2t9g2
> at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
> ~[?:?]
> at java.net.InetAddress.getAllByName0(InetAddress.java:1524) ~[?:?]
> at java.net.InetAddress.getAllByName(InetAddress.java:1382) ~[?:?]
> at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]
> at java.net.InetAddress.getByName(InetAddress.java:1256) ~[?:?]
>
> IMO, I need to find some way in which the pod name resolves to pod ip. If
> you have any recommendations, let me know.
>
> Regards,
> Prateek Jain
>
>
> --------------------------------------------------------------
> EXPECTATION : Causes all troubles......
> --------------------------------------------------------------
>
>
> On Wed, Mar 15, 2023 at 2:23 PM Justin Bertram <jb...@apache.org>
> wrote:
>
>> ActiveMQ Artemis offers JGroups integration for cloud use-cases such as
>> yours, but, generally speaking, we don't really provide support for
>> JGroups
>> in the ActiveMQ community since we don't maintain the JGroups code-base
>> and
>> don't know its inner workings.
>>
>> That said, I would suggest that the JGroups configuration using FILE_PING
>> shipped in the examples may not be the best for your use-case. We chose
>> the
>> FILE_PING configuration because it was the simplest for the use-cases in
>> the examples (i.e. where all the nodes are running on the same machine on
>> different ports with access to the same disk). However, something like
>> KUBE_PING [1] would probably be a better fit for your use-case since you
>> are, in fact, using Kubernetes.
>>
>> In any event, it's hard to comment on what may be happening without more
>> information. You've only provided a snippet of the stack-trace for the
>> UnknownHostException so it's not clear what class is actually attempting
>> that operation.
>>
>>
>> Justin
>>
>> [1] https://github.com/jgroups-extras/jgroups-kubernetes
>>
>> On Wed, Mar 15, 2023 at 7:27 AM prateekjainaa@gmail.com <
>> prateekjainaa@gmail.com> wrote:
>>
>> > Hi All,
>> >
>> > How are you? I am a bit confused about the errors reported by artemis
>> slave
>> > instance (running in clustered mode using jgroups). I might be missing
>> > something here so please bear with me.
>> >
>> > Deployment details: I am running 2 pods of artemis in master/slave
>> > configuration in k8s. It supports NFS v4. I am using jgroups
>> configuration
>> > (same file which is supplied in examples of artemis) and I can see that
>> > correct pod names and IPs are updated in share directory
>> >
>> > -------- output from jgroups file ----------------
>> > root@jms-01-deployment-6d7dcb8ccd-2t9g2:/opt# cat
>> >
>> >
>> /usr/share/artemis-jgroups/active_broadcast_channel/ab62408d-6a7a-4894-9923-2649d8bf3b28.jms-01-deployment-6d7dcb8ccd-2t9g2-29095.list
>> >
>> > jms-01-deployment-6d7dcb8ccd-2t9g2-29095
>> > ab62408d-6a7a-4894-9923-2649d8bf3b28 192.168.48.154:7800 T
>> > jms-01-deployment-6d7dcb8ccd-j7pxl-34353
>> > 5b3d099b-8e5f-4dd0-ab43-604ad1ae87c3 192.168.199.2:7800 F
>> > --------------------------------------------------------
>> >
>> > POD "jms-01-deployment-6d7dcb8ccd-2t9g2" is the master and another is
>> > slave. Correct POD ips are listed there. But in log/artemis.log file of
>> > slave pod, I see these logs
>> >
>> > -------------------------------
>> > 2023-03-15 11:29:51,519 ERROR [org.apache.activemq.artemis.core.client]
>> > AMQ214016: Failed to create netty connection
>> > java.net.UnknownHostException: jms-01-deployment-6d7dcb8ccd-2t9g2
>> > at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) ~[?:?]
>> > at java.net.InetAddress.getAllByName0(InetAddress.java:1524) ~[?:?]
>> > at java.net.InetAddress.getAllByName(InetAddress.java:1382) ~[?:?]
>> > -----------------------------------
>> >
>> > jgroups is correctly creating and updating files. IPs of master/slave
>> pods
>> > are visible from each other. I have tested them using command
>> >
>> > "./artemis check node".
>> > Right now I have created deployment objects. Is it necessary to create
>> > stateful sets?
>> >
>> > Am I missing something here? Let me know if you need more information on
>> > this.
>> >
>> >
>> > Regards,
>> > Prateek Jain
>> >
>> >
>> > --------------------------------------------------------------
>> > EXPECTATION : Causes all troubles......
>> > --------------------------------------------------------------
>> >
>>
>
Re: artemis slave errors | jgroups | k8s
Posted by "prateekjainaa@gmail.com" <pr...@gmail.com>.
Hi Justin,
Thanks for replying, I think artemis is failing to lookup POD by its name
because we cannot just get POD IP just by its name. Given below is the
error which is getting repeated in logs:
2023-03-15 10:04:06,611 INFO [org.apache.activemq.artemis] AMQ241002:
Artemis Jolokia REST API available at
http://jms-01-deployment-6d7dcb8ccd-j7pxl:8161/console/jolokia
2023-03-15 10:04:06,611 INFO [org.apache.activemq.artemis] AMQ241004:
Artemis Console available at
http://jms-01-deployment-6d7dcb8ccd-j7pxl:8161/console
2023-03-15 10:04:10,868 ERROR [org.apache.activemq.artemis.core.client]
AMQ214016: Failed to create netty connection
java.net.UnknownHostException: jms-01-deployment-6d7dcb8ccd-2t9g2
at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
~[?:?]
at java.net.InetAddress.getAllByName0(InetAddress.java:1524) ~[?:?]
at java.net.InetAddress.getAllByName(InetAddress.java:1382) ~[?:?]
at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]
at java.net.InetAddress.getByName(InetAddress.java:1256) ~[?:?]
at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:156)
~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:153)
~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at
io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:153)
~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:41)
~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:61)
~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:53)
~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55)
~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:31)
~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:106)
~[netty-resolver-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:206)
~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:46)
~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:180)
~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:166)
~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105)
~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:990)
~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:516)
~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429)
~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486)
~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:406)
~[netty-transport-classes-epoll-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
~[artemis-commons-2.28.0.jar:?]
2023-03-15 10:04:10,868 ERROR [org.apache.activemq.artemis.core.client]
AMQ214016: Failed to create netty connection
java.net.UnknownHostException: jms-01-deployment-6d7dcb8ccd-2t9g2
at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
~[?:?]
at java.net.InetAddress.getAllByName0(InetAddress.java:1524) ~[?:?]
at java.net.InetAddress.getAllByName(InetAddress.java:1382) ~[?:?]
at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]
at java.net.InetAddress.getByName(InetAddress.java:1256) ~[?:?]
IMO, I need to find some way in which the pod name resolves to pod ip. If
you have any recommendations, let me know.
Regards,
Prateek Jain
--------------------------------------------------------------
EXPECTATION : Causes all troubles......
--------------------------------------------------------------
On Wed, Mar 15, 2023 at 2:23 PM Justin Bertram <jb...@apache.org> wrote:
> ActiveMQ Artemis offers JGroups integration for cloud use-cases such as
> yours, but, generally speaking, we don't really provide support for JGroups
> in the ActiveMQ community since we don't maintain the JGroups code-base and
> don't know its inner workings.
>
> That said, I would suggest that the JGroups configuration using FILE_PING
> shipped in the examples may not be the best for your use-case. We chose the
> FILE_PING configuration because it was the simplest for the use-cases in
> the examples (i.e. where all the nodes are running on the same machine on
> different ports with access to the same disk). However, something like
> KUBE_PING [1] would probably be a better fit for your use-case since you
> are, in fact, using Kubernetes.
>
> In any event, it's hard to comment on what may be happening without more
> information. You've only provided a snippet of the stack-trace for the
> UnknownHostException so it's not clear what class is actually attempting
> that operation.
>
>
> Justin
>
> [1] https://github.com/jgroups-extras/jgroups-kubernetes
>
> On Wed, Mar 15, 2023 at 7:27 AM prateekjainaa@gmail.com <
> prateekjainaa@gmail.com> wrote:
>
> > Hi All,
> >
> > How are you? I am a bit confused about the errors reported by artemis
> slave
> > instance (running in clustered mode using jgroups). I might be missing
> > something here so please bear with me.
> >
> > Deployment details: I am running 2 pods of artemis in master/slave
> > configuration in k8s. It supports NFS v4. I am using jgroups
> configuration
> > (same file which is supplied in examples of artemis) and I can see that
> > correct pod names and IPs are updated in share directory
> >
> > -------- output from jgroups file ----------------
> > root@jms-01-deployment-6d7dcb8ccd-2t9g2:/opt# cat
> >
> >
> /usr/share/artemis-jgroups/active_broadcast_channel/ab62408d-6a7a-4894-9923-2649d8bf3b28.jms-01-deployment-6d7dcb8ccd-2t9g2-29095.list
> >
> > jms-01-deployment-6d7dcb8ccd-2t9g2-29095
> > ab62408d-6a7a-4894-9923-2649d8bf3b28 192.168.48.154:7800 T
> > jms-01-deployment-6d7dcb8ccd-j7pxl-34353
> > 5b3d099b-8e5f-4dd0-ab43-604ad1ae87c3 192.168.199.2:7800 F
> > --------------------------------------------------------
> >
> > POD "jms-01-deployment-6d7dcb8ccd-2t9g2" is the master and another is
> > slave. Correct POD ips are listed there. But in log/artemis.log file of
> > slave pod, I see these logs
> >
> > -------------------------------
> > 2023-03-15 11:29:51,519 ERROR [org.apache.activemq.artemis.core.client]
> > AMQ214016: Failed to create netty connection
> > java.net.UnknownHostException: jms-01-deployment-6d7dcb8ccd-2t9g2
> > at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) ~[?:?]
> > at java.net.InetAddress.getAllByName0(InetAddress.java:1524) ~[?:?]
> > at java.net.InetAddress.getAllByName(InetAddress.java:1382) ~[?:?]
> > -----------------------------------
> >
> > jgroups is correctly creating and updating files. IPs of master/slave
> pods
> > are visible from each other. I have tested them using command
> >
> > "./artemis check node".
> > Right now I have created deployment objects. Is it necessary to create
> > stateful sets?
> >
> > Am I missing something here? Let me know if you need more information on
> > this.
> >
> >
> > Regards,
> > Prateek Jain
> >
> >
> > --------------------------------------------------------------
> > EXPECTATION : Causes all troubles......
> > --------------------------------------------------------------
> >
>
Re: artemis slave errors | jgroups | k8s
Posted by Justin Bertram <jb...@apache.org>.
ActiveMQ Artemis offers JGroups integration for cloud use-cases such as
yours, but, generally speaking, we don't really provide support for JGroups
in the ActiveMQ community since we don't maintain the JGroups code-base and
don't know its inner workings.
That said, I would suggest that the JGroups configuration using FILE_PING
shipped in the examples may not be the best for your use-case. We chose the
FILE_PING configuration because it was the simplest for the use-cases in
the examples (i.e. where all the nodes are running on the same machine on
different ports with access to the same disk). However, something like
KUBE_PING [1] would probably be a better fit for your use-case since you
are, in fact, using Kubernetes.
In any event, it's hard to comment on what may be happening without more
information. You've only provided a snippet of the stack-trace for the
UnknownHostException so it's not clear what class is actually attempting
that operation.
Justin
[1] https://github.com/jgroups-extras/jgroups-kubernetes
On Wed, Mar 15, 2023 at 7:27 AM prateekjainaa@gmail.com <
prateekjainaa@gmail.com> wrote:
> Hi All,
>
> How are you? I am a bit confused about the errors reported by artemis slave
> instance (running in clustered mode using jgroups). I might be missing
> something here so please bear with me.
>
> Deployment details: I am running 2 pods of artemis in master/slave
> configuration in k8s. It supports NFS v4. I am using jgroups configuration
> (same file which is supplied in examples of artemis) and I can see that
> correct pod names and IPs are updated in share directory
>
> -------- output from jgroups file ----------------
> root@jms-01-deployment-6d7dcb8ccd-2t9g2:/opt# cat
>
> /usr/share/artemis-jgroups/active_broadcast_channel/ab62408d-6a7a-4894-9923-2649d8bf3b28.jms-01-deployment-6d7dcb8ccd-2t9g2-29095.list
>
> jms-01-deployment-6d7dcb8ccd-2t9g2-29095
> ab62408d-6a7a-4894-9923-2649d8bf3b28 192.168.48.154:7800 T
> jms-01-deployment-6d7dcb8ccd-j7pxl-34353
> 5b3d099b-8e5f-4dd0-ab43-604ad1ae87c3 192.168.199.2:7800 F
> --------------------------------------------------------
>
> POD "jms-01-deployment-6d7dcb8ccd-2t9g2" is the master and another is
> slave. Correct POD ips are listed there. But in log/artemis.log file of
> slave pod, I see these logs
>
> -------------------------------
> 2023-03-15 11:29:51,519 ERROR [org.apache.activemq.artemis.core.client]
> AMQ214016: Failed to create netty connection
> java.net.UnknownHostException: jms-01-deployment-6d7dcb8ccd-2t9g2
> at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) ~[?:?]
> at java.net.InetAddress.getAllByName0(InetAddress.java:1524) ~[?:?]
> at java.net.InetAddress.getAllByName(InetAddress.java:1382) ~[?:?]
> -----------------------------------
>
> jgroups is correctly creating and updating files. IPs of master/slave pods
> are visible from each other. I have tested them using command
>
> "./artemis check node".
> Right now I have created deployment objects. Is it necessary to create
> stateful sets?
>
> Am I missing something here? Let me know if you need more information on
> this.
>
>
> Regards,
> Prateek Jain
>
>
> --------------------------------------------------------------
> EXPECTATION : Causes all troubles......
> --------------------------------------------------------------
>