You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Ge...@gta-travel.com on 2009/12/11 18:32:27 UTC
Strange characters seen in RR and Cd columns on jkserver-status page after upgrading modjk to .28
Hi
We have upgraded our modjk from .26 to .28
using mod_jk-1.2.28-httpd-2.2.X.so obtained from
http://../tomcat-connectors/jk/binaries/linux/jk-1.2.28/x86_64/
We are running
Server version: Apache/2.2.3
Server built: Nov 12 2008 10:40:14
CentOS release 5.2 (Final)
After upgrading we are seeing random characters in the RR an Cd columns on the jkserver-status page.
eg
o%2B
o+
����
Upgrade were done by stopping httpd.
Copying new modjk to /etc/httpd/modules/
Editing /etc/httpd/conf.d/modjk.conf to reflect new version.
Requests are being served but the strange characters and there origin a bit troubling.
Our workers.properties looks as follows:
( There is actually 12 workers but I have removed most to simplify config)
# Worker list
worker.list=xml-gta,jkstatus
# Worker definitions
# main loadbalancer worker
worker.xml-gta.type=lb
worker.xml-gta.max_reply_timeouts=10
#worker.xml-gta.sticky_session=false
worker.xml-gta.retries=6
worker.xml-gta.method=Busyness
worker.xml-gta.balance_workers=worker1, worker2
# status page worker
worker.jkstatus.type=status
# Worker Reference worker.reference.port=8009
worker.reference.type=ajp13
worker.reference.lbfactor=1
worker.reference.socket_timeout=0
worker.reference.socket_keepalive=true
worker.reference.connect_timeout=5000
worker.reference.prepost_timeout=0
worker.reference.reply_timeout=0
worker.reference.retries=6
worker.reference.recovery_options=27
worker.reference.connection_pool_timeout=600
worker.reference.connection_pool_size=2
worker.reference.ping_mode=A
worker.reference.ping_timeout=5000
worker.reference.retry_interval=60
# Balance workers
worker.worker1.reference=worker.reference
worker.worker1.host=10.10.10.10
worker.worker2.reference=worker.reference
worker.worker2.host=10.10.10.11
Regards
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
RE: Strange characters seen in RR and Cd columns on jkserver-status page after upgrading modjk to .28
Posted by Ge...@gta-travel.com.
-----Original Message-----
From: André Warnier [mailto:aw@ice-sa.com]
Sent: 11 December 2009 17:43
To: Tomcat Users List
Subject: Re: Strange characters seen in RR and Cd columns on jkserver-status page after upgrading modjk to .28
Gerhardus.Geldenhuis@gta-travel.com wrote:
> Hi
>
> We have upgraded our modjk from .26 to .28
> using mod_jk-1.2.28-httpd-2.2.X.so obtained from
> http://../tomcat-connectors/jk/binaries/linux/jk-1.2.28/x86_64/
>
> We are running
> Server version: Apache/2.2.3
> Server built: Nov 12 2008 10:40:14
>
> CentOS release 5.2 (Final)
>
> After upgrading we are seeing random characters in the RR an Cd columns on the jkserver-status page.
> eg
> o%2B
> o+
> ����
>
That kind of thing triggers an immediate suspicion regarding a
mismatched character set/encoding somewhere.
I am not familiar at all with JSP pages, but could it be that during the
update, you also customized some JSP pages, with an editor ?
Are they really random characters (e.g. noise), or would-be real
characters being replaced by the cabalistic signs above ?
It does seem to be random... i have seen a lot of different characters appearing in these fields which change occasionaly as I refresh th jkserver-status page.
I am going to hunt down character encodings on Monday. It baffles me because modjk is the only change that has happend. Maybe the .28 is just better at pointing out mistakes thats been there in the past.
Best Regards
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
Re: Strange characters seen in RR and Cd columns on jkserver-status
page after upgrading modjk to .28
Posted by André Warnier <aw...@ice-sa.com>.
Gerhardus.Geldenhuis@gta-travel.com wrote:
> Hi
>
> We have upgraded our modjk from .26 to .28
> using mod_jk-1.2.28-httpd-2.2.X.so obtained from
> http://../tomcat-connectors/jk/binaries/linux/jk-1.2.28/x86_64/
>
> We are running
> Server version: Apache/2.2.3
> Server built: Nov 12 2008 10:40:14
>
> CentOS release 5.2 (Final)
>
> After upgrading we are seeing random characters in the RR an Cd columns on the jkserver-status page.
> eg
> o%2B
> o+
> ����
>
That kind of thing triggers an immediate suspicion regarding a
mismatched character set/encoding somewhere.
I am not familiar at all with JSP pages, but could it be that during the
update, you also customized some JSP pages, with an editor ?
Are they really random characters (e.g. noise), or would-be real
characters being replaced by the cabalistic signs above ?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Strange characters seen in RR and Cd columns on jkserver-status
page after upgrading modjk to .28
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
All,
On 12/14/2009 12:43 PM, Christopher Schultz wrote:
> Rainer, is this an encoding problem, or an instrumentation problem
> (counter out-of-range, etc.)? Can you please post a reference to the bug
> you mentioned earlier?
Looks like this is
https://issues.apache.org/bugzilla/show_bug.cgi?id=46503
I've added a link to this thread in the comments.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAksmeuoACgkQ9CaO5/Lv0PBFKQCgsHbS6WklF2maGxmlcrlOJHt1
31YAoLLXUYVD8kE6M5rfbkvzf/7HmsrV
=QGz7
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Strange characters seen in RR and Cd columns on jkserver-status
page after upgrading modjk to .28
Posted by Rainer Jung <ra...@kippdata.de>.
On 14.12.2009 18:43, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> All,
>
> On 12/14/2009 12:34 PM, Christopher Schultz wrote:
>> I have a development server with 18 workers defined. I'll enable the
>> status page and check to see what happens.
>
> Hmm... I don't see any RR or Cd stuff... probably because I'm not in a
> clustered environment. Sorry.... I won't be able to offer any empirical
> data to this discussion.
It seems it is not straightforward to reproduce. Clustering shouldn't be
necessary and I think some (or all?) who observed the problem are not
using the domain and redirect attributes. It seems you only need a
loadbalancer plus some not yet understood magic sauce.
> Rainer, what character encoding does mod_jk attempt to use when writing
> the output page?
>
> The XML document for the status page says:
> <?xml version="1.0" encoding="ISO-8859-1"?>
>
> The Content-Type of the HTTP response specifies no charset, to
> ISO-8859-1 ought to be the default. I am not forcing any character set
> for this particular URI.
>
> Rainer, is this an encoding problem, or an instrumentation problem
> (counter out-of-range, etc.)? Can you please post a reference to the bug
> you mentioned earlier?
I expect it's a problem with uninitialized memory. It shouldn't have to
do with encoding, unfortunately, it's really garbage we see. I thought
it has to do with reusing shared memory during graceful restart, but
since Gerhardus also gets it immediately after start, I guess I'm wrong.
I need to analyze, whther the garbage is coming from the shared memory
or is only part of the in memory worker data. I'll get back with
questions or whatever as sonn as I start working on it, which is
definitely before 1.2.29.
And thanks to already finding the issue and adding the link :)
Regards,
Rainer
> Thanks,
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAksmeUEACgkQ9CaO5/Lv0PCtdgCePYu9CYoGFI1DX33AGQ3K7nHH
> X28AnRPo5FoEjKdshYdqAzGDsN7BO/FL
> =P9xJ
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Strange characters seen in RR and Cd columns on jkserver-status
page after upgrading modjk to .28
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
All,
On 12/14/2009 12:34 PM, Christopher Schultz wrote:
> I have a development server with 18 workers defined. I'll enable the
> status page and check to see what happens.
Hmm... I don't see any RR or Cd stuff... probably because I'm not in a
clustered environment. Sorry.... I won't be able to offer any empirical
data to this discussion.
Rainer, what character encoding does mod_jk attempt to use when writing
the output page?
The XML document for the status page says:
<?xml version="1.0" encoding="ISO-8859-1"?>
The Content-Type of the HTTP response specifies no charset, to
ISO-8859-1 ought to be the default. I am not forcing any character set
for this particular URI.
Rainer, is this an encoding problem, or an instrumentation problem
(counter out-of-range, etc.)? Can you please post a reference to the bug
you mentioned earlier?
Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAksmeUEACgkQ9CaO5/Lv0PCtdgCePYu9CYoGFI1DX33AGQ3K7nHH
X28AnRPo5FoEjKdshYdqAzGDsN7BO/FL
=P9xJ
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Strange characters seen in RR and Cd columns on jkserver-status
page after upgrading modjk to .28
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Gerhardus and Rainer,
On 12/14/2009 6:32 AM, Gerhardus.Geldenhuis@gta-travel.com wrote:
>> -----Original Message-----
>> From: Rainer Jung [mailto:rainer.jung@kippdata.de]
>> It's a known problem. There's an open Bugzilla for that issue. It seems
>> you need at least around 8 worker for it to happen and I suspect it has
>> to do with webserver restarts (apachectl graceful or apachectl
>> restart).
>> Unfortunately I don't yet fully understand the reasons, but noone has
>> reported an operational problem resulting our of it, like stickyness or
>> load balancing being broken. Of course we need to fix it ...
>>
>> Can you confirm it happens only after restarting? And yes, it would be
>> nice if your could build from the sources yourself and double check the
>> problem is still there.
[snip]
> We do not see the problem on a similar configured httpd server with
> only 5 workers which seems to corroborate the 8 worker threshold.
I have a development server with 18 workers defined. I'll enable the
status page and check to see what happens.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAksmdzMACgkQ9CaO5/Lv0PD+ewCeOmGEJRI8FfLqSQP7G5rd4+Jy
BSMAoK037Zp8XrFrAWBaQTZVYOLar+MR
=bzRv
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Strange characters seen in RR and Cd columns on jkserver-status page after upgrading modjk to .28
Posted by Ge...@gta-travel.com.
> -----Original Message-----
> From: Rainer Jung [mailto:rainer.jung@kippdata.de]
> Sent: 12 December 2009 15:51
> To: Tomcat Users List
> Cc: Gerhardus Geldenhuis (GTA-LON)
> Subject: Re: Strange characters seen in RR and Cd columns on jkserver-
> status page after upgrading modjk to .28
>
> On 11.12.2009 18:32, Gerhardus.Geldenhuis@gta-travel.com wrote:
> > Hi
> >
> > We have upgraded our modjk from .26 to .28
> > using mod_jk-1.2.28-httpd-2.2.X.so obtained from
> > http://../tomcat-connectors/jk/binaries/linux/jk-1.2.28/x86_64/
> >
> > We are running
> > Server version: Apache/2.2.3
> > Server built: Nov 12 2008 10:40:14
> >
> > CentOS release 5.2 (Final)
> >
> > After upgrading we are seeing random characters in the RR an Cd
> columns on the jkserver-status page.
> > eg
> > o%2B
> > o+
> > ����
> >
> > Upgrade were done by stopping httpd.
> > Copying new modjk to /etc/httpd/modules/
> > Editing /etc/httpd/conf.d/modjk.conf to reflect new version.
> >
> > Requests are being served but the strange characters and there origin
> a bit troubling.
> >
> > Our workers.properties looks as follows:
> > ( There is actually 12 workers but I have removed most to simplify
> config)
> >
> >
> > # Worker list
> > worker.list=xml-gta,jkstatus
> >
> > # Worker definitions
> >
> > # main loadbalancer worker
> > worker.xml-gta.type=lb
> > worker.xml-gta.max_reply_timeouts=10
> > #worker.xml-gta.sticky_session=false
> > worker.xml-gta.retries=6
> > worker.xml-gta.method=Busyness
> >
> > worker.xml-gta.balance_workers=worker1, worker2
> >
> >
> > # status page worker
> > worker.jkstatus.type=status
> >
> > # Worker Reference worker.reference.port=8009
> > worker.reference.type=ajp13
> > worker.reference.lbfactor=1
> > worker.reference.socket_timeout=0
> > worker.reference.socket_keepalive=true
> > worker.reference.connect_timeout=5000
> > worker.reference.prepost_timeout=0
> > worker.reference.reply_timeout=0
> > worker.reference.retries=6
> > worker.reference.recovery_options=27
> > worker.reference.connection_pool_timeout=600
> > worker.reference.connection_pool_size=2
> >
> > worker.reference.ping_mode=A
> > worker.reference.ping_timeout=5000
> > worker.reference.retry_interval=60
> >
> > # Balance workers
> >
> > worker.worker1.reference=worker.reference
> > worker.worker1.host=10.10.10.10
> >
> > worker.worker2.reference=worker.reference
> > worker.worker2.host=10.10.10.11
> >
> > Regards
>
> It's a known problem. There's an open Bugzilla for that issue. It seems
> you need at least around 8 worker for it to happen and I suspect it has
> to do with webserver restarts (apachectl graceful or apachectl
> restart).
> Unfortunately I don't yet fully understand the reasons, but noone has
> reported an operational problem resulting our of it, like stickyness or
> load balancing being broken. Of course we need to fix it ...
>
> Can you confirm it happens only after restarting? And yes, it would be
> nice if your could build from the sources yourself and double check the
> problem is still there.
>
> Regards,
>
> Rainer
>
Thank you for all the replies.
I upgraded the httpd server to the latest CentOS version which is:
Name : httpd
Arch : x86_64
Version: 2.2.3
Release: 31.el5.centos.2
I also did a custom compile of modjk.
The problem persist despite these changes and is not related to only restarts but persists even if the httpd service is completely stopped and then started again.
We do not see the problem on a similar configured httpd server with only 5 workers which seems to corroborate the 8 worker threshold.
If there is any debugging scripts or log files that I can attach to the bugzilla then please let me know and I will happily do so.
Regards
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
Re: Strange characters seen in RR and Cd columns on jkserver-status
page after upgrading modjk to .28
Posted by Rainer Jung <ra...@kippdata.de>.
On 11.12.2009 18:32, Gerhardus.Geldenhuis@gta-travel.com wrote:
> Hi
>
> We have upgraded our modjk from .26 to .28
> using mod_jk-1.2.28-httpd-2.2.X.so obtained from
> http://../tomcat-connectors/jk/binaries/linux/jk-1.2.28/x86_64/
>
> We are running
> Server version: Apache/2.2.3
> Server built: Nov 12 2008 10:40:14
>
> CentOS release 5.2 (Final)
>
> After upgrading we are seeing random characters in the RR an Cd columns on the jkserver-status page.
> eg
> o%2B
> o+
> ����
>
> Upgrade were done by stopping httpd.
> Copying new modjk to /etc/httpd/modules/
> Editing /etc/httpd/conf.d/modjk.conf to reflect new version.
>
> Requests are being served but the strange characters and there origin a bit troubling.
>
> Our workers.properties looks as follows:
> ( There is actually 12 workers but I have removed most to simplify config)
>
>
> # Worker list
> worker.list=xml-gta,jkstatus
>
> # Worker definitions
>
> # main loadbalancer worker
> worker.xml-gta.type=lb
> worker.xml-gta.max_reply_timeouts=10
> #worker.xml-gta.sticky_session=false
> worker.xml-gta.retries=6
> worker.xml-gta.method=Busyness
>
> worker.xml-gta.balance_workers=worker1, worker2
>
>
> # status page worker
> worker.jkstatus.type=status
>
> # Worker Reference worker.reference.port=8009
> worker.reference.type=ajp13
> worker.reference.lbfactor=1
> worker.reference.socket_timeout=0
> worker.reference.socket_keepalive=true
> worker.reference.connect_timeout=5000
> worker.reference.prepost_timeout=0
> worker.reference.reply_timeout=0
> worker.reference.retries=6
> worker.reference.recovery_options=27
> worker.reference.connection_pool_timeout=600
> worker.reference.connection_pool_size=2
>
> worker.reference.ping_mode=A
> worker.reference.ping_timeout=5000
> worker.reference.retry_interval=60
>
> # Balance workers
>
> worker.worker1.reference=worker.reference
> worker.worker1.host=10.10.10.10
>
> worker.worker2.reference=worker.reference
> worker.worker2.host=10.10.10.11
>
> Regards
It's a known problem. There's an open Bugzilla for that issue. It seems
you need at least around 8 worker for it to happen and I suspect it has
to do with webserver restarts (apachectl graceful or apachectl restart).
Unfortunately I don't yet fully understand the reasons, but noone has
reported an operational problem resulting our of it, like stickyness or
load balancing being broken. Of course we need to fix it ...
Can you confirm it happens only after restarting? And yes, it would be
nice if your could build from the sources yourself and double check the
problem is still there.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Strange characters seen in RR and Cd columns on jkserver-status
page after upgrading modjk to .28
Posted by André Warnier <aw...@ice-sa.com>.
Christopher Schultz wrote:
...
>
>> After upgrading we are seeing random characters in the RR an Cd
>> columns on the jkserver-status page.
>> eg
>> o%2B
>> o+
>> ����
>
> Is this only appearing on the mod_jk status page? In think André is
> asking about other pages (like those generated by JSP code). The status
> page is served directly by mod_jk (within httpd) and so there's no
> Tomcat interaction at all.
>
Ooops. Pid, we're even.
I should have thought about that. Duh.
>
> What is the Content-Type header for the jkserver-status page? Are there
> any <META> tags in the page?
>
> It's possible that your httpd is simply advertising a Content-Type
> encoding that is different from what jkserver-status is actually using.
>
But I see we're staying within character set/encoding issues.
A small consolation for me.
Looks like I'm going to have to finally look at that mod_jk status page
after all.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Strange characters seen in RR and Cd columns on jkserver-status
page after upgrading modjk to .28
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Gerhardus,
On 12/11/2009 12:32 PM, Gerhardus.Geldenhuis@gta-travel.com wrote:
> We have upgraded our modjk from .26 to .28
> using mod_jk-1.2.28-httpd-2.2.X.so obtained from
> http://../tomcat-connectors/jk/binaries/linux/jk-1.2.28/x86_64/
Any chance of compiling it yourself? Binary distros are convenient, but
building it yourself (simple: ./configure && make) gives you a 100%
compatible binary.
> We are running
> Server version: Apache/2.2.3
> Server built: Nov 12 2008 10:40:14
>
> CentOS release 5.2 (Final)
Is 2.2.3 the latest version available for CentOS?
> After upgrading we are seeing random characters in the RR an Cd
> columns on the jkserver-status page.
> eg
> o%2B
> o+
> ����
Is this only appearing on the mod_jk status page? In think André is
asking about other pages (like those generated by JSP code). The status
page is served directly by mod_jk (within httpd) and so there's no
Tomcat interaction at all.
> Upgrade were done by stopping httpd.
> Copying new modjk to /etc/httpd/modules/
> Editing /etc/httpd/conf.d/modjk.conf to reflect new version.
>
> Requests are being served but the strange characters and there
> origin bit troubling.
I agree.
What is the Content-Type header for the jkserver-status page? Are there
any <META> tags in the page?
It's possible that your httpd is simply advertising a Content-Type
encoding that is different from what jkserver-status is actually using.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAksir5QACgkQ9CaO5/Lv0PAx/QCgwpEB+Ie6GbtfN19aeSxNvMRM
SQQAnRFqdsBu8BC1/gj0lq5Ad689AoaD
=8KLg
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org