You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Mikolaj Rydzewski <mi...@ceti.pl> on 2006/12/01 13:40:54 UTC
Web spiders - disabling jsessionid
Hi,
As you may know url rewriting feature is not a nice thing when spiders
come to index your site -
http://gabrito.com/post/javas-seo-blunder-jsessionid.
There are a few solutions I'm thinking of:
* configuration at Tomcat / web.xml level to disable/enable url
rewriting (unfortunately Tomcat doesn't support it)
* do not use <c:url, <html:link, and similiar tags - could be quite
expensive when working with Strurs or other framework
* extend classes for <c:url, <html:link and other tags not to make
use of url rewriting - seems quite a good solution, but could be a
problem with legacy applications
* apply a filter in web.xml to remove ;jsessionid=XXX from the
generated output - I'm afraid it'd be quite resource expensive
(lots of string operations)
* apply a filter in web.xml to inject custom HttpServletResponse
implementation with empty encodeURL method - seems quite interesting
* apply a similiar filter at the apache level (I assume one is uing
apache as a fronted to tomcat) - I didn't benchmark it, but I
suppose that a perl filter would be faster than similiar Java one
What are your opinions?
--
Mikolaj Rydzewski <mi...@ceti.pl>
RE: [OT] Web spiders - disabling jsessionid
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Leon Rosenberg [mailto:rosenberg.leon@googlemail.com]
> Subject: Re: Web spiders - disabling jsessionid
>
> It's completely OT, but once a customer of mine has placed a
> direct-login link to the public accessible test-system for the newest
> project on a crawled site, so that google not only logged in, but also
> deleted about 400 entries from the cms.
Sometimes I wonder why we go to all the effort to make our software
secure when all it takes is one human to undo everything.
"Badges? We ain't got no badges. We don't need no stinkin' badges..."
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Web spiders - disabling jsessionid
Posted by Leon Rosenberg <ro...@googlemail.com>.
On 12/1/06, Caldarale, Charles R <Ch...@unisys.com> wrote:
> > From: Len Popp [mailto:len.popp@gmail.com]
> > Subject: Re: Web spiders - disabling jsessionid
> >
> > On my site (as on many others) you can browse the site without a
> > session, but if you want to log in (to add content or to use
> > personalized settings) you need a session.
>
> O.k., now I'm confused.
>
> 1) Sessions are not created unless logged in.
>
> 2) Bots don't like the session id they're given.
>
> From the above, I have to conclude the bots are logging in, which sounds
> pretty ambitious, even for Google. If the bots aren't logging in, then
> they won't have a session, so there's no need to filter out what isn't
> there.
It's completely OT, but once a customer of mine has placed a
direct-login link to the public accessible test-system for the newest
project on a crawled site, so that google not only logged in, but also
deleted about 400 entries from the cms.
Everything imaginable happens somewhere...
regards
Leon
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Web spiders - disabling jsessionid
Posted by brycenesbitt <br...@obviously.com>.
Rashmi Rubdi wrote:
>
> I don't know because this problem doesn't happen in my case, on 2
> different web applications.
>
> Bryce should really test his case by setting cookies="true" or remove the
> cookies attribute and test his links with Xenu to see if he still gets
> jsessionid with Xenu.
>
A quick google search will show this happens to many other people -- even if
your webapps are magically immune. http://www.citycarshare.org/ is
definitely affected.
None of the three cookie options do it for me... not "true", not "false",
and not leaving it at the default (which should be true). This appears to
be something Tomcat just can't do. While it's a search robot bug (not
parsing out the ";jessionid=", it's my problem.
Next I'll try:
http://randomcoder.com/articles/jsessionid-considered-harmful
Adding a filter like:
HttpServletResponseWrapper wrappedResponse
= new HttpServletResponseWrapper(httpResponse)
{
public String encodeRedirectUrl(String url) { return url; }
public String encodeRedirectURL(String url) { return url; }
public String encodeUrl(String url) { return url; }
public String encodeURL(String url) { return url; }
};
--
View this message in context: http://www.nabble.com/Web-spiders---disabling-jsessionid-tf2737558.html#a7670111
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Web spiders - disabling jsessionid
Posted by brycenesbitt <br...@obviously.com>.
Caldarale, Charles R wrote:
>
>> From: brycenesbitt [mailto:bryce1@obviously.com]
>> Subject: Re: Web spiders - disabling jsessionid
>> Creating semicolon-based URL strings is the default in
>> Tomcat/Struts.
>
> I don't know about Struts, but that's not true for Tomcat. Look at the
> cookies attribute of the <Context> tag:
> http://tomcat.apache.org/tomcat-5.5-doc/config/context.html
> - Chuck
>
When "Cookies" is true, and the browser does not keep cookies, Tomcat
rewrites the URL. "true" is the default.
--
View this message in context: http://www.nabble.com/Web-spiders---disabling-jsessionid-tf2737558.html#a7661072
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Web spiders - disabling jsessionid
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: brycenesbitt [mailto:bryce1@obviously.com]
> Subject: Re: Web spiders - disabling jsessionid
>
> Creating semicolon-based URL strings is the default in
> Tomcat/Struts.
I don't know about Struts, but that's not true for Tomcat. Look at the
cookies attribute of the <Context> tag:
http://tomcat.apache.org/tomcat-5.5-doc/config/context.html
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Web spiders - disabling jsessionid
Posted by brycenesbitt <br...@obviously.com>.
Christopher Schultz-2 wrote:
>
> Perhaps that is the /quickest/ solution, but I would argue that the best
> solution is not to create a session if you don't actually need one.
>
The problem in many cases is the author does not care about sessions at all!
Creating semicolon-based URL strings is the default in Tomcat/Struts. We
get session ID's not because we want a session, but because we can't figure
out how to turn them off!
--
View this message in context: http://www.nabble.com/Web-spiders---disabling-jsessionid-tf2737558.html#a7660951
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Web spiders - disabling jsessionid
Posted by brycenesbitt <br...@obviously.com>.
Rashmi Rubdi wrote:
>
> So the solution for Bryce would be to leave the session on on each JSP
> page, and omit the cookies attribute of <Context which defaults it to
> true.
> This should solve the problem of jsessionid for bots.
> From my observation search bots support cookies otherwise I would have the
> problem of jsessionid appended to URLs too.
>
I'm just not getting it.
Can someone take a look at this site, and maybe give some insight?
http://www.citycarshare.org/howitworks.do
Or at 216.93.188.140 you can see a test intance which has the following
ROOT/META-INF/context.xml
<?xml version='1.0' encoding='UTF-8'?>
<Context path='/' cookies="false">
</Context>
I can share with you lots of log lines showing the JSESSIONID, including
crawls by Google, Alexa and Exalead. A quick scan of the Google index shows
cached pages with JSESSIONID. 2224 of the 9273 log lines from today have
JSESSIONID. I have thousands on thousands of crawls of the same content, on
the same day, with different JSESSIONID's.
Here are some examples:
69.106.42.228 - - [01/Dec/2006:16:02:58 -0800] "GET
/images/events/CCS_5_Icon.png;jsessionid=6DC390F0ADC7569009CB60C98378919D
HTTP/1.1" 200 4960 "http://www.citycarshare.org/" "Mozilla/5.0 (Windows; U;
Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0"
193.47.80.51 - - [30/Nov/2006:04:37:56 -0800] "GET
/press.do;jsessionid=E49722F6235A31A3627A6C62753A7CDB HTTP/1.1" 200 22020
"-" "Exabot/3.0"
193.47.80.51 - - [30/Nov/2006:04:59:56 -0800] "GET
/press.do;jsessionid=1407BA083FB2123469A4E544C3F26DFC HTTP/1.1" 200 22020
"-" "Exabot/3.0"
193.47.80.51 - - [30/Nov/2006:06:16:01 -0800] "GET
/press.do;jsessionid=5FAFDDABFF42C82F3C766377F5AC9F44 HTTP/1.1" 200 22020
"-" "Exabot/3.0"
193.47.80.51 - - [30/Nov/2006:06:31:36 -0800] "GET
/press.do;jsessionid=EAFB1F3DB5B7D47DFF4212A66911754F HTTP/1.1" 200 22020
"-" "Exabot/3.0"
193.47.80.51 - - [30/Nov/2006:07:00:45 -0800] "GET
/press.do;jsessionid=ADF4E609E38901897648ABD6C7BF4E57 HTTP/1.1" 200 22020
"-" "Exabot/3.0"
193.47.80.51 - - [30/Nov/2006:07:20:54 -0800] "GET
/press.do;jsessionid=5049AD9757D8C7BAA599C2837EBFB3BE HTTP/1.1" 200 22020
"-" "Exabot/3.0"
193.47.80.51 - - [30/Nov/2006:07:37:42 -0800] "GET
/press.do;jsessionid=F53FC49BCAD98F4181F05DAC7D7A65C4 HTTP/1.1" 200 22020
"-" "Exabot/3.0"
193.47.80.51 - - [30/Nov/2006:07:49:13 -0800] "GET
/press.do;jsessionid=15FCF8DCE01CBD47DAB1A8D668EF9F38 HTTP/1.1" 200 22020
"-" "Exabot/3.0"
193.47.80.51 - - [30/Nov/2006:07:59:28 -0800] "GET
/press.do;jsessionid=E717438CB2746895BFF9C16DE6A72F28 HTTP/1.1" 200 22020
"-" "Exabot/3.0"
I am 1000% certain that not all bots browse with cookies, at least not all
the time. How can I stop these bots from crawling me so often? It is over
25% of my bandwidth just to the duplicate crawls, never mind the regular bot
traffic.
--
View this message in context: http://www.nabble.com/Web-spiders---disabling-jsessionid-tf2737558.html#a7667574
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Web spiders - disabling jsessionid
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Len Popp [mailto:len.popp@gmail.com]
> Subject: Re: Web spiders - disabling jsessionid
>
> On my site (as on many others) you can browse the site without a
> session, but if you want to log in (to add content or to use
> personalized settings) you need a session.
O.k., now I'm confused.
1) Sessions are not created unless logged in.
2) Bots don't like the session id they're given.
>From the above, I have to conclude the bots are logging in, which sounds
pretty ambitious, even for Google. If the bots aren't logging in, then
they won't have a session, so there's no need to filter out what isn't
there.
What am I missing?
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Web spiders - disabling jsessionid
Posted by Len Popp <le...@gmail.com>.
On 12/1/06, Christopher Schultz <ch...@christopherschultz.net> wrote:
> Mikolaj,
>
> Back to the original question...
>
> Mikolaj Rydzewski wrote:
> > As you may know url rewriting feature is not a nice thing when spiders
> > come to index your site -
> > http://gabrito.com/post/javas-seo-blunder-jsessionid.
>
> So, the problem is that your URLs contain ";jsessionid=...", right? When
> does that become a problem?
>
> That becomes a problem when google (or whomever) crawls your site on
> different days and sees the same content with "different" URLs. Well, I
> have a couple of thoughts about that.
>
> 1. A semi-colon is listed in the HTTP specification as being a valid
> delimiter, despite pretty much every major web server out there
> ignoring it and thinking that it's part of the path.
> This is partially the crawler's fault for not following the HTTP
> specification. The ";" character is not technically a valid URL
> character outside of it's role as a delimiter, just like "&" or "?".
Whether or not you consider it part of the URL, Google treats it that
way, and so we have to live with it.
> 2. If you strip-off the jsessionid argument for all of these URLs,
> you will end up with thousands of sessions being created for
> each URL requested by the google bot. Do you think that's a good
> idea?
As far as I can see, that's not a problem - I don't get anywhere near
a thousand live sessions from Google. In fact, Google's crawler seems
to limit itself to about one page per minute (according to my logs) so
there won't be more than a few dozen sessions at most.
> 3. If you don't want googlebot to get a session, why are you allocating
> one? If you need sessions to manage site navigation, then you
> cannot turn them off and have things work correctly... can you?
On my site (as on many others) you can browse the site without a
session, but if you want to log in (to add content or to use
personalized settings) you need a session. Sessions aren't required
for site navigation or crawling, but they are required for other
reasons.
> 4. Consider instructing googlebot not to crawl certain portions of your
> site (those which require a session) by using a robots.txt file.
Not an option if that would mean not indexing the interesting parts of the site.
The best solution I could find is to use a filter and
HttpServletResponse wrapper, as others have described. An
implementation of the wrapper class can be found here:
http://mail-archives.apache.org/mod_mbox/struts-user/200311.mbox/%3c3FBA8A39.8010907@fiskars.com%3e
The result is that to login to the site you need a browser that
supports session cookies, but I can accept that. And, Google can now
index the site without crawling all over it repeatedly with different
jsessionids. Yay.
--
Len
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Web spiders - disabling jsessionid
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mikolaj,
Back to the original question...
Mikolaj Rydzewski wrote:
> As you may know url rewriting feature is not a nice thing when spiders
> come to index your site -
> http://gabrito.com/post/javas-seo-blunder-jsessionid.
So, the problem is that your URLs contain ";jsessionid=...", right? When
does that become a problem?
That becomes a problem when google (or whomever) crawls your site on
different days and sees the same content with "different" URLs. Well, I
have a couple of thoughts about that.
1. A semi-colon is listed in the HTTP specification as being a valid
delimiter, despite pretty much every major web server out there
ignoring it and thinking that it's part of the path.
This is partially the crawler's fault for not following the HTTP
specification. The ";" character is not technically a valid URL
character outside of it's role as a delimiter, just like "&" or "?".
2. If you strip-off the jsessionid argument for all of these URLs,
you will end up with thousands of sessions being created for
each URL requested by the google bot. Do you think that's a good
idea?
3. If you don't want googlebot to get a session, why are you allocating
one? If you need sessions to manage site navigation, then you
cannot turn them off and have things work correctly... can you?
4. Consider instructing googlebot not to crawl certain portions of your
site (those which require a session) by using a robots.txt file.
Just my .02.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFFcHDf9CaO5/Lv0PARAtp5AKCgPVdAXu80zADXifTx6AJOYfpupACfXcPb
nEEqn4rpvDVatwSRf/XPScg=
=Y5KA
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Web spiders - disabling jsessionid
Posted by Mikolaj Rydzewski <mi...@ceti.pl>.
Leon Rosenberg wrote:
> google uses this 3rd agent to check your site from another ip adress,
> whether you do some ugly seo stuff, like cloacking etc.
Seems possible.
> so please don't do it, if you rely on being found.
I think that just removing ;jsessionid=XXX for the first one won't make
much harm.
--
Mikolaj Rydzewski <mi...@ceti.pl>
Re: Web spiders - disabling jsessionid
Posted by Leon Rosenberg <ro...@googlemail.com>.
Hello Christopher,
When I check my access logs I could imagine
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
beeing a google.bot.
Of course I don't know it for sure, cause I'm don't do any seo
cloaking here, and don't care. But one could go to seo boards, pick
the posted ip-adresses for cloaked google-bot and check one's
logfiles.
my access stats:
http://moskito.anotheria.net/moskitodemo/mui/mskShowProducer?pProducerId=UserAgentFilter
regards
Leon
On 12/1/06, Christopher Schultz <ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Leon,
>
> Leon Rosenberg wrote:
> > you believe everything you've been told ?:-)
>
> Well, I've been told by you, and I don't believe you. ;)
>
> > google has 3 (at least 3 known) user agents : google, mozzila with
> > google-bot in the agent string (the one you sent) and another one,
> > which is just Mozilla/5.0.
> >
> > google uses this 3rd agent to check your site from another ip adress,
> > whether you do some ugly seo stuff, like cloacking etc.
> >
> > If it detects that you deliver different content to his
> > mozilla-disguised bot, your chances to be thrown out of the index are
> > pretty high.
>
> This sounds pretty plausible. Unfortunately, my empirical data suggests
> otherwise. Allow me to post a portion of webalizers "top user agents"
> list for a small site I maintain for the month of November:
>
> # Hits User Agent
> 1 26529 48.64% Googlebot/2.1
> 2 12077 22.14% MSIE 6.0
> 3 5285 9.69% Yahoo! Slurp
> 4 3353 6.15% Mozilla/5.0
>
> There are 11 more user agents which are all pretty much irrelevant. As
> you can see, "googlebot" appears with a plurality of the hits (yeah,
> it's not a really popular site). That's a /lot/ of hits compared to the
> others. In fact, if you agree that "MSIE 6.0" is not google-in-disguise,
> then it is not possible for the remaining user agent stats to sum to a
> value even close to what googlebot says.
>
> Webalizer can be configured to "collapse" different user agent strings
> into one single user agent (say, anything containing MSIE into a single
> MSIE in order to get an aggregate MSIE usage number). No such
> aggregation is being used, here, so what you see is what you get.
>
> If your assertion was correct, I would have expected to see a large
> number (perhaps 1/3 or 1/2 of the googlebot hits) to come from "Mozilla"
> as google-incognito, but that's not the case.
>
> Can you give a reference to where you discovered this "fact"?
>
> - -chris
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.5 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iD8DBQFFcEZ/9CaO5/Lv0PARAuvqAJ41d+SbmskQIDH1xW5obI2f2xQWTwCfavcf
> ed8ZaktgYzFpjfk2lli4vns=
> =HZmV
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Web spiders - disabling jsessionid
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Leon,
Leon Rosenberg wrote:
> you believe everything you've been told ?:-)
Well, I've been told by you, and I don't believe you. ;)
> google has 3 (at least 3 known) user agents : google, mozzila with
> google-bot in the agent string (the one you sent) and another one,
> which is just Mozilla/5.0.
>
> google uses this 3rd agent to check your site from another ip adress,
> whether you do some ugly seo stuff, like cloacking etc.
>
> If it detects that you deliver different content to his
> mozilla-disguised bot, your chances to be thrown out of the index are
> pretty high.
This sounds pretty plausible. Unfortunately, my empirical data suggests
otherwise. Allow me to post a portion of webalizers "top user agents"
list for a small site I maintain for the month of November:
# Hits User Agent
1 26529 48.64% Googlebot/2.1
2 12077 22.14% MSIE 6.0
3 5285 9.69% Yahoo! Slurp
4 3353 6.15% Mozilla/5.0
There are 11 more user agents which are all pretty much irrelevant. As
you can see, "googlebot" appears with a plurality of the hits (yeah,
it's not a really popular site). That's a /lot/ of hits compared to the
others. In fact, if you agree that "MSIE 6.0" is not google-in-disguise,
then it is not possible for the remaining user agent stats to sum to a
value even close to what googlebot says.
Webalizer can be configured to "collapse" different user agent strings
into one single user agent (say, anything containing MSIE into a single
MSIE in order to get an aggregate MSIE usage number). No such
aggregation is being used, here, so what you see is what you get.
If your assertion was correct, I would have expected to see a large
number (perhaps 1/3 or 1/2 of the googlebot hits) to come from "Mozilla"
as google-incognito, but that's not the case.
Can you give a reference to where you discovered this "fact"?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFFcEZ/9CaO5/Lv0PARAuvqAJ41d+SbmskQIDH1xW5obI2f2xQWTwCfavcf
ed8ZaktgYzFpjfk2lli4vns=
=HZmV
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Web spiders - disabling jsessionid
Posted by Leon Rosenberg <ro...@googlemail.com>.
you believe everything you've been told ?:-)
google has 3 (at least 3 known) user agents : google, mozzila with
google-bot in the agent string (the one you sent) and another one,
which is just Mozilla/5.0.
google uses this 3rd agent to check your site from another ip adress,
whether you do some ugly seo stuff, like cloacking etc.
If it detects that you deliver different content to his
mozilla-disguised bot, your chances to be thrown out of the index are
pretty high.
so please don't do it, if you rely on being found.
regards
Leon
On 12/1/06, Tim Funk <fu...@joedog.org> wrote:
> Wrong. Google is very clear about not hiding user agent - as well as a
> the other major bots.
>
> Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html
>
> Just just for Googlebot in the user-agent header.
>
> -Tim
>
> Leon Rosenberg wrote:
> > On 12/1/06, Tim Funk <fu...@joedog.org> wrote:
> >> The easiest is the filter and custom HttpServletResponse which overrides
> >> encodeURL() to do nothing.
> >>
> >> It could be made one step smarter by checking if the User agent is a
> >> search engine bot to selectively execute or not.
> >>
> >
> > How do you want to achieve that?
> > They disguise themself as normal browsers....
> > Leon
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Web spiders - disabling jsessionid
Posted by Tim Funk <fu...@joedog.org>.
Wrong. Google is very clear about not hiding user agent - as well as a
the other major bots.
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html
Just just for Googlebot in the user-agent header.
-Tim
Leon Rosenberg wrote:
> On 12/1/06, Tim Funk <fu...@joedog.org> wrote:
>> The easiest is the filter and custom HttpServletResponse which overrides
>> encodeURL() to do nothing.
>>
>> It could be made one step smarter by checking if the User agent is a
>> search engine bot to selectively execute or not.
>>
>
> How do you want to achieve that?
> They disguise themself as normal browsers....
> Leon
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Web spiders - disabling jsessionid
Posted by Leon Rosenberg <ro...@googlemail.com>.
On 12/1/06, Tim Funk <fu...@joedog.org> wrote:
> The easiest is the filter and custom HttpServletResponse which overrides
> encodeURL() to do nothing.
>
> It could be made one step smarter by checking if the User agent is a
> search engine bot to selectively execute or not.
>
How do you want to achieve that?
They disguise themself as normal browsers....
Leon
> -Tim
>
> Mikolaj Rydzewski wrote:
> > Hi,
> >
> > As you may know url rewriting feature is not a nice thing when spiders
> > come to index your site -
> > http://gabrito.com/post/javas-seo-blunder-jsessionid.
> >
> > There are a few solutions I'm thinking of:
> >
> > * configuration at Tomcat / web.xml level to disable/enable url
> > rewriting (unfortunately Tomcat doesn't support it)
> > * do not use <c:url, <html:link, and similiar tags - could be quite
> > expensive when working with Strurs or other framework
> > * extend classes for <c:url, <html:link and other tags not to make
> > use of url rewriting - seems quite a good solution, but could be a
> > problem with legacy applications
> > * apply a filter in web.xml to remove ;jsessionid=XXX from the
> > generated output - I'm afraid it'd be quite resource expensive
> > (lots of string operations)
> > * apply a filter in web.xml to inject custom HttpServletResponse
> > implementation with empty encodeURL method - seems quite interesting
> > * apply a similiar filter at the apache level (I assume one is uing
> > apache as a fronted to tomcat) - I didn't benchmark it, but I
> > suppose that a perl filter would be faster than similiar Java one
> >
> > What are your opinions?
> >
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Web spiders - disabling jsessionid
Posted by Tim Funk <fu...@joedog.org>.
The easiest is the filter and custom HttpServletResponse which overrides
encodeURL() to do nothing.
It could be made one step smarter by checking if the User agent is a
search engine bot to selectively execute or not.
-Tim
Mikolaj Rydzewski wrote:
> Hi,
>
> As you may know url rewriting feature is not a nice thing when spiders
> come to index your site -
> http://gabrito.com/post/javas-seo-blunder-jsessionid.
>
> There are a few solutions I'm thinking of:
>
> * configuration at Tomcat / web.xml level to disable/enable url
> rewriting (unfortunately Tomcat doesn't support it)
> * do not use <c:url, <html:link, and similiar tags - could be quite
> expensive when working with Strurs or other framework
> * extend classes for <c:url, <html:link and other tags not to make
> use of url rewriting - seems quite a good solution, but could be a
> problem with legacy applications
> * apply a filter in web.xml to remove ;jsessionid=XXX from the
> generated output - I'm afraid it'd be quite resource expensive
> (lots of string operations)
> * apply a filter in web.xml to inject custom HttpServletResponse
> implementation with empty encodeURL method - seems quite interesting
> * apply a similiar filter at the apache level (I assume one is uing
> apache as a fronted to tomcat) - I didn't benchmark it, but I
> suppose that a perl filter would be faster than similiar Java one
>
> What are your opinions?
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Web spiders - disabling jsessionid
Posted by Andrew Stepanenko <an...@gmail.com>.
Hello,
we use filter in web.xml as you said: StripSessionIdFilter. Works
fine. Not that much overhead.
Regards
Andrew Stepanenko,
Ternopil, Ukraine
http://unf.tane.edu.ua
On 12/1/06, Mikolaj Rydzewski <mi...@ceti.pl> wrote:
> Hi,
>
> As you may know url rewriting feature is not a nice thing when spiders
> come to index your site -
> http://gabrito.com/post/javas-seo-blunder-jsessionid.
>
> There are a few solutions I'm thinking of:
>
> * configuration at Tomcat / web.xml level to disable/enable url
> rewriting (unfortunately Tomcat doesn't support it)
> * do not use <c:url, <html:link, and similiar tags - could be quite
> expensive when working with Strurs or other framework
> * extend classes for <c:url, <html:link and other tags not to make
> use of url rewriting - seems quite a good solution, but could be a
> problem with legacy applications
> * apply a filter in web.xml to remove ;jsessionid=XXX from the
> generated output - I'm afraid it'd be quite resource expensive
> (lots of string operations)
> * apply a filter in web.xml to inject custom HttpServletResponse
> implementation with empty encodeURL method - seems quite interesting
> * apply a similiar filter at the apache level (I assume one is uing
> apache as a fronted to tomcat) - I didn't benchmark it, but I
> suppose that a perl filter would be faster than similiar Java one
>
> What are your opinions?
>
> --
> Mikolaj Rydzewski <mi...@ceti.pl>
>
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Web spiders - disabling jsessionid
Posted by brycenesbitt <br...@obviously.com>.
Rashmi Rubdi wrote:
>
> As discussed previously in this thread you can turn jsessionid in the URL
> off easily by setting the cookies attribute of <Context to true.
> http://tomcat.apache.org/tomcat-5.0-doc/config/context.html
>
No option seems to match the need:
true -- uses URL-rewriting if the browser does not support cookies. this is
exactly the problem, as spiders don't use cookies.
false --- uses only URL-rewriting.
Nowhere there an option to disable URL-rewriting totally.
Perhaps you're confused with Resin-- which offers an option to solve this
problem.
--
View this message in context: http://www.nabble.com/Web-spiders---disabling-jsessionid-tf2737558.html#a7661033
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Web spiders - disabling jsessionid
Posted by brycenesbitt <br...@obviously.com>.
Google's index has 33.4 million pages with a jsessionid:
http://www.google.com/search?hl=en&lr=&q=inurl%3Ajsessionid&btnG=Search
Many of those are duplicates (no different other than the jsessionid).
-Bryce Nesbitt
--
View this message in context: http://www.nabble.com/Web-spiders---disabling-jsessionid-tf2737558.html#a7660959
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Web spiders - disabling jsessionid
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Bryce,
brycenesbitt wrote:
>
> Caldarale, Charles R wrote:
>> Try turning off cookies in your browser.
>>
>
> Sorry for the lack of clarity. I can't force jessionid to show up even with
> cookies off in the browser.
My guess is that there is a page that you missed on your site that it
still creating a session. I would recommend writing a session listener
and dumping a stack trace when one is created. If you really do not need
sessions, you shouldn't be creating them in the first place.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFFdDTO9CaO5/Lv0PARApliAJwOCBzpyoKqIF6UcC5ShcunBFMRXwCcCmyK
aJIGLchEbvAg61C/y20kaO0=
=i4tf
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Web spiders - disabling jsessionid
Posted by brycenesbitt <br...@obviously.com>.
Caldarale, Charles R wrote:
>
> Try turning off cookies in your browser.
>
Sorry for the lack of clarity. I can't force jessionid to show up even with
cookies off in the browser.
Using wget from the unix command line (no cookies!) I get a jsessionid for
images:
/images/logo.gif;jsessionid=C09497FCD52DC8B5A09FAA23F999FB2A
But not for links:
/aboutus.do
And those pesky jsessionid's keep showing up in my logs -- with the
attendent misinterpretation by Alexa and Exalead. Google seems a bit less
stupid about them, but still crawls the same pages over and over with
different cached and stale session ID's.
--
View this message in context: http://www.nabble.com/Web-spiders---disabling-jsessionid-tf2737558.html#a7661062
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Web spiders - disabling jsessionid
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: brycenesbitt [mailto:bryce1@obviously.com]
> Subject: Re: Web spiders - disabling jsessionid
>
> I can't force a ";" based jsessionid to show in Firefox.
Try turning off cookies in your browser.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Web spiders - disabling jsessionid
Posted by brycenesbitt <br...@obviously.com>.
Mikolaj Rydzewski-2 wrote:
>
> Hi,
> As you may know url rewriting feature is not a nice thing when spiders
> come to index your site -
> http://gabrito.com/post/javas-seo-blunder-jsessionid.
>
I'm having trouble with JSESSIONID with search engines Google, Accoona,
Alexa and Exalead.
My approach was to contact each firm, and ask them fix thier bugs. Each
engine
crawls each page of my site 5-50 times a day, presumably because they don't
understand the ";" in the URL. For example:
93.47.80.51 - - [28/Nov/2006:16:05:36 -0800] "GET
/awards.do;jsessionid=68B86DFF8E4A8597B210531C3431965D HTTP/1.1" 200
17195 "-" "Exabot/3.0"
193.47.80.51 - - [28/Nov/2006:16:17:30 -0800] "GET
/awards.do;jsessionid=0621414681C92E1A00A9428A7800AC30 HTTP/1.1" 200
17195 "-" "Exabot/3.0"
193.47.80.51 - - [28/Nov/2006:17:00:36 -0800] "GET
/awards.do;jsessionid=0079FCD91ED8E5B86902228D285CCEEF HTTP/1.1" 200
17195 "-" "Exabot/3.0"
193.47.80.51 - - [28/Nov/2006:20:41:50 -0800] "GET
/awards.do;jsessionid=DE9B61384D3D75DE9EB38A21F066E433 HTTP/1.1" 200
17195 "-" "Exabot/3.0"
While I have received responses from all, and three have promised action,
none has fixed the issue to date. So I tried to fix it myself.
This was easy under Resin. But under Tomcat I have had no luck.
I added <% <at> page session="false" %> each of my struts tiles templates,
but I still see some session ID's in the logs. Now I'm confused. My
application does not care about sessions. I can't force a ";" based
jsessionid to show in Firefox. Yet somehow the darn robots (and some other
browsers) are seeing them. What's up?
71.146.168.171 - - [02/Dec/2006:23:29:20 -0800] "GET
/images/events/CCS_5_Icon.p ng;jsessionid=D5912D8983A86FF2BCF3381DB454D54A
HTTP/1.1" 200 4960 "http://www.ci tycarshare.org/" "Mozilla/5.0 (Macintosh;
U; Intel Mac OS X; en) AppleWebKit/418 .9 (KHTML, like Gecko) Safari/419.3"
The website in question is http://www.citycarshare.org/
--
View this message in context: http://www.nabble.com/Web-spiders---disabling-jsessionid-tf2737558.html#a7660925
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org