You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "S.L" <si...@gmail.com> on 2014/09/22 06:43:14 UTC
jsessionid not being remvoed from the url
The jsessionid on the cralwed URL is not being removed ,even though a regex
URL normalizer is beign specifiied, can someonle please let me know the
issue here ?
I have already set the following
*nutch-site.xml*
<property>
<name>plugin.includes</name>
<value>protocol-http|urlfilter-regex|parse-(html|tika)|index-(basic|anchor|more)|scoring-optic|urlnormalizer-(pass|regex|basic)
</value>
</property>
<property>
<name>urlnormalizer.order</name>
<value>org.apache.nutch.net.urlnormalizer.basic.BasicURLNormalizer
org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer
</value>
<description>Order in which normalizers will run. If any of these
isn't
activated it will be silently skipped. If other normalizers not
on the
list are activated, they will run in random order after the
ones
specified here are run.
</description>
</property>
<property>
<name>urlnormalizer.regex.file</name>
<value>regex-normalize.xml</value>
<description>Name of the config file used by the RegexUrlNormalizer
class.
</description>
</property>
*And the regex-normalize.xml file has this entry*
<regex>
<pattern>(?i)(;?\b_?(l|j|bv_)?(sid|phpsessid|sessionid)=.*?)(\?|&|#|$)</pattern>
<substitution>$4</substitution>
</regex>
Looks like this should have been removed , is the regex in
regex-normalize.xml correct ?
Re: jsessionid not being remvoed from the url
Posted by Sebastian Nagel <wa...@googlemail.com>.
Hi,
if testing trunk via
echo 'http://www.xyz.com/...' \
| bin/nutch org.apache.nutch.net.URLNormalizerChecker
the jsessionid is properly removed. The resulting URL is:
http://www.xyz.com/site/hosa-technology-3-5mm-trs-to-1-4-trs-adapter/8561415.p?id=1208561582654&skuId=8561415&st=categoryid$abcat0207000&cp=1&lp=8
Although, I didn't test it, Nutch 1.7 should behave identical in this point.
Best,
Sebastian
2014-09-22 22:29 GMT+02:00 S.L <si...@gmail.com>:
> Sebastian , I am using Nutch 1.7 and a specific example in this case is
> this.
>
>
> http://www.xyz.com/site/hosa-technology-3-5mm-trs-to-1-4-trs-adapter/8561415.p;jsessionid=7936CA95263E9C78B735E5EBE827BDDA.bbolsp-app04-163?id=1208561582654&skuId=8561415&st=categoryid$abcat0207000&cp=1&lp=8
>
>
>
> On Mon, Sep 22, 2014 at 4:12 PM, Sebastian Nagel <
> wastl.nagel@googlemail.com
> > wrote:
>
> > > Looks like this should have been removed , is the regex in
> > > regex-normalize.xml correct ?
> > >
> >
> > Yes. It removes various session ids, see
> > src/plugin/urlnormalizer-regex/sample/regex-normalize-default.test
> >
> > Can you give a concrete example of a session id not removed?
> > Which Nutch version is used?
> >
> > Thanks,
> > Sebastian
> >
> > On 09/22/2014 06:43 AM, S.L wrote:
> > > The jsessionid on the cralwed URL is not being removed ,even though a
> > regex
> > > URL normalizer is beign specifiied, can someonle please let me know the
> > > issue here ?
> > >
> > > I have already set the following
> > >
> > > *nutch-site.xml*
> > >
> > > <property>
> > > <name>plugin.includes</name>
> > >
> > >
> >
> <value>protocol-http|urlfilter-regex|parse-(html|tika)|index-(basic|anchor|more)|scoring-optic|urlnormalizer-(pass|regex|basic)
> > > </value>
> > > </property>
> > >
> > > <property>
> > > <name>urlnormalizer.order</name>
> > >
> > <value>org.apache.nutch.net.urlnormalizer.basic.BasicURLNormalizer
> > > org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer
> > > </value>
> > > <description>Order in which normalizers will run. If any of
> these
> > > isn't
> > > activated it will be silently skipped. If other normalizers
> > not
> > > on the
> > > list are activated, they will run in random order after the
> > > ones
> > > specified here are run.
> > > </description>
> > > </property>
> > >
> > >
> > > <property>
> > > <name>urlnormalizer.regex.file</name>
> > > <value>regex-normalize.xml</value>
> > > <description>Name of the config file used by the
> > RegexUrlNormalizer
> > > class.
> > > </description>
> > > </property>
> > >
> > >
> > > *And the regex-normalize.xml file has this entry*
> > > <regex>
> > >
> > >
> >
> <pattern>(?i)(;?\b_?(l|j|bv_)?(sid|phpsessid|sessionid)=.*?)(\?|&|#|$)</pattern>
> > > <substitution>$4</substitution>
> > > </regex>
> > >
> > >
> > > Looks like this should have been removed , is the regex in
> > > regex-normalize.xml correct ?
> > >
> >
> >
>
Re: jsessionid not being remvoed from the url
Posted by "S.L" <si...@gmail.com>.
Sebastian , I am using Nutch 1.7 and a specific example in this case is
this.
http://www.xyz.com/site/hosa-technology-3-5mm-trs-to-1-4-trs-adapter/8561415.p;jsessionid=7936CA95263E9C78B735E5EBE827BDDA.bbolsp-app04-163?id=1208561582654&skuId=8561415&st=categoryid$abcat0207000&cp=1&lp=8
On Mon, Sep 22, 2014 at 4:12 PM, Sebastian Nagel <wastl.nagel@googlemail.com
> wrote:
> > Looks like this should have been removed , is the regex in
> > regex-normalize.xml correct ?
> >
>
> Yes. It removes various session ids, see
> src/plugin/urlnormalizer-regex/sample/regex-normalize-default.test
>
> Can you give a concrete example of a session id not removed?
> Which Nutch version is used?
>
> Thanks,
> Sebastian
>
> On 09/22/2014 06:43 AM, S.L wrote:
> > The jsessionid on the cralwed URL is not being removed ,even though a
> regex
> > URL normalizer is beign specifiied, can someonle please let me know the
> > issue here ?
> >
> > I have already set the following
> >
> > *nutch-site.xml*
> >
> > <property>
> > <name>plugin.includes</name>
> >
> >
> <value>protocol-http|urlfilter-regex|parse-(html|tika)|index-(basic|anchor|more)|scoring-optic|urlnormalizer-(pass|regex|basic)
> > </value>
> > </property>
> >
> > <property>
> > <name>urlnormalizer.order</name>
> >
> <value>org.apache.nutch.net.urlnormalizer.basic.BasicURLNormalizer
> > org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer
> > </value>
> > <description>Order in which normalizers will run. If any of these
> > isn't
> > activated it will be silently skipped. If other normalizers
> not
> > on the
> > list are activated, they will run in random order after the
> > ones
> > specified here are run.
> > </description>
> > </property>
> >
> >
> > <property>
> > <name>urlnormalizer.regex.file</name>
> > <value>regex-normalize.xml</value>
> > <description>Name of the config file used by the
> RegexUrlNormalizer
> > class.
> > </description>
> > </property>
> >
> >
> > *And the regex-normalize.xml file has this entry*
> > <regex>
> >
> >
> <pattern>(?i)(;?\b_?(l|j|bv_)?(sid|phpsessid|sessionid)=.*?)(\?|&|#|$)</pattern>
> > <substitution>$4</substitution>
> > </regex>
> >
> >
> > Looks like this should have been removed , is the regex in
> > regex-normalize.xml correct ?
> >
>
>
Re: jsessionid not being remvoed from the url
Posted by Sebastian Nagel <wa...@googlemail.com>.
> Looks like this should have been removed , is the regex in
> regex-normalize.xml correct ?
>
Yes. It removes various session ids, see
src/plugin/urlnormalizer-regex/sample/regex-normalize-default.test
Can you give a concrete example of a session id not removed?
Which Nutch version is used?
Thanks,
Sebastian
On 09/22/2014 06:43 AM, S.L wrote:
> The jsessionid on the cralwed URL is not being removed ,even though a regex
> URL normalizer is beign specifiied, can someonle please let me know the
> issue here ?
>
> I have already set the following
>
> *nutch-site.xml*
>
> <property>
> <name>plugin.includes</name>
>
> <value>protocol-http|urlfilter-regex|parse-(html|tika)|index-(basic|anchor|more)|scoring-optic|urlnormalizer-(pass|regex|basic)
> </value>
> </property>
>
> <property>
> <name>urlnormalizer.order</name>
> <value>org.apache.nutch.net.urlnormalizer.basic.BasicURLNormalizer
> org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer
> </value>
> <description>Order in which normalizers will run. If any of these
> isn't
> activated it will be silently skipped. If other normalizers not
> on the
> list are activated, they will run in random order after the
> ones
> specified here are run.
> </description>
> </property>
>
>
> <property>
> <name>urlnormalizer.regex.file</name>
> <value>regex-normalize.xml</value>
> <description>Name of the config file used by the RegexUrlNormalizer
> class.
> </description>
> </property>
>
>
> *And the regex-normalize.xml file has this entry*
> <regex>
>
> <pattern>(?i)(;?\b_?(l|j|bv_)?(sid|phpsessid|sessionid)=.*?)(\?|&|#|$)</pattern>
> <substitution>$4</substitution>
> </regex>
>
>
> Looks like this should have been removed , is the regex in
> regex-normalize.xml correct ?
>