You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Alex Basa <al...@yahoo.com> on 2009/02/16 15:54:22 UTC
regex for a folder only crawl
Hi guys,
I'm trying to make a regex to only crawl a folder. So if I was crawling 123.com/456/789
I would only want to crawl
^http://123.com/456/789/(.*)/
I tried
^http://123.com/456/789/*\.*
but there are many web pages with no file extensions.
I'm not sure how to specify only one forward slash after in the regex. Any ideas?
Thanks as always in advance,
Alex
AW: regex for a folder only crawl
Posted by Koch Martina <Ko...@huberverlag.de>.
Try this one:
^http://123.com/456/789/[^/]+/$
With that you allow all characters except / between the forward slashes.
Besides, a good web page for online regular expression testing can be found at:
http://www.fileformat.info/tool/regex.htm
Kind regards,
Martina
-----Ursprüngliche Nachricht-----
Von: Alex Basa [mailto:alex_basa@yahoo.com]
Gesendet: Montag, 16. Februar 2009 18:08
An: nutch-user@lucene.apache.org
Betreff: Re: regex for a folder only crawl
I actually tried that but it also picks up
http://123.com/456/789/folderA/folderB/
http://123.com/456/789/folderA/folderB/folderC/
what I really need is something to say the first slash after the previous one.
--- On Mon, 2/16/09, Cool The Breezer <te...@yahoo.com> wrote:
> From: Cool The Breezer <te...@yahoo.com>
> Subject: Re: regex for a folder only crawl
> To: nutch-user@lucene.apache.org
> Date: Monday, February 16, 2009, 9:47 AM
> Try
> ^http://123.com/456/789/.*/$, which says end should be /
>
> - RB
>
> --- On Mon, 2/16/09, Alex Basa <al...@yahoo.com>
> wrote:
>
> > From: Alex Basa <al...@yahoo.com>
> > Subject: regex for a folder only crawl
> > To: nutch-user@lucene.apache.org
> > Date: Monday, February 16, 2009, 9:54 AM
> > Hi guys,
> >
> > I'm trying to make a regex to only crawl a folder.
> So
> > if I was crawling 123.com/456/789
> >
> > I would only want to crawl
> > ^http://123.com/456/789/(.*)/
> >
> > I tried
> > ^http://123.com/456/789/*\.*
> >
> > but there are many web pages with no file extensions.
> >
> > I'm not sure how to specify only one forward slash
> > after in the regex. Any ideas?
> >
> > Thanks as always in advance,
> >
> > Alex
Re: regex for a folder only crawl
Posted by Cool The Breezer <te...@yahoo.com>.
ohh then
^http://123.com/456/789/\\w*/$ that means it will allow only a word(a-zA-Z_0-9) between http://123.com/456/789/ and /.
- RB
--- On Mon, 2/16/09, Alex Basa <al...@yahoo.com> wrote:
> From: Alex Basa <al...@yahoo.com>
> Subject: Re: regex for a folder only crawl
> To: nutch-user@lucene.apache.org
> Date: Monday, February 16, 2009, 12:08 PM
> I actually tried that but it also picks up
>
> http://123.com/456/789/folderA/folderB/
> http://123.com/456/789/folderA/folderB/folderC/
>
> what I really need is something to say the first slash
> after the previous one.
>
> --- On Mon, 2/16/09, Cool The Breezer
> <te...@yahoo.com> wrote:
>
> > From: Cool The Breezer
> <te...@yahoo.com>
> > Subject: Re: regex for a folder only crawl
> > To: nutch-user@lucene.apache.org
> > Date: Monday, February 16, 2009, 9:47 AM
> > Try
> > ^http://123.com/456/789/.*/$, which says end should be
> /
> >
> > - RB
> >
> > --- On Mon, 2/16/09, Alex Basa
> <al...@yahoo.com>
> > wrote:
> >
> > > From: Alex Basa <al...@yahoo.com>
> > > Subject: regex for a folder only crawl
> > > To: nutch-user@lucene.apache.org
> > > Date: Monday, February 16, 2009, 9:54 AM
> > > Hi guys,
> > >
> > > I'm trying to make a regex to only crawl a
> folder.
> > So
> > > if I was crawling 123.com/456/789
> > >
> > > I would only want to crawl
> > > ^http://123.com/456/789/(.*)/
> > >
> > > I tried
> > > ^http://123.com/456/789/*\.*
> > >
> > > but there are many web pages with no file
> extensions.
> > >
> > > I'm not sure how to specify only one forward
> slash
> > > after in the regex. Any ideas?
> > >
> > > Thanks as always in advance,
> > >
> > > Alex
Re: regex for a folder only crawl
Posted by Alex Basa <al...@yahoo.com>.
I actually tried that but it also picks up
http://123.com/456/789/folderA/folderB/
http://123.com/456/789/folderA/folderB/folderC/
what I really need is something to say the first slash after the previous one.
--- On Mon, 2/16/09, Cool The Breezer <te...@yahoo.com> wrote:
> From: Cool The Breezer <te...@yahoo.com>
> Subject: Re: regex for a folder only crawl
> To: nutch-user@lucene.apache.org
> Date: Monday, February 16, 2009, 9:47 AM
> Try
> ^http://123.com/456/789/.*/$, which says end should be /
>
> - RB
>
> --- On Mon, 2/16/09, Alex Basa <al...@yahoo.com>
> wrote:
>
> > From: Alex Basa <al...@yahoo.com>
> > Subject: regex for a folder only crawl
> > To: nutch-user@lucene.apache.org
> > Date: Monday, February 16, 2009, 9:54 AM
> > Hi guys,
> >
> > I'm trying to make a regex to only crawl a folder.
> So
> > if I was crawling 123.com/456/789
> >
> > I would only want to crawl
> > ^http://123.com/456/789/(.*)/
> >
> > I tried
> > ^http://123.com/456/789/*\.*
> >
> > but there are many web pages with no file extensions.
> >
> > I'm not sure how to specify only one forward slash
> > after in the regex. Any ideas?
> >
> > Thanks as always in advance,
> >
> > Alex
Re: regex for a folder only crawl
Posted by Cool The Breezer <te...@yahoo.com>.
Try
^http://123.com/456/789/.*/$, which says end should be /
- RB
--- On Mon, 2/16/09, Alex Basa <al...@yahoo.com> wrote:
> From: Alex Basa <al...@yahoo.com>
> Subject: regex for a folder only crawl
> To: nutch-user@lucene.apache.org
> Date: Monday, February 16, 2009, 9:54 AM
> Hi guys,
>
> I'm trying to make a regex to only crawl a folder. So
> if I was crawling 123.com/456/789
>
> I would only want to crawl
> ^http://123.com/456/789/(.*)/
>
> I tried
> ^http://123.com/456/789/*\.*
>
> but there are many web pages with no file extensions.
>
> I'm not sure how to specify only one forward slash
> after in the regex. Any ideas?
>
> Thanks as always in advance,
>
> Alex